Rating for Relevance
Goal of Task:
No pin evaluation
Relevance: This is established on the following: matching suggestions, intent, and distance/prominence.
Determining how well suggestion match the user’s potential intent:
Query and Category Suggestions: (Research steps same as above)
AutoComplete Comments
Comments are mandatory for any relevance rating of Good or below as well as for any Name and Address Accuracy issues not rated Correct and should explain the reasoning behind your rating. Be sure to:
Include user intent: It’s usefuI to indicate what you think the user intent is. In Autocomplete the cleery is sometimes too short to indicate the precise intent.
Refer to the guideline:>: If the demotion is due to a guideline instruction, note the {guideline section that guided your decision
Include sources: If you find incorrect information in a suggestion, your comment should include the correct information and its source. Be sure that any links you include lead directly to the information you provide. If they don’t, include a link to the source you used and explain how to get to the information frcmi there.
Use URL shorteners when inse•rting long links into your comments.
Be concise: Don’t write elaborate comments. Be short and specific.
Comment in English: Your comments will sometimes be reviewed or used by people who are not familiar with your test locale or its language. Comments in English are understandable to the widest possible audience.
2.3.2 Implicit Location
If the user’s location intent is not expressly stated in the query, for example [chinese] or [479 margarita ave], use the user location, viewport location, and viewport age to determine the area of expected results.
4.2 User Intent
Example 1
Firstly, we need to understand the likely user intent – in Autocomplete tasks the query string may be incomplete, so the user query ‘home’ could be for
the user’s home location, an incomplete category or business query (e.g. Home Bargains or Homebase) or for an address, such as this suggestion we are rating for ‘Home Farm’. It’s unclear if this address physically exists, as a Royal Mail search for the full address does not return a match, however here is a similar address nearby ‘Home Farm, Marston Montgomery, Ashbourne’ which may be the correct address. However, as per the AC GLs 8.1.1 even though this address cannot be confirmed to exist it should be rated independently of any data issues.
Research
As the user is outside of the fresh viewport, we use the fresh viewport as location intent. We now need to determine what possible real-world options are nearby, to assess how relevant the suggestion satisfies the query. Copying the viewport centre co-ordinates into a map application and searching nearby for ‘home’ shows there are multiple options nearby.
We can see that the nearest suggestion to the viewport is Home Bargains, with our likely Home Farm location next nearest, followed by a Homebase store further away. AC GLs 4.3.2 and 4.3.3 tells us that the further away and less prominent the suggestion is, the less desirable it becomes, so this
suggestion for ‘Home Farm’ should be demoted to Good for distance/prominence issue, as Home Bargains is a nearer and more prominent suggestion that better satisfies the query.
Example 2
In this task, the query is a category query for ‘vintage tearooms’ with an incomplete location modifier which is likely the nearby location of Hartlepool. This suggestion for ‘Dawn P’s Vintage Teas’ is relatively nearby but checking street imagery using a map application shows that this appears to be a residential address with no visible signage for the business. A web search returns a claimed Yelp page https://www.yelp. com/biz/dawn- ps-vintage-teas-
hartlepool which indicates this is most likely a home-based catering supplier operating from a residential address, rather than a vintage tearoom that a user would visit.
Research
AC GLs 7.1.3 tells us that back-office locations and home-based businesses should be rated Bad for relevance, because they are not useful to the user and have no maps intent.
Example 3
For this task, the viewport is stale, so we use the user location as location intent. The official website for the POI https://www.academymusicgroup.com/o2forumkentishtown/ shows that the correct name for the POI is ‘O2 Forum Kentish Town’. AC GLs 4 tells us that we rate the relevance of a suggestion before considering the accuracy of the name or address, so although the suggestion is not a direct match for the correct POI name, it does satisfy the query.
However, it is quite distant compared to one suggestion nearby, and less prominent than the locality Kentish Town, so should be demoted to Acceptable or Bad for distance.
Example 4
In this task, the user is located just outside the fresh viewport, so we use the viewport as location intent. Copying the viewport centre co-ordinates to a map application and searching nearby for the query ‘napie’ shows there are multiple options nearby, including three Edinburgh Napier University campus locations.
The official university website suggests that the Sighthill Campus (which is nearest to the fresh viewport) is the main campus for the university and is listed as the contact address on the official website https://www.napier.ac.u k/about-us/contact-us
Research
The suggestion we are rating for the Craiglockhart Campus is further away and less prominent, so should be demoted to Good for distance/prominence issue.
Example 5
In this task, the user is located inside the fresh viewport, so suggestions are expected near the user location. The query could be an incomplete query for chain/brand ‘Victoria’s Secret’ but searching a map application shows the nearest store is 30km away in Milton Keynes (the map application’s drawing tools can be used to measure the distance).
Research
Our suggestion for Victoria Coach Station is another possible match for the query string but is further away (45km) – it is still a relatively prominent
transport POI, so a rating of Good or Acceptable is the best option. AC GLs 4.3.4 explains the relationship between distance and prominence in the context of all possible suggestions in the real world, so for this task a rating of Excellent would be too lenient, as the Victoria’s Secret stores are nearer; however, a rating of Bad is likely too harsh, as this is still a well-known transport POI and is not unreasonably far away.
Example 6
This query suggestion for ‘Beignet’ (a French doughnut or fritter) matches the query string but for it to be useful to the user, it must return useful options in or nearby the fresh viewport, as per AC GLs 4.13. Copying the viewport centre co-ordinates to a map
application and searching nearby for ‘Beignet’ returns multiple cafes and bakeries nearby offering
similar products such as doughnuts and bagels. A rating of Excellent or Good is accepted as these options may not be an exact match for the query but are closely related and likely to satisfy user intent.
Research
Example 7
In this task, the user is located outside the fresh viewport, so we use the viewport as location intent. Searching for ‘brook house’ on a map application returns multiple options nearby, which are consistent with the suggestions returned by TryRating. Our suggestion for ‘Brookhouse Farm Riding School’ is further away than multiple options but is also a more prominent POI, so a rating of Good or Acceptable is correct in this case.
Research
The prominence of a suggestion should always be considered, just because it’s further away doesn’t necessarily mean it should receive a lower rating, as per AC GLs 4.3.3.
Example 8
For this task, the user is located outside the fresh viewport, so we use the viewport as location intent. Copying the viewport centre co-ordinates to a map application and search for ’40 ki’ shows us there are multiple possible options, including one address nearby not returned by TryRating – it’s important to consider all real-world options when rating, not just those returned. This suggestion should be demoted to Acceptable or Bad for distance issue, as there are multiple options nearer to the fresh viewport
Research
