Rating for Relevance
Goal of Task:
No pin evaluation
Relevance: This is established on the following: matching suggestions, intent, and distance/prominence.
Determining how well suggestion match the user’s potential intent:
Query and Category Suggestions: (Research steps same as above)
AutoComplete Comments
Issue Identified | Expeictations | Example Rating and Comment |
|---|---|---|
Relevance Demotions | The comment should explain the reasoning behind your rating. For example, a demotion to a suggestion may be due to distance from the user’s location compared to similar suggestions/ potential suggestions, to secondary intent, and so on. | Relevance -> Good (Distance/ Prominence Issue) Comment: Demoted -1 for a few closer real world locations in the Middletown/Anchorage area |
Name and Address Issues | It is important to include a reason when rating Name or Address something other than Correct. This also applies to Business closed/does not exist. — | Address Accuracy -> Incorrect Comment: Correct street number: 332 W El Camino Real, Sunnyvale, CA ^ 87, see official webpage wmy.starbucks.coni |
Comments are mandatory for any relevance rating of Good or below as well as for any Name and Address Accuracy issues not rated Correct and should explain the reasoning behind your rating. Be sure to:
Include user intent: It’s usefuI to indicate what you think the user intent is. In Autocomplete the cleery is sometimes too short to indicate the precise intent.
Refer to the guideline:>: If the demotion is due to a guideline instruction, note the {guideline section that guided your decision
Include sources: If you find incorrect information in a suggestion, your comment should include the correct information and its source. Be sure that any links you include lead directly to the information you provide. If they don’t, include a link to the source you used and explain how to get to the information frcmi there.
Use URL shorteners when inse•rting long links into your comments.
Be concise: Don’t write elaborate comments. Be short and specific.
Comment in English: Your comments will sometimes be reviewed or used by people who are not familiar with your test locale or its language. Comments in English are understandable to the widest possible audience.
General Rating Rules
Rating Relevance – Autocomplete
If the user’s location intent is not expressly stated in the query, for example [chinese] or [479 margarita ave], use the user location, viewport location, and viewport age to determine the area of expected results.
Viewport | User | Location Intent |
Fresh | Inside Viewport | When the user is within the fresh viewport, take the user location as location intent. Suggestions are generally expected in or near the viewport and suggestions inside the area cannot be rated Bad for distance alone. |
Outside Viewport | Suggestions ere expected in or near the viewport area. All relevant suggestions inside the viewport are eligible for a rating of Excellent. If no suggesticns can be found in or near the viewport, use the user location as ii secondary location intent. | |
Missing | When the user is missing, the viewport remains the location intent. | |
Stale | Inside Viewport | When the viewport is stale, consider only the user location as location intent. |
Outside Viewport | ||
Missing | Use the stale viewport as location intent when the user is missing. . | |
Viewport Age Missing | Present or Missing | Consider the viewport fresh when the viewportageis missing. |
Missing entirely | Present |
The user location sets th«s location intent when the viewport is not present. |
Missing | When the user and viewport are missing, the test locale becomes location intent with a strong focus on prominent suggestions. |
User Intent | Definition | Highest Initial Rating |
Primary Intent | A suggestion that is useful to the user, holds significant prominence, or directly satisfies a possible category intent of the query string. There can be more>than one primary user intent suggestion per query string. | Excellent |
Secondary Intent | A sugg estion that is ntit as prominent as other potential suagestions or satisfies int< jnt of a possible category query stringtoalesser degree. | Good (User Intent) |
Unlikely Intent | A suggestion which matches the query but is very unlikely to be useful to the user. | Acceptable (User Intent) |
Non-Relevant Intent | There are issues that make the suggestion useless for the user, such as no match to the query string. | Bad (User Intent) |
Query String | Suggestion | Intent | Highest Initial Rating | Explanation |
[Hollywoo] User located in NY | Hollywood, CA | Primary | Excellent | This is an internationally known locality and should be considered a primary intent suggestion. |
The Hollywood Sign Los Angeles | This is an internationally known landmark and should be considered a primary intent suggestion. | |||
Hollywood Heights, CA | Secondary | Good | This suggestion is for a s lightly less well-known reighborhood and not as prominent as other potential suggestions. | |
Hollywood Fwy | Unlikely Intent | Acceptable | Although this suggestion is a major highway in the area it is not internationally or nationally known. This suggestion would unlikely be useful to the user located on the other side of the country. | |
Arclight Hollywood 6360 W Sunset Blvd, | NonReleva nt Inkmt |
Bad | This suggestion matches the query string on the location modifier part of the POI name which is nonessential. Since the core brand name is ‘Arclight’ this suggestion should not be considered a match. |
The rating determined by deciding how well potential intent is met is called Initial Rating. Distance
and prominence will be considered next in order to establish the final rating.
The user has issued a misspelled and slightly ambiguous query. Research shows us that Croke Park is a large sports stadium in Dublin city and is not a
locality but still a location modifier. We can interpret ‘ho’ in several ways but the most likely is Hotel. Further research shows us there is a hotel called The
Croke Park Hotel adjacent to the stadium. A highly likely interpretation of this query is that this user is either looki for this particular hotel or other hotels nearby.
We are evaluating result 2. This is a category suggestion for Hotels with the instruction ‘Search nearby’. As this is our primary interpretation of this query, we can assign a rating of Excellent to this result.
This is an incomplete query and a little ambiguous at first glance. There are many localities in Ireland that begin this way. On examining the area inside the fresh viewport though we see it is centered on the village of Killenard, so it is highly likely the user is looking for results in this locality as it’s matching the query but not in the expected order, it could still be what the user is looking for and this should therefore be rated Good or Acceptable but Excellent also accepted.
Review the query string. The suggestion ”Annacotty” matches the query string for the query “anna”. Understanding User Intent. Before evaluating the
suggestions, you will need to understand what the user wanted to find when they made the query. To determine user intent, use a combination of your own research and local knowledge plus the information provided in the rating interface, such as user location, user viewport and viewport age.
The viewport is fresh, and the user is outside the fresh viewport. According to 2.3.2 Implicit Location “Suggestions are expected in or near the viewport area.”
Place the VP center coordinates into an online map application then zoom out to give a wider area to search.
Copy the query from the rating interface and paste into the map
application. The dropdown list will show suggestions close to the viewport
and gives a clue to possible user intent.
According to 4.3.2.Distance: Generally, the farther away the suggestion is from the location intent, the less desirable it becomes. When the user is outside the fresh viewport, the fresh viewport should be used as the location intent. All ratings made should be based on the fresh viewport location.
This is matching the query but it’s very far away from where the user is looking and it’s not specifically prominent, this should therefore be rated Acceptable or Good.
Example 4
Review the query string. The suggestion ”Bridgetown” matches the query string for the query “bridge”.
Before evaluating the suggestions, you will need to understand what the user wanted to find when they made the query. To determine user intent, use a combination of your own research and local knowledge plus the information provided in the rating interface, such as user location, user viewport and viewport age.
The viewport is fresh, and the user is inside the fresh viewport. According to 2.3.2 Implicit Location “When the user is within the fresh viewport, take the user location as location intent. .”
Place the Centre VP center co-ordinates into the map application and zoon out to give a wider area view.
Center User location
Center Resul
Research
Copy the query from the rating interface and paste into the map application. The dropdown list will show suggestions close to the viewport and gives a clue to possible user intent.
According to 4.3.2.Distance: Generally, the farther away the suggestion is from the location intent, the less desirable it becomes. When the user is inside the fresh viewport, the user should be used as the location intent. All ratings made should be based on the user location.
This result is quite far from the viewport and not the most prominent but it’s sill matching, this should therefore be rated Acceptable or Good.
Research
Place the Centre Viewport co-ordinates into an online map application and Zoom out to give a wider area to search.
Copy the query from the rating interface and paste into the map application. The dropdown list will show suggestions close to the viewport and gives a clue to possible user intent.
According to 4.3.2.Distance: Generally, the farther away the suggestion is from the location intent, the less desirable it becomes. When the user is outside the fresh viewport, the fresh viewport should be used as the location intent. All ratings made should be based on the fresh viewport location.
This is matching the query and could still be what the user is looking for. It’s very far away but since it’s matching it should be rated Acceptable or possibly Bad.
Example 7
Review the query string. The suggestion ”Irishtown Stadium” matches the query string for the query “irish”. Before evaluating the suggestions, you will need to understand what the user wanted to find when they made the query. To determine user intent, use a combination of your own research and local knowledge plus the information provided in the rating interface, such as user location, user viewport and viewport age.
The viewport is fresh, and the user is outside the fresh viewport.
Location “Suggestions are expected in or near the viewport area.” Place r the Centre Viewport co- * ordinates into an online map application and zoom out to give a wider search
Copy the query from the rating interface and paste into the map application. The dropdown list will show suggestions close to the viewport and gives a clue to possible user intent.
According to 4.3.2.Distance: Generally, the farther away the suggestion is from the location intent, the less desirable it becomes. When the user is outside the fresh viewport, the fresh viewport should be used as the location intent. All ratings made should be based on the fresh viewport location.
This is a fairly clear query, where we can assume what the user is looking for. This could be what the user is looking for and should therefore be rated Excellent or Good.