This document describes some of the most common mistakes observed in AC relevance tasks. It provides important information and useful tips on how to avoid making the same mistakes.
The goals of this task are:
● Todetermine how well the suggestions satisfy the user’s query (relevance)
● Tocheck the data accuracy (name and address) of the suggestion
Relevance for each suggestion is found by assessing:
● Matching Suggestion
● User Intent
● Distance/Prominence
General Rating Rules:
● Rate each suggestion individually and demote based on relevance and distance.
● Always rate against the real world: If there is a better suggestion but it is not shown, demote the existing suggestion(s) in relation to the missing one(s).
● Ignore the order of the suggestions and evaluate each one individually
● Ignore duplicate suggestions and rate them individually. Do not demote for repetition.
● If there are suggestions that are extremely inappropriate or illegal, rate Bad. Identify such content by determining if the suggestion is likely to be embarrassing if seen by users.
1.Determine whether the suggestion meets the primary intent
| User intent | Definition | Highest Initial Rating |
| Primary Intent | A suggestionthat isuseful to theuser, holds significant prominence or directly satisfies a possible category intent of the query string. There can be more than one primary user intent suggestion perquery string. | Excellent |
| Secondary Intent | A suggestion that is not as prominent as other potential suggestions or satisfies the intent of a possible category query string to a lesser degree. | Good(UserIntent) |
| Unlikely Intent | A suggestion that matches the query but is very unlikely to be useful to the user. | Acceptable(UserIntent) |
| Non-RelevantIntent | There are issues that make the suggestion useless for the user, such as no match to the query string. | Bad(UserIntent) |
1.1 Example 1
Query: 大金
Viewport Age: Fresh
Suggestion 1: 大金新苑
Suggestion 2: 大金新苑(公交站
In this example, the user is not within the viewport, and all two result suggestions are within the viewport. Suggestion 1 matches the query and it is rated as Excellent. Suggestion 2 is a bus stop of this POI, which meets the user’s secondary transit intent, so it is rated as Good.
2. Consider the suggestion in the context of all possible suggestions in the real world
2.1 Example 1
Query: wu xiao
User Location: 38.47412109375,106.23779296875
Viewport Age: Fresh
Viewport Centre: 38.4668639567695,106.249786936918
Suggestion 1: 五小所
Suggestion 2: 西夏区第五小学
Suggestion 3: 武小二肝子凉皮
Suggestion 4: 吴小红棋牌室
In this example, there is a result near the viewport that matches the query but did not return, which is the fifth primary school in Jinfeng District (金凤区第五 小学). Therefore, when rating the other results, it is necessary to consider this non-returned result. For example, suggestion 3 is also relatively close to the
viewport but is not as prominent as this school, so it can be demoted to Good. Suggestion 2 and Suggestion 4 can also be demoted to Acceptable considering the presence of closer results.
2.2 Example 2
Query: 新世纪cai
User Location: 34.73876953125,113.66455078125
Viewport Age: Fresh
Viewport Centre: 34.7560711162156,113.684871370157
Suggestion 1: 新世纪菜市场
In this example, at first glance, Suggestion 1 seems to be the nearest result that matches the query. However, upon closer inspection, we discover that there is a prominent result within the viewport: 新世界百货.We have to take this result into consideration as it is possible that the user’s query was an accidental input of “新世纪bai”. Therefore, although Suggestion 1 follows the original query, it has to be demoted due to its distance and prominence.
2.3 Example 3
Query: wei feng
User Location: 34.453125,120.234375
Viewport Age: Fresh
Viewport Centre: 34.4515505075418,119.545966504907
Suggestion 1: 圩丰镇
Suggestion 2: 圩丰镇人民政府
Typically, Chinese users won’t look for local government with initial input starting with the locality name. However, in this case, the town and viewport are small so user intent is more likely to be matched. Good or Excellent is accepted for Suggestion 2
2.4 Example 4
Query: 吴江全纳
User Location: 31.13525390625,120.65185546875
Viewport Age: Fresh
Viewport Centre: 31.1617541350159,120.680709988969
Suggestion 2: 苏州全纳铝业科技有限公司
Considering there are very few results in the real world, Suggestion 2 is eligible for Excellent as the location modifier matches.