According to section 1.3.3. Result Relevance Rating, the relevance rating will take into consideration user intent as well as distance/prominence. Relevance is always rated independently of any data (name, address, or pin) inaccuracies under the assumption that the result exists and is correct. Relevance ratings can be given starting from Navigational (if applicable) to Bad.
Remember that for any rating below Good, we need to select either User Intent issue or Distance/Prominence issue. And please don’t forget that leaving a comment is mandatory.
Steps for Rating Relevance
First, we need to understand the user’s intent. To do this we need to use research and local knowledge and the information provided such as the user viewport, location and viewport age. We also need to decide what the location intent is for the query. Is there an implicit or explicit location modifier? We can use official resources like businesses’ official websites, national postal service websites, and government websites to confirm the user intent.
Decide what query type it is. Is it an address query, POI, Business, or Category query?
Finally, we can review the result of the query to understand if it would satisfy the user’s intent. By doing this we would need to use the information we have gathered from the first step and use real world research to determine the relevance of the result that is returned.
If the user’s location intent is not expressly stated in the query, for example [chinese] or [479 margarita ave], use the user location, viewport location, and viewport age to determine the area of expected results.
Viewport | User | Location Intent |
Fresh | Inside Viewport | When the user is within a fresh viewport, take the user location as location intent. Results are generally expected in or near the viewport, and results inside the area cannot be rated Bad because of distance alone. |
Outside Viewport | Results are expected in or near the viewport area. All relevant results inside the viewport are eligible for a rating of Excellent. If no results can be found in or near the viewport, consider the user location a secondary location intent. | |
Missing | When the user is missing, the viewport is the location intent. | |
Stale | Inside Viewport | When the viewport is stale, consider only the user location as location intent. |
Outside Viewport | ||
Missing | Use the stale viewport as location intent when the user location is missing. | |
Viewport Age Missing | Present or Missing | Consider the viewport fresh when the viewport age is missing. |
Missing entirely | Present | _ The user location sets the location intent when the viewport is not present. |
Missing | When the user location and viewport are missing, the test locale becomes the location intent with a strong focus on prominent results. |
5.1.8. Lack of Connection |
Previously the guidelines contained the concept of ‘Associated Entities’. The concept of Associated Entities has been removed and no longer applies. This means that a result cannot satisfy user intent only because of its containment relationship with the POI/business in the query. Therefore, according to the guidelines section 5.1.8. Lack of Connection, if the result does not satisfy the user intent either because there is no relationship between the query and the result or because the connection between the query and the result will not be immediately obvious to the user, rate the relevance Bad. | [airport] | Santa Cruz Boardwalk Santa Cruz | None | Does not satisfy user intent: Research shows that there is no connection between the query and the result. Rate Bad. |
[Raging Waters 2333 South White Rd San Jose] | 2333 South White Rd San Jose | None | Does not satisfy user intent: Result is for the correct address, but does not include the business named in the query, so users will have no way of knowing if this is truly connected to the business or not. Rate Bad. | |
[Raging Waters] | 2333 South White Rd San Jose | None | Does not satisfy user intent: Query is the name of a business and result is for the correct address, but does not include the business name. Users will have no way of knowing whether or not this address is connected to the business they asked for. Rate Bad. | |
[valley fair mall] | Macy’s 2801 Stevens Creek Blvd, Santa Clara | None | Does not satisfy user intent: Result is for a store, not a mall. This store is inside the requested mall, but does not satisfy the intent of the query, which is the whole mall. Rate Bad. | |
[macy’s] | Westfield Valley Fair 2855 Stevens Creek Blvd, Santa Clara | None | Does not satisfy user intent: Result is for a mall, not a store. This mall contains the requested store, but this result is the whole mall, not the single store the user asked for. Rate Bad. | |
[costco] | Costco Gasoline 1601 Coleman Ave, Santa Clara | None | Does not satisfy user intent: Result is for a Costco gas station, not a Costco store. The store and the station are at the same address and share a brand name, but the result is not what the user asked for and does not satisfy the query intent. Rate Bad. | |
[costco gas]
| Costco 1601 Coleman Ave, Santa Clara
| None
| Does not satisfy user intent: Result is for a Costco store, not a Costco gas station. The store and the station are at the same address and share a brand name, but the result is not what the user asked for and does not satisfy the query intent. Rate Bad.
|
Example 1 – 5.12. Partial Address Does Not Exist
Viewport Centre: 1.355960,103.931809
User Location: 1.354908,103.933897
Query: Blk 302 Tampines
Result: 302 Tampines Avenue 4 Singapore
This is an address query, where the result does not match exactly to the query. In this case, we need to refer to official resources, like OneMap https://www.onemap.sg/main/v2/ to verify the result address.
Location and Real-World Result
With some research on we can confirm that the result address “302 Tampines Avenue 4 Singapore” does not exist. The only existing address that matches this query is 302 TAMPINES STREET 32 SINGAPORE 520302, which is further away from the user;
https://share.onemap.sg/aUolWb. According to Guidelines 5.12. Partial Address Does Not Exist; we can see that example [2373 Chestnut Street, Redwood City] is a non-existent address. However, it gets a rating of Excellent, as it is closer to the user location than the other address that does exist [2373 Chestnut Street, San Francisco]. (Please note that name, address and pin accuracy issues do not affect the relevance rating). So, applying the same concept, this result should be rated Excellent for relevance.
Example 2 – 10.6.3. Chain Business with Location Modifier
Viewport Centre: 1.315225,103.794974
User Location: 1.300408,103.838401
Query: genki sushi orchard
Result: Genki Sushi 201 Victoria Street Singapore 188067
The user has queried for “genki sushi orchard”, a chain business with location modifier.
According to Guidelines 10.6.3. Chain Business with Location Modifier, we should ignore the user and viewport locations in this task.
Orchard is a General Location Modifier, as it is a locality/area (Guidelines 10.6.3.1). Any result that is inside the specified location should get either a Navigational or Excellent rating (based on the number of results), and results outside the requested location are demoted based on the distance to the location, and how many outlets of the chain business exist in the real world.
Location and Real-World Results.
On the business’ official website https://www.genkisushi.com.sg/lo cate-us/, we can see that there are only two outlets within Orchard, which are at Orchard Central and Ngee Ann City. This result is in Bugis, which is slightly outside of Orchard area. As per the guidelines, if there are only a few results within the specified area, results outside it should not be demoted too harshly. Hence this result is demoted (-1) to Good (Please see Aldi waco tx example).
Example 3 – 5.7. Few Possible Results
Viewport Centre: 1.303596,103.780591
User Location: 1.311339,103.720011
Query: Jack’s Place
Result: Jack’s Place 50 Jurong Gateway Road Singapore 608549
The user, who is outside the fresh viewport, has queried for “Jack’s Place”, a chain restaurant. As per Guidelines 2.3.2. Implicit Location, we will use the viewport as the location intent. The result here is located outside the viewport.
Location and Real-World Results
We must first check for other possible real-world results in the area.
Through research on the business’ official website https://www.jacksplace.com.sg/locati ons/ and mapping services, we can confirm that there are two outlets located within the fresh viewport. These outlets are eligible for a rating of Excellent (Guidelines 2.3.2). And according to Guidelines 5.7. Few Possible Results, if the query has only a few results within the fresh viewport, then we should be more lenient when considering distance. As this result is just outside the fresh viewport; it should be demoted (-1) to Good for distance.
Example 4 – 5.14. Unexpected Results
Viewport Centre: 1.323581,103.845110
User Location: 1.325912,103.844833
Query: Serangoon
Result: Serangoon Garden Market & Food Centre
49A Serangoon Garden Way
Singapore 555945
The user’s query is for Serangoon, a specific neighbourhood or MRT station located in the North-East Region of Singapore. And the result returns a
market and hawker centre managed by National Environment Agency (NEA); https://www.nea.gov.sg/our- services/hawker-management/overview. Although the result is located at Serangoon, it is rated Bad, as this is a random business unrelated to the query and is very unlikely to satisfy the user intent. Please see Guidelines 5.14. [Soho Mildreds Restaurant] example.
Example 5 – 5.4. Distance
Viewport Centre: 1.309601,103.853063
User Location: 1.320210,103.860027
Query: DBS bank branch
Result: DBS Bank – Plaza Singapura
Branch
68 Orchard Road
#B1-25 Plaza Singapura
Singapore 238839
The user is within the fresh viewport; hence user location is the location intent.
As the query is for DBS bank branch, we should log on to the bank’s branch locator page https://www.dbs.com.sg/index/locator.page so as to determine their locations.
Location and Real-World Results
Research shows that there are a few branches located near to the user, and the
nearest ones are the Towner Road and Toa Payoh branches. This result is slightly further away than those two branches, therefore it is demoted to Acceptable according to its distance to the user location.
Unlike the Bank of America example (Guidelines 5.18), it is not a query of the general bank brand but is specifically looking for bank branches.
Example 6 – 10.6.3. Chain Business with Location Modifier
Viewport Centre: 1.309601,103.853063
User Location: 1.320210,103.860027
Query: DBS bank branch Result: DBS Bank
520 Lorong 6 Toa Payoh Singapore 310520
This is another result from the same task as above. In this case research on the bank’s branch locator page https://www.dbs.com.sg/index/locator.pa ge shows that this result is the second nearest DBS bank branch to the user location, therefore this should be rated as Good.
Example 7 – 5.1.8. Lack of Connection
Viewport Centre: 1.840103,103.811312 User Location: 1.384277,103.864746 Query:
Mayflower Secondary School 2 Ang Mo Kio Street 21
Singapore 569384
Result: 2 Ang Mo Kio Street 21
Singapore 569384
The user has queried for a specific POI with its full address. The result returns with the queried address “2 Ang Mo Kio Street 21” but does not include the POI name “Mayflower Secondary School”. Due to this missing information, the user will have no way of knowing if this result address is truly connected to the POI. And when the result seems to have no connection to the query, it becomes irrelevant to the user. Hence, it is rated Bad for relevance. Please see Guidelines 5.1.8. Lack of Connection [Raging Waters 2333 South White Rd San Jose] example; and 10.6.1. NonChain Businesses [Gary Danko] example.
Example 8 – CSGL 3.2.1. Rating rules for locality queries
Viewport Centre: 1.336086,103.794366
User Location: 1.337585,103.793335
Query: Holland Village Result: Holland Village Singapore
The user’s query is for Holland Village, and the result points to the Holland Village neighbourhood. As it is a locality query, we should always determine whether there is a MRT station with an exact same name. Through further research, we can confirm that there is one station named “Holland Village MRT station”;
http://www.smrt.com.sg/Journey-with- Us/Trains/NetworkMap/HollandVillage.
Hence, as per CSGL 3.2.1. Rating rules for locality queries, both the locality and the MRT station should get an Excellent rating. This is due to the prominence of MRT stations in Singapore.