Always rate against the real world
When evaluating Relevance, we first need to determine the user intent. Then, we conduct research to find the best matching results for the query, and evaluate their distance to the user intent. Once we have this information, we can determine if there are closer results that match the user intent. If better results are not shown, we demote the existing result(s) in relation to the missing one(s), GL 1.3.3. Result Relevance Rating.
Query: Marionnaud
Viewport: Fresh, over 8200 Gleisdorf
Result: Marionnaud, Bürgergasse 8, 8330 Feldbach, Österreich
This is a chain business query. The viewport is fresh; results are expected inside/near the viewport, GL 2.3.2. Implicit Location. Our result is the only one listed in the task. As explained above, we rate against the real world and conduct
research to determine the relevance of our result. To do so, we copy the viewport coordinates into a maps application. Now, we can establish our Excellent, Good, and Acceptable zones based on the number and distribution of possible results in
the real world and the environment, GL 5.4. Distance.
We can see that there are possible real world results not listed in the task. One of these results is inside the viewport, which would be rated Excellent. Moreover,
there are several other results roughly the same distance from the viewport; our result is among them. In this semi-rural environment, we are more lenient in terms of distance, so we include all of these results in the Good zone. Final Relevance rating: Good. The store to the top right falls in the transition zone between Good and Acceptable, so both ratings would be viable.
In urban environments with many possible results (GL 5.6), we are less lenient in terms of distance and distance differences as compared to scenarios with fewer possible results or such in rural environments (GL 5.7 to 5.9). In an urban environment with many possible outcomes, we apply a stricter standard to distances than would be the case with few outcomes.
Query: Tankstelle
Viewport: Fresh with user inside
This is a category query. The viewport is fresh with the user inside: Results are expected near the user, GL 2.3.2. Implicit Location. The task contains three results rather close to the user.
As usual, we determine where the user is located by clicking on the user icon and copying the coordinates into a maps application. We can see that there are many possible results available. One of these is closer to the user than any other.
While there is not a large difference in distance to the user, it’s significant enough
in this scenario to demote our results to Good. Final Relevance rating for all three results: Good.
When there is an explicit location mentioned in the query, we ignore the user and viewport locations. The user has told us exactly where they want to find results, GL 2.3.1. Explicit Location.
E59
Query: woolworth mürzzuschlag
Viewport: Fresh
Result: Woolworth, Ruster Straße 82-104, 7000 Eisenstadt, Österreich
This is a chain business query with a general location modifier, GL 10.6.3.1.
General Location Modifier. Researching the query, we find one Woolworth store in the requested location. Since it’s the only store in this location, it receives a
Navigational rating. In this scenario, results outside the requested location are demoted from Good to Bad. There are few stores outside the requested location, so we are more lenient in terms of distance. Two stores are roughly the same distance from the requested location; these would be rated Good. Our result, although far away, is still a valid result in this scenario. Final Relevance rating: Acceptable.
When the viewport is stale and the user location is present, consider only the user
location as location intent, GL 2.3.2. Implicit Location.
Query: Tankstellen
Viewport: Stale
The user is looking for gas stations. The viewport is stale and the query does not contain a location modifier; thus, the user location is the location intent. We establish our zones as usual. There are only marginal differences in distance to the user for the three closest results, which all fall in the Excellent zone. The results to the top right and bottom left both fall in the Good zone.
When the user is within a fresh viewport, take the user location as the location intent. Results inside the area cannot be rated Bad because of distance alone, GL 2.3.2. Implicit Location. However, this does not mean that results outside the fresh viewport cannot be rated Bad even if there are results inside the viewport that are rated Acceptable while being farther away from the user.
Query: Lebensmittel
Viewport: Fresh with user inside
In this scenario, result #1 is significantly farther away from the user than result #2. However, since it’s still inside the fresh viewport and matches the query, it is rated Acceptable. Result #2 outside the fresh viewport would be rated Bad, though, because there are plenty possible results closer to the user; the exception for results inside the fresh viewport does not apply to it.
Two factors determine the final name rating: Name of the POI/business and classification. When the classification is wrong, the final Name Accuracy rating is always Incorrect. However, if the classification is missing or listed as N/A, the final rating should not be demoted, GL 6.3. Result Classification.
Partially Correct Name
According to the section 6.2.2. of the guideline, a partially correct name differs from the official versions but can still be recognized by the user. The name can have minor/moderate misspelling, service level mismatches and missing or unnecessary name parts, including legal entity names (like GmbH, AG, KG, OHG,
GmbH & Co. KG, etc.), as well as unexpected use of upper/lower case. The table below shows some examples for minor/moderate misspellings that are Partially Correct. Please note that in some cases, they might not be easy to catch, for example the difference between i and l, especially if you lose focus. Please be sure to take regular breaks when rating.
| Result Name | Official Business name | Classification | Reason |
|---|---|---|---|
Lldl | Lidl | Discounter | Result name uses l instead of i |
Hofer | HOFER | Missing | Business consistently uses all caps |
ElectronicPartner GmbH | ElectronicPartner | Elektronik | Corporate structure not used for stores |
Clever fit | clever fit or CLEVER FIT | N/A | Unexpected use of upper/lower case |
DM drogerie markt | dm drogerie markt | Apotheke | While the name is Partially Correct due to unexpected use of upper/lower case, the overall rating is Incorrect since the classification is wrong |
Street names
Misspellings in the street name should be rated Incorrect, GL 7.1.3. Street Name.This includes minor misspellings, including missing hyphens or dots, as well as incorrect street directions and types.
Non-mandatory address components
The table in CSGL 3.1 Address Expectations for Results lists the address requirements for the various result types. Not all of the components are mandatory for all of the result types. If non-mandatory components are returned, they need to be correct, however. This also applies to POIs without an expected address.
Example: Linz Hbf, 4010 Linz, Österreich
A full street address is not required for transit stations. The locality must be returned whenever a POI is small enough to fit entirely within it, GL 8.3.2.1. Minimum Address Component. On the official website, we can verify that the nonmandatory postal code is incorrect. The correct postal code is 4020; this is also confirmed by geoportal.de. Final Address Accuracy rating: Incorrect – postal code.
Postal codes and (sub-)localities There are different result expectations for postal codes. We differentiate between localities with only a single postal code and ones with multiple ones. If the query is for a locality, which has only one postal code, we need to return the postal code in the result. If we do not show the locality with the correct postal code, the address accuracy should be rated Incorrect – postal code.
This also applies to smaller, independent villages that share a postal code with other localities. As long as the locality only has one postal code, it needs to be returned.
Result Address | Standard Address | Rating |
Fiss Tirol Österreich | 6533 Fiss Österreich | Incorrect – Postal Code: The locality has a single postal code, which we should return in our result. |
Klagenfurt Kärnten Österreich | Klagenfurt Österreich | Correct: Klagenfurt has multiple postal codes, so it is correct to not return one in our result. |
9928 Heinfels Tirol Österreich | 9919 Heinfels Österreich | Incorrect – Postal Code: We display the incorrect postal code for the locality. |
For sub-localities (or neighborhoods in larger cities), we also require the postal code to be included in the result. We need to return the name of the sub-locality and the associated postal code with the larger locality.
Result Address | Standard Address | Rating |
Pradl Innsbruck Tirol Österreich | Pradl 6020 Innsbruck Österreich | Incorrect – Postal Code: The locality has a single postal code, which we should return with the larger locality in our result. |
Heiligkreuz Sölden Tirol Österreich | Heiligkreuz 6450 Sölden Österreich | Correct: Heiligkreuz is a smaller village within Sölden. We need to include the postal code for the larger locality. |
Meidling Österreich
| Meidling 1120 Wien Österreich — | Incorrect – Locality, Postal Code: We should include the larger locality, as . . . . , . Meidling is a part of Wien, and include the postal code, since Meidling only has a singular postal code. |
Sometimes, it can be difficult to determine Next Door and Approximate pin ratings. In particular, this concerns scenarios with multiple street numbers under a shared rooftop, and pinpointing the exact location of stores in shopping malls. We always use the best available evidence in such cases. The more evidence that can be found
to verify a feature’s location, the more precise the pin’s location must be in order to be rated Perfect. Examples of strong evidence are, e.g., street imagery or floor plans.
Result Type | Perfect | Next Door |
Single Rooftop | Pin falls on the intended rooftop | Pin falls on the next door property |
Example type: Multiple street numbers under one rooftop without shared parking, GL 9.2. Single Rooftop | ||
Result Type | Perfect | Approximate |
Store in a Shopping Mall | With floor plans, street imagery or other strong evidence, the exact location of the result under the shared rooftop can be confirmed | The rest of the connected rooftop and parcel, including all other rooftops that are not the intended one, if they exist |
Example type: One building with single street number and multiple businesses under the rooftop, GL 9.2. Single Rooftop | ||
Incorrect address listed in the result
In some cases, the listed address might contain an incorrect street number.
According to GL 9. Pin Accuracy, the pin should be a reflection of the result: The location represented by the pin should reflect the result. Pin ratings should be rated individually and not influenced by address or other data. A pin can be correct when other rating components are wrong. In other words: The pin should always fall on the actual location of the POI.
Example: POI is listed as located on Hauptstraße 2. Research shows that the correct address is Hauptstraße 4. If the pin falls on Hauptstraße 2, it is rated Next Door. If it falls on Hauptstraße 4 – the correct address – it is rated Perfect.