Always rate against the real world
When evaluating Relevance, we first need to determine the user intent. Then, we conduct research to find the best matching results for the query, and evaluate their distance to the user intent. Once we have this information, we can determine if there are closer results that match the user intent. If better results are not shown, we demote the existing result(s) in relation to the missing one(s), GL 1.3.3. Result Relevance Rating.
Query: Rossmann
Viewport: Stale
Result: Rossmann, Helene-Weigel-Weg 9, 48165 Münster
This is a chain business query. Since the viewport is stale, results are expected near the user, GL 2.3.2. Implicit Location. Our result is the only one listed in the task. As explained above, we always rate against the real world and conduct research to determine the relevance of our result. To do so, we first determine where exactly the user is located. We click on the user icon and copy the coordinates into a maps application. We can see that there are possible real world results closer to the user.
Now, we can establish our Excellent, Good, and Acceptable zones based on the number and distribution of possible results in the real world, as well as the environment, GL 5.4. Distance.
There is one store next to the user in the Excellent zone. Two more are located inside the Good zone, with a third one in the transition zone between Good and Acceptable. Along with several other stores, our result is in the Acceptable zone. Final Relevance rating: Acceptable.
In urban environments with many possible results (GL 5.6), we are less lenient in terms of distance and distance differences as compared to scenarios with fewer possible results or such in rural environments (GL 5.7 to 5.9). In an urban environment with many possible outcomes, we apply a stricter standard to distances than would be the case with few outcomes.
Query: Tankstelle
Viewport: Fresh with user inside
This is a category query. The viewport is fresh with the user inside: Results are expected near the user, GL 2.3.2. Implicit Location. The task contains three results rather close to the user.
As usual, we determine where the user is located by clicking on the user icon and copying the coordinates into a maps application. We can see that there are many possible results available. One of these is closer to the user than any other.
While there is not a large difference in distance to the user, it’s significant enough in this scenario to demote our results to Good. Final Relevance rating for all three results: Good.
When there is an explicit location mentioned in the query, we ignore the user and viewport locations. The user has told us exactly where they want to find results, GL 2.3.1. Explicit Location.
Query: woolworth rotenburg an der fulda
Result: Woolworth, Obere Königsstraße 13, 34117 Kassel
This is a chain business query with a general location modifier, GL 10.6.3.1. General Location Modifier. Researching the query, we find one Woolworth store in the requested location. Since it’s the only store in this location, it receives a Navigational rating. In this scenario, results outside the requested location are demoted from Good to Bad. There are several stores outside the requested location that are closer to it. Final Relevance rating: Acceptable.
Result: Bahnhof Niederstotzingen, Niederstotzingen
A n on-transit POI can be associated with a transit POI with the same name. You can establish the re laitioriship between the two entities by using local knowledge, researching pnominenee,and understanding how well the result satisfies user intent. This way you will learn whether the; query has a distinct navigational intent or if the prominence of the two POIs that share a similar name is so strong that both can be the intent.
The criteria described above should be applied to understand the transit queries. Additionally, if it is determined that a query has a clear navigational intent, all other results will be Bad (see [12th st Oakland bart] example below). If a result is promoted to Navigational, other results that could potentially satisfy the user intent should be demoted further for distance and prominence (see [BART daly city] and [stockport station] examples below). In general, the fewer the results that satisfy the user intent, the farther away the results can be and still be considered relevant. The more available results that can satisfy the user intent, the closer they need to be.
Query: Bahnhof Niederstotzingen
Result: Bahnhof Niederstotzingen, Niederstotzingen
The user is looking for train stations in/near Niederstotzingen. There is only one station in the requested locality. Final Relevance rating: Navigational.
Query: Bahnhof Niederstotzingen
Result: Bahnhof Rammingen, Rammingen
As explained above, if a result is promoted to Navigational, other results that could potentially satisfy the user intent should be demoted further for distance and prominence. Our result is located in a neighboring town and provides a choice of stations for the user. Final Relevance rating: Good.
Two factors determine the final name rating: Name of the POI/business and classification. When the classification is wrong, the final Name Accuracy rating is always Incorrect. However, if the classification is missing or listed as N/A, the final rating should not be demoted, GL 6.3. Result Classification.
Business/POI Name | Classification | Name Accuracy Rating |
Correct | Correct or Missing or N/A | Correct |
Correct | Incorrect | Incorrect |
Partially Correct | Correct or Missing or N/A | Partially Correct |
Partially Correct | Incorrect | Incorrect |
Incorrect | Correct, Incorrect, Missing or N/A | Incorrect |
Partially Correct Name
According to the section 6.2.2. of the guideline, a partially correct name differs from the official versions but can still be recognized by the user. The name can have minor/moderate misspelling, service level mismatches and missing or unnecessary name parts, including legal entity names (like GmbH, AG, KG, OHG, GmbH & Co. KG, etc.), as well as unexpected use of upper/lower case. The table below shows some examples for minor/moderate misspellings that are Partially Correct. Please note that in some cases, they might not be easy to catch, for example the difference between i and l, especially if you lose focus. Please be sure to take regular breaks when rating.
Result Name | Official Business Name | Classification | Reason |
Lldl | Lidl | Discounter | Result name uses l instead of i |
Aldi | ALDI | Missing | Business consistently uses all caps |
ElectronicPartner GmbH | ElectronicPartner | Elektronik | Corporate structure not used for stores — |
Street names
Misspellings in the street name should be rated Incorrect, GL 7.1.3. Street Name. This includes minor misspellings, including missing hyphens or dots, as well as incorrect street directions and types.
Listed street name | Official street name | Rating |
Dr-Abele-Weg | Dr.-Abele-Weg | Incorrect – Street Name |
Georg Ring-Weg | Georg-Ring-Weg | Incorrect – Street Name |
Kolonenweg | Kolonnenweg | Incorrect – Street Name |
Marktstraße | Marktplatz | Incorrect – Street Name |
Friedenspaltz | Friedensplatz | Incorrect – Street Name |
Saarstr | Saarstr. | Incorrect – Street Name |
Non-mandatory address components
The table in CSGL 3.1 Address Expectations for Results lists the address requirements for the various result types. Not all of the components are mandatory for all of the result types. If non-mandatory components are returned, they need to be correct, however. In particular, this applies to sub-localities for business/POI, full address, and street results, as well as postal codes for street results.
Example 1: Saturn, An der Römervilla 3, 56070 Koblenz, Niederberg
The official website does not list a sub-locality. Checking geoportal.de, we find that the correct sub-locality is Bubenheim. Final Address Accuracy rating: Incorrect – sub-locality.
Example 2: Siebenbrückenweg 1, 84034 Landshut, Achdorf
Checking geoportal.de, we find that the correct sub-locality is Nikola. Final Address
Accuracy rating: Incorrect – sub-locality.
Example 3: Dortmunder Straße, 48527 Nordhorn
Checking geoportal.de, we find that the correct postal code is 48529. Final Address
Accuracy rating: Incorrect – postal code.
Sometimes, it can be difficult to determine Next Door and Approximate pin ratings. In particular, this concerns scenarios with multiple street numbers under a shared rooftop, and pinpointing the exact location of stores in shopping malls. We always use the best available evidence in such cases. The more evidence that can be found to verify a feature’s location, the more precise the pin’s location must be in order to be rated Perfect. Examples of strong evidence are, e.g., street imagery or floor plans.
Result Type | Perfect | Next Door |
Single Rooftop | Pin falls on the intended rooftop | Pin falls on the next door property |
Example type: Multiple street numbers under one rooftop without shared parking, GL 9.2. Single Rooftop | ||
Result Type | Perfect | Approximate |
Store in a Shopping Mall | With floor plans, street imagery or other strong evidence, the exact location of the result under the shared rooftop can be confirmed | The rest of the connected rooftop and parcel, including all other rooftops that are not the intended one, if they exist |
Example type: One building with single street number and multiple businesses under the rooftop, GL 9.2. Single Rooftop | ||
It can be difficult to determine whether a business/POI has been closed/does not exist, or if it has been misplaced.
The Rater Portal contains a helpful document with examples for each of these scenarios. Please have a look at http://tinyurl.com/2s4dnwvz. While the branch is temporarily closed, http://tinyurl.com/ymu3km9u, we treat it as if it were open, GL 4.2. Business/POI is Closed or Does not Exist.