Austria – Rating tips for search tasks

The relevance rating will take into consideration user intent as well as distance/prominence. Relevance is always rated independently of any data (name/classification, address, or pin) inaccuracies. This means that when rating relevance we always assume that the result exists (even if research reveals the location is closed) and that the data presented is correct.

A good rating practice always includes a search to cover all possible real world results. Depending on the query type (category or chain business) conduct a map search (Google, Bing, etc.) for real world results not listed in the TryRating map. For chain business results, always use the official source (if available) for other possible real world results.

How to rate Relevance when the real world results are not in the task results

When evaluating Relevance, the first thing to do is to determine the user intent. Then, we need to research what the best matching results for the query are, and determine their distance to the fresh viewport/user location. Once we have this information, we can determine if the returned result is the closest possible result matching the user intent. If there is a better result but it is not shown, demote the existing result(s) in relation to the missing one(s).

Always rate against the real world: If there is a better result available but it is not shown,
demote the existing result(s) while considering the missing one(s).

Here, we have an example of a chain business query with many possible outcomes in an urban environment. The user is searching for BILLA stores and the user is inside the Fresh Viewport. We base our ratings according to where the user is located since the user is inside the fresh viewport. Results are expected to be close to the user.

The displayed result is a BILLA store located on Franzensbrückenstraße 20, 1020 Wien. As usual, we first determine where exactly the user is located. We click on the user icon and copy the coordinates into a maps application.

Example 1 – Chain Businesses Queries

After searching the BILLA stores on the user’s location, we discover that there is a branch in the immediate vicinity of the user and this nearest store was not listed among the results in the task. Since we always rate based on the real world, the branch that is closest to the user is rated Excellent.

In an urban environment with many possible outcomes, we apply a stricter standard to distances than would be the case with few outcomes. We see a whole set of stores that are a bit further away and that are all within the same zone. Each of these branches gets Good for Relevance. Thus, we group results based on a distance pattern that matches the scenario at hand. We see another set of branches further away which would be rated Acceptable. Therefore, the correct rating for our BILLA result located on ‘Franzensbrückenstraße 20, 1020 Wien’ will be Acceptable.

uas Kiai

In this scenario, the query is for dm. The user (red dot) is inside the fresh viewport. TryRating (left) shows only one result (green circle) close to the user’s location. Now we have to check at the official dm location page (right) for possible other real world results close to the user. We see that there are other results close to the user. These results change the initial rating for results further away (GL Sections 5.4 to 5.6, and10.6.2).

Example 2 – Chain Businesses Queries

In this scenario, the query is for dm. The user (red dot) is inside the fresh viewport. TryRating (left) shows only one result (green circle) close to the user’s location. Now we have to check at the official dm location page (right) for possible other real world results close to the user. We see that there are other results close to the user. These results change the initial rating for results further away (GL Sections 5.4 to 5.6, and10.6.2).

Example 3 – Category Queries

For Category queries, we use map services to conduct a general map search. In this example the query is for [restaurants]. The user (red dot) is outside the Fresh Viewport. TryRating (left) shows no results inside the Fresh Viewport (location intent), but one result near the user (green dot). The map search shows many real world results (yellow) inside the Viewport. As explained above, these results change the initial rating of results further away (GL Sections 5.4 to 5.6, and 10.7).

.

.

2. Name Accuracy 

.The Name Accuracy rating of any Search Task consists of two crucial components

A partially correct name differs from the official versions but can still be recognized by the user.

Partially correct names can include minor and moderate misspellings, service level mismatches, and missing or unnecessary name parts, including holding names/corporate structures. When the business name on the storefront does not include the corporate status (Ltd., Inc., etc.) seen in the result, it is partially correct.

Such errors would be minor misspellings, unnecessary or redundant name parts (e.g.

corporate structure GesmbH), and missing special characters or ALL CAPS writing (e.g. IKEA or BILLA).

2. Name Accuracy 

An incorrect result name is one that can’t be recognized because of severe misspelling or ambiguous or unnecessary/missing parts in the name.

When the classification is wrong, the final Name Accuracy rating is always Incorrect. This is true whether the result name is Correct, Partially Correct, or Incorrect.

We have to check both components for possible errors. First check the Name for spelling errors, then check the Classification.

If Name OR Classification are incorrect, the final Name Accuracy rating is Incorrect

Name Accuracy: How to rate if the result name is misspelled or incorrect or if the classification is incorrect or missing

The Name Accuracy rating consists of two components, which together will lead to the final name rating: Name of the POI/business, and Classification. When the Classification is wrong, the final Name Accuracy rating is always Incorrect. Keep in mind, though, that not all task results will have a classification and that you should not demote a result that has a missing classification.

Partially Correct Name

According to the GL Section 6.2.2, “a partially correct name differs from the official versions but can still be recognized by the user”. The name can have minor/moderate misspellings, service level mismatches and missing or unnecessary name parts, including legal entity names (like GmbH, AG, KG, OG, e.U., GmbH&Co KG etc. in Austria). The table below shows some examples of Partially Correct ratings.

2. Name Accuracy – Partially Correct Name

Always use official resources to determine Name Accuracy. We can confirm from the official website and the business logo that supermarket chains “BILLA” and “HOFER” use capital letters to communicate their company brand name to consumers. “KG” (Kommanditgesellschaft, limited partnership) is a legal corporate structure in German speaking countries. According to the GL, corporate structures do not need to be displayed.

Research shows that CLEVER FIT uses either all uppercase letters (CLEVER FIT) or all lowercase letters (clever fit) for their company brand name.

Lastly, the Lidl supermarket chains name was misspelled, but can still be recognized by the user.

Incorrect Name

According to GL Section 6.2.3, “an incorrect result name is one that can’t be recognized because of severe misspelling or ambigious or unnecessary/missing parts in the name.” When the classification is wrong, the final Name Accuracy rating is always Incorrect.

.

Users won’t be able to associate the name IEA with the IKEA brand. Therefore, Name Accuracy is Incorrect.

Similarly, users can’t recognize the “clever fit” line of gyms by the name “Clever” because there is discount brand called “clever” in Austria.

Lastly, the correct Classification for BIPA is “Drogerie” (drugstore).

Examples

Partially Correct:

Result name is bILLA – Classification: Lebensmittel→ Correct name is BILLA / Classification is correct → final rating – Partially Correct

Result name is Müller HandelsgmbH – Classification: Drogerie → Correct name is Müller (no corporate structure) / Classification is correct → final rating – Partially Correct

Result name is McDonalds – Classification: N/A→ Correct name is McDonald’s/ No Classification→ final rating – Partially Correct

Incorrect:

Result name is dm drogerie markt GmbH – Classification: Schuhe → Partially Correct name / Wrong Classification → final rating – Incorrect

Result name is Douglas – Classification: U-Bahn → Correct name / Wrong Classification → final rating – Incorrect

Distinction between localities with a single postal code and those with multiple postal codes

Postal codes are one of the sources of common mistakes in the Austrian market. There are different result expectations for postal codes. According to CSG 3.3, if a locality has a single postal code, the result should display this particular postal code. If the postal code is missing or incorrect, Address Accuracy is rated Incorrect – Postal Code. This also applies to smaller, independent villages that share a postal code with other localities. As long as the locality has only one postal code, it needs to be returned.

If the result has multiple postal codes, it is correct not to return a postal code.

Please note that postal codes are required for Search results; Autocomplete tasks do not require any postal code (CSG 2.1).

Examples:

Result is Mattsee, Österreich – single postal code not returned (5163) – Incorrect – Postal code

Result is Innsbruck, Österreich – no postal code returned; locality has multiple postal codes – Correct

Result is 3871 Gars am Kamp, Österreich – wrong postal code returned – Incorrect – Postal code

Sub-Localities

For sub-localities (or neighborhoods in larger cities), we also require the postal code to be included in the result. The result should return the name of the sub-locality, the associated postal code, and the larger locality.

Examples:

Result is Favoriten, Österreich – single postal code (1100) & larger locality (Wien) not returned – Incorrect – Postal code, – Incorrect – Locality

Result is 4040 Urfahr, Linz, Österreich – single postal code and larger locality returned – Correct

Result is Maxglan, Salzburg, Österreich – single postal code (5020 not returned) – Incorrect – Postal code

How can we identify the correct postal code?

We can use the official Austrian Post Office (Österreichische Post) postal code/locality list to determine postal codes for localities and sub-localities in Austria. All postal codes of localities and sub-localities are listed in an Excel Sheet. To download the excel sheet, please visit the Austrian Post Office Postal Code and Locality/Sub- Locality List and download the file “PLZ Bestimmungsort” at the bottom of the page.

Column A lists all postal codes, column B lists all localities, column D lists their sub-localities. To search a particular locality or sub-locality, you can use <Strg>+F to easily find the postal code of a locality.

Other helpful resources for localities, sub-localities and postal codes:

Geodata map at https://www.geoland.at/

List of postal codes, localities and sub-localities provided by the Umweltbundesamt at https://tinyurl.com/3x8vv9xm

Detecting if a locality is missing

Another common mistake in the Austrian market is the missing locality of a sub-locality result. If TryRating lists just the sub-locality, this would be rated Incorrect – Locality since the larger locality is missing. According to CSG Section 2.1, both Search and Autocomplete tasks require the larger locality as mandatory component of sub-locality results. If the locality is missing or false, we need to rate Incorrect – Locality.

How can we identify the correct locality of a sub-locality?

Essentially, we can use the same resources as explained above for postal codes:

On geoland.at, simply search for a sub-locality name. On the result page, click on “Verwaltungsgrenzen” on the left side and click on the checkbox “Gemeindegrenzen”. The map application will then display the locality of the sub-locality.

Example

Aisting, Österreich (48.263432,14.576961)

To rate the correctness of the address, we need to determine whether Aisting is a sub-locality or locality, and whether it has a single postal code or multiple postal codes. In the “PLZ Bestimmungsort” Excel file of the Austrian Post Office (see above), we find that “Aisting” is a sub-locality of “Schwertberg”, and that it has a single postal code (4311). Both the locality and postal code are missing. Address Accuracy is therefore Incorrect, with the ‘Locality’ and ‘Postal code’ checkboxes selected.

.

Search “Aisting” using <Strg>+F. We can see that the Aisting is a sub-locality of Schwertberg, and that it has a single postal code.

To verify the correctness of the locality name, we search for Aisting on geoland.at, and select the checkbox “Gemeindegrenze” in “Verwaltungsgrenzen”. The locality “Schwertberg” will be displayed on the map.

.

.4. Pin Rating Next Door / Approximate

Always try to determine the boundaries of the feature (fences, walls, bodies of water etc.). In dense areas it may be difficult to find boundaries between features. In these cases, divide the space between the street or next building in half to create a boundary (Half ‘n’ Half Rule). Use this rule to determine the Approximate and Next Door pin locations (GL Section 9.1.3.1).

4. Example 1 

.The result is a HOFER store located on “Bahnhofstraße 36, 4810 Gmunden”.

Perfect > The pin is on the rooftop (green area)

Approximate > The pin is within the boundaries of the POI (parking lot and half of the street according to the Half ‘n Half rule (yellow area)

Next door > The pin is on the property immediately next to the intended feature that is located on the same street (purple area)

Wrong> The pin falls outside of the property boundaries or Next Door property (everything else)

A pin is considered Next Door if it drops on the immediate property next to the intended one. A Next Door pin must be:

The TryRating pin (red) lands on the rooftop next to the actual result address (green). Since the properties are next to each other and on the same street and street side, the correct pin rating is Next Door.

4. Example 3

The TryRating pin (red) is placed on a rooftop within a shared space. The result address (green) is within the same shared space complex. Since there is no Next Door rating for shared spaces, the correct pin rating is Approximate.

No Next Door for Shared Spaces

A feature cannot be Next Door to another
feature within the same property
boundaries. This means that two buildings
in the same shared parking lot or parcel
can never be rated as Next Door to one
another.

There will also be no Next Door ratings
made outside the parcel or shared space.
Any pin falling outside of the Approximate
area will be marked Wrong.

The pin lands within the result property’s boundary. For a rating of Perfect, the pin should fall on the rooftop. Therefore, a rating of Approximate is correct.

In dense areas it may be difficult to find
boundaries between parcels. In these
cases, divide the space
between the street or next building in half
to create a boundary.

Use this “Half ‘n Half” rule to
determine the Approximate and Next
Door pin locations.

Campuses and business complexes do
not have Next Door or Approximate pin
ratings

.

.