This document is intended to provide guidance and best practices for rating Autocomplete tasks in the Israeli market. Therefore, understanding the differences between Autocomplete and other task types and specific considerations for the Israeli market is essential.
The goal of Autocomplete is to save users time by providing relevant content as they type. Unlike search tasks, the user query may not be complete, and the analyst must rate how likely the user is looking for the suggested result based on the query and location intent.
To better understand the differences between Autocomplete and search tasks, refer to the following table:
Autocomplete | Search |
Evaluate: Suggestions | Evaluate: Results |
Relevance Ratings: Excellent, Good, Acceptable, Bad | Relevance Ratings: Navigational, Excellent, Good, Acceptable, Bad |
Data Accuracy: Name, Address | Data Accuracy: Name, Address, Pin |
It is important to note that the high-quality suggestions in Autocomplete are rated as Excellent, while Navigational relevance ratings are not used. Relevance is always rated independently of any data accuracy.
Since the user query may not be complete, the query string may not have a clear user intent. A combination of research, local knowledge, and information in the rating interface are essential to determine the user’s intent. The suggestions returned in Autocomplete are divided into four types: name/title and address details.
In the Israeli market, autocomplete tasks have a few unique considerations to take into account:
Navigational station or stations with a location modifier. It can be challenging to determine the exact user intent in many train station queries, so a lenient approach should be taken when evaluating the Relevance rating. Hence, train stations should only be demoted by distance rather than as described in the global guidelines in section 5.16.1: Transit Queries. It’s important to note that the demotion is based on distance, not the train station’s location on the line. The examples in section 7.3.1 of the Country Specific Guidelines for Israel provide further clarity on evaluating the Relevance of train stations in the Israeli market.
Address Accuracy: According to section 6.2 of the Country Specific Rating Guidelines (CSGL) for Israel, if a POI or Business claims an address that differs from the official sources, the returned address should be evaluated against both the claimed address and the official sources. If the returned address matches either the claimed address or the official source’s address, the Address Accuracy should be rated as Correct. On the other hand, if the address does not exist in either official sources, the address can still be verified if there is clear, recent street-level imagery or if the business claims the address on its official website. If there is no clear evidence pointing to the existence of the address, it should be rated as Incorrect – Address Does Not Exist. However, it should be noted that it may be difficult to verify the address in small localities with no available street-level imagery and a single postal code. In such cases, Address Accuracy should be rated as Can’t Verify.
“רחוב קדיש לוז, קריית מוצקין” :1 Suggestion –
Explanation: The first suggestion is a partial street name in Kiryat Motzkin. The result’s relevance is excellent, as the street is located within the FVP, which is the location intent.
רחוב קדיש לוז 37, קריית ” :2 Suggestion “מוצקין
“רחוב קדיש לוז, באר שבע” :3 Suggestion
Explanation: The third suggestion is a matching street in Be’er Sheva, which is far from the FVP, and there are at least five other matching streets nearer the FVP.
Query: “מחלף הסירה“
“מחלף הסירה, תחנת אוטובוס, הרצליה” :Suggestion
The user is searching for the HaSira Interchange located in Herzliya. Given that the bus stop at the interchange is quite well-known, with many bus lines passing through it, and that it matches the user’s query, it should be rated as Excellent according to the guidelines outlined in section 7.1 of the CSGL.
Rating According to ProminenceQuery: “תל “
Suggestion 1: “תל יצחק” (user location is in yellow, VP is fresh)
Explanation: The user is outside an FVP, so the FVP is the location intent. The query is partial, and the user is likely searching for “Tel Yitzhak,” as this result matches the query and is closest to the FVP, making it an excellent match.
Suggestion 2: “תל אביב“
Explanation: Although the suggested locality is relatively far from the FVP (the location intent), it is a highly prominent city in Israel and shouldn’t be demoted to Good. According to the CSGL, the Relevance rating for a result should be based on its proximity to the FVP, as it reflects the user’s location intent. However, in cases where the suggested locality is far from the FVP, it can still be rated as Excellent if it is a highly prominent city in Israel.
Suggestion 3: “תל שבע“
Explanation: The suggested locality is far from the location intent (the FVP), isn’t prominent, and there are many other matching options closer to the FVP, so it should be rated as bad.
Query: “שדה הת” (the user is in Ramat-Gan, and the VP is stale above Eilat)
Suggestion 1: “נמל התעופה בן-גוריון“
Explanation: The query is likely looking for a category, airports. The VP is stale, so the user location is the location intent. The suggested airport is the primary airport in Israel and is relativity close to the user, which is an Excellent suggestion.
Suggestion 2: “נמל התעופה שדה דב“
Status: PERMANENT_CLOSURE
Business/POI is closed/does not exist
Explanation: The query is likely looking for a category, airports. The VP is stale, so the user location is the location intent. First, we should notice that the result has a status of PERMANENT_CLOSURE. Such a result requires special treatment when rating relevance.
According to 5.19.2 in GL, it is unexpected to see results with the status of
PERMANENT_CLOSURE when there are also open and operational results in the same area of the intended location that could fully satisfy the user’s intent without requiring any modifications to the original query. Therefore, unexpected permanently closed results should be demoted by 2 for relevance, meaning the highest rating starts at Acceptable.
The suggested airport is the closest option to the user (the location intent). However, it is permanently closed, and another operational result (Ben- Gurion Airport) fully satisfies the user’s intent, so this is an acceptable suggestion (due to User Intent).
Suggestion 3: “נמל התעופה הבינלאומי ע”ש אילן ואסף רמון“
Explanation: The query is likely looking for a category, airports. The VP is stale, so the user location is the location intent. The suggested airport is one of two international airports in Israel. It is far from the user’s location, but as there aren’t many options, this is an Acceptable result (due to distance).