Best Practices
“Comments are mandatory for any rating below Excellent for Relevance or below
Correct/Perfect for Data Accuracy”
(Guidelines Section 1.3.6.).
Comments should be included in any
of the following situations.
“Comments are mandatory for any rating
below Excellent for Relevance or below
Correct/Perfect for Data Accuracy”
(Guidelines Section 1.3.6.).
• A Relevance rating below Excellent.
• Data Accuracy (e.g.
Name/Address/Pin) which is not
rated Correct or Perfect.
• Anything else unusual relating
to the result.
Why leave comments?
Data analysts use your comments to understand your rating choices when evaluating your tasks. By leaving meaningful comments, you enable them to efficiently:
• Process this data
• Find potential issues
When high quality comments are included, you rating accuracy can be verified much more efficiently!
What makes a good comment?
Your comment should explain the reasoning behind your rating choice. Without looking at the task in any great detail, your rating should be easily understood after reading your comment!
You should also include any research resources and links that are relevant for the task and support your rating. Without adding the correct information and the source it came from, a reviewer will have to redo all your research.
At the same time, your comments should be concise and to the point. Don’t waste your time writing long comments – Only include relevant information!
All your comments have to be in English! The person analyzing your tasks may not necessarily speak your local language
Possible Elements of a Good Comment
Some noteworthy attributes that point towards a high quality comment
include:
• References to the specific user intent and location intent.
• Explanations for issues with the user intent or distance/prominence.
• The inclusion of data corrections and links to the sources of these
corrections.
• Any locale-specifc knowledge that impacted the final rating.
• Examples/quotes from the guidelines supporting the analyst’s ratings.
Remember to always use a URL shortener such as bit.ly or
tinyurl.com!
Examples of Good Comments
“User is looking for a specific apartment complex – http://saladoatwalnutcreek.com/. This is a random intersection in a different city and not relevant to intent.”
Explanation: The user intent is not obvious from the query and therefore specifically stated in the
comment. The comment also explains why the result is unhelpful.
“This pharmacy is located inside the intended Walmart. Rated Bad because the result doesn’t satisfy
the user’s intent based on its containment relation with the intended POI alone according to section 5.1.8. of the guidelines.”
Explanation: The comment explains the rationale behind the rating choice by explaining the demotion
that has been applied.
“This is a random business result that partially matches query but is not related to the quiered chain
of stores “Target”. Thus not relevant to intent.”
Explanation: The comment states the user intent and explains why it is not met by the result, which
happens to randomly match parts of the query.
“Across the road from the correct building, should be 40.153721, 74.698486.
https://tinyurl.com/ybcrtuyt”
Explanation: The comment states the correct Pin location and why it’s incorrect. It also includes the
sources and a URL shortener is used for the link.
“Correct postal code is 96813 according to the official website https://bit.ly/2MYpEGT”
Explanation: The comment states the correct data and where it was confirmed. A URL shortener was
used for the link.
Examples of Unhelpful Comments
“Pin is approximate”
Explanation: This comment is unhelpful as it only restates the rating that was submitted. There is no explanation WHY the pin rating is Approximate and where the pin should be dropped instead.
“The user intent is a Motel.”
Explanation: The comment correctly states the user intent, but it doesn’t explain why the result is not
satisfying this user intent.
“Perm closed.”
Explanation: This comment is missing the source from where the information is taken. Also,
abbreviations should be avoided, unless they are very commonly used.
“Das Resultat ist mehr als 20km vom User entfernt und es gibt viele nähere Resultate.”
Explanation: The comment is in German, which Is not helpful for the data analysts who don’t speak this
language. All comments must be written in English.
“The query appears most likely to be a category search for a business type called internet café. The result is an internet café, as asked for in the query, but it is extremely far from the fresh viewport and the user. The fresh viewport is over Funabashi city, where also the user is located. In Funabashi city, there are many intent cafés, so the user would not want to travle so far to the result. Three examples for internet cafés in Funabashi city are: http://mangakissa24.com – http://www.media café.ne.jp/tenpo/funabashi/ – https://www.facebook.com/netsumochiba”
Explanation: This comment is unnecessarily long. Internet cafés are extremely popular in Japan, which
is why we don’t need to specify some of the many examples that would fit the query. A more concise
comment would be “Query is for an internet café. Result is such business, but extremely far away while
many results exist within the viewport”.
“Street number is 22500 https://www.tripadvisor.com/Restaurant_Review-g35489-d4614768 Reviews-Pizza_Hut-Idaho.html#RESTAURANT_DETAILS”
Explanation: Apart from the fact that this is not an official source and can’t be used alone for data
verification, a URL shortener should always be used