This document provides useful tips and guidance to help you be better prepared to rate Result Satisfaction tasks. Please make sure to read this document carefully before rating Result Satisfaction tasks.
First of all, check the task locale!
Please make sure to always check the task locale given for each task, as locales are not always the same, and ignoring them will have a direct impact on your quality ratings.
The task locale is displayed at the top left corner of your screen during annotation time:
Task
Locale: [fr-FR]
In this task, you’ll be presented with an input image and a result. The result may have more than one part. Please consider all parts guidelines first if you are not familiar with visual search result satisfaction grading tasks. Please follow these steps while grading.
o If yes, grade each result based on common user’s intent and result satisfaction. For more details refer to the grading gu o If no but the result is still relevant/useful to one of the secondary object, it can be SS; otherwise, NS.
Find below some of the task locales you might encounter, and the expected languages for each one of them:
Locales:
Check the type of result you are rating, and evaluate the importance of its components (title/name, description, thumbnail, linked website, etc.).
Once you have verified the task locale, and identified the object in the input image, check what type of result is the one that you are evaluating. Overall, when assessing the result satisfaction, you should consider the name/title, description, thumbnail, linked website, and locale together, nonetheless, those components may have different importance depending on the result type you are evaluating. Demotions are applied differently depending on the type of result or category.
See below some examples of result components and its importance:
• The landing page for ‘Show More’ -when given- must be in the country of the task locale.
PLEASE NOTE:
For Books/Songs/Albums: The language of the linked website is critical. If the linked website is in a different language/locale, or the book/song/album is given in a different language than the task locale -and a version of the same exists in the correct language-, the result should be demoted to Moderately Satisfying. The reason is because the user intent is to potentially purchase/stream/read it, so a result in a different language would not meet that intent.
Example 1:
Plus de détails
The Album result is accurate however, it shows a partial language match, and the
linked website takes the user to the US Apple Store Website, it also shows the English
version of the album, which is not the language of the task locale.
Because the language of the landing website is wrong, and it’s critical for
Books/Songs/Albums results, this should be demoted to
Moderately Satisfying.
PLEASE ALSO NOTE:
For Knowledge results or Books/Songs/Albums: Pay close attention to words in another language in the title/name and/or description, some words are really close in English and in French but if there is a word in another language, it will be enough to demote to “Moderately Satisfying”.
For Knowledge results: The linked website is not so critical. The focus should be on the information shown on the TryRating result itself. If the title/name and/or description are given in a different language, you should demote the result to Moderately Satisfying automatically, regardless of the language/locale of the linked website.
See example below:
Task
Locale: [fr-FR]
base of the plant. It grows to 18 in (45 cm) tall by
This result is accurate, and contains all the critical components for a plant result,
however, the result is given in English, and the task locale is fr-FR (results are
expected in French). The result should be rated
Moderately Satisfying.
The linked website is not so critical for knowledge results, as the user intent is to
identify the object and learn more about it. This user intent can be technically met if an
accurate name, description and thumbnail is displayed in the result.
Landing websites are not critical, but still important in this type of results.
Maps results do not include a link or thumbnail, but this is normal, they can still be rated S or HS.
Maps results should be judged by the information displayed in the result itself. If the map result correctly identifies the primary object in the image, and the title and description (address) are correct, and given in the correct language, the rating can be “Satisfying” or “Highly Satisfying”.
PLEASE NOTE:
There is no link or thumbnail in a maps result, this is completely normal and expected, maps results should not be demoted for this reason.
Example:
This result is accurate, and given in the correct language, there is no
link/thumbnail/further description in the maps result, but this is normal, and
expected. The result should be rated
Highly Satisfying
Key Reminders:
o Task/language locale: Task locales are constantly changing! Always check what is the expected language (Task Locale) of each task, before providing any ratings!
o Maps Results: Remember that maps results do not include links/description, this is normal, and expected, do not demote maps results for this reason! Rate based on what is presented.
o Research Links. Always do research using the links provided, even if the image seems obvious, click on the link to confirm your first impression.
o Linked websites : Always click on linked websites -if given- and evaluate the language/locale of the landing website. Demote results as appropriate, considering how important is this component for the type of result you are evaluating.