Entity Intent Guidelines

1. Introduction

Welcome to Entity Intent rating.
For each task you will be presented with a query. This is a short piece of text users have typed in a web search box or spoken to a smart assistant. These queries can be about any topic and may or may not take the form of a question:

• [how old is pitbull]
• [mascot of university of kentucky]
• [how deep is lake erie]
• [kimberly williams paisley]

You must first decide whether or not the query intent is clear: Can you identify the information the user is looking for?
No: If the query intent is not clear you will select Unclear and be prompted to select Not Listed as the Subject Entity.
Yes, but the intent is not listed: If the query intent is clear but is not included in the list of options, you will select Not Listed and be prompted to select Not Listed as the Subject Entity.

Remember: If the query intent is unclear, or is clear but not included in the list of options, you must Select Not Listed as the Subject Entity even if you think you see the correct entity in the list of options.

Yes and the intent is listed: If the query intent is clear and you find it in the list of options, you will answer two questions:

1. What best describes the query’s intent? (Query Intent)
2. What are the subject entities? (Subject Entity)

You must choose the most precise answer from a list of options for each question.
Your task is not to guess or find the answer to the user queries, but to select the best answer(s) for each test question presented.

1.1. The Rating Tool

This is the web based tool used to rate Entity Intent. It’s divided into three sections:

• Query Intent
• Subject Entity
• Query Features

1.2. The Query

Read the query carefully. If you don’t understand it, research it using the search link provided or other online resources. This research is an important first step. If a query appears to make no sense, say it out loud or speak it into a smart assistant to see how it is interpreted there.
The query may have come from spoken words, so there may be obvious errors, like “clothes” transcribed as “close.” In cases like this, base your rating on the corrected version if the intent is obvious enough to be clear.
A query can have a clear intent even if the query itself has no answer. As long as you can understand what the user wanted to know—even if you’re not sure what entity they wanted to know it about—the query intent is clear. A query can also have a clear intent even if it is incomplete, is missing words, contains repeating or misspelled words or poor grammar.
If, even after research, you cannot understand what the user wanted to know, select Unclear.
You will be prompted to select Not Listed for the Entity before moving to the next query.

2. Query Intent: What Does the User Want to Know?

Your first task is to identify the information the user is looking for. What does the user say they want to know? (Remember, you don’t need to know what entity they wanted to know it about in order for the query to be clear.)
Once you have determined what the user wants to know, you must choose the most precise intent from a list of options. The intents are presented in the form of a word or linked words followed by a description. Read the descriptions carefully as they provide specific information about the meaning of each intent.
If you can identify the information the user is looking, for but none of the intent options are correct, select Not Listed for the Intent. You will then be prompted to select Not Listed for the Subject Entity.
If you cannot understand what information the user is looking for, select Unclear. You will then be prompted to select Not Listed for the Subject Entity.

An intent may occasionally be shown with an incorrect description or with no description at all. If you see either one of these, ignore the missing or incorrect explanation and base your rating on the intent name.
Incorrect Intent explana on:
ChildrenOf: Query asking for the current head of government
Missing Intent Explanation:
ChildrenOf:

2.1. Query Intent is Clear

The intent of a query is considered clear when you can understand what the user wanted to know and can identify the information they are looking for.
A query can have a clear intent even if the query itself has no answer. As long as you can understand what the user wanted to know—even if you’re not sure what entity they wanted to know it about—the query intent is clear. A query can also have a clear intent even if it is incomplete, is missing words, contains repeating or misspelled words or poor grammar.

2.1.1. SubjectInformation

Sometimes the user has asked for very general facts or information about an entity, or has not explicitly asked for any facts or information at all. In cases like these, SubjectInformation may be the most appropriate rating option.

Candidates for the SubjectInformation rating include, but are not limited to, queries like:

•Celebrity or band names
•Movie, book, song, plr poem titles
•Names of memes
•Names of historical events
•Names of health conditions
•Names of video games
•What is [entity]
•Who is [entity]
•Describe [entity]
•Find/Search/Search for/Show me [entity]

2.1.2. Query Intent is Clear but Not Listed

If the query intent is clear but none of the intent options apply, select Not Listed. If you select Not Listed for the Query Intent, you will must select Not Listed for the Subject Entity.

2.1.3. Specific Intent for Specific Query

If a query is more specific, and more specific intent options are offered, choose the most specific one instead of SubjecctInformation. Be as specific as you can given the wording of the query and the intents listed.

2.1.4. Query Intent is Clear, Entity is Missing or Incomplete

Sometimes you’ll be able to understand what a user wants to know, but not who or what they want to know it about. For instance, in the query [who is the mayor of], it’s clear that the user’s intent is to find out about a mayor. What’s not known is the name of the place the user wants this information about. In cases like this, the query intent is still clear and can be rated.

2.2. Query Intent is Unclear

Rate queries Unclear when you cannot tell what the user wants to know. This is true even if you can identify the entity the user wants to know it about.
If you select Unclear for the Query Intent, you must select Not Listed for the Subject Entity.

2.3.Clear vs. Unclear Queries

Use this reference chart to help you decide whether a query is clear or unclear.

Entity: What Entity Do They Want to Know it About?

Once you have identified the query intent (what information the user wants to know) you’ll answer the question: What are the subject entities? In other words: To which entity does the query intent apply? (There is often only one entity to which the intent applies, but in some cases there may be more than one.)
You must choose the most precise entity from a list of options associated with phrases from the query. Other entities may also be plausible, but for most unambiguous cases the task is to pick the single most plausible intent. If several entities seem equally plausible, break the tie by doing a web search and selecting the top result.

Remember: If the query intent is unclear, or is clear but not listed, you must always select Not Listed as the Subject Entity. This is true even if you think you see the correct entity in the list of options

Each entity option should include a description. The entity name is also linked to a page with information about that entity. The link is provided to give a full description of the meaning of each entity. Don’t assume you know what the description of the entity will say — read the description on the linked page. (This will be especially helpful if the description is missing from the tool.)
Note that the wording of the phrase from the query and the name of the entity option do not have to be a perfect match. Mismatches between the wording of the phrase and the name of the entity option are acceptable as long as the linked page refers to an exact definition of the phrase in the context of the query.

To qualify for rating, a page must refer to and describe one entity and not be a disambiguation page or a page that offers no description of the entity or page that is not in your market language.
Choose the most precise entity or entities based on what’s mentioned in the query. Partial entities are not the most precise, as the example entity ”bed” in the query “what is the size of a full size bed” shows.
If you cannot tell what entity the user wants to know about, choose Unclear.
If you know what the entity is but do not see it in the list of options, choose Not Listed.

Check your entity ratings by doing this quick test to see if the intent paired with the entity results in the query: Intent + Entity = Query
Consider the query: [what is drake’s net worth]

• NetWorth + Drake = What is Drake’s net worth
• NetWorth + Net ≠ What is Drake’s net worth
  
  Note: The rating platform will now show you this formula so that you can check your ratings.

3.1. Multiple Entities

Sometimes there will be more than one entity to which the query intent can be applied. In these cases, select all the appropriate entities.

3.2. Entity is Not Listed

When you can understand what a user wants to know, and you also know what entity they want to know it about, but don’t see that entity in the list of options, rate the entity as Not Listed.
You will must also choose Not Listed when the Query Intent is Unclear or Not Listed.

3.3. Entity is Unclear

When you can understand what a user wants to know, and have selected an intent, but you don’t know who or what they want to know it about, rate the entity as Unclear.

4. Putting it All Together

Follow the 3 columns in this reference chart to help make your ratings.
Remember, when the query intent is unclear, or is clear but not listed, you must Select Not Listed for the Subject Entity. This is true even if you think you see the correct entity in the list of options.

5. Query Features

You will be asked to identify different aspects of the query:

• Can the query can be answered with Yes or No?
• Can the query be answered by counting the results of the intent?
• Is the query intent asked about during a specific time period?
  Sometimes one of more of these aspects will be present, and other times none of them will be present.

5.1. Yes/No Answer

Some queries, like [is mars a planet], can be answered with a simple Yes or No (or with Yes or No followed by an explanation). Other queries, like [what’s the plot of game of thrones], cannot be answered with Yes or No (or by Yes or No followed by an explanation).
You must decide whether or not the query can be answered with Yes or No.
Note: The answer to the query will not be shown. You must make the rating based on your research about the query.

5.1.1. Yes/No Answer Expected
Choose this checkbox if the answer to the query could be either Yes or No and the user would be happy with either this short answer or an answer that began with Yes or No followed by an explanation.

4.1.2. Yes/No Answer Not Expected

If a query cannot be answered with a Yes or No (or a Yes or No followed by an explanation), do not select this box.

5.2. Counting Intent Results

Some questions, like [how many number one songs did abba have] can be answered by counting the results of the intent (number one songs). Others, like [how many people live in alaska] cannot. Select the Counting Intent Results checkbox when the answer to the query can be determined by counting the results of the intent.
Note: The answer to the query will not be shown. You must make a decision based on your
research about the query.

5.2.1. Answer Can be Determined by Counting Results of Intent

5.2.2. Answer Cannot be Determined by Counting Results of Intent

Do not check this box if the answer cannot be determined by counting the results of the query intent.
Remember, just because the answer to a query is a number, this does not necessarily mean that the answer can be determined by counting the results of the intent.

5.3. Time Period Intent

Some queries refer to a particular year or event. Choose the appropriate checkbox if the query intent is asked about during a specific time period.

5.3.1. In a Particular Year

Select the In a particular year option when the query intent is asked about during a specific year.
Be sure to read the query carefully. Just because a year appears in a query does not mean the query intent is asked about during that year.
Do not choose this checkbox if no year is mentioned in the query, even if the answer is a year
(for instance [what year did Elvis die]).

4.3.2. During/In Event

Select During/In Event when the answer to the query will change based on the explicitly mentioned time period or event in the query

6. Comments

Leave a comment when:

• You have chosen Not Listed for either Intent or Entity
• Intent: Explain what intent is expected
• Entity: Name the missing entity
• The intent description is incorrect, missing or unclear
• Begin your comment with the words “Incorrect Explanation” or “Missing Explanation” or
  “Unclear Explanation”