1. Introduction

Welcome to Entity Recognition Evaluation.

For each task you will be presented with a query. This is a short piece of text that people have typed in a web search box or spoken to a smart assistant. You must first decide whether the query intent is clear.

If the query intent is not clear, indicate this and move to the next task. You do not need to evaluate the phrase if the query intent is not clear.

If the query intent is clear, you must decide whether a phrase taken from the query has meaning and represents an entity in the context of the query.

If the phrase has no meaning or does not represent an entity in the query context, indicate this and move to the next task.

If the phrase has meaning in the query context, you will see one or more links to pages about the phrases and must decide whether any of the pages provides a plausible interpretation of the phrase in the context of the query.

1.1. The Rating Tool

This is the web based tool used to rate Entity Recognition. It’s divided into three sections: Query, Phrase, and Evaluation Links.

The first time you are presented with this task you’ll be asked to select a default search engine. After you make this selection, your choice will be saved and the Select a search engine text will show as Web Search. Clicking Web Search will now open your preferred search engine pre-populated with the query or phrase. To change your preferred search engine, choose one of the options shown and the new selection will now be saved.

1.2. Workflow

Use the following workflow when rating.

2. Query Intent

The first judgment is always about the query. You must decide whether or not the query intent is clear.

Read the query carefully. If you don’t understand it, research it using the search links provided or other online resources. If a query appears to make no sense, say it out loud or speak it into a smart assistant to see how it is interpreted there.

The query may have come from spoken words, so there may be obvious errors, like “clothes” transcribed as “close.” In cases like this, base your rating on the corrected version if the intent is obvious enough to be clear.

A query can have a clear intent even if the query itself has no answer. As long as you can understand what the user wanted to know, the query intent is clear. A query can also have a clear intent even if it is incomplete, is missing words, contains repeating or misspelled words, or contains poor grammar.

Always research the queries, and pay attention to grammar, wording, and other details as they can make a difference to the meaning and context of a query. Compare the different meanings of these two very similar queries:

• [how long is the titanic]: This query is asking about the length of the ship RMS Titanic
• [how long is titanic]: This query is asking about the length of the movie Titanic

If, even after research, you cannot understand what the user wanted to know, select Query intent is unclear. No further rating will be required and you can submit your rating and move to the next query.

If a query is not in the language of your market, select Query intent is unclear unless the query would be widely understood in your market. No further rating will be required and you can submit your rating and move to the next query. If you can understand what a user with this query wanted to know, you will move to the next step: reading, researching, and rating the phrase

2.1. Clear Queries

If you can understand what the user wants to know, the query intent is clear. When the query intent is clear, do not select the Query intent is unclear box.

Note that a query does not have to take the form of a question to have a clear intent. Users may say the name of a place because they want to learn more about it or enter a person’s name to find out more about them.

Personal questions for the smart assistant, like “Do you believe in Santa“ can also be clear queries when they are about concepts or entities that exist outside the device (in this case, Santa Claus).

News headlines like these are always clear queries:

• Larrys Mabiala scores twice to lead Portland Timbers to 3-2 win over Seattle Sounders
• Man arrested with 22 gold bars
• Nearly 900 GP projects completed via infrastructure fund
• Brad Pitt set to ‘produce a film on Harvey Weinstein investigation’
• Lack of sleep ‘costs UK £40bn a year’
• BRING IT! LIVE! Comes to NJPAC This Summer

Commands for the smart assistant, including [Find/Find me/Show me/Show me pictures of/ Search/ Search for/Do you know/Play/Delete/Pause/Create a playlist/Go to (entity)],are considered clear queries as long as they refer to an entity that exists outside the device.

Query	Rating and Explanation
[how expensive is spinal decompression]	Query intent is unclear
	Even though the answer to this query will depend on where the user is and the technique and doctor they chose, the intent of the query is clear.
[Who is the president of the united states of]	Query intent is unclear
	The word “america” is missing from the country name “United States of America, “ but even without this word, it’s possible to understand what the user wants to know.
[what is the minimum required length of a buoyant heaving line for all pleasure craft und4 m]	Query intent is unclear
	This is a long query with a very specific intent.
[Show me a slime  is cat]	Query intent is unclear
	By using context clues (the word “cat”) and saying this query out loud, it can be understood as “Show me a siamese cat.”
[what is the function of a microscope field of you]	Query intent is unclear
	“Field of view” is written as “field of you,” but the query intent is very clear.
[How much carbs in bread]	Query intent is unclear
	This query has poor grammar, but the user’s intent can be  clearly understood.
[lego]	Query intent is unclear
	This one-word query has a very clear intent and refers to an entity: the toy blocks.

Query	Rating and Explanation
[orange]	Query intent is unclear
	This query could have several possible intents, including the color and the fruit.
[zimbabwe]	Query intent is unclear
	The intent of this query is to find information about a specific country.
[can you tell me who is the lead singer of the sacred mushrooms]	Query intent is unclear
	The intent of this query is to find the name of the lead singer for a specific band
[how old is mike meyers]	Query intent is unclear
	The user wants to know the age of a specific celebrity.
[simple green]	Query intent is unclear
	The user is looking for information about a specific brand of cleaning products.
[who was Kim kardashian’s first husband]	Query intent is unclear
	The user wants to know the name of this celebrity’s first spouse.
[la vie en rose]	Query intent is unclear
	This query is foreign for an English-language market, but is a well- known song and movie title that English speakers will recognize.
[What’s the formula for finding the circumference of a circle]	Query intent is unclear
	This is a query with a clear intent and is not considered a math question.
[How much is 100 Euro in Lira]	Query intent is unclear
	This is a query with a clear intent and is not considered a math question.
[show me pictures of bigfoot]	Query intent is unclear
	This query is addressed to the smart assistant and has a clear intent about an entity (Bigfoot).

Query	Rating and Explanation
[when is your birthday]	Query intent is unclear
	This query is addressed to the smart assistant and has a clear intent about an entity (a birthday).
[tell me a joke]	Query intent is unclear
	This query is addressed to the smart assistant and has a clear intent about an entity (a joke).
[play finnish speed metal]	Query intent is unclear
	This query is addressed to the smart assistant and has a clear intent about an entity that exists outside the device (a kind of music).
[i want to hear sam smith]	Query intent is unclear
	This query is addressed to the smart assistant and has a clear intent about an entity that exists outside the device (Sam Smith).
[play banda music in the master bedroom]	Query intent is unclear
	This command to the smart assistant includes the entity “banda music” that exists outside of the device. It is a specific genre of Mexican music.
[play me songs to get me in the mood for thanksgiving]	Query intent is unclear
	This command to the smart assistant includes the entity “thanksgiving,“ which is not unique to the user or the device.
[Shop the Season’s Best Swimsuits for Every Personality]	Query intent is unclear
	News headlines are always clear queries
[Why this year has been so dangerous for Mount Everest’s climbers]	Query intent is unclear
	News headlines are always clear queries

2.2.Unclear Queries

Select Query intent is unclear when you cannot understand what the user is looking for.

2.2.1. Special Types of Unclear Queries

Certain types of queries should always be rated unclear, even if their intent is obvious and you know exactly what the user wanted to find out.

2.2.1.1. Math and Number Questions

Math questions contain mostly or only numbers and/or references to the mathematical processes to be applied to them. The numbers and processes do not represent an entity in the context of the query.

Note that conversion questions like [how much is 100 euro in lira] or [how many inches in a foot] are not considered math questions.

Queries that consist of numbers only are generally unclear unless the number itself is widely known to represent an entity, like [911], which is the emergency number for North America as well as the name of a TV show.

2.2.1.2. Commands for Smart Assistant That Don’t Refer to an Entity Outside the Device or User

When commands for the smart assistant are queries about things that assistant can do within itself or through an app and that do not refer to an entity outside the device, they should be rated unclear.

These kinds of unclear commands include requests about things that are unique to the device or the user and whose meaning or identity will change depending on device and user, like “my playlist” or “the master bedroom” (since every user will have a different playlist and a different room called the master bedroom).

Note that command queries like [Find me/Show me/Show me pictures of/Search/ Search for/Do you know/Play/Delete/Pause/Create a playlist/Go to (entity)] are considered clear queries as long as they refer to an entity that exists outside the device and is not unique to the user or device.

Query	Rating and Explanation
[turn off the lights please]	Query intent is unclear	☑
	This is a command made to a smart assistant about an action it can perform within itself on something that is unique to the user (this specific user’s lights), not a question about an entity.
[show me the home button]	Query intent is unclear	☑
	This is a command made to a smart assistant about an action it can perform within itself, not a question about an entity.
[remind me to make an appointment]	Query intent is unclear	☑
	This is a command made to a smart assistant about an action it can perform within itself, not a question about an entity.
[shuffle my songs]	Query intent is unclear	☑
	This is a command made to a smart assistant about an action it can perform within itself on something that is unique to the user (the user’s songs), not a question about an entity.
Query	Rating and Explanation
[pause the track on the speakers in the living room]	Query intent is unclear	☑
	This is a command made to a smart assistant about an action it can perform within itself on something that is unique to the user (this user’s living room), not an entity. Note that “living room” in this query refers to an entity that is unique to this user, while “living room” in the query [what is a living room] refers to an entity that would have the same meaning to every user.
[please delete my rainy days playlist]	Query intent is unclear	☑
	This is a command made to a smart assistant about an action it can perform within itself (deleting) on something that is unique to the user (this user’s rainy days playlist), not an entity.

Commands that Refer to Entities Both Inside and Outside the Device or User

Some commands refer to entities that exist both inside and outside the device or user. In cases like this, rate the query as clear.

Query	Rating and Explanation
[play 80s rock in the office]	Query intent is unclear
	This query contains a reference to an entity that exists outside the device or user (80s rock) and to one that’s unique to the user (the user’s office). Since a mix of both types of entities is present, this is a clear query.
[play contemporary blues music in the master bedroom]	Query intent is unclear
	This query contains a reference to an entity that exists outside the device or user (contemporary blues music) and to one that’s unique to the user (the user’s master bedroom). Since a mix of both types of entities is present, this is a clear query.

2.2.1.3. Song Lyrics

Queries that are song lyrics should always be rated unclear.

Note that song titles, like book, movie, and other media titles, are valid queries.

2.2.1.4. Definition/Spelling/Thesaurus/Rhyming/Translation and Other Queries About Words

Questions about the definitions, spellings, and properties of words (including requests for synonyms and antonyms), requests for rhymes and translations, or any other type of query where the word in question does not represent an entity should always be rated unclear. If a single-word query does not represent an entity, rate it unclear, but if it does represent an entity it is clear and will be a valid phrase.

Query	Rating and Explanation
[How do you spell triskaidekaphobia]	Query intent is unclear	☑
	This query is about the spelling of the word “triskaidekaphobia,” not about anything related to the entity (fear of the number 13)
[define nebulous]	Query intent is unclear	☑
	This is a request for a definition of a word that does not stand for   an entity. Note that [define elephant] is a clear query since “elephant” is an entity
[nebulous]	Query intent is unclear	☑
	This is a request for a information about a word that does not stand for an entity. Note that [elephant] is a clear query since “elephant” is an entity.
[synonym for castle]	Query intent is unclear	☑
	T This is a request for a word that means the same thing as castle. Requests    for antonyms, or words that mean the opposite of the specified word, will also be rated unclear.
[plural of leaf]	Query intent is unclear	☑
	This is a request for the plural version of a word, not a query about the entity “leaf.”

3. Phrase Validity

If a query has a clear intent, you will rate a phrase taken from it. A phrase is a word or words within the query that names an entity. An entity can be a person, place, thing, concept, or event.

There may be more than one potential phrase within each query.

You must decide whether each phrase presented to you is valid or invalid in the context of the query. A phrase is only rated in the context of a specific query. It may or may not be valid in a different query.

If the phrase is invalid in the context of the query, select the Phrase is invalid checkbox. No further rating will be required and you can submit your ratings and move to the next query. If the phrase is valid in the context of the query, you will move to the next step: evaluating pages about the phrases to see if they refer to a plausible meaning of the phrase in the context of the query.

Note that there does not need to be a Wikipedia entry about a phrase in order for it to be valid.

3.1. Valid phrases

Phrases are valid if they meet all three of the following conditions:

• They refer to an entity that exists outside the device or user and
• Theyhave meaning within the context of the query and
• They are as specific as possible, meaning they contain all parts necessary to retain meaning in the context of the query and no extra parts. In the query [lifespan of a panda bear] the valid phrase refers to the entire entity “panda bear” not the more generic “bear” and does not include the extra word “a.”

Phrases in foreign languages are valid when they commonly used and widely understood in the market and meet all three of the conditions above.

A phrase does not need to be the most ideal phrase or the most important to the meaning of the query to be valid, it just needs to have meaning in the context of the query. The phrase “lifespan” from the query [lifespan of a panda bear] would also be a valid phrase.

Entity vs. Description of an Entity

In the query: “Does the person flying the plane need a license,” the words “person flying the plane” describe, but do not name, the pilot and are an invalid phrase. It is important to identify descriptions because they need to be broken into the valid phrases they contain, in this case “person” and “plane.” “License” is also a valid phrase in the query. Do research to determine if a phrase is an entity or a description of it.

The table below shows how different parts of a query can create both valid and invalid phrases. A valid phrase contains all necessary parts present in the query without losing its original meaning. The table also shows invalid phrases that are too short or too long. Phrases that are too short have a different, or no, meaning in the context of the query. Phrases that are too long may include other valid phrases or create a phrase that does not refer to an entity.

Note: It is possible for valid phrases to have more than one possible meaning in the context of the query. An example is given below for the phrase “halloween”.

Query	Phrase	Explanation
[what can i do to keep my dog from chewing on his leg]	dog	Phrase is invalid
		“Dog” has a meaning within the context of the query, refers to an entity, and is the fullest and most specific phrase present in the context of this query.
[how long is halloween]	halloween	Phrase is invalid
		This phrase is as full and specific as possible and has two possible meaning in the context of this query: the holiday and the movie.
[What’s a hermit crab]	hermit crab	Phrase is invalid
		This is a full and specific phrase referring to a type of crustacean and has a meaning within the context of the query.
[Is Oprah married]	oprah	Phrase is invalid
		The fullest and most complete phrase available in the context of the query is “oprah,” and research shows there is a dominant intent for this phrase. Note that the phrase “Oprah Winfrey” is not present in this query, so “oprah” is the most complete phrase available.

Query	Phrase	Explanation
[what language do they speak in argentina]	language	Phrase is invalid
		“Language” has a meaning within the context of the query.
	argentina	Phrase is invalid
		“Argentina” has a meaning within the context of the query.
[who wrote the book catch 22]	book	Phrase is invalid
		“Book” has a meaning within the context of the query.
	catch 22	Phrase is invalid
		“Catch 22” is the book’s title and has meaning within the context of the query.
[how long do you boil corn on the cob]	corn on the cob	Phrase is invalid
		“Corn on the cob” is the fullest and  most specific phrase that can be found in the context of the query, and has a clear meaning in this context. Note that both “corn” and “cob” also have meanings in the context of then query, but are not as full and specific as they could be given the wording of the query.

Query	Phrase	Explanation
[who was Kim kardashian’s first husband]	kim kardashian’s	Phrase is invalid
		“Kim Kardashian’s” refers to the celebrity and has meaning within the context of the query. Even though there is a ‘s at the end of her name, this is a clear reference to the entity “kim kardashian”
	husband	Phrase is invalid
		“Husband” has a meaning within the context of the query. Note that even though it’s possible to find out the name of Kim Kardashian’s first husband, the purpose of this task is to identify the entities in the query, not to answer it. The reason “kim kardashian’s first husband” is not a valid phrase is because these words are a description that contains two entities.
[giraffe]	giraffe	Phrase is invalid
		This one-word query produces a valid one-word phrase.
[what are the birds and the bees]	the birds and the bees	Phrase is invalid
		The phrase “the birds and the bees” refers to a specific concept.
[road sign for cul de sac]	cul de sac	Phrase is invalid
		While “cul de sac” is a term in French, it is widely used in the US market, which makes the phrase valid.
[capital city of new mexico]	capital city	Phrase is invalid
		This query about the name of a city contains two entities. “Captial city” is one of them.
	new mexico	Phrase is invalid
		This query about the name of a city contains two entities. “New Mexico” is the other one.

Query	Phrase	Explanation
[hot wings recipe]	hot wings	Phrase is invalid
		This query about cooking directions contains two entities. “Hot wings” is one of them.
	recipe	Phrase is invalid
		This query about cooking directions contains two entities. “Recipe” is the other one.
[when is your birthday]	birthday	Phrase is invalid
		This query to the smart assistant about its date of birth contains the entity “birthday.”
[define triangle]	triangle	Phrase is invalid
		The phrase “triangle” refers to an entity and is a valid phrase. Note that the query “define triangular” would be unclear since “triangular” does not refer to an entity.
[who is wonder woman’s mom]	wonder woman	Phrase is invalid
		“Wonder Woman’s” refers to the superhero and has meaning within the context of the query.
	mom	Phrase is invalid
		“Mom” has a meaning within the context of the query. Note that even though it’s possible to find out the name of Wonder Woman’s mom, the purpose of this task is to identify the entities in the query, not to answer it.
[i want to hear sam smith]	sam smith	Phrase is invalid
		This command to the assistant refers to the entity “Sam Smith,” which has meaning in the query context.
Query	Phrase	Explanation
[Trump Appeasement Fails, So Trudeau Takes the Gloves Off]	Trudeau	Phrase is invalid
		“Trudeau” is a valid phrase in this news headline because the entire name “Justin Trudeau” does not appear in the query. Note that if the entire name ““Justin Trudeau” did appear in the query, “Trudeau” alone would be an invalid phrase.
[world cup scores this week]	World Cup	Phrase is invalid
		“World Cup” is a valid phrase in this sports headline.

Product and Brand Names

When a query contains a reference to a brand or product, the fullest expression of the entity present together is one phrase. When the brand and product are separated by additional words not belonging to the brand or product description, consider product/model and brand as separate valid entities. The model year (if present) is always part of the model. The category or class of product (for instance, “car,” “shoe,” or “tablet”) are always separate valid phrases.

Query	Phrase	Explanation
[price of 2018 Honda Civic EX Hatchback]	2018 Honda Civic EX Hatchback	Phrase is invalid
		The phrase includes the full product specifications needed to be a valid phrase, including the model year.
[how wide is the catalina 39mkts coachmen rv]	catalina 39mkts coachmen	Phrase is invalid
		The valid phrase includes product and brand.
	rv	Phrase is invalid
		The category “RV” is its separate valid phrase.

Political Positions, Titles,  Organizations, and Departments

Titles of elected and other officials as well as names of departments, offices, and other organizations are valid phrases. This includes titles like “chancellor of germany,” “president of indonesia,” “mayor of san francisco,“ and so on.

Query	Phrase	Explanation
[who is the president of the united states]	president of the united states	Phrase is invalid
		The president of the united states is a political position and is a valid phrase.
[who are the members of the santa cruz city council]	santa cruz city council	Phrase is invalid
		This is the title of a public board and considered one entity.
[Michigan Department of Treasury]	michigan department of treasury	Phrase is invalid
		The full department name is a valid phrase.
[Supreme Court of New York]	supreme court of new york	Phrase is invalid
		This refers to a specific entity and is a valid phrase.

Events

Phrases that refer to events  are valid when they are the fullest expression of the entity present in the query.  In the query [what year did the spanish american american war start]  the valid phrase is “spanish american war” since it is fuller and more specific than “war.

Query	Phrase	Explanation
[when was woodstock]	woodstock	Phrase is invalid
		In the context of this query, the phrase”woodstock” refers to a specific concert
[What was the cause of the Upper Canada Rebellion]	upper canada rebellion	Phrase is invalid
		The phrase “upper canada rebellion” refers to a specific event.
[who sang at live aid]	live aid	Phrase is invalid
		The phrase “live aid” refers to the 1985 concert that raised funds for famine relief.

Years

Years that appear within queries are considered valid phrases unless they are part of a product/ brand phrase. This means that in the query [what was the summer of 1967 called] “1967” is a valid phrase, while in the query [how much would it cost to buy a 1967 volkswagen beetle today] “1967” is an invalid phrase as it is missing the brand/product information. (The valid phrase would be “1967 volkswagen beetle.”)

Query	Phrase	Explanation
[what was the summer of 1967 called]	1967	Phrase is invalid
		The year “1967″ appears within this query and is a valid phrase.
[what battle was fought in 1066]	1066	Phrase is invalid
		The year “1066” appears within this query and is a valid phrase.
[World Asthma Day 2018: 6 asanas to ease the effects of asthma]	World Asthma Day	Phrase is invalid
		“World Asthma Day” is an entity within this headline.
	2018	Phrase is invalid
		The year “2018” appears in the headline and is a different entity from “World Asthma Day.”
[2004 Olympics]	2004	Phrase is invalid
		The year “2004” appears in the query and is a different entity from “Olympics.”
	Olympics	Phrase is invalid
		“Olympics” is an entity within this query.

Phrases in Commands that Refer to Entities Both Inside and Outside the Device or User

When a command to the assistant refers to entities both inside and outside the device or user, only the phrase(s) referring to the entity or entities that exists outside the device or user are valid

Query	Phrase	Explanation
[play 80s rock in the office]	80s rock	Phrase is invalid
		The phrase “80s rock” appears within this query and is a valid phrase referring to an entity that exists outside the user or device. Note that “the office” is not a valid phrase since it refers to something unique to this particular user (the user’s office).
[i want to listen to janis joplin in all the rooms]	janis joplin	Phrase is invalid
		The phrase “janis joplin” appears within this query and is a valid phrase referring to an entity that exists outside the user or device. Note that “rooms” is not a valid phrase since it refers to something unique to this particular user (all the rooms in this user’s house).

Sports Teams and Terms

Words like “score,” “schedule,” “team,“ “uniform,” and “game” are separate entities even when they appear as part of a longer series of words in a query.

Partial teams names are often used in headlines and by users with questions for voice assistants. As long as the complete team name is not available in the headline or query and the partial name has meaning in the context of the query, partial team names are valid phrases.

Query	Phrase	Explanation
[ncaa women’s basketball score]	ncaa women’s basketball	Phrase is invalid
		The score this user is asking about is for “ncaa women’s basketball.” “NCAA” alone is not a complete phrase in this context because it refers to  the organization only, not to a specific sport within the organization.

Query	Phrase	Explanation
	score	Phrase is invalid
		“Score” is an entity in the context of this query.
[wichita state men’s basketball schedule 2018]	wichita state men’s basketball	Phrase is invalid
		The team this user is asking about is “wichita state men’s basketball.” “Wichita State” by itself is not a complete phrase in this context because it refers only to the school, not the specific team.
	Schedule	Phrase is invalid
		“Schedule” is an entity in the context of this query.
	2018	Phrase is invalid
		The year “2018” appears in the query and is an entity. Learn more about years here.
[Major League Baseball Roundup: Smoak Lifts Jays Over Tigers]	Smoak	Phrase is invalid
		In the context of this headline, the phrase “Smoak” refers to baseball player Justin Smoak and is a valid phrase since the full name does not appear in the query.
	Jays	Phrase is invalid
		In the context of this headline, the phrase “Jays” refers to the Toronto Blue Jays baseball team (not the birds) and is a valid phrase, since the full team name does not appear in the query.
	Tigers	Phrase is invalid
		In the context of this headline, the phrase “Tigers” refers to the Detroit Tigers baseball team (not the big cat) and is a valid phrase, since the full team name does not appear in the query.
Query	Phrase	Explanation
[does georgia have a new basketball coach]	Georgia	Phrase is invalid
		In the context of this headline, the phrase “Georgia” refers to the University of Georgia (not the state) and is a valid phrase since the school’s full name does not appear.

Acronyms and Abbreviations

Acronym and abbreviations (like NATO, NASCAR, or “temp” for “temperature” in Englishlanguage markets) can be valid phrases as long as they have meaning in the context of the query.

Sometimes headlines or other types of queries will contain unusual or unfamiliar acronyms and abbreviations, so always do research. Even if an acronym or abbreviation is unexpected or something you’ve never seen before, it is still valid as long as it has meaning in the query’s context.

Query		Phrase	Explanation
[FANG is Back in Favor]	FANG		Phrase is invalid
			Research shows that FANG (Facebook, Amazon, Netflix, and Google) is a fairly new acronym, but it still has meaning in the context of this headline and is a valid phrase.
[SC summer recess to begin before final ruling]	SC		Phrase is invalid
			Research shows that in this headline, “SC” stands for Supreme Court of the United States. This unusual acronym has meaning in the context of the query and is a valid phrase
[Dems looking for 2020 inspiration]	Dems		Phrase is invalid
			In this headline, “Dems” stands for “Democrats.” This shortened form of the word is a valid phrase with meaning in the context of the query.

3.2. Invalid Phrases

A phrase can be invalid for any of the following reasons:

• It has no meaning in context of the query
• It is not an entity
• It is a partial phrase (one or more words in a multi-word phrase have not been included)
• Partial phrases lose context within the query
• Partial phrases are not as specific as the full phrases found in the query ( for instance,

“spaniel” in the query [pictures of cocker spaniel] rather than “cocker spaniel”)

• It is only part of the full name given in the query
• The phrase contains words that are not part of the entity
• The phrase contains words that are not in the query

3.2.1. No Meaning in Context of Query

Some phrases are meaningless in the specific context of the query (even though they may have meaning in other contexts). These phrases are invalid.

3.2.2. Not an Entity

A valid phrase refers to an entity (person, place, thing, concept, or event). If a phrase does not refer to an entity, it is invalid.

If the query is a valid command for the smart assistant that refers to entities both inside and outside the device or user, a phrase referring to an entity that’s unique to the device or user is invalid since it does not have the same meaning for all users.

3.2.3. Partial Phrases when a Full Phrase is Available

Partial phrases are smaller sections of phrases made of more than one word that only refer to a part of an entity. They are not as long and specific as the complete phrase present in the query. Generally, the best phrase is the longest, most specific one seen in the query that refers to a given entity.

In the query [what’s the minimum wage in texas]  both “wage” and “minimum wage” could be phrases. The partial phrase “wage” is invalid for this query even though it is an entity and has a meaning within the query. This is because a more specific phrase is also present in the query. “Minimum wage“ is a longer and more specific entity, making it a valid phrase in the context of this query. “Texas” is also an entity in this query.

3.2.4. Partial Names

When a query contains a person’s full name (first and last) the phrase must consist of all parts of the name used in the query to be valid. In cases where a person is well known by one name and that single name alone appears in the query, the single name is a valid phrase. However, if both this person’s names appear in the query, the single name alone is invalid as a phrase because the query makes a more complete version available.

3.2.5. Contains an Article

Phrases that contain articles are valid only if the name of the entity they refer to in the query also contains an article. In these cases, if the article is missing, the phrase is invalid.

3.2.6.Extra Words or Entities in Phrase

A valid phrase refers to one entity and does not contain additional words that are not related to the entity or that refer to other entities.

Query	Phrase	Rating and Explanation
how to remove an oil stain	stain	Phrase is invalid	☑
		The valid phrase is “oil stain.” “Oil”or “stain “on their own are referring to separate entities that are outside of the specific query context.
[I want to hear 7 rings by ariana grande]	7 rings by ariana grande	Phrase is invalid	☑
		The query contains 2 valid phrases: The song title “7 rings” and the artist “ariana grande.” Putting both entities together in the same phrase makes it invalid.
[i want to hear some free music from india]	music from india	Phrase is invalid	☑
		“Music” and “india” are two separate entities. Combining them creates an invalid phrase.

3.2.7. Contains Words Not Found in the Query

A valid phrase contains only words that can be found in the query.

The purpose of this task is to identify the valid phrases within the query, not to answer the question. Even if the answer appears as the phrase, if it is not part of the query, it is not valid.

4. Rating Entity Links

If the query intent is clear and the phrase is valid, you will rate one or more links to pages about the phrase. These pages can come from a variety of sources, but will all be rated in the same way.

Check each page to see whether or not it refers to a plausible meaning of the phrase in the context of the query. There are several pages that refer to the phrase “Raptor,” for example, and each one refers to a different entity. There are also several links that lead to pages containing recordings of Cher’s song “Believe” (studio version, live version, concert version, remixes), and they all refer to different versions of exactly the same entity. Depending on the context of the query, all, some, or none of the pages may be plausible and rated Yes.

Sometimes a phrase can have more than one possible meaning in the context of a query This could be because the query is very general or because the query and the valid phrase are the same. Rate all links that provide a plausible explanation of the phrase in context of the query Yes. This is true even if an intent is rare or unlikely, as long as it has some degree of plausibility in the context of the query.

The broadness or narrowness of a phrase’s meaning in context of the query must be matched by broadness or narrowness of the entry. The phrase “hunting dogs” in the query [how much do hunting dogs cost] refers to a very specific type of dog.

• An article about all dogs discusses every type of dog, has a much more general meaning, and would be rated No.
• The phrase “dog” in the query [teach a dog to sit] refers to a very general meaning of the word “dog,” so an article about hunting dogs would be too specific for the phrase in this context and be rated No.

Note that the wording of the phrase and the wording of the page’s title do not have to be an exact match. Mismatches between the wording of the phrase and title of page are acceptable as long as the page refers to exact definition of the phrase in the context of the query.

RateYes when:

• The phrase is valid and the link has a plausible connection to the phrase in the context of the query

It’s possible that all of the pages shown will have a plausible connection to the phrase in the context of the query, and they will all get a rating of Yes.

• The page is not in the language of your market or or is not available/does not open in your market.
• The page does not refer to the meaning of the phrase in the context of the query.
• The phrase’s meaning in the context of the query is very general and the page refers to a very specific aspect of this meaning, or the phrase’s meaning in the context of the query is very specific and the page refers to a more general aspect of this meaning.
• The page provides a list of possible intents for a phrase and does not explain any single meaning. These pages can be disambiguation pages: https://en.wikipedia.org/wiki/Raptor, search engine results pages: https://tinyurl.com/raptorsresultspage or other types of pages that refer to many possible meanings of the phrase.
• The page does not lead to any information about the entity: https://itunes.apple.com/ artist/367611351

It’s possible that none of the pages shown will have a plausible connection to the phrase in the context of the query, and they will all get a rating of No.

Query	Phrase	Link	Rating and Explanation
[How much would I weigh on Mars]	mars	https:// en.wikipedia.org/wiki/ Mars	Plausible connection	Yes
			Even though this Wikipedia page does not answer the question “How much would I weigh on Mars” it is about the phrase “mars” used in the query, which refers to the planet Mars.
[Can you freeze squash]	squash	https:// en.wikipedia.org/wiki/ Cucurbita	Plausible connection	Yes
			This Wikipedia page is about the squash family of plants
		https:// en.wikipedia.org/wiki/ Squash_(sport)	Plausible connection	No
			The sport of squash cannot be frozen, so a Wikipedia page referring to this meaning of the phrase is not a plausible  connection in the context of the query.
[Score between Giants and Yankees]	giants	https:// en.wikipedia.org/wiki/ San_Francisco_Giants	Plausible connection	Yes
			Research shows that the Yankees in the query are a baseball team, so the Giants in the context of this query must be another baseball team.
		https:// en.wikipedia.org/wiki/ New_York_Giants	Plausible connection	No
			Research shows that the Yankees in the query are a baseball team, so the Giants must be another baseball team. This Wikipedia page is about a football team named the New York Giants. This meaning of the word “Giants” is implausible in the context of the query.

Query	Phrase	Link	Rating and Explanation
[What is the capital of Argentina]	argentina	https:// en.wikipedia.org/wiki/ Argentina	Plausible connection	Yes
			This Wikipedia page refers to the country of Argentina.
		https:// en.wikipedia.org/wiki/ Buenos_Aires	Plausible connection	No
			Even though Buenos Aires is the correct answer to the query, this page does not refer to the phrase “Argentina” that was used in the query.
[when does mary queen of scots come out]	mary queen of scots	https:// en.wikipedia.org/wiki/ Mary_Queen_of_Scots_ (2018_film)	Plausible connection	Yes
			The wording of the query makes it clear that the phrase refers to the film by this name.
		https:// en.wikipedia.org/wiki/ Mary,_Queen_of_Scots	Plausible connection	No
			The wording of the query makes it clear that the phrase refers to the film by this name, not the person.
[Orange]	orange	https:// en.wikipedia.org/wiki/ Orange_(colour)	Plausible connection	Yes
			Color is a frequently used meaning of this phrase.
		https:// en.wikipedia.org/wiki/ Orange_(fruit)	Plausible connection	Yes
			The fruit is a frequently used meaning of this phrase.
		https:// en.wikipedia.org/wiki/ Orange,_New_South_Wales	Plausible connection	Yes
			A small town in southwest Australia is an unlikely intent and not something most users are likely to ask about, but is still plausible.

Query	Phrase	Link	Rating and Explanation
[who was Kim kardashian’s first husband	kim kardashian’s	https:// en.wikipedia.org/wiki/ Kim_Kardashian	Plausible connection	Yes
			This page clearly refers to the same entity as the phrase.
	husband	https:// en.wikipedia.org/wiki/ Husband	Plausible connection	Yes
			This page defining “husband” has a plausible connection to the phrase used in the query.
		https:// en.wikipedia.org/wiki/ The_Underdogs_(prod uction_team)	Plausible connection	No
			This page about the production company of Kardashian’s first husband, Damon Thomas, contains the answer to the query, but does not have a plausible connection to the phrase “husband”used in the query.
[ancient greece]	ancient greece	https:// en.wikipedia.org/wiki/ Ancient_Greece_and_w ine	Plausible connection	No
			The query is a general one about ancient Greece while the page is about a very specific aspect of it (wine). When the query and phrase are this general, the page must be just as general to be plausible.
		https:// en.wikipedia.org/wiki/ Ancient_Greece	Plausible connection	Yes
			This page about all aspects of ancient Greece matches the general nature of the query.
		https:// www.google.com/ searchq=ancient+greece	Plausible connection	No
			The link goes to a Google results page that refers to many meanings of this phrase instead of to a page that refers to only one meaning. It is rated No.

Query	Phrase	Link	Rating and Explanation
[niagara falls]	niagara falls	https:// en.wikipedia.org/wiki/ Niagara_Falls	Plausible connection	Yes
			This article about all the waterfalls in this location is plausible in  the context of this query.
		https:// en.wikipedia.org/wiki/ Niagara_Falls,_Ontario	Plausible connection	Yes
			This article about a well-known city by this name is plausible in  the context of this query.
		https:// en.wikipedia.org/wiki/ Niagara_Falls,_New_Yor k	Plausible connection	Yes
			This article about another well-known city by this name is plausible in the context of this query.
		https:// en.wikipedia.org/wiki/ Niagara_Falls_(provinci al_electoral_district)	Plausible connection	Yes
			This article is about the provincial electoral district with this name. It refers to an unlikely query intent, but is still plausible.
		https:// pl.wikipedia.org/wiki/ Niagara_Falls_(Kanada)	Plausible connection	No
			This article about Niagara Falls is in Polish and implausible in any Englishspeaking market.

Query	Phrase	Link	Rating and Explanation
[play justin beiber]	justin beiber	Justin Beiber https:// itunes.apple.com/ artist/367611351	Plausible connection	No
			Page does not contain any information
		https:// en.wikipedia.org/wiki/ Justin_Bieber	Plausible connection	Yes
			The Wikipedia page refers to the singer Justin Beiber and is plausible.
		Justin Beiber (feat King Gordy) https:// itunes.apple.com/ album/547303596? i=547304294	Plausible connection	Yes
			This is a link to a song called “Justin Beiber” and is an unlikely but still plausible, intent intent.