Task Goal: Identify if the Search Ad present, has a relationship to the user intent.
What Makes an Ad Relevant? Ads can be relevant to the user intent in several different ways. They can be exactly what the user was looking for, but they can also be something that the intent implied by the user’s query suggests they might be interested in.
Research – The best way to research a query is to type it into a search engine and add words like “app”, “game”, “iphone” etc. if the original query isn’t specific enough. Use the search engine links in the tool to start your search. If you have access to an iPhone or iPad, you can also try searching the App Store directly.
Relevance Selections:
- Excellent – the app has a strong relationship to the query intent and is among the most likely to be of interest to the user.
- Good – the app has some relation to the query intent and is quite likely to be of interest to the user, but other apps might be more compelling. The weakness might be in the app’s function, themes, and/or perceived target audience.
- Acceptable – the app has a slight relation to the query intent, and the user would not be surprised to see it as an ad, but they aren’t that likely to be interested.
- Bad – the app doesn’t have any relation to the user intent, and/or seems likely to surprise the user and make their experience worse.
Game Queries- The relevance of a game ad should be based on factors like:
- Gameplay – what sort of game is this? What are the skills it requires, or activities it requires the player to do? For example, a puzzle game that requires slow careful thought versus an action game that rewards quick reactions
- Presentation – what does the game look like? What sort of themes does it center around? For instance, a game themed around World War II featuring realistic weapons, versus a cartoonish farm building game.
- Audience – who is this game for? Does it seem to target particular users? For example, a game themed around colorful cartoon characters intended for small children, vs a shooting game intended for more mature users
Common Mistakes
1.) Apps from the same developer cannot be rated below Acceptable even if there is no similarity between the query and the result app.
Example:
a.) Query: Family link
Result: Gmail – Email by Google
Comment: Acceptable-The apps have nothing in common except the the fact that both are developed by Google. Therefore, the correct rating would be Acceptable.
2.) Excellent rating should only be given to apps that are either directly providing the user with what they’ve Queried, or a direct competitor of the queried app. Ads of apps that have only a slight relation to the queried app should NOT be considered as Excellent.
Example:
a.) Query: houseparty
Result: Gaze – The Live Video Chat App
Comment: Good/Acceptable- Both apps are for video calling, however, Gaze is created to have a video call with strangers and make new friends. On the other hand, Houseparty is a proper video calling app for Friends.
3.) There’s no need to be lenient for apps that have no relationship, matching functionality, nor same developer.
Example:
a.) Query: cisco webex meeting
Result: LinkedIn: Job Search & News
Comment: Bad- Although both the above apps may be used in business environment, the functionalities of both apps are very different. Therefore, Bad would be an appropriate rating in such examples.
4.) On the other hand, please do not straight away rate Bad for results that are different from the query. It is important to conduct enough research to decide if there is ANY relation between the two apps. If there is even a slight connection between the two, you should choose between Good-Acceptable depending on the extent of connection.
Example:
a.) Query: youtube for ipad
Result: Uplive-Live it Up (Live Streaming App)
Comment: Good/Acceptable – Youtube is a Video sharing platform which also has an option of LIVE streaming. On the other hand, Uplive is limited to LIVE streaming. The user would not be totally surprised to see an ad for this app.
5.) Please take notice if the user would be interested in the ad, based on the query.
Example:
a.) Query: Class Dojo
Result: IXL – Math, English
Comment: Good or Acceptable- Class Dojo, is a communication method between parents and teachers. IXL are online classes, which is likely to be of interest to the user with this query as the app serves parents of students It should be not be Excellent, as the app is not in the same exact category as the queried app.
Example
b.) Query: snapchat
Result: Hashtag Generator
Comment: Acceptable: This ad app, may not be supplemental to Snapchat, but in this case, the user has indicated an interest in using social media, and while SC does not use hashtags, the ad would not be surprising to the user with this type of social-media-related query. A rating of Acceptable fits
this situation. Please note that this explanation is specific to this case only, and other cases must be considered on an individual basis.