Playing the Google Game as a Search Engine Evaluator

google.ratersEver heard of making money online with a company called LeapForce?   They hire people as independant contractors and the work is about helping google collect online data.  Besides Leapforce, there are other similar companies like Lionsbridge,  Butler Hill and Work Force Logic.   Depending on the company, this work of google raters might be referred to as an “internet assessor”, “search engine evaluator” or “google quality rater”.

Think of it as a form of the old adage ‘judge a book by its cover’ type of evaluation.   How quickly can one determine relevance and quality content after arriving at a web page?   As you’ll see below, these ratings are done VERY quickly, so it is helpful to know what the evaluators must see to give a site higher rank.  This article only refers to organic search results, however these evaluators also rate Google ads (AdWords), videos, etc.

How to Apply  To do this kind of work, you do not apply with Google.  One must submit an application and resume to one of the companies who contract with Google and hire independent contractors (named above).   This is not a new practice for google, as they’ve been contracting this type of assistance since at least 2004.  They pay monthly and workers must keep detailed records of work completed and submit them to the site.   The pay rate begins at $13.50/hour on the low end with higher pay for some tasks and also some opportunities for bonus pay.

Best Candidates  Internet marketers are not necessarily the population sought for this position. Rather they are seeking normal google users who are preferably highly educated, and also proficient in their abilities with written language, organization skills and working independantly.  Those speaking more than one language may have the advantage of more work available, since google of course has a worldwide market.

Preparation and Entrance Exams   If accepted, the next step is a series of entrance exams.    Preparation for these exams and time invested taking them is clearly no small task.   The exams are quite lengthy and do require a large investment of unpaid time to complete.   One reviewer posted that there are “more than 50 different documents for different functions of the rating task”  

Section 1 of the exam consists of 24 theory questions. Section 2 is much longer and consists of 270 practical exams (site evals that simulate the actual ratings).  Beyond that, no information about the exam is available because applicants are required to sign a nondisclosure agreement regarding this information.

Why the Need for Google Raters?  These companies are contracted by google to help evaluate their algorithms and determine how well they get relevant results for search engine queries.  So these search engine evaluators do not directly affect a ranking (porn sites being a possible exception). Their evaluations will not directly effect rankings, but indirectly they may certainly effect them if their evaluations result in algo changes (more on that below).

The Impact of the Data  The work of these evaluators can have an effect on the algorithms used by google.   The data they report benefits Google in determining how well changes to their algorithms perform.  For one example, if an algorithm change results in a spam site appearing near the top of Google search rankings, then the evaluators will quickly flag it as such.   This alerts programmers that a change is needed in the algorithm.

What Do Google Search Quality Raters Do?

Raters for google search organic results are given two types of rating assignments.

1) Keyword Relevance Raters are given one keyword and one url with the task to rate the relevance of the url to the intent that it matches the keyword search.   The quality raters handbook describes the ‘intent’ of the searcher as a keyword being a DO, KNOW or GO type keyword. So the rater decides if the keyword is relevant to something that searcher would want to:

  • DO –  buy X, like X or watch X
  • KNOW – how to X, X reviews, or X info
  • GO – go to X (link)

Then the rater looks at the url or domain name and rates whether it is relevant to that specific search query.  Here there’s also an opportunity to mark that url as spam.  This acually DOES include the option for the evaluator to rate a URL as both (spam AND relavant).   While this may seem strange, keep in mind what spam triggers might include – such as heavy loads of ads and especially banner ads. For example, if the keyword is ‘buy chain saw” and the site does indeed offer ways to buy a chain saw, but also includes lots of banner ads and other ‘spam indicators’ – that site might get the double rating (not good).

2) Comparison Rank The second kind of assignment for these google raters is to evaluate or compare two sets of search results.  This is based on first page google search results for any particular keyword query. The two sets of results are compared, both before & after a google algo change. So the raters are asked to assess or rate which set of results is better or more relavant, in their opinion.  These assignments are referred to as SXS (side by side) tasks.

Matt Cutts of Google’s Webspam Team Explains the Role of Google Search Raters

Performance Requirements for a Google Rater

Quality raters are paid by the hour, so there are minimum performance levels required to maintain. This is based on the number of urls or website domains they rate per hour.   For those assignments to rate a single URL & keyword, the minimum per hour APPEARS to be about 30. Consider the comment below:

“I believe that 30 is the minimum expected for rating, within a few weeks after you’re hired (not including the test week). You’ll get an email if you fall below standards. They stress quality over quantity, however, so that overall, a slower rater who rates really well may be preferable to a fast one who misses the mark more often. “May” is the operative word, we don’t know what the criteria are for judging our work.”

The above and other comments were selected from an online forum where raters from all of these companies post about their experiences.  You can read them all at this google raters forum.

Raters receive progress reports regularly on their performance on a star-rating type system. However, the progress reports themselves appear to be ambiguous.  One rater commented on the forum that after getting a 2nd progress report, their productivity was rated at only at 3 stars after rating between 45 to 50 websites per hour.

This begs a few questions like:

So a worker doing 45-50 website ratings per hour is given only 3 stars?!

How can these evaluators accurately ‘rate’ any website within only a minute or two???

Another google rater commented:  “It’s not possible to do 20 SxS (side-by-side) tasks an hour if you click through to each result. It’s pretty easy on the occasions when the sides are nearly identical and you just have to decide whether you’d rather have A or B somewhere. When I was doing 20 I did a lot based just on snippets… “

IMPORTANT:  Therefore your site is often judged simply by the snippet in the search results! It’s quite possible no one even LOOKS at your actual website at all, only the meta description!

Google’s Engineering Director, Scott Huffman was interviewed about this practice in 2009.  When asked:

“Does this data have any effect on the site page ranks or pay-per-click and Ad Word bids?”

Reply from Scott Huffman:  “We don’t use any of the data we gather in that way. I mean, it is conceivable you could. But the evaluation site ratings that we gather never directly affect the search results that we return. We never go back and say, “Oh, we learned from a rater that this result isn’t as good as that one, so let’s put them in a different order.” Doing something like that would skew the whole evaluation by-and-large. So we never touch it.”

These human raters are just one piece of the puzzle for google to determine relevant ranking status. Another important step in evaluating these algorithm changes is then to release these results to a test population of actual human searchers, those who have some actual vested interest in the topic.   See below an infographic direct from google about the big picture of what goes into a decision regarding algorithm changes.

Note that the section in dark blue on the right side saying ‘send differences to external raters’ is only one piece of the Google algorithm puzzle.


How Google Makes Algorithm Change Decisions

google search testing

Important take-aways to remember:

Web publishers who seek to improve organic traffic results from google might consider the following.

1) Evaluate intent and relevance. Consider your target keyword(s). What might YOU expect to find after typing that phrase in at google? Is your content (and URL) truly relevant to the intent of the keyword used to find your web page?
First impressions an any website DO matter BIG TIME!  Within a few seconds of landing on any given web page, we all tend to cast judgement, to either click out or maybe stick around. The rater is doing just that, or rather looking for relavance.   So is it crystal clear what your site content is about?
2) Double check your snippet (the meta description) to make it as relevant to your target query (keyword) as possible.  Google can and does auto-generate these, still DO try and control what you can.
3) Be Proactive – Evaluate your own site as a CONSUMER might, not as a marketer. As a consumer, would you stick around? Would YOU possibly click on and subscribe or maybe buy something?
4) Don’t shoot the messenger. Remember that these raters are just people trying to earn some money online from home.  They have no vested interest at all in what happens to your web page.  Rather they just are trying to do a job, one where the expectation appears to often be left ambiguous.  In many cases they clearly are not sure what is expected of them, given many of the forum comments.  Likely many do not know or simply don’t care how their daily actions fit into the BIG picture.

Last, one who seeks a bridge job (needs income to supplement online solopreneur income), might indeed find this to be a worthwhile opportunity.  While earning a decent hourly wage from home, the evaluator also gets an inside look on a regular basis at specifically how keywords and other variables effect SEO rankings, crucial info for most web developers.  There’s a bit of unpaid work involved to get started, same as many jobs, still good potential.

I have not become a google rater, but after researching this role heavily it is on my list of roles to consider for legitimate work from home opportunities.  I do know a few people who give Leapforce positive reviews.

In the past, one of my income streams has come from affiliate marketing and since my state was recently hit by the Amazon Nexus Laws, that income has dwindled considerably.  I’ve already begun a few new projects to replace that income, so my main focus will be on firing those up for the next few months.

Is understanding this ‘google game’ info helpful toward small changes you might make to increase SEO and organic traffic to your website?   Would you possibly consider being a search engine evaluator as a bridge job to earn extra income?  Those who invest the time and effort to train as a Google Rater would be in a position to publish that ranks high. Please comment below.

Carolan Ross

CFO(Chief Fun Officer) at SoloSpark
Freelance writer, creative soul, solopreneur and former teacher who networks with rebels, visionaries, artists and other square pegs in round holes. I support solopreneurs with copywriting and juggling life and business to SHINE ONLINE!

Leave a Reply

Your email address will not be published. Required fields are marked *