[HN Gopher] 'Rater' is paid $10 hourly to teach Google's algorithm
___________________________________________________________________
'Rater' is paid $10 hourly to teach Google's algorithm
Author : exz
Score : 70 points
Date : 2022-05-06 14:39 UTC (1 days ago)
(HTM) web link (finance.yahoo.com)
(TXT) w3m dump (finance.yahoo.com)
| Justin_K wrote:
| Another prime example of how ML and AI aren't mature. With all
| the data google collects, they have to rely on minimum wagers to
| "find" the best results.
| xnx wrote:
| In the sense that ML and AI aren't yet AGI, yes, but this
| doesn't mean that Google's ML and AI aren't very useful and
| benefit from having another source of training and validation
| data.
| dfdz wrote:
| This criticism seems unfair.
|
| There is nothing wrong with semi-supervised learning.
| akomtu wrote:
| There is your "AI".
| ggm wrote:
| Surprisingly badly paid. I wonder how strongly quality would
| track pay? Probably they act as a CAPTCHA mutually cross checking
| and so QA is by volume, and not coincidentally demands they work
| in isolation.
|
| Sad story.
| echen wrote:
| Quality control is often indeed by volume. When I was at Google
| / Facebook, search evaluations would typically have 5 raters
| each. One of the difficulties, though, is that search
| evaluation can be quite subjective. And honestly, most of the
| search engine raters we used were low quality and did a bad job
| (so the aggregate answer was often nonsense).
| civilized wrote:
| Dumb question - why can't I just upvote and downvote pages, and
| Google enhances its ranking for me based on that?
|
| Maybe that's why the search "X reddit" is so popular - it's got a
| bit of that social validation.
| lordnacho wrote:
| Who do you reckon would cast the most votes?
|
| Correct, SEO bots.
| civilized wrote:
| Collaborative filtering: not just for movie recommendations
| anymore?
| jancsika wrote:
| If Google can't use it's vast array of user data to tell the
| difference between SEO bots and non-bot users, what's even
| the point of it collecting all that fucking data in the first
| place?
| rednalexa wrote:
| I would assume that if Google is paying 10$/hour for humans
| to rank content, someone else is paying 11$/hour to fool it
| with a human generating content, clicks, etc.
|
| Additionally, people writing bots learn just like Google
| learns, and start generating what appears to be legitimate
| interactions. Same algorithms that fit to critique between
| 'bot' and 'non-bot' class can be used to create bots that
| fall into the 'non-bot' category.
|
| It's a losing battle - but what we get is better than we
| would get if they collected less data.
| [deleted]
| bryanrasmussen wrote:
| Good point, but I have my reasons for suspecting they can't
| tell the difference
| https://news.ycombinator.com/item?id=31296953
| scoopertrooper wrote:
| Maybe that's part of how they train it?
| arthurcolle wrote:
| this used to exist but they got rid of it like 10 yrs ago IIRC
| fshbbdssbbgdd wrote:
| Google already has similar signals:
|
| 1. Do you click on the link in the results page?
|
| 2. After clicking on that link, do you go back to the search or
| not?
|
| 3. How long do you stay there before coming back?
|
| Adding an explicit up/downvote would capture some additional
| information, but it would overlap with the above, and most
| users would ignore the button. I'd guess Google has at some
| point experimented with this and found it wasn't worth keeping.
| echen wrote:
| Note that it's often difficult to tell if these signals are
| good or bad. For example:
|
| 1. Clicking on a link is often a negative signal. If you're
| searching for "when was barack obama born?", hopefully you
| get the answer on the search results page itself, so you
| never have to click. Or, the caption for each search result
| should have a snippet containing the answer.
|
| 2. If you stay a long time on the search result you click on,
| is it because the information is hard to find or because the
| page is deep and useful?
|
| I used to work on Search measurement at YouTube and Facebook,
| and run a human evaluation platform now. Here's an example
| what they can look like when you use high-quality raters:
| https://demos.surgehq.ai/google-vs-bing/
| civilized wrote:
| I'm sure they have all sorts of fancy stuff but at the end of
| the day, there's a lot of SEO garbage at the top of the
| rankings that I would really love to downvote.
| hypothesis wrote:
| That's assuming that the goal is to make search better and
| not say make more money.
| remus wrote:
| > Adding an explicit up/downvote would capture some
| additional information, but it would overlap with the above,
| and most users would ignore the button. I'd guess Google has
| at some point experimented with this and found it wasn't
| worth keeping.
|
| Sounds pretty ripe for abuse too.
| EugeneOZ wrote:
| It obviously doesn't work, otherwise stackoverflow ad-filled
| copies wouldn't exist.
| bmacho wrote:
| Can you give me some?
|
| I love stackoverflow content, but hate the surface, e.g.
| how it loads jquery from google (???) and if I disable it,
| then the content breaks, because obviously the only way to
| implement a toggleable div is when you send your users
| browsing history to google directly..
| EugeneOZ wrote:
| You can find much more than just "some" here :)
| https://github.com/quenhus/uBlock-Origin-dev-filter
| vgeek wrote:
| This is 100% a race to the bottom. Years ago the Leapforce (now
| Appen as mentioned in the article) and Lionsbridge "contractors"
| could average over $18/hr. They would _only_ work on Google tasks
| and adhere to Google 's policies. The work seemed inconsistent
| (people would have 5 hours of work some days, none the next) and
| about what you would expect for AI training. No worker
| protections, benefits, etc.-- all while doing work, albeit
| opaquely, for one of the richest companies of all time. Why can't
| companies at least try to take care of their workers?
| Tenoke wrote:
| I worked for Leapforce from the UK in 2015-16 and I definitely
| didn't reach 18$/hr and the availability of work wasnt too
| great. Probably not bad if you have another gig you can do in
| the mean time though.
| daniel-cussen wrote:
| > all while doing work, albeit opaquely, for one of the richest
| companies of all time.
|
| It doesn't work that way. Contrariwise.
| Proven wrote:
| How much should they be paid?
|
| It's not like you need a PhD to confirm if a search result sucks.
| Results are supposed to be good enough for the average user, so
| you need common sense and some on-job training to do a good job.
|
| And having 10 common-sense guys churning through user feedback
| @$11/h is better than 2 geniuses @$55/h.
|
| As one comment says, it probably beats fliping burgers.
| closedloop129 wrote:
| Where is the limit? Why not 20 guys in a low-cost country
| @$5.50/h?
|
| They cannot rate all results anyway, so why not get the best
| possible evaluations?
|
| Shouldn't quality data beat quantity if you extrapolate from
| the evaluated samples to the entire index?
|
| Spammers target the lower part of the bell curve. If you let
| that part evaluate the quality of the results, do you think
| that they will recognize spam?
| echen wrote:
| It can be suprisingly difficult. A lot of the difficulty is in
| understanding the user intent behind a search query (since
| you're usually rating other people's queries, not your own).
|
| For example, here are some queries in my history - would the
| average person understand what I'm looking for and what a good
| search result would be?
|
| "neovim worktrees"
|
| "django models through"
|
| "pandas econdb"
| rcktmrtn wrote:
| On a related note, am I right in understand that the "Google
| Experts" [1] do it for free?
|
| [1] https://developers.google.com/community/experts/
| ocdtrekkie wrote:
| Yes, Google has tricked a massive army of people into doing
| free work for one of the most profitable companies in the
| world. Things like "Google Local Guides" give people a gold
| star (and maybe a pair of branded socks) for doing unpaid
| labor. It's kinda hilarious, but also sad.
|
| I imagine there are Google executives in a room somewhere
| laughing their full heads off every time they think about how
| there are people who do free work for Google who think they are
| making the world a better place or something.
| omoikane wrote:
| I can see how the gamification aspect of points and levels
| might be more motivating for some people, but mostly I see
| Local Guides as not too different from people posting
| pictures and reviews to various social networks, except that
| it is more discoverable by people using Maps. I submit
| contents to Maps for mostly the same reasons why I use social
| networks and expect nothing in return.
| vba616 wrote:
| >laughing their full heads off every time they think about
| how there are people who do free work for Google who think
| they are making the world a better place or something.
|
| So when you go to a store that Google Maps says is open, and
| find it's closed, you refuse to correct the hours and go
| "MUHAHAHAHA...I _refuse_ to help Google, take _that_ ,
| corporate scum!"
| ocdtrekkie wrote:
| Why would I bother checking Google Maps? But basically,
| yeah. If Google wants my expertise and data collection
| ability, they can pay me for it. As it stands now they just
| cold call businesses and ask them with an AI thing.
| nine_k wrote:
| What other source would you rather check?
| ocdtrekkie wrote:
| I always check the actual website of the business in
| question. Search engines are wrong like 40% of the time
| in my experience.
| pietervdvn wrote:
| OpenStreetMap?
| nine_k wrote:
| Okay! I just opened OsmAnd and looked up my local
| supermarket. Sadly, no work hours. And this is a very
| popular supermarket right next to a train station in
| central Brooklyn, NY.
|
| The thing is that business owners do update the info on
| Google Maps, and have an incentive to do so, but only
| volunteers update OSM.
|
| OSM shines where businesses don't care but volunteers do:
| trails in a park, non-commercial POIs, etc.
| acheron wrote:
| What? Why on earth would I correct it?
| Dylan16807 wrote:
| I'm sure you can figure it out. I believe in you. (And
| that response is because I'm 100% treating your question
| in good faith and not "I know why I would, but also I
| know why I wouldn't and prefer that, so I'll pretend I
| don't know why I would".)
| rurban wrote:
| When they tried to lure me into a "Local Guide" I was offered
| a free flight, hotel and food at Menlo Park. Once, but still
| good enough for no work at all.
| washadjeffmad wrote:
| This is probably one of the more embarrassing and shameful
| parts of Google's labor ecosystem.
|
| These "Google $title Experts" work as uncompensated customer
| support (a legal requirement that they treat as such, if the
| quality is any indicator) on various social media outlets like
| Reddit and Twitter. They don't just provide free labor, they
| also go to bat for Google's reputation, acting as marketing and
| PR liaisons to prevent backlash from users who've been failed
| by their official support processes.
|
| One 'expert' I spoke with justified working for free by the
| "connections and access" they received. When I asked if Google
| would be flying them out to the events they have access to to
| meet with the people they have connections with, they said they
| were told they would have to cover their own travel and
| lodging. I suppose if Google doesn't pay them, they never have
| to worry about any Experts connecting and starting a union.
|
| We're talking about a multi trillion dollar company. Absolutely
| ridiculous.
| kurupt213 wrote:
| Like the Reddit super mods
| rurp wrote:
| > They don't just provide free labor, they also go to bat for
| Google's reputation, acting as marketing and PR liaisons to
| prevent backlash from users who've been failed by their
| official support processes.
|
| This is so true. When I try to find a fix for some broken
| behavior in a major Google app the most common place I end up
| is a support forum with zero solution, no comment from any
| Google employee, and a volunteer support person saying how
| the defect isn't actually a problem but that you should
| submit feedback within the app.
|
| These volunteers can't be helping at this point. Google's
| disdain for users is pretty widely recognized by now and
| seeing a whole bunch of frustrated users being blown off yet
| again doesn't help.
| Gravyness wrote:
| Definitely explains the quality of search as of late.
| amelius wrote:
| Those raters must really love Pinterest ...
| TheGoddessInari wrote:
| They (and everyone else) have been doing this forever, if not
| obvious
| nyhwtt wrote:
| Better than working at Wendys!
| Crosseye_Jack wrote:
| Depends on how you look at it, The wage fits in around the
| rough ave pay for a Wendys "crew memeber", however you are
| responsable for your own costs while working (computer,
| internet bill, electricity bill, heating) but on the flip side,
| its part-time, no comute, no long hours standing, no dress
| code.
| hobo_mark wrote:
| Ha, so they still do it, that is how I paid rent for a few months
| back in 2012.
| advisedwang wrote:
| I thought the management theory was to in-source your core-
| competencies and out-source everything else. Surely quality
| search results is one of Google's core competencies!
| kurupt213 wrote:
| If you use Google you know the search results have been getting
| worse and worse
| ocdtrekkie wrote:
| Google's core competency is ads. Any time you click a search
| result, it's a failure: Google wanted you to click the ad above
| it.
| kappuchino wrote:
| Ex Google Search quality Rater here, based in Germany. From 2004
| to 2006 I worked on SQR and "special projects", with a much
| better pay (maybe because of german circumstances). I've been
| tracking the SQR-guide since then (public version here
| https://static.googleusercontent.com/media/guidelines.raterh...),
| so I have still some current understanding. If you have
| questions, I'd be happy to try to answer them since my NDA has
| expired ;-).
| demadog wrote:
| When those reports come out that "The average Google employee
| makes $220k a year" or whatever, it's totally misleading since it
| doesn't account for all the outsourced labor at $10 an hour that
| is critical to tuning their algorithms.
| Nexxxeh wrote:
| I did voice training tasks for Google through a 3rd party I
| can't remember the name of. I was recruited via /r/beermoney or
| something. I enjoyed it, and it was easy work that fit around
| my schedule. It also gave a feeling of improving the lives of
| users, which was cool.
___________________________________________________________________
(page generated 2022-05-07 23:01 UTC)