I started Turking a few months ago, made a quick couple hundred dollars, and then learned how MTurk's system leaves workers too often out in the cold. The site is not well maintained, and it functions far below Amazon's standards of customer service and user optimization. A Turker faces seemingly random blocks with little recourse. Spam tasks are not well screened. The interface needs basic improvements. Development is frustratingly slow. Amazon has heard these criticisms many times over but mysteriously has done little in response.
The point: a company as sophisticated and customer-service oriented as Amazon should pay more attention to its flagship crowdsourcing operation. MTurk should be nimble and trustworthy. It should not treat its best workers the same as its worst. Users should be given the benefit of the doubt, or an opportunity to establish a meaningful reputation. Site search should be robust. Task and profit tracking should be personalized and detailed. The site should be easy and efficient to use, and new features should be added regularly in response to user input.
The purpose of this blog is not to badmouth Mechanical Turk or Amazon.com, but to help them make a better system which will attract more people to use it, contribute to it, and benefit from it. Let me know what you think...
Four out of ten HITs are spam. Surprise.
So says NYU business professor, Panos Ipeirotis. Here's his finding:
Here's the link to Panos' full post describing his research: http://behind-the-enemy-lines.blogspot.com/2010/12/mechanical-turk-now-with-4092-spam.html
The results were disturbing. Out of the total of 5841 HITs, a total of 2390 HITs, or 40.92% were marked as spam HITs.Turkers have a feel for what to avoid. If it sounds too good to be true, if it looks fishy, if the requester is unknown and offering $10.13 for a ''quick HIT'', we try it once, and don't try it ever again. But why are these tasks still on the site at all? They're painfully obvious to notice (except for newcomers), and they bring down the efficiency and reputation of the site. Maybe Amazon will do something about it.
Here's the link to Panos' full post describing his research: http://behind-the-enemy-lines.blogspot.com/2010/12/mechanical-turk-now-with-4092-spam.html
Subscribe to:
Posts (Atom)