Beware the block (Part 2)

Here's the second installment from the great MTurkforum.com post.  I'm jumping in mid way through because the first half are full of real screen shots from the requester interface.  Worth checking out.  Here's the advice he gives:

So what do I do if I feel I've been blocked unfairly or by mistake?

First, contact the requester as soon as you realize you can't do their HITs anymore. Don't waste any time, because the longer you delay the more chance they'll forget the reason. They could keep a copy of the reason in their records, but they don't have to.

Be polite. Explain that you were surprised at being rejected and blocked after submitting what you thought was good work. Ask them to please tell you why the work was rejected, and if it was your mistake then tell them you're sorry and you'd like to do better. Ask them if they would please consider unblocking you and giving you a chance to prove yourself again.

If it turns out they blocked you by mistake, whether a careless error or whatever reason, ask them to please remove the block so that you can continue submitting the good work you've always enjoyed giving them.

But what if they don't respond?

In that case, you should contact Amazon. But be aware that Amazon may at first tell you that they can't get involved because it's a matter between you and the requester. They don't consider it their job to judge whether your block was fair. If they take that attitude, contact them again and ask for a supervisor to review your account and assess the true situation.

If that still doesn't get you anywhere, then please write about your experience here. I am willing to contact the requester and Amazon on your behalf if you can't get anywhere on your own. I don't have any special influence with Amazon at all, and I've never been blocked as far as I know, but I'm willing to go to bat for workers regarding the issue of having more rights.

In part 3, he explains what happens when bad blocks linger.  I'll give you a hint... BAM!

Beware the block (Part 1)

I'm ripping this straight from MTurkforum.com.  It's an awesome 3 part description of what happens on the requester side which makes the blocking policy so unfair for Turkers, asymmetrical, if you will.

Beware the Block

Let me start by saying that I believe that most people who get blocked by a requester probably deserve it. In many cases, people submit poor work, receive a rejection with a note explaining why the work was rejected, then go on to submit more poor work. I can't blame a requester for putting a block on a worker who does that.

However, I've also learned recently that sometimes people are blocked who don't deserve to be. Sometimes despite submitting good work consistently over a period of time, a worker gets blocked suddenly and without explanation. As I'll show in this series, the blocking system is deeply flawed and stacked against the worker, and has the potential to harm even the honest among us. Please review with me some excerpts from the instructions Amazon provides to requesters about processing HITs.

He goes on to explain some shocking facts:

•  Requesters can block up to 100,000 people
• They don't have to give a reason
• You don't get an email when it happens
• They can block all of your work in a batch after looking at only one HIT

...and there's more you must check out yourself here.

Transparency is good for business

This commentary from the MTurk.com forum got right at the point.  Transparency is good for business and anything else is... not:

I wonder why they can't lay out the violations for workers in as much detail? Or is it somewhere written where I haven't looked yet? I have read the participation agreement and apart from "don't use bots or scripts" it doesn't tell me much about what I'm allowed to do and what not. I have been reading here and in the other forum of workers who's account was suspended, apparently for violating the TOS, but they all have no clue what they did to be punished so harshly, just like Nisha here. And MTurk isn't telling them either. It's just a very weird and rude way of dealing with people, not at all fitting for a 21st century business, and it leaves many workers tense and wondering whether they do everything right according to their mysterious terms, or not.  link

It happened in India

Mechanical Turk is not surprisingly used by people all over the world, especially where the exchange rate makes the HIT payments seem relatively rich.  Unfortunately, good workers from anywhere can be banned without explanation,  Don't mind the language differences, it's the same story:

---

Hai Friends,
I came to know about this forum today. I am working for a year and had no problems till March 31. On March 31 while I was working suddenly I got a message that My account was suspended. I asked for explanation. They replied Our system detected activity on your Amazon Mechanical Turk account that we believe violates the Participation Agreement/Conditions of Use, and have terminated your MTurk account . I don't know which one was violated by me and I asked this several times but didn't get any reply. At the time there was about 5000rs in my account and I lost that money. This was a great shock for me because I have a 4 year old child and has hearing problems. He was in treatment. The treatment and our daily life expenses was meeting with the income from mturk. when they suspended the account I am in such a situation how to find money for these. I am not able to go for work because of my son's treatment. Could any one help me in reopening the suspended account.  link

Welcoming the newcomers

A classic.  This guy works in a college admissions office.  He wasn't treated like it:

Hello, I am new to Mturk and have only been working for about a week, I have around 4,000 submitted hits with only 2 rejections which gives me a 99.9% approval rating. I have not been banned from working from any requesters nor have I received any communication as to anything I was doing wrong. However last night Mturk suspended my account, in the email they sent me they stated it was because I have a low approval rating. I sent them an e-mail asking them to explain to me how I have a low approval rating, and they sent me an email back saying they were forced to suspend my account because of a consistently low approval rating, and that I am terminated, and may never open another account again, and that I will not be paid for any hits I had pending... This all seems a little extreme for just two rejected hits. I am a newbie so maybe I misread and everyone is supposed to maintain a 100 percent approval rating all the time or be suspended.. I am just very very confused as to why this happened, and curious as to whether this is normal practice for Mturk. I was also wondering if there is any possible way anyone knows that I could get my account reinstated.  link

It turned out this guy poured a full week's worth of energy into categorizing college courses, right up his alley.  But he happened to stumble upon one of the less reliable requesters on the site and BAM.  Low approval rating = suspended = no reinstatement.  Ever.  On Amazon Turk, one bad batch of HITS can get you banned from the site.  Permanently.  Really?  Really.

When the bell tolls... for you.

This is from TurkerNation.com:

Finally it is happening, after worked for MTurk for two years I am banned for no reason.

This is the reply from the customer support:

Hello,

Our system detected activity on your Amazon Mechanical Turk account that we believe violates the Participation Agreement/Conditions of Use, and we have terminated your Amazon Mechanical Turk account effective immediately.

Additionally, we do not disburse funds for suspended accounts.

And then I fight back, what did I violate? I worked by myself and approval rate is more than 99%, what's wrong?

And here is the second mail:

Hello,

I'm sorry for any concerns but your account was closed due to a violation of our Participation Agreement and cannot be reopened.

Any funds that were remaining on the account are forfeited, and we will not be able to provide any additional insight or action.

You may review the Participation Agreement/Conditions of Use at this URL:

http://www.mturk.com/mturk/conditionsofuse

Thank you for trying Amazon Mechanical Turk.

Well, I think what this guy said is true: http://blog.kyloo.net/2010/05/16/why-mec....how-to-save-it/

MTurk treats workers like s***, and please go on banning ppl for no reason and steal our money, without serious workers, who will work for you?

Suggestions for workers: Transfer every penny you earned.  link

Not good.  Not good.

Why is this necessary?

"If you want help with a mTurk suspension, please PM spamgirl & include your email address or Worker ID & a summary of what happened. THIS DOES NOT GUARANTEE YOU WILL GET YOUR ACCOUNT BACK! If you have done something to DESERVE the suspension, DO NOT PM!"  link

That's what you see at the top of the most popular Mechanical Turk forum website.  Really.  What would it mean if the top of forums.coca-cola.com was a message about getting refunds for bad bottles, or if the top of forums.bmw.com was about returning lemons, or if... you get the idea.  This is a problem so pervasive that it's literally pinned to the top of the message board in flashing, bold, letters.

Amazon, are you paying attention?  Jeff, can you hear us???

A simple idea: reward good workers

One of the problems which immediately comes up in any open system is reputation--trust.  On the one hand, Amazon keeps track of your HIT approval rating, how many tasks you do are accepted vs. rejected.  That lets requesters know if you have a history of good work.  Oddly, sadly, that rating is not considered when Amazon decides to suspend your account.  That means a two-day old Turker who goes on a scamming binge is treated the same way as a multi-year Turker who accidentally accumulates one or two blocks before BAM, account gone, reputation irrelevant.  


Here are some telling comments from the TurkerNation.com forum:

• Yeah they don't consider ratings since those can be scammed by doing work for people who never reject but they should consider the whole account. I also had a lot of exclusive trusted worker type qualifications. So even if a couple requesters did block me and not want me working for them any more there were many that trusted me, liked my work, and wanted me to keep working for them. But they only saw I had blocks and banned me. If that is even the real reason since they changed their reason after a few emails.

• Everyone always lists their approval ratings when they get banned and I bet amazon doesn't consider it. Lets say you have been turking for a year, 99% and over 100,000 approved. You can scam the system and get 50,000 rejected and still be around 75%, which is where some new turkers fall by doing bad hits. Obviously that's a good way to scam the system. So, amazon probably looks at the past 7 days for low quality. If you have 500 rejections and 500 approvals in the past 7 days, thats pretty horrible work. And i bet amazon doesn't consider rotten requesters that will mass reject for no reason. The system is already set up that way that requesters can do no harm, and can create as many accounts as they want, but workers have 3 strikes and they won't tell us a damn thing.

• Lately I've only been doing hits I'm really sure about and I looked carefully at just those two. I only did 2 where the color in the description matched the picture, the size was stated in the description, etc just to be sure. And then I get warned over 2 hits......give me a break. After accumulating over 350,000 approvals with only a little over a hundred and thirty rejections I can't be doing that low of quality work. And about a hundred of those rejections were from produniverse when several posters were having probs with them. Its a shame anyone with a 98 or 99 approval rating could be banned so easily, but it seems like it could happen to anyone at any time. 

•  Really a few blocks shouldn't matter under the current system as iambrendalee said. Unless they send in along with the block something horrible "I want them banned" "Obvious scammer" then it should be treated as it is, a block from just that worker and not the whole site.  My past blocks were very old(over a year) and all misunderstandings but because of Amazon not making it easy for Requesters they didn't know how to lift the block. Even then an older answer from Amazon said even lifted ones do count against you.  If they are going to count so badly against you the least they could do is send a warning email when you are blocked but I never got any. 

• I say, unless someone is clearly abusing the system, then they shouldn't get banned. Requesters block people randomly, their approval / rejection systems go on the fritz and randomly reject, bad requesters reject for no reason at all...those things really shouldn't have an effect on our accounts. In what world should a sleazy (or dumb) requester or two...or three be able to determine whether I can continue working or not?  The thing is, we're disposable to mturk, so it's all moot. It's like they're looking for any excuse to get rid of us...which is just plain weird, considering they need us. 

• On a side note, stories like this are what make me paranoid as a turker. I have done this, on and off, since March of '08. It worries me that even with many thousands of hits completed properly, and an approval rating of over 99%, that a person can just arbitrarily be banned with absolutely no recourse at all. I know MTurk isn't a large priority for Amazon, but have they ever actually come out and explained why there is so little support for their work force? 

• Maybe you should take blocking more seriously. I mean, it only takes three irresponsible requesters blocking you before you are completely banned from mTurk. There isn't room for requesters who block even by mistake. 

I didn't post links to these, but you can just go to the search box and type in 'banned' and you'll see hundreds of posts come up.  Maybe, instead of scaring the pants off of loyal users, Amazon could leverage the reputation data that they have famously developed at Amazon.com and start treating Turkers more the customers and less like the commodities.

Close calls and near misses

There was another trainwreck posted on the boards.  This one was escaped, but barely, and not without the usual panic, confusion, and hand-wringing.

Stage 1:  Surprise... You're Banned!

"Although I’m new to these forums, I have been turking for about 7 months. I used my Mturk account without issue in the morning and when I turned my computer on in the afternoon and tried to log in, instead of my dashboard I had a message stating my account was suspended by the Mturk team.

During my time on Mturk I have had about 7,000 HITs approved (many of these being transcription HITs for both CastingWords and SpeechInk) with only 50 rejections for an approval rate of 99.4%. I had just passed the $1100 mark a few days ago. I have searched through these forums trying to find information by reading posts by other workers who had their accounts suspended, but I still have a few questions I was hoping to get some help with.

Upon trying to log in and noticing that my account was suspended I checked my email account hoping to find an email with some sort of explanation or at least a notification that the account was suspended, but I have not received any type of notification from Mturk. Do they usually send an email as soon as your account is suspended or must you contact them first if you hope to receive any type of explanation as to a reason for the suspension?

...I have used the contact form on the Mturk homepage in hopes of getting some more information on my suspension, but other than that I’m not sure there is anything else I can do."  link

 Stage 2:  Beg, Borrow, and Plead... Thank Spamgirl

"Just checked my email and my account has been reinstated
Thank you to spamgirl for anything she was able to do to get my suspension reviewed.
I still hope that the PED Research block situation is resolved for everyone that had that block put on their account unfairly."  link

This one turned out ok, but was it really necessary?  If this is what can happen to even the best Turkers, what does that say about Amazon's sense of customer loyalty and of rewarding trust?

A Turker horror story

It's almost Halloween, so a good fright tale is in order.  It began one morning on Mechanical Turk.

Greetings from Mechanical Turk.

We regret to inform you that we have suspended your Mechanical Turk account effective immediately. This action was taken because we received repeated complaints from Requesters regarding the quality of your work. In particular, you were blocked by several Requesters.

You may withdraw any remaining balance on your account by logging on to your account on http://payments.amazon.com If you have an Indian worker account your balance has already been automatically remitted via check.

The Mechanical Turk Team  link

That's how the story begins.  Here's the horror part:

This is my only source of income. I've done everything by the f****** book. I'm losing money now because of this. 

Classify It blocked me and refused to respond.
PED Research blocked me cuz they didn't know what to do.
Now I probably got a block from drawing boxes around people. 

Nothing else. I've drawn plenty of boxes around so I don't see how I could get it wrong. Did I miss someone in the distance maybe? Does that merit a block cuz I couldn't see a tiny person somewhere? I don't know. i don't even know if they blocked me since I didn't get another block email.

Now, i have zero income until this gets resolved, if it gets resolved. Being unemployeed is hard enough but at least I had mturk so Iw ouldn't have to borrow money for food or anything. Now I get to go back to being unemployeed until this straightens out.  link

I have some sympathy for Mechanical Turk.  I'm sure they get a lot of bad workers who try and scam the site.  But what about the people who do little to nothing wrong, who have been working for months, doing thousands of successful tasks, only to have a few aberrations result in this.

Is Jeff Bezos listening?  Is this what Amazon.com sounds like?

Silence... Scary.

Spamgirl the superhero

The mark of a mature system, whether in politics or technology, is that it doesn't require special pleading.  It works as designed; you don't have to call your congressman, or start a blog, or go see Spamgirl.  Who is Spamgirl?  Don't be fooled, her name is ironic.  She's the supermoderator at the TurkerNation.com forums and when the ban-monster strikes, she is the only one who can fix it.  Seriously.  Listen to Spamgirl:

• "Listen, I emailed them... it takes days to get a response, they're kind of inundated with this sort of thing. We just have to wait and see."  link

Spamgirl has received so many please-help-me emails that she put instructions on how to deal with it on top of the Turker Nation forum.  Banning complaints happen all of the time and generally, no one cares.  Sure, some of the blocks are warranted.  But I really have to ask about the system that Amazon has set up to handle complaints.  It looks something like this:

1. Do a bunch of hits
2. Get an email notifying you your account has been suspended
3. Email Amazon in a panic
4. Get a canned response that you were blocked too many times
5. Email the requesters who blocked you
6. Find out it was a mistake
7. Go tell Amazon
8. Get canned response that Amazon doesn't interfere between Turkers and Requesters
9. Go ask requester to contact Amazon directly
10. Wait, wait, wait
11. Go get Spamgirl

Seriously, this is what happens daily over at TurkerNation.com.  Does it mean that no one deserves it?  Of course not.  But on a completely anonymous site where thousands of workers are completing thousands of tasks, you would think that Amazon would make it a little easier to deal with these kinds of issues.  After all, it only takes 3 requesters blocking you to be permanently banned from the site.  Much of the time, the blocks are accidents or misunderstandings.  But recourse is slim and next to impossible.  Unless you have superpowers.

Unaffiliated with any blog, site, forum, Turker, poster... as always.

Why does this blog exist?

This blog exists to improve the functioning of Amazon.com's Mechanical Turk.  The site is a crowdsourced, microtask nexus, very web 2.0.  Simply put, it's where piece work lives on the internet.  Pennies, nickels, dimes, quarters... no amount is too small and no task too menial for the site.  That's not a criticism; the availability of a place where companies can find intelligent humans to complete repetitive tasks is an enormous asset.  The problem is that this site is not run anywhere near Amazon.com's reputation for great user interfaces and unparalleled customer service.

The purpose of this blog is to do one thing and one thing only:  Get Amazon to improve Mechanical Turk so that it is up to the quality standards for its users as the rest of their sites.  When that is done, or on track, this blog will go dark.

Here is the main complaint:

Amazon has a well-publicized no-interference policy on Mechanical Turk.  This means that what goes on between a Turker (freelance task-doer) and a Requester (task-poster) is not governed by Amazon.  Aside from fraud or blatant terms-of-service violations, what happens between a Turker and a Requester happens behind closed doors.  Except it doesn't.

If for any reason a requester issues a ban on a particular Turker, which can happen due to mistakes, accidental problems, misunderstandings, or sheer malice (in addition to the legitimate reasons), the ban gets marked down in Amazon's central database.  If enough bans are logged (say, 3), the account of a Turker is permanently banned with no recourse and no consideration for how long they have been 'Turking' or how high their ratings were prior to the ban.

What does that look like?

* A user who had completed 60,000 HITS with an over 99% approval rating was summarily banned because of an admittedly mistaken review from one requester and an otherwise normal weekend of work.

What this is getting at is that Amazon.com is letting its most trusted users of the Mechanical Turk site be tossed with indifference.  When the matters are brought to their attention, the canned response is:  this is between you and the requester; we can't do anything about it; it doesn't matter how long you've been there or how great your work has been; we'll pass this on to our developers.  Well, until the developers fix it, this site will exist to document every banning of a quality user, users who could have been this site's ambassadors but instead are being spurned with bureaucratic dismissal.  Does that sound like a site Jeff Bezos would be proud of?  Does that sound consistent with the mission to be the world's best technology and customer service company?  It doesn't to me.

But this blog is not a hate site or even a protest site.  It is a call for action from one of the greatest companies in the world to live up to its billing and refine its practices.  The sooner this happens, the sooner a black mark from Amazon's noteworthy suite of internet successes will be erased.  This blog welcomes that day...