PBS has an article about the the rising use of Mechanical Turk — Amazon's marketplace for low-paying, one-shot jobs — for academic research surveys. In the old days, psychology students were the primary subjects. Not anymore.
Research shows that the median "Turker" has completed 300 academic studies. Jenny Marder at PBS writes about the problems this causes:
First, there's the question of dropout rates. Turkers are more likely to drop out mid-study, and that can skew the results. Then there's the question of environmental control. Read: There is none. In the lab, it's easy to monitor survey takers; not so online. Who's to say they're not watching reality television while working, or drinking a few beers on the job? To guard against this, researchers test a worker's focus by planting "attention checks" in their surveys. "Have you ever eaten a sandwich on Mars," a question might read. Or "Have you ever had a fatal heart attack?" But the attention check questions are often recycled, and experienced workers spot them immediately. ("Whenever I see the word vacuum, I know it's an attention check," Marshall has said.)
But it's the absence of gut intuition from experienced workers that concerns Rand the most.
A person's gut response to a question is an important measurement in many social psychology studies. It's common to compare the automatic, intuitive part of the decision-making brain with the part that's rational and deliberate. But a psychologist testing for this among professional survey takers may very well be on a fool's errand.
... It could be argued that the qualities that make these subjects natural and fallible, the very things that make a human human, get swallowed up by experience.
They do point out that some kinds of tests won't be affected by experience as the effect they're testing is too strong to be undone by knowledge. However, knowing what kind of surveys can and can't be put onto Mechanical Turk, given its rising prevalence in research, is very important.
Read the rest of the article — which is fascinating — at PBS.