Pineapple Express: Someone Is Wrong on the Internet?


log in or register to remove this ad

Cadence said:
Just saw this about Amazon's Mechanical Turk:
link.springer.com

Why you shouldn’t trust data collected on MTurk - Behavior Research Methods

Several prior studies have used advanced methodological techniques to demonstrate that there is an issue with the quality of data that can be collected on Amazon’s Mechanical Turk (MTurk). The goal of the present project was to provide an accessible demonstration of this issue. We administered...
link.springer.com link.springer.com

If you haven't heard of this MTurk before, a post on using it for surveys is at:

A Bluesky thread on the "Why you shouldn't..." article is at:
https://bsky.app/profile/johnholbein1.bsky.social/post/3mbwu3cuwlk2i

Looking at footnote 5, some folks seem to have figured out that hooking LLMs up to it isn't hard:
"The data for the four studies also includes responses to a question asking participants whether they are human and a question asking participants whether they are a large language model. The responses to these questions are not directly relevant to the present project, but it is notable that, in response to the question of whether they are a large language model, 37.00% of the participants in the open MTurk sample said “yes”; 1.75% said “very large”; 3.00% provided some definition of a large language model (e.g., “A type of artificial intelligence (AI) program that can recognize and generate text, among other tasks”); 27.25% said “English” (perhaps because they interpreted the question as asking them what language they spoke); and 0.25% said “Spanish” (see prior parenthetical)."

Asking ChatGPT (for what that's worth)...

Can you easily be hooked up to Mechanical Turk to provide it answers?

Short answer: not easily, and not legitimately—at least not in the way Mechanical Turk (MTurk) is designed to work.

Here’s the breakdown.


1. How Mechanical Turk is designed​

MTurk is built around human workers (“Turkers”) completing Human Intelligence Tasks (HITs). Requesters expect:

  • A real person
  • With human judgment, perception, or lived experience
  • Who can be held accountable under MTurk’s terms
Using an AI to answer HITs while presenting it as a human generally violates MTurk’s Terms of Service.


2. Technical feasibility vs. permission​

Technically​

Yes, someone could:

  • Write a script that pulls HITs
  • Sends prompts to a language model (like me)
  • Submits the generated answers
This is technically straightforward for text-based HITs.

Practically / legally​

But:

  • MTurk has bot-detection, quality checks, and attention tests
  • Many HITs are designed to catch automation
  • Accounts found automating work are commonly suspended or banned
  • Requesters may reject work if AI-generated responses are detected
So while it’s possible, it’s fragile, risky, and often short-lived.


3. Ethical and research issues​

Many HITs exist to:

  • Collect human opinions
  • Measure human behavior
  • Train or validate models
Using an LLM in place of a human can:

  • Corrupt datasets
  • Bias research results
  • Undermine the platform’s purpose
That’s why most academic and commercial requesters explicitly forbid AI-generated responses.


4. Legitimate ways AI​

There are acceptable models:

✔ Human-in-the-loop​

  • AI suggests drafts
  • A human reviews, edits, and takes responsibility
  • Requester explicitly allows assistance tools

✔ As a requester tool​

  • Researchers use LLMs to:
    • Design HITs
    • Pretest instructions
    • Analyze MTurk results afterward

✔ Separate “AI worker” platforms​

Some platforms explicitly allow or are built for:

  • Synthetic respondents
  • Simulated users
  • AI evaluations
MTurk is not one of them.


5. Bottom line​

  • Technically: Yes, it’s easy to connect an LLM to MTurk
  • Contractually: Usually not allowed
  • Practically: Likely to get caught over time
  • Ethically: Often inappropriate unless explicitly disclosed
If you’re asking out of curiosity or for research design, I can help explore proper alternatives (e.g., synthetic panels, agent-based simulations, or human-AI hybrid workflows).


Which is the hardest to hook you up to, MTurk, Prolific, or Connect?

Short answer: Prolific is the hardest to hook me up to, CloudResearch Connect is next, and MTurk is the easiest (though still not safe or allowed).

Here’s why, from hardest → easiest.


1. Prolific — hardest​

Prolific is explicitly designed to exclude bots and AI respondents.

Why it’s hard:

  • Strong identity verification (phone, email, payment, demographics consistency)
  • Longitudinal profiling: your answers are checked across studies over time
  • High-quality attention and consistency checks
  • Researcher expectations: Prolific is heavily used for academic research, where human-only data is critical
  • Explicit policy against automated or AI-generated participation
Bottom line:
Even if you technically automated submissions, statistical inconsistency and profiling would expose it quickly. Prolific is built to reject synthetic respondents.


2. CloudResearch Connect — medium-hard​

Connect sits between Prolific and MTurk.

Why it’s harder than MTurk:

  • Uses research-grade participants
  • Stronger quality scoring and reputation systems
  • Increasing use of AI-detection and response-pattern analysis
  • More studies with open-ended reasoning and manipulation checks
Why it’s still easier than Prolific:

  • Less extensive longitudinal profiling
  • Smaller ecosystem → fewer cross-study consistency signals
Bottom line:
Automation is possible but unstable, and quality checks would likely flag AI-like behavior over time.


3. MTurk — easiest (but still not legitimate)​

MTurk was not originally designed with modern AI in mind.

Why it’s comparatively easy:

  • Older infrastructure
  • Many low-effort, short, text-based HITs
  • Less centralized participant profiling
  • Some requesters still use weak validation
Why it still fails long-term:

  • Bot-detection scripts
  • Honey-pot questions
  • Requester-side rejection patterns
  • Account bans are common once automation is suspected
Bottom line:
It’s the easiest to automate, but also the most polluted and surveilled, making it unreliable and risky.


Overall ranking (hardest → easiest)​

  1. Prolific 🥇
  2. CloudResearch Connect 🥈
  3. MTurk 🥉

Important clarification​

In all three cases, using me (or any LLM) as a participant while claiming to be human:

  • Violates platform rules
  • Undermines research validity
  • Risks permanent bans
If your interest is methodological (e.g., “Could AI replace human panels?”), that’s a legitimate research question—and I can help you design proper synthetic-subject experiments that don’t abuse these platforms.

Just say the word.
Click to expand...
I'm still trying to understand the value added by this thing. You're trying to outsource tasks that require many humans to participate in e.g. a survey. I'm sure MTurk has safety measures, but it's not hard to imagine bad actors taking advantage of the loopholes that a system like this introduces.

It's an interesting concept though.
 

gnarlygninja said:
Whew, sometimes I come across a post by a person and think "That really explains a lot about you"
Click to expand...
I never let a single post, without context of that person's life or of their circumstances when they wrote that post, from a person I have never met, define them for me.

For example, as distasteful as I find it, I am not letting this singular post of yours define who you are to me.
 







MoonSong said:
In a video meeting. Normally I would take it inside, but today it's not an option.
Click to expand...
I was about to say this: "I don't care how cold your thumbs are, let's hear it. Right now, I'm thinking up something crazy."

This sounds a bit more down to earth though. Yet, I'm super curious. Why are you outside for a video meeting? What could you possibly do for a living??? Marine biologist?
 

Similar Threads

Snarf Zagyg
The First Rule of David Fincher is You Don't Rank David Fincher
2 3 4
Replies
31
Views
5K
GMMichael
GMMichael
Snarf Zagyg
History Rhymes: Another Gygax Lawsuit
2 3 4 5
Replies
49
Views
14K
Some Dude
Some Dude
Snarf Zagyg
King Kubrick: Ranking Stanley's Best Films
4 5 6 7 8
Replies
75
Views
11K
The Soloist
T
Snarf Zagyg
Civilizations, RANKED!
6 7 8 9 10
Replies
96
Views
25K
Jasperak
Jasperak
Snarf Zagyg
Seven Views of Mount Cannoli: Subjective Preferences in Geek Media
5 6 7 8 9
Replies
84
Views
11K
Dannyalcatraz
Dannyalcatraz

Recent & Upcoming Releases

Remove ads

Top