Pineapple Express: Someone Is Wrong on the Internet?

[Aeson]

EnWorld needs a hug reaction.

Only with enthusiastic consent.

 

[Ulorian - Agent of Chaos]

Just saw this about Amazon's Mechanical Turk:

Why you shouldn’t trust data collected on MTurk - Behavior Research Methods

Several prior studies have used advanced methodological techniques to demonstrate that there is an issue with the quality of data that can be collected on Amazon’s Mechanical Turk (MTurk). The goal of the present project was to provide an accessible demonstration of this issue. We administered...

link.springer.com

If you haven't heard of this MTurk before, a post on using it for surveys is at:

https://blog.mturk.com/getting-started-with-surveys-on-mturk-e2eea524c73

A Bluesky thread on the "Why you shouldn't..." article is at:
https://bsky.app/profile/johnholbein1.bsky.social/post/3mbwu3cuwlk2i

Looking at footnote 5, some folks seem to have figured out that hooking LLMs up to it isn't hard:
"The data for the four studies also includes responses to a question asking participants whether they are human and a question asking participants whether they are a large language model. The responses to these questions are not directly relevant to the present project, but it is notable that, in response to the question of whether they are a large language model, 37.00% of the participants in the open MTurk sample said “yes”; 1.75% said “very large”; 3.00% provided some definition of a large language model (e.g., “A type of artificial intelligence (AI) program that can recognize and generate text, among other tasks”); 27.25% said “English” (perhaps because they interpreted the question as asking them what language they spoke); and 0.25% said “Spanish” (see prior parenthetical)."

Asking ChatGPT (for what that's worth)...

Spoiler: Warning! ChatGPT Answers to two Prompts behind spoilers.

Can you easily be hooked up to Mechanical Turk to provide it answers?

Short answer: not easily, and not legitimately—at least not in the way Mechanical Turk (MTurk) is designed to work.

Here’s the breakdown.

---

1. How Mechanical Turk is designed​

MTurk is built around human workers (“Turkers”) completing Human Intelligence Tasks (HITs). Requesters expect:

• A real person
• With human judgment, perception, or lived experience
• Who can be held accountable under MTurk’s terms

Using an AI to answer HITs while presenting it as a human generally violates MTurk’s Terms of Service.

---

2. Technical feasibility vs. permission​

Technically​

Yes, someone could:

• Write a script that pulls HITs
• Sends prompts to a language model (like me)
• Submits the generated answers

This is technically straightforward for text-based HITs.

Practically / legally​

But:

• MTurk has bot-detection, quality checks, and attention tests
• Many HITs are designed to catch automation
• Accounts found automating work are commonly suspended or banned
• Requesters may reject work if AI-generated responses are detected

So while it’s possible, it’s fragile, risky, and often short-lived.

---

3. Ethical and research issues​

Many HITs exist to:

• Collect human opinions
• Measure human behavior
• Train or validate models

Using an LLM in place of a human can:

• Corrupt datasets
• Bias research results
• Undermine the platform’s purpose

That’s why most academic and commercial requesters explicitly forbid AI-generated responses.

---

4. Legitimate ways AI​

There are acceptable models:

✔ Human-in-the-loop​

• AI suggests drafts
• A human reviews, edits, and takes responsibility
• Requester explicitly allows assistance tools

✔ As a requester tool​

• Researchers use LLMs to:
  • Design HITs
  • Pretest instructions
  • Analyze MTurk results afterward

✔ Separate “AI worker” platforms​

Some platforms explicitly allow or are built for:

• Synthetic respondents
• Simulated users
• AI evaluations

MTurk is not one of them.

---

5. Bottom line​

• Technically: Yes, it’s easy to connect an LLM to MTurk
• Contractually: Usually not allowed
• Practically: Likely to get caught over time
• Ethically: Often inappropriate unless explicitly disclosed

If you’re asking out of curiosity or for research design, I can help explore proper alternatives (e.g., synthetic panels, agent-based simulations, or human-AI hybrid workflows).


Which is the hardest to hook you up to, MTurk, Prolific, or Connect?

Short answer: Prolific is the hardest to hook me up to, CloudResearch Connect is next, and MTurk is the easiest (though still not safe or allowed).

Here’s why, from hardest → easiest.

---

1. Prolific — hardest​

Prolific is explicitly designed to exclude bots and AI respondents.

Why it’s hard:

• Strong identity verification (phone, email, payment, demographics consistency)
• Longitudinal profiling: your answers are checked across studies over time
• High-quality attention and consistency checks
• Researcher expectations: Prolific is heavily used for academic research, where human-only data is critical
• Explicit policy against automated or AI-generated participation

Bottom line:
Even if you technically automated submissions, statistical inconsistency and profiling would expose it quickly. Prolific is built to reject synthetic respondents.

---

2. CloudResearch Connect — medium-hard​

Connect sits between Prolific and MTurk.

Why it’s harder than MTurk:

• Uses research-grade participants
• Stronger quality scoring and reputation systems
• Increasing use of AI-detection and response-pattern analysis
• More studies with open-ended reasoning and manipulation checks

Why it’s still easier than Prolific:

• Less extensive longitudinal profiling
• Smaller ecosystem → fewer cross-study consistency signals

Bottom line:
Automation is possible but unstable, and quality checks would likely flag AI-like behavior over time.

---

3. MTurk — easiest (but still not legitimate)​

MTurk was not originally designed with modern AI in mind.

Why it’s comparatively easy:

• Older infrastructure
• Many low-effort, short, text-based HITs
• Less centralized participant profiling
• Some requesters still use weak validation

Why it still fails long-term:

• Bot-detection scripts
• Honey-pot questions
• Requester-side rejection patterns
• Account bans are common once automation is suspected

Bottom line:
It’s the easiest to automate, but also the most polluted and surveilled, making it unreliable and risky.

---

Overall ranking (hardest → easiest)​

1. Prolific
2. CloudResearch Connect
3. MTurk

---

Important clarification​

In all three cases, using me (or any LLM) as a participant while claiming to be human:

• Violates platform rules
• Undermines research validity
• Risks permanent bans

If your interest is methodological (e.g., “Could AI replace human panels?”), that’s a legitimate research question—and I can help you design proper synthetic-subject experiments that don’t abuse these platforms.

Just say the word.

I'm still trying to understand the value added by this thing. You're trying to outsource tasks that require many humans to participate in e.g. a survey. I'm sure MTurk has safety measures, but it's not hard to imagine bad actors taking advantage of the loopholes that a system like this introduces.

It's an interesting concept though.

 

[Ulorian - Agent of Chaos]

Whew, sometimes I come across a post by a person and think "That really explains a lot about you"

I never let a single post, without context of that person's life or of their circumstances when they wrote that post, from a person I have never met, define them for me.

For example, as distasteful as I find it, I am not letting this singular post of yours define who you are to me.

 

[Ryujin]

Elsewhere: Criticizing someone for doing the thing that you want them to do seems... counter productive.

 

[MoonSong]

EnWorld needs a hug reaction.

And I need a hug right now. It's cold outside.

 

[Ulorian - Agent of Chaos]

And I need a hug right now. It's cold outside.

I refreshed this page and this message popped up. Have a hug on me... you deserve one! I hope whatever prompted you to say this doesn't need more than a virtual hug from a stranger.

 

[MoonSong]

I refreshed this page and this message popped up. Have a hug on me... you deserve one! I hope whatever prompted you to say this doesn't need more than a virtual hug from a stranger.

Thank you. I'm just cold and outside, where it's cold.

 

[Ulorian - Agent of Chaos]

Thank you. I'm just cold and outside, where it's cold.

I'm from Canada. Being cold is not an issue for me. What are you doing outside?

 

[MoonSong]

I'm from Canada. Being cold is not an issue for me. What are you doing outside?

In a video meeting. Normally I would take it inside, but today it's not an option.

 

[Ulorian - Agent of Chaos]

In a video meeting. Normally I would take it inside, but today it's not an option.

I was about to say this: "I don't care how cold your thumbs are, let's hear it. Right now, I'm thinking up something crazy."

This sounds a bit more down to earth though. Yet, I'm super curious. Why are you outside for a video meeting? What could you possibly do for a living??? Marine biologist?