Menu
News
All News
Dungeons & Dragons
Level Up: Advanced 5th Edition
Pathfinder
Starfinder
Warhammer
2d20 System
Year Zero Engine
Industry News
Reviews
Dragon Reflections
White Dwarf Reflections
Columns
Weekly Digests
Weekly News Digest
Freebies, Sales & Bundles
RPG Print News
RPG Crowdfunding News
Game Content
ENterplanetary DimENsions
Mythological Figures
Opinion
Worlds of Design
Peregrine's Nest
RPG Evolution
Other Columns
From the Freelancing Frontline
Monster ENcyclopedia
WotC/TSR Alumni Look Back
4 Hours w/RSD (Ryan Dancey)
The Road to 3E (Jonathan Tweet)
Greenwood's Realms (Ed Greenwood)
Drawmij's TSR (Jim Ward)
Community
Forums & Topics
Forum List
Latest Posts
Forum list
*Dungeons & Dragons
Level Up: Advanced 5th Edition
D&D Older Editions, OSR, & D&D Variants
*TTRPGs General
*Pathfinder & Starfinder
EN Publishing
*Geek Talk & Media
Search forums
Chat/Discord
Resources
Wiki
Pages
Latest activity
Media
New media
New comments
Search media
Downloads
Latest reviews
Search resources
EN Publishing
Store
EN5ider
Adventures in ZEITGEIST
Awfully Cheerful Engine
What's OLD is NEW
Judge Dredd & The Worlds Of 2000AD
War of the Burning Sky
Level Up: Advanced 5E
Events & Releases
Upcoming Events
Private Events
Featured Events
Socials!
EN Publishing
Twitter
BlueSky
Facebook
Instagram
EN World
BlueSky
YouTube
Facebook
Twitter
Twitch
Podcast
Features
Million Dollar TTRPG Crowdfunders
Most Anticipated Tabletop RPGs Of The Year
Tabletop RPG Podcast Hall of Fame
Eric Noah's Unofficial D&D 3rd Edition News
Top 5 RPGs Compiled Charts 2004-Present
Adventure Game Industry Market Research Summary (RPGs) V1.0
Ryan Dancey: Acquiring TSR
Q&A With Gary Gygax
D&D Rules FAQs
TSR, WotC, & Paizo: A Comparative History
D&D Pronunciation Guide
D&D in the Mainstream
D&D & RPG History
About Morrus
Log in
Register
What's new
Search
Search
Search titles only
By:
Forums & Topics
Forum List
Latest Posts
Forum list
*Dungeons & Dragons
Level Up: Advanced 5th Edition
D&D Older Editions, OSR, & D&D Variants
*TTRPGs General
*Pathfinder & Starfinder
EN Publishing
*Geek Talk & Media
Search forums
Chat/Discord
Menu
Log in
Register
Install the app
Install
Upgrade your account to a Community Supporter account and remove most of the site ads.
Enchanted Trinkets Complete--a hardcover book containing over 500 magic items for your D&D games!
Community
General Tabletop Discussion
*Geek Talk & Media
Pineapple Express: Someone Is Wrong on the Internet?
JavaScript is disabled. For a better experience, please enable JavaScript in your browser before proceeding.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an
alternative browser
.
Reply to thread
Message
<blockquote data-quote="Cadence" data-source="post: 9834749" data-attributes="member: 6701124"><p>Just saw this about Amazon's Mechanical Turk:</p><p>[URL unfurl="true"]https://link.springer.com/article/10.3758/s13428-025-02852-7[/URL]</p><p></p><p>If you haven't heard of this MTurk before, a post on using it for surveys is at:</p><p>[URL unfurl="true"]https://blog.mturk.com/getting-started-with-surveys-on-mturk-e2eea524c73[/URL]</p><p></p><p>A Bluesky thread on the "Why you shouldn't..." article is at:</p><p>[URL unfurl="false"]https://bsky.app/profile/johnholbein1.bsky.social/post/3mbwu3cuwlk2i[/URL]</p><p></p><p>Looking at footnote 5, some folks seem to have figured out that hooking LLMs up to it isn't hard:</p><p>"The data for the four studies also includes responses to a question asking participants whether they are human and a question asking participants whether they are a large language model. The responses to these questions are not directly relevant to the present project, but it is notable that, in response to the question of whether they are a large language model, 37.00% of the participants in the open MTurk sample said “yes”; 1.75% said “very large”; 3.00% provided some definition of a large language model (e.g., “A type of artificial intelligence (AI) program that can recognize and generate text, among other tasks”); 27.25% said “English” (perhaps because they interpreted the question as asking them what language they spoke); and 0.25% said “Spanish” (see prior parenthetical)."</p><p></p><p>Asking ChatGPT (for what that's worth)...</p><p></p><p>[SPOILER="Warning! ChatGPT Answers to two Prompts behind spoilers."]</p><p><em><span style="font-size: 26px"><strong>Can you easily be hooked up to Mechanical Turk to provide it answers?</strong></span></em></p><p></p><p>Short answer: <strong>not easily, and not legitimately</strong>—at least not in the way Mechanical Turk (MTurk) is designed to work.</p><p></p><p>Here’s the breakdown.</p><p></p><p>[HR][/HR]</p><h2>1. How Mechanical Turk is designed</h2><p>MTurk is built around <strong>human workers</strong> (“Turkers”) completing Human Intelligence Tasks (HITs). Requesters expect:</p><p></p><ul> <li data-xf-list-type="ul">A <strong>real person</strong></li> <li data-xf-list-type="ul">With human judgment, perception, or lived experience</li> <li data-xf-list-type="ul">Who can be held accountable under MTurk’s terms</li> </ul><p>Using an AI to answer HITs <strong>while presenting it as a human</strong> generally violates MTurk’s Terms of Service.</p><p></p><p>[HR][/HR]</p><h2>2. Technical feasibility vs. permission</h2><h3>Technically</h3><p>Yes, someone <em>could</em>:</p><p></p><ul> <li data-xf-list-type="ul">Write a script that pulls HITs</li> <li data-xf-list-type="ul">Sends prompts to a language model (like me)</li> <li data-xf-list-type="ul">Submits the generated answers</li> </ul><p>This is <strong>technically straightforward</strong> for text-based HITs.</p><p></p><h3>Practically / legally</h3><p>But:</p><p></p><ul> <li data-xf-list-type="ul">MTurk has <strong>bot-detection, quality checks, and attention tests</strong></li> <li data-xf-list-type="ul">Many HITs are designed to catch automation</li> <li data-xf-list-type="ul">Accounts found automating work are commonly <strong>suspended or banned</strong></li> <li data-xf-list-type="ul">Requesters may reject work if AI-generated responses are detected</li> </ul><p>So while it’s <em>possible</em>, it’s <strong>fragile, risky, and often short-lived</strong>.</p><p></p><p>[HR][/HR]</p><h2>3. Ethical and research issues</h2><p>Many HITs exist to:</p><p></p><ul> <li data-xf-list-type="ul">Collect <strong>human opinions</strong></li> <li data-xf-list-type="ul">Measure <strong>human behavior</strong></li> <li data-xf-list-type="ul">Train or validate models</li> </ul><p>Using an LLM in place of a human can:</p><p></p><ul> <li data-xf-list-type="ul">Corrupt datasets</li> <li data-xf-list-type="ul">Bias research results</li> <li data-xf-list-type="ul">Undermine the platform’s purpose</li> </ul><p>That’s why most academic and commercial requesters explicitly forbid AI-generated responses.</p><p></p><p>[HR][/HR]</p><h2>4. Legitimate ways AI</h2><p>There <em>are</em> acceptable models:</p><p></p><h3>✔ Human-in-the-loop</h3> <ul> <li data-xf-list-type="ul">AI suggests drafts</li> <li data-xf-list-type="ul">A human reviews, edits, and takes responsibility</li> <li data-xf-list-type="ul">Requester explicitly allows assistance tools</li> </ul><h3>✔ As a requester tool</h3> <ul> <li data-xf-list-type="ul">Researchers use LLMs to:<ul> <li data-xf-list-type="ul">Design HITs</li> <li data-xf-list-type="ul">Pretest instructions</li> <li data-xf-list-type="ul">Analyze MTurk results afterward</li> </ul></li> </ul><h3>✔ Separate “AI worker” platforms</h3><p>Some platforms explicitly allow or are built for:</p><p></p><ul> <li data-xf-list-type="ul">Synthetic respondents</li> <li data-xf-list-type="ul">Simulated users</li> <li data-xf-list-type="ul">AI evaluations</li> </ul><p>MTurk is <strong>not</strong> one of them.</p><p></p><p>[HR][/HR]</p><h2>5. Bottom line</h2> <ul> <li data-xf-list-type="ul"><strong>Technically:</strong> Yes, it’s easy to connect an LLM to MTurk</li> <li data-xf-list-type="ul"><strong>Contractually:</strong> Usually not allowed</li> <li data-xf-list-type="ul"><strong>Practically:</strong> Likely to get caught over time</li> <li data-xf-list-type="ul"><strong>Ethically:</strong> Often inappropriate unless explicitly disclosed</li> </ul><p>If you’re asking out of curiosity or for research design, I can help explore <strong>proper alternatives</strong> (e.g., synthetic panels, agent-based simulations, or human-AI hybrid workflows).</p><p></p><p></p><p><em><strong><span style="font-size: 26px">Which is the hardest to hook you up to, MTurk, Prolific, or Connect?</span></strong></em></p><p></p><p><strong>Short answer:</strong> <strong>Prolific</strong> is the hardest to hook me up to, <strong>CloudResearch Connect</strong> is next, and <strong>MTurk</strong> is the easiest (though still not safe or allowed).</p><p></p><p>Here’s why, from <em>hardest → easiest</em>.</p><p></p><p>[HR][/HR]</p><h2>1. Prolific — hardest</h2><p>Prolific is explicitly designed to <strong>exclude bots and AI respondents</strong>.</p><p></p><p><strong>Why it’s hard:</strong></p><p></p><ul> <li data-xf-list-type="ul"><strong>Strong identity verification</strong> (phone, email, payment, demographics consistency)</li> <li data-xf-list-type="ul"><strong>Longitudinal profiling</strong>: your answers are checked across studies over time</li> <li data-xf-list-type="ul"><strong>High-quality attention and consistency checks</strong></li> <li data-xf-list-type="ul"><strong>Researcher expectations</strong>: Prolific is heavily used for academic research, where <em>human-only</em> data is critical</li> <li data-xf-list-type="ul"><strong>Explicit policy</strong> against automated or AI-generated participation</li> </ul><p><strong>Bottom line:</strong></p><p>Even if you technically automated submissions, <strong>statistical inconsistency and profiling would expose it quickly</strong>. Prolific is built to <em>reject synthetic respondents</em>.</p><p></p><p>[HR][/HR]</p><h2>2. CloudResearch Connect — medium-hard</h2><p>Connect sits between Prolific and MTurk.</p><p></p><p><strong>Why it’s harder than MTurk:</strong></p><p></p><ul> <li data-xf-list-type="ul">Uses <strong>research-grade participants</strong></li> <li data-xf-list-type="ul">Stronger <strong>quality scoring and reputation systems</strong></li> <li data-xf-list-type="ul">Increasing use of <strong>AI-detection and response-pattern analysis</strong></li> <li data-xf-list-type="ul">More studies with <strong>open-ended reasoning and manipulation checks</strong></li> </ul><p><strong>Why it’s still easier than Prolific:</strong></p><p></p><ul> <li data-xf-list-type="ul">Less extensive longitudinal profiling</li> <li data-xf-list-type="ul">Smaller ecosystem → fewer cross-study consistency signals</li> </ul><p><strong>Bottom line:</strong></p><p>Automation is <strong>possible but unstable</strong>, and quality checks would likely flag AI-like behavior over time.</p><p></p><p>[HR][/HR]</p><h2>3. MTurk — easiest (but still not legitimate)</h2><p>MTurk was not originally designed with modern AI in mind.</p><p></p><p><strong>Why it’s comparatively easy:</strong></p><p></p><ul> <li data-xf-list-type="ul">Older infrastructure</li> <li data-xf-list-type="ul">Many low-effort, short, text-based HITs</li> <li data-xf-list-type="ul">Less centralized participant profiling</li> <li data-xf-list-type="ul">Some requesters still use weak validation</li> </ul><p><strong>Why it still fails long-term:</strong></p><p></p><ul> <li data-xf-list-type="ul">Bot-detection scripts</li> <li data-xf-list-type="ul">Honey-pot questions</li> <li data-xf-list-type="ul">Requester-side rejection patterns</li> <li data-xf-list-type="ul">Account bans are common once automation is suspected</li> </ul><p><strong>Bottom line:</strong></p><p>It’s the <strong>easiest to automate</strong>, but also the <strong>most polluted and surveilled</strong>, making it unreliable and risky.</p><p></p><p>[HR][/HR]</p><h2>Overall ranking (hardest → easiest)</h2> <ol> <li data-xf-list-type="ol"><strong>Prolific</strong> <img class="smilie smilie--emoji" alt="🥇" src="https://cdn.jsdelivr.net/joypixels/assets/8.0/png/unicode/64/1f947.png" title="1st place medal :first_place:" data-shortname=":first_place:" loading="lazy" width="64" height="64" /></li> <li data-xf-list-type="ol"><strong>CloudResearch Connect</strong> <img class="smilie smilie--emoji" alt="🥈" src="https://cdn.jsdelivr.net/joypixels/assets/8.0/png/unicode/64/1f948.png" title="2nd place medal :second_place:" data-shortname=":second_place:" loading="lazy" width="64" height="64" /></li> <li data-xf-list-type="ol"><strong>MTurk</strong> <img class="smilie smilie--emoji" alt="🥉" src="https://cdn.jsdelivr.net/joypixels/assets/8.0/png/unicode/64/1f949.png" title="3rd place medal :third_place:" data-shortname=":third_place:" loading="lazy" width="64" height="64" /></li> </ol><p>[HR][/HR]</p><h2>Important clarification</h2><p>In <strong>all three cases</strong>, using me (or any LLM) <em>as a participant while claiming to be human</em>:</p><p></p><ul> <li data-xf-list-type="ul">Violates platform rules</li> <li data-xf-list-type="ul">Undermines research validity</li> <li data-xf-list-type="ul">Risks permanent bans</li> </ul><p>If your interest is <strong>methodological</strong> (e.g., <em>“Could AI replace human panels?”</em>), that’s a legitimate research question—and I can help you design <strong>proper synthetic-subject experiments</strong> that don’t abuse these platforms.</p><p></p><p>Just say the word.</p><p>[/SPOILER]</p></blockquote><p></p>
[QUOTE="Cadence, post: 9834749, member: 6701124"] Just saw this about Amazon's Mechanical Turk: [URL unfurl="true"]https://link.springer.com/article/10.3758/s13428-025-02852-7[/URL] If you haven't heard of this MTurk before, a post on using it for surveys is at: [URL unfurl="true"]https://blog.mturk.com/getting-started-with-surveys-on-mturk-e2eea524c73[/URL] A Bluesky thread on the "Why you shouldn't..." article is at: [URL unfurl="false"]https://bsky.app/profile/johnholbein1.bsky.social/post/3mbwu3cuwlk2i[/URL] Looking at footnote 5, some folks seem to have figured out that hooking LLMs up to it isn't hard: "The data for the four studies also includes responses to a question asking participants whether they are human and a question asking participants whether they are a large language model. The responses to these questions are not directly relevant to the present project, but it is notable that, in response to the question of whether they are a large language model, 37.00% of the participants in the open MTurk sample said “yes”; 1.75% said “very large”; 3.00% provided some definition of a large language model (e.g., “A type of artificial intelligence (AI) program that can recognize and generate text, among other tasks”); 27.25% said “English” (perhaps because they interpreted the question as asking them what language they spoke); and 0.25% said “Spanish” (see prior parenthetical)." Asking ChatGPT (for what that's worth)... [SPOILER="Warning! ChatGPT Answers to two Prompts behind spoilers."] [I][SIZE=7][B]Can you easily be hooked up to Mechanical Turk to provide it answers?[/B][/SIZE][/I] Short answer: [B]not easily, and not legitimately[/B]—at least not in the way Mechanical Turk (MTurk) is designed to work. Here’s the breakdown. [HR][/HR] [HEADING=1]1. How Mechanical Turk is designed[/HEADING] MTurk is built around [B]human workers[/B] (“Turkers”) completing Human Intelligence Tasks (HITs). Requesters expect: [LIST] [*]A [B]real person[/B] [*]With human judgment, perception, or lived experience [*]Who can be held accountable under MTurk’s terms [/LIST] Using an AI to answer HITs [B]while presenting it as a human[/B] generally violates MTurk’s Terms of Service. [HR][/HR] [HEADING=1]2. Technical feasibility vs. permission[/HEADING] [HEADING=2]Technically[/HEADING] Yes, someone [I]could[/I]: [LIST] [*]Write a script that pulls HITs [*]Sends prompts to a language model (like me) [*]Submits the generated answers [/LIST] This is [B]technically straightforward[/B] for text-based HITs. [HEADING=2]Practically / legally[/HEADING] But: [LIST] [*]MTurk has [B]bot-detection, quality checks, and attention tests[/B] [*]Many HITs are designed to catch automation [*]Accounts found automating work are commonly [B]suspended or banned[/B] [*]Requesters may reject work if AI-generated responses are detected [/LIST] So while it’s [I]possible[/I], it’s [B]fragile, risky, and often short-lived[/B]. [HR][/HR] [HEADING=1]3. Ethical and research issues[/HEADING] Many HITs exist to: [LIST] [*]Collect [B]human opinions[/B] [*]Measure [B]human behavior[/B] [*]Train or validate models [/LIST] Using an LLM in place of a human can: [LIST] [*]Corrupt datasets [*]Bias research results [*]Undermine the platform’s purpose [/LIST] That’s why most academic and commercial requesters explicitly forbid AI-generated responses. [HR][/HR] [HEADING=1]4. Legitimate ways AI[/HEADING] There [I]are[/I] acceptable models: [HEADING=2]✔ Human-in-the-loop[/HEADING] [LIST] [*]AI suggests drafts [*]A human reviews, edits, and takes responsibility [*]Requester explicitly allows assistance tools [/LIST] [HEADING=2]✔ As a requester tool[/HEADING] [LIST] [*]Researchers use LLMs to: [LIST] [*]Design HITs [*]Pretest instructions [*]Analyze MTurk results afterward [/LIST] [/LIST] [HEADING=2]✔ Separate “AI worker” platforms[/HEADING] Some platforms explicitly allow or are built for: [LIST] [*]Synthetic respondents [*]Simulated users [*]AI evaluations [/LIST] MTurk is [B]not[/B] one of them. [HR][/HR] [HEADING=1]5. Bottom line[/HEADING] [LIST] [*][B]Technically:[/B] Yes, it’s easy to connect an LLM to MTurk [*][B]Contractually:[/B] Usually not allowed [*][B]Practically:[/B] Likely to get caught over time [*][B]Ethically:[/B] Often inappropriate unless explicitly disclosed [/LIST] If you’re asking out of curiosity or for research design, I can help explore [B]proper alternatives[/B] (e.g., synthetic panels, agent-based simulations, or human-AI hybrid workflows). [I][B][SIZE=7]Which is the hardest to hook you up to, MTurk, Prolific, or Connect?[/SIZE][/B][/I] [B]Short answer:[/B] [B]Prolific[/B] is the hardest to hook me up to, [B]CloudResearch Connect[/B] is next, and [B]MTurk[/B] is the easiest (though still not safe or allowed). Here’s why, from [I]hardest → easiest[/I]. [HR][/HR] [HEADING=1]1. Prolific — hardest[/HEADING] Prolific is explicitly designed to [B]exclude bots and AI respondents[/B]. [B]Why it’s hard:[/B] [LIST] [*][B]Strong identity verification[/B] (phone, email, payment, demographics consistency) [*][B]Longitudinal profiling[/B]: your answers are checked across studies over time [*][B]High-quality attention and consistency checks[/B] [*][B]Researcher expectations[/B]: Prolific is heavily used for academic research, where [I]human-only[/I] data is critical [*][B]Explicit policy[/B] against automated or AI-generated participation [/LIST] [B]Bottom line:[/B] Even if you technically automated submissions, [B]statistical inconsistency and profiling would expose it quickly[/B]. Prolific is built to [I]reject synthetic respondents[/I]. [HR][/HR] [HEADING=1]2. CloudResearch Connect — medium-hard[/HEADING] Connect sits between Prolific and MTurk. [B]Why it’s harder than MTurk:[/B] [LIST] [*]Uses [B]research-grade participants[/B] [*]Stronger [B]quality scoring and reputation systems[/B] [*]Increasing use of [B]AI-detection and response-pattern analysis[/B] [*]More studies with [B]open-ended reasoning and manipulation checks[/B] [/LIST] [B]Why it’s still easier than Prolific:[/B] [LIST] [*]Less extensive longitudinal profiling [*]Smaller ecosystem → fewer cross-study consistency signals [/LIST] [B]Bottom line:[/B] Automation is [B]possible but unstable[/B], and quality checks would likely flag AI-like behavior over time. [HR][/HR] [HEADING=1]3. MTurk — easiest (but still not legitimate)[/HEADING] MTurk was not originally designed with modern AI in mind. [B]Why it’s comparatively easy:[/B] [LIST] [*]Older infrastructure [*]Many low-effort, short, text-based HITs [*]Less centralized participant profiling [*]Some requesters still use weak validation [/LIST] [B]Why it still fails long-term:[/B] [LIST] [*]Bot-detection scripts [*]Honey-pot questions [*]Requester-side rejection patterns [*]Account bans are common once automation is suspected [/LIST] [B]Bottom line:[/B] It’s the [B]easiest to automate[/B], but also the [B]most polluted and surveilled[/B], making it unreliable and risky. [HR][/HR] [HEADING=1]Overall ranking (hardest → easiest)[/HEADING] [LIST=1] [*][B]Prolific[/B] 🥇 [*][B]CloudResearch Connect[/B] 🥈 [*][B]MTurk[/B] 🥉 [/LIST] [HR][/HR] [HEADING=1]Important clarification[/HEADING] In [B]all three cases[/B], using me (or any LLM) [I]as a participant while claiming to be human[/I]: [LIST] [*]Violates platform rules [*]Undermines research validity [*]Risks permanent bans [/LIST] If your interest is [B]methodological[/B] (e.g., [I]“Could AI replace human panels?”[/I]), that’s a legitimate research question—and I can help you design [B]proper synthetic-subject experiments[/B] that don’t abuse these platforms. Just say the word. [/SPOILER] [/QUOTE]
Insert quotes…
Verification
Post reply
Community
General Tabletop Discussion
*Geek Talk & Media
Pineapple Express: Someone Is Wrong on the Internet?
Top
