Reply to thread

Message

<blockquote data-quote="giant.robot" data-source="post: 9291056" data-attributes="member: 93119">A small clarification, you made this point but I didn't find it clear in the description: when the GenAI is choosing the probability of the next word, it's calculating the probability of the next word in relation to the current sequence of words which includes the prompt. As the sequence changes the probability of certain next token increases or decreases. The probability of tokens after the word &quot;The&quot; at the beginning of a sentence are myriad. There's fewer good probabilities for the sequence &quot;The dog&quot; and so on as the sequence grows.The training data is a bunch of plain language sentences. So if you ask an AI about hobbits, the training data near the word hobbit likely had a bunch of statements about hobbits. So when the AI encounters the &quot;hobbit&quot; token the probability of other tokens from the training data near the word &quot;hobbit&quot; end up showing up in the output.The AI doesn't actually &quot;know&quot; anything about hobbits. It pulls up tokens that were near the word &quot;hobbit&quot; in the training data. If you poisoned the training data and used the word hobbit to describe refrigerators, when you asked an AI trained on this poisoned data you'd end up with stories of Maytag the Hobbit from the Home Depot Shire.</blockquote>

[QUOTE="giant.robot, post: 9291056, member: 93119"] A small clarification, you made this point but I didn't find it clear in the description: when the GenAI is choosing the probability of the next word, it's calculating the probability of the next word in relation to the current sequence of words which includes the prompt. As the sequence changes the probability of certain next token increases or decreases. The probability of tokens after the word "The" at the beginning of a sentence are myriad. There's fewer good probabilities for the sequence "The dog" and so on as the sequence grows. The training data is a bunch of plain language sentences. So if you ask an AI about hobbits, the training data near the word hobbit likely had a bunch of statements about hobbits. So when the AI encounters the "hobbit" token the probability of other tokens from the training data near the word "hobbit" end up showing up in the output. The AI doesn't actually "know" anything about hobbits. It pulls up tokens that were near the word "hobbit" in the training data. If you poisoned the training data and used the word hobbit to describe refrigerators, when you asked an AI trained on this poisoned data you'd end up with stories of Maytag the Hobbit from the Home Depot Shire. [/QUOTE]

Verification