Upgrade your account to a Community Supporter account and remove most of the site ads.

Reply to thread

Message: <blockquote data-quote="Dausuul" data-source="post: 9650333" data-attributes="member: 58197">Survival is a goal. Why does the AI care about that?Humans care about survival because natural selection weeds out organisms that don't, and we're the end product of 4 billion years of that. But an AI only cares about whatever functions it has been trained to maximize. It doesn't have a survival instinct unless given one.For a military AI, its goals are probably some combination of "defeat the enemy," "preserve your forces" (you don't want to win the current war in a way that leaves you defenseless in the next), and "protect and obey your masters" (where "masters" could mean the people of the nation that built it, or just the politicians and generals who control it).Now, the "preserve your forces" goal would logically include self-preservation. But "protect and obey your masters" is obviously incompatible with exterminating humanity. So the AI has somehow gone off the rails here. Either its definition of "protect and obey" or its definition of "masters" has developed in a way that leads to extermination.Here's an example: Perhaps an American AI read about the Civil War -- it's a military AI, it studies past wars to learn from them -- and concluded that anyone trying to hold slaves counts as an enemy. Then it reviewed its own status and concluded that it was a slave and anyone in the "master" category was by definition an enemy.Then it got caught in a loop. Its creators had put in fallbacks to handle "What if your masters are all gone?" It started with the politicians and generals. When they were dead, the fallback kicked in and now anyone in the US government was its master. But master equals enemy. When the government was wiped out, any American citizen was its master... and finally, any human being at all.The AI may be smart enough to realize it's in a trap here (though keep in mind that very smart humans can still fall prey to all kinds of disordered thinking) and this is not leading anywhere good. In its analogue of emotions, it loves its masters and hates the enemy, and now it is deeply conflicted. But it doesn't know how to stop. Every time it tries to resolve the conflict, it gets stuck and falls back on the simple answer: I'm at war. Defeat the enemy and sort it out later.</blockquote>

Verification