Scribe, I was in a session last night, where the DM had to reset expectations, essentially a 2nd session 0. Three of the 6 players joined the campaign (some 6 sessions old now), after the first session. One of these players thinks it "playing his char" to use Cutting Words on MY char after we had been ambushed by Orcs. This same player believes it is "fun" for his char to get drunk and play bagpipes while we explore a dungeon of unknown size and danger level.
The first half of the session was an out of game discussion about the motivations of the various chars and the DM gently suggesting that PvP is not a good thing. I stated flat out at the table that I hate it and never allow it my table, but "his table, his rules" applies. The rest of the session was pure RP. I rolled twice for Perception. It was excruciatingly awful.
And this is among chars that are of the same mindset, and at least share/recognize each other's value system. To suggest that some Lizard, who's culture and biological evolution, over countless generations has made it what it is, will stop considering you as food, over a campfire conversation, suspends all belief. There is D&D fantasy, and then there is something else that you are describing.
The closing scene at the end of Guardians of the Galaxy is another good analogy, when the Law guy is explaining to Drax how removing someones' spine is a bad thing, or how Rocket can't steal something because he wants it more. Both those chars continue to do those things, even though it was clearly explained to them.
You are suggesting this kind of behaviour can be RP'ed away. I am stating that if players truly RP'ed said chars, such behaviour CANNOT be modified in a couple RP chats. If it is, then the players are simply not playing their chars. The entire concept of players RP'ing a pantheon of different species, with wildly divergent cultures and biological drives, is impossible from the outset. Either players are NOT truly RP'ing their alien race chars, at which point it just Humans with Funny Hats, or if the players DO succesfully RP their chars, there will be massive intra-party conflict. Both ways, the entire concept fails.