So alien you can't even comprehend it is hard. Hard, doable, but tricky is "alien mindset that's almost, but not quite, like what you were accepting." In C.J. Cherryh's
Foreigner books, the Atevi race is wired differently than humans as far as how social relationships work. It looks like people liking other people, caring in the same way a human would, but it isn't quite, and in those books Cherryh works hard to illustrate this. (The main character is the paidhi-aiji (lord translator) between the humans and the atevi.) Anyway, it all comes down to really subtle things. A human develops attachments to a number of friends, and those attachments can override any sort of sense of duty. A human can deeply care about people on both sides of some sort of feud. Atevi evolved on a world where pack structure is the main rule of the dominant species: there's you, and there's your alpha.
So when something bad happens, and a human jumps to be near a given person, a human would interpret that to mean "Ahh, he cares about that person and wants to stay close." You'd probably infer that the person was important to the human--perhaps a close friend or loved one. Perhaps someone the human was charged to protect. Perhaps the one to whom the human feels loyalty. An atevi would say "Ahh, that is where his man'chi lies", and would infer that the jumper respects the target of his jump as his ultimate authority. If a human hesitated between two people, another human might see it as a conflict of loyalties--do I protect my mother, or my liege lord? And we would expect any human in that position to have that kind of reaction. An atevi would see it as a crisis of man'chi--the one who hesitates temporarily does not know where his man'chi lies. This is a momentary aberration, before the person settles again and knows where his man'chi lies. An atevi who was unable to confirm his man'chi would be considered insane. (And in fact, a crisis of man'chi is mostly the atevi discovering that while he consciously believe he had man'chi in one direction, the instinctual man'chi in his hind brain was actually leading him elsewhere.)
Forgive me if I'm unconvincing that it's an alien mindset--it actually took me three times reading the full set of six books until I really started to grasp it.
The whole point is a mindset that's not about taking some part of human behavior and making a caricature of it (like Klingons or Vulcans from Star Trek, say, where one is a caricature of uncontrolled passion, and the other a caricature of passion tightly controlled--perhaps too tightly). It's taking some aspect of behavior that we rely on, and making it not apply any more--in a way that's internally consistent and difficult to discern, except by a tingling sensation in the back of your head that says "that's not right..."
Oh, and in that series of books, the ease with which each side (humans and atevi) fell into assuming their new neighbor was the same lead to a cross-species war, which the humans lost. (And which lead to the creation of the office of the paidhi--the one human allowed to be in contact with the atevi.) (And no, that's not a spoiler.)
For the "so alien it's incomprehensible" thing, well... That's a hard one to model in any case. Especially because there should always be an internal system, which is something that human intellect is good at figuring out. I'd say that the best goal is one which a person *needs* to apply intellect to figure out. An example would be a race in which each member keeps a "score" that it assigns to each individual it knows. Depending on your score, your reception might ignore you, help you, or sneer at you. And the thing is, as you converse it constantly adjusts your score based on what you say. If you ever pass a "threshold", the creature's attitude changes instantly. If you choose the triggers for scoring well, this could result in behavior that outwardly seems extremely erratic and inexplicable. But there's a definite logic to it. And of course, you have to choose triggers that aren't what a human would expect. Perhaps kindness, charity, etc. don't matter at all. Perhaps the only thing that matters is specific topics of conversation. If you mention sunlight, you get 2 points. If you mention clouds, you lose 1 point. If you call treasure "treasure" you lose 2 points. If you call treasure "swag" you gain 1 point. (Okay, this example is pretty silly, but you get the idea.) *Then* add the following stipulation: only if the creature is grumpy will he help you with correct information about people. Only if the creature is happy will he help you with correct information about places. And of course, if your conversation is nonsensical or boring, he'll shoo you off.
The whole point is to make a system that the players (or their characters) can figure out, but will always have to think about, even once they know how the system works. "How do I converse with him while leading him to the state I need him in to answer my questions?"
Then once they have it figured out you start throwing in loops. Like the set of key concepts is slightly different from one tribe to the next. Just different enough that you could make a mis-step without realizing it.
What other alien motivations could we have... How about a strange devil that is creating a work of art. The work of art is the expression a complicated set of numbers as murders. pi, say. This demo wants to write pi in the language of murder. So the first day, he aims to kill three people. The second, he aims to kill 1 person, and so on. This is perhaps a bad example, because the players will almost certainly not be able to figure it out. But it gives an example of an entirely alien (or psychopathic) motivation in a more clear-cut situation than the weird talking rules above. (And, of course, psychopathic *is* essentially alien. A madman follows rules. They just happen to be rules divorced from those most of us follow. And most likely divorced from reality.)
