To answer these questions...
I don't think that the difference between
D&D and
Settlers of Catan in how much people roleplay is a mechanical difference exactly. There is a difference in how much people roleplay in those games, but it doesn't have a lot to do with the limitations or mechanical options in either. Rather, I think it has a lot to do with the expectations of people going in to the games. People go into
D&D to roleplay while they play the game, and people typically don't go into
Settlers of Catan expecting to roleplay. In this regard, you might say that the act of creating a character for
D&D might be considered a mechanical incentive to roleplay, since it puts you in the mindset of playing a role (your character), but that might be the extent of it.
As for chess and such... for some reason a minor scene in C. S. Lewis'
Voyage of the Dawn Treader comes to mind, where mouse knight Reepicheep keeps losing at chess because he is too busy replaying grand stories of valorous knights to remember to try to win. You don't see roleplaying in chess because, well, it gets in the way of winning, and chess is an extremely structured and competitive game with clear good and bad moves (clear to skilled players, that is, not necessarily clear to me

). I think there is a certain point where, if roleplaying and winning directly conflict, winning will almost always be put at a higher priority. That said, I don't think this applies very much to any RPG I have ever seen, because such games almost always allow numerous choices that are all equally valid routes to victory (unless you have a stingy railroading DM, which is fortunately rare).
Put simply, there
are factors that affect how much people roleplay, but they come up in the comparison of RPGs to other forms of entertainment far more than in comparisons between different RPGs.
I actually think this is a very good example.
If you want to promote roleplaying, it must create a game reward and a way to success.
For example, there are two "rules" in most RPGs that I use as an example.
1) Never split the party.
It's too dangerous. A single character has less chances to survive if he enters any kind of combat, and even other types of conflicts get either if there is help. But, there are actually valid reasons for PCs to go Solo. The Rogue that goes scouting, the Bard that seduces an NPC to get aid.
So basically there needs to be a "rule" in the game that helps doing this.
There is another problem with this - RPGs are played with multiple players, and if an individual player has the spotlight for a longer time, it tends to get boring for other players. So,you need something to "fight" against this. For example, the game might allow PCs to take over NPCs in any given situation (without it "breaking" the game, too.)
2) No inter party conflicts.
That's something that we all avoid, because it leads to splitting the party and weakening the overall group. But realistically, sometimes this can happen. But most people avoid it - and it also hurts the enjoyment of the game if sometimes, inter party conflicts turn into conflicts between players.
If you want to promote something like this, you need to create rewards that help all sides and compensate for any disadvantages the conflict creates in other situations.
---
I think the "guiding" principle I see in these two examples is that you need to distinguish between "character award" and "player award".
A character isn't rewarded if he falls in love with an NPC and the NPC is later held hostage or killed. It's a terrible experience for the PC, and it gave his enemies power over him. He might lose money, his life or his love, he will fail at other in-game goals. But you can give the player a reward for going through all this, because he roleplayed how his character would act in this situation.
If you want the PCs to run "solo", sometimes the other players will have to take over NPCs or other opposing elements. The rules will need to reward that player, too, while his character is entirely unaffected by the event.
The biggest "disadvantage" of this approach is, that it lacks something some people really like, the "simulation" aspect of the rules. The rewards the players get don't model anything in the real world. They represent something outside the game world.
But they will still affect the game world - the player can use this rewards to shape the course of the campaign - set stakes, take control over game elements, and so on.