An xp system that somewhat rewards those who take the risk over those who do not might encourage someone to take the plunge, yes.
Sometimes there's alternate means of achieving the same end. Sometimes there are not. And sometimes the party only think there are not because they've flat-out failed to notice that there are.
Ideally none, but it's probably unavoidable that it'll happen now and then. That said, if the way that looks like it'll produce more xp also has - or appears to have - more risk attached then the choice both in and out of character simply becomes one of whether or not to take the risk.
"That said, if the way that looks like it'll produce more xp also has - or appears to have - more risk attached then the choice both in and out of character simply becomes one of whether or not to take the risk."
In character "this option is riskier"
Out of character " this way is riskier but provides more xp"
Thats not two choices about taking risk as much as its hoping players make in game in character choices for out of game reasons. In many cases, that is not normally deemed a desirable role playing system goal.
"Lets take the riskier choice" would parabus etcetera be consider the bad choice, not the one deserving of out of character reward (exception noted for character personality etc)
Now, that is of course different from in-game, in character having the riskier goal **also** result in more in-game results, like say rescuing hostages and stealing the macgufgin, where you have the in-gane rewards as the in-character yemptation and no XP bonus is needed to "encourage" obe course over the others. Maybe they take a, maybe they take b... each provides different levels of risk and outcome the character can weigh without need for the player to step out of character to decide.
I have found with advancement totally divorced from choices and decisions and paths... Its no problem at all as GM to provide tough choices, rish-v-reward, etc etc etc given the characters and settings having motivations, goals, objectives, needs, energencies and even fears - even if it does for some boil down to greed.
I would suggest that your obe way vital door whammy debate/arguement is more a case of no in-game ties between that vital door and character specific interest. Add in an NPC one character wants to impress or a clear case of "best guy for the job" ( Joe is resistant to that damage), a history of blessings in-game given by gods of courage and sacrifice or even just a setup of "will get first crack at whats on the other side" and a reason that would scratch a character or twod itch...
I recall a discussion with the designer of Amber, diceless, where he was asked "how resolve equal scores" and his response was "why give yourself that headache. Dont give equal scores. Problem avoided."
So, similarly, if the need for xp out of character bonuses is to encourage one to step up to resolve impasses where the situation is both vital **and** has no in-character elements to encourage one or the other... I suggest the better approach is to save that development time and game by game implementation time for the sub-system and use it to add more "personal to characters" flavor to these situations.
I have myself rarely if ever found the need to resort to put of game incentives to push outcomes along ways i want.