Their RP style simply does not work with D&D.
Its rather that they role play their characters and the roles they have chosen goes against the default metagame of D&D.
I
strongly disagree with this. I don't believe there is a default gaming style for D&D; I think it's a broad and very flexible family of systems that has, IME, led to great variation in gaming styles. Which is part of the problem here; the other players' style would have been perfectly at home in many 1E campaigns.
That said, I think there's a happy medium. In the stories that I like, the heroes do try to have plans, but are so motivated to accomplish their goals that they do not simply give up and flee whenever circumstances forces them to act without a perfect plan. They have the courage to take risks, but they know those are risks -- they don't assume "Well, nothing bad can happen to me because I'm a main character/I have too many HP/they're just minions." Heroes take risks because they have to. When Sam and Frodo jump off the bridge in Mordor, they don't know how far down it is, but they hope they
might survive; getting caught is certain death, and the situation is just that dire. Their leap of faith is rewarded, but they never begin thinking of themselves as invincible.
So on the one hand, I can totally see the paranoid players not wanting to play in the style of "Kick in the door and we know we'll stomp them because the DM wouldn't put us against anything we couldn't handle." That style, where PCs are metagame-aware of their invulnerability, usually holds little appeal for me. But on the other hand, taking 20 to Search every 5' square is boring. So I imagine one compromise might be if the players agree "We will make characters who can be compelled by the plot or by personal goals enough to take risks that they see as unwise but necessary." They can keep on acting as if things are risky as their PCs would perceive them, but be motivated enough to push on even if it isn't the perfectly pragmatic thing to do.
Or, as an alternative, give them a vague in-game understanding of their near-invulnerability (an 11th-level PC
is legendary) so it is no longer meta-gaming, and make it a plot point as they try to cope with their overnight transformation into something freakishly powerful. How is it that they have mastered arcane powers in a few months that take most mortals a lifetime to understand? How is it that arrows and swords usually leave only scratches in their flesh? Is is demonic possession, reincarnation of a great hero, or divine/aberrant ancestry? If they know in-character they can kick in the door and slaughter normal people without any real risk, how they respond becomes character-defining.