The fact that you can face anything from CR1 to CR20 does mean you will keep losing; the lower your level, the higher the chance you'll face something higher than you.
Except that you start at lvl 12, meaning you're 60% of the way to the top. Meaning off the bat you've got a 60% chance of facing lower lvl, and gaining exp from beating them. Meaning you have a much greater chance of going up lvls than down lvls, especially since not even 80% of losses should involve death.
spellcasters might HOLD you and take off with the key
Rogues/bards could steal the key from you while hidden
Monks could stun you and take off with the key.
Paladins would refrain from killing you, instead trying to incapacitate you.
Plus there's the whole 'unconscious at 0, dead at -10' thing, meaning you've got a 10 HP zone where you're beat but not dead (Down but not Out)
And then there's the fact that your characters, due to the fact that you made them specifically for a game like this, are more likely to be victorious than even other people of their own lvl. (I have little doubt you could each take on a CR12 Collosal Zombie or the like with ease.)
And finally - If you do die, and loose a lvl then I have 2 things to say about that:
1) Should've bought insurance.
2) It's not supposed to be a cakewalk.