Conan, Fafhrd and other pulp heroes didn't actually have to face any obstacles. They were all pre-written predicaments with pre-written solutions.
If you want to be that literal, you're still probably incorrect. From what I read from genre authors about their writing processes, they aren't that deterministic - having written the situation, then asks themselves, "Okay, so how is my hero going to get out of this one". Authors sometimes speak of "characters writing themselves", and stories frequently go places the authors didn't plan at the beginning. There's some role-playing going on there, but the rules involved include more of the rules of popular fictional structure and pacing than simulations of detailed combat actions.
Be that as it may, I'll rephrase the question: Why are the obstacles in the classic modules chosen so that the stories that result after play typically bear so little resemblance to the stories that were supposedly the inspiration for the game in the first place?
My proposed answer is that the game authors were more familiar with the fiddly bits of battle simulation rules, and less facile with the rules of fictional dramaic structure, content, tension and pacing. Rules that facilitate drama-focus came later, from other designers, standing on the shoulders of their predecessors.
Are you saying that only the type of obstacles from the stories should be included in game scenarios?
No. Moreover, I don't see how that logically or reasonably follows. It is as if I asked, "Why is there so much pepperoni on this pizza, and so little cheese?" and you respond with, "Are you saying no pizza should ever have any pepperoni?"
Classic modules, as goofy as some of them were, had one thing going for them; the authors were clever enough to know that any challenge worthy of being called such was meant for the player sitting at the table rather than a pile of doodles and numbers scribbled on a piece of paper.
I think the authors of today's modules are just as clever - it is just the fashion of what kind of challenges the players like has changed over time.