Two things. First, planning is not the enemy of spontaneity. Far from it. However, in planning, it sounds like you are overthinking and overplanning. You have realized that when you improvise, your confidence level is higher and your ideas are less tethered to your preceonceptions. That is a useful insight. I think it's worthwhile to often plan less (but still plan as much as you need to). It does take nerve to plan the unexpected, but it is rewarding; if you find yourself wondering, "I wonder how my players are going to respond to this?" then you are on the right track.
Second, the talespinning process in RPGs is completely different than the novelistic or poetic approach. Storytelling, in an RPG, means put together interesting situations and choices, then experiencing something novel and perhaps a little unsettling, but within a shared understanding of the characters involved and the story elements. The upshot is that in general, I think the best RPG scenario is a simple one. When well-played, with honesty, courage, and creativity, even the most basic scenario is more than complex enough. If the players respond really immersively to an NPC prisoner, or something unsettling, or the death of an NPC, not only could it easily take half a session to play out, but the experience will be far more complex and interesting than plot tinker toys.
Basic advice:
- Put down the plot tinker toys. Try instead to come up with one or two really interesting situations that do not require contrivance to put into play.
- Trust the players. It may well be that you have a rotten session. But it's better for players to take responsibility for boring (or TPKking) themselves than for you to do all the heavy lifting and it's still not fun for anyone. The upshot is that players are full of wonderful surprises.
- Try not to use more the number of elements you can count on your fingers. 3-5 main plot points. 3-5 set pieces or encounters. 3-5 NPCs interacting significantly. 3-5 main types of adversary per fight. This will maximize your cognitive capacity, freeing up the creative side of your mind.
- Neither prepare too little, nor plan that which does not need to be planned.
- Never decide how it's all going to end. Definitely think about how you would prefer it to end, but let go of the insistence it has to go that way. I call this the Anakin-Luke principle: when it comes time to wrap up the campaign, the players should have a legitimately free hand to "join the Dark Side" or be heroes or whatever other plausible choice they want. Become kings, ride off into the sunset, disband the group acrimoniously, get married, whatever. It's better if the players write their own endings; emotionally, they have enough simply accepting and responding to what you have offered them. Resist the urge to manipulate, even in the service of some "lesson."