I want to run a fun game with a coherent storyline, but whatever "creativity" or "improvising" I come up with at the table - whatever new idea catches my interest this week - almost always derails the game.
<snip>
I don't feel confident improvising on the fly with the basic plot of the game. My players are having fun, but I feel like what would make the game more fun for me would take away from their enjoyment.
<snip>
What is a reasonable limit or guideline for including fresh ideas every week?
It's a bit hard to answer without knowing what counts as a "new idea"/"fresh idea".
I don't think every session can have a PC-related reveal at the level of "Luke, I am your father!" That would tend to leave the players with too little traction on the identify of their PCs and their place in the gameworld.
But I tend to think that every session
can have a situation-related twist/reveal at the level of "Lando sold us out to the empire": one salient aspect of the storyline has something about it that was not what the players (and perhaps the GM, if you're improvising) expected up until now.
In my own game, I tend to focus twists/reveals on NPC backstory/motivations (eg it turns out that A is connected to B in an unexpected way) and on opposing forces (eg there's an enemy we didn't expect!). I personally think it is better if these twists/reveals are linked to the PCs in some way (eg A is the cleric's friend and B is the chief immortal enemy of the cleric's god; the enemy is, or is in the service of, some important aspect of a PC's backstory), but I know some groups prefer to downplay those sorts of PC-oriented links in their games.
I would also strongly advise you to be open to the players introducing their own twists/reveals. For example, if they decide to bargain with or befriend some NPC or monster rather than fight it, you shold be open to that and see where it takes things.
