Jester David
Hero
"Because the players expect it" is not a narrative reason. I mean a story reason, an in world reason this spellcaster has access to unique magic or unlearnable secrets.There's always a narrative reason! The players in my game expect monsters, and NPC wizards, to be able to do weird stuff they haven't encountered before, and I've never found it disrupting. Dare I say that's part of what makes it magic?
One of the tropes of the game is scribing scrolls and copying spellbooks. If fighting a wizard NPC, a wizard PC will be excited at the idea of access to a new spellbook. Unique spells throw a monkey wrench in that convention and player expectations. It's okay sometimes, but not every single time.
Really, there's also only so many magical effects we need. We don't need a dozen >slightly< different area effect attacks that deal fire damage. If a monster is using fireball just say fireball. We all know what a fireball looks like and acts like so it's easier to describe and visualize at the table. There's a shared narrative. There's already hundreds of spells with a history in the game; each of the 300+ monsters in a single Monster Manual do not need 2-3 unique snowflake powers. (Plus, when making a monster, you shouldn't have to look through 600-900 powers to see if there's a monster that does the same thing only better/worse.)
Likewise, there's the fighter. If the skilled level 5 fighter, the master of the longsword, so special they're an adventurer. There shouldn't be many ways to swing a sword they cannot learn.
*
Little tangent here.
I’ve been re-reading some forum posts I made in the lead-up and release of 4e. My philosophy has really changed in the intervening years. I was really gung-ho for a lot of the 4e changes. It’s really weird; it’s like reading posts by people I’ve spent the last two years arguing against. I do recall my opinions were more middle-ground but I focused on the opposite position to be the Devil’s Advocate in a strongly 4e-wary community. And I was coming off a long stint playing Living Greyhawk, which really skewed my perceptions of 3e.
Back in 2007-8 I was a huge proponent of the idea that monsters were just meant to show up, doing their cool thing, and die within 2-4 rounds. And that anything in their statblock above and beyond that was wasted; that everything in the statblock should pertain to combat. Basically, the 4e monster design.
Since then, my opinions have really changed. Reversed really. I did the 4e thing for a time and found the combat-only monster entries problematic. I missed the lore, the story ideas, and monsters having a place in the world. I missed monsters that had powers useful for things other than combat, who had powers that could justify plots and serve as adventure hooks. Monsters who could do things outside of combat that didn’t amount of DM fiat or only having phenomenal cosmic power when off camera.
*
I think there's room for a middle ground. Because, like most problems, the answer is not an either/or. The best compromise would be the highest level or most thematic/combat useful spells/powers have a full description. Other spells are just listed, so the DM can treat them as rituals (usable outside of combat) or, if they're feeling adventurous, skim through the books.