Exactly. And with the swapping of weapons, and all the situational modifiers...
My current idea is:
1/
One buff per person. Most buffs become "auras" like
bless, and are always centered on the caster.
2/ Most Abjuration spells become Immediate actions ("counters").
3/ Players allocate their Hero Points ("essentia") dynamically, with certain limits. But this allocation is where
most of your benefits come from, rather than buffs.
The benefits:
1/ All party members (mostly) have the same buffs. Easier to remember.
2/ Walking around without buffs up is no longer insane.
3/ Yeah, it's complicated, but it's not actually hard. Props -- software or just poker chips -- will help a lot.
Still a bit of a pipe dream.

Cheers, -- N