I don’t think the rules require more than a single free hand for every material component and the same free hand for somatic components. Am I mistaken?
You are correct by the rules as written. If you think about what is needed, you're fishing for components with one hand, while your other hand is full. Then waving the component through the air, performing the somatic portion of the spell. Followed by putting away the material component you just used, again with one hand.
For me at least, this breaks the number of free object interactions that you have on one turn: swap two hands to one-handed hold on your weapon; open component pouch; grab component; put component back in pouch; swap back to two-handed hold on your weapon. At the least, maybe you'd need to drop the component, rather than stash it away again.
I have less issues in this scenario when there's a spell focus on a chain around your neck. In that case, when you "drop" the focus, as it is on the chain, you retain it without needing the additional free object interaction to stash it.
Can compare this juggling of components or focus with two-weapon fighting. If a spell caster can do this much object interaction while spell casting, why can't a two-weapon fighter draw two weapons on the same turn without having the Dual Wielder feat to quick draw both?
In my opinion, D&D rules over-complicate this. I would allow relatively straightforward mechanisms (say 200gp additional cost maximum for the construction or enchantment) to have a spell casting focus as a weapon or shield that you are wielding, without requiring the attunement of the Ruby of the Warmage. Likewise, I would permit the dual wielder to draw both weapons without the feat, just as a longbow user can draw multiple arrows from a quiver when using extra attack without having a feat to enable it.
To put it differently, does D&D really intend to game-balance combat mechanics for spell casters to have a hand free to cast a spell? I don't think they do, at least not any more, because there are so many ways around this, e.g., holy symbol emblazoned on shield for Paladins and Clerics to use as a divine focus, various ways of having a weapon as a spell casting focus, etc.
The complication of V,S components (i.e., when there is no material component) requiring a hand free unless you have the War Caster feat is also an unnecessarily complicated game mechanic. It's not that the spells with V,S components are significantly more powerful than those with V,S,M components; on the contrary, as the majority of spells have V,S,M components, the more powerful spells number among them.