By the book nothing stops you casting subtly.
The verbal component of spellcasting requires chanting, and a specific pitch and resonance, neither of which encompass volume. You can't whisper, because that's definitely a difference in pitch and resonance, but you can talk quietly enough that someone standing nearby can't hear you.
And for all that somatic components say they might be forceful gesticulation or intricate gestures, they only require free use of a single hand. Which means you could literally do them with that one hand tied behind your back.
And if a spell requires a material component or a focus, then you can cast it with one hand tied behind your back holding a 4lb staff in that hand.
The rules don't stop stealthy spellcasting, and there's at least one spell that makes zero sense if it's impossible (message), plus others whose fiction doesn't really work.
But I think mostly it shuts off possibilities for teamwork. If the team spellcasters can't cast spells any time they are in a city because they have to bellow them at the top of their voice and do full body breakdancing, that makes clever plans more difficult and dropping back to murder-hoboism more likely.
If your scenario is defeated by one spellcaster's loadout, while the party fighter has nothing at all he could possibly contribute, then I think there are deeper issues than whether spellcasting is concealable.
The argument that subtle spell is somehow a sorceror's raison d'etre is ridiculous. Apart from letting you cast while bound, gagged, drowning, silenced, and allowing you perfect stealth on your spellcasting without needing to roll, it also is a 100% protection against counterspelling. That's pretty good for 25% of one class ability.