I saw a DM handle this by requiring the caster to have +1 free hands. That is to say, spells with a V component needed one free hand instead, and spells with an S and V component needed 2 free hands OR a spell focus and a free hand. It was still a net benefit for the silent spellcaster, but...