Saying that no, you can't just whisper the verbal components this time, isn't any different than saying, no you can't just fly because you are flapping your arms.
Exactly. And the DM is free to rule that you CAN fly just because you are flapping your arms.
PHB p.193 said:The only limits to the actions you can attempt are your imagination and your character’s ability scores.
...
When you describe an action not detailed elsewhere in the rules, the DM tells you whether that action is possible and what kind of roll you need to make, if any, to determine success or failure.
As DM, you can decide whether disguised spellcasting is possible in your games, and also whether it's possible to fly by flapping one's arms, whether a PC can swim up a waterfall, or shoot an arrow and hit the moon, etc. There are genres where these sorts of actions are appropriate; maybe a more mythical/legendary approach to gaming, or maybe more comical (for the arm-flapping scenario). Likewise, in some games, it may be interesting to let mages cast their spells subtly (at some risk or cost), and in others, it may be better to take a hard line and just say, nope, all magic is obvious unless you've got the meta-magic mojo to cast subtly.