You're not the dealer in a casino, or the host of a television game show. Your function is not to just tell players "that's not in the rules, you can't do it." Your job is to run the game, because you're the Dungeon Master. That means if it isn't in the rules, you make it up.
I appreciate the sentiment behind this line of thought, but there is always an inherent danger in having to simply "make it up," particularly as it relates to a universal capacity to disrupt casters. If the made up rule is effective and consistent in any capacity, it becomes a de facto strategy to shut down mid-to-high level casters.
Given your scenario above, a savvy party would gladly keep a rogue around (perhaps with Expertise in Arcana) to constantly keep attacks readied when facing a potent spellcaster. It's not difficult to find a way for the rogue to obtain advantage, so there's often going to be sneak attack damage added to any readied attack. If this same strategy was applied against the PCs, I doubt it'd be enthusiastically embraced by the players.