Less than 1 spell per spell level over bards, and half that over clerics and druids is hardly what I would call "quite a bit."
Well, compared to Sorcerers and Warlocks, Wizards eventually would know twice as many spells (30 vs only 15), and more than 35% more than Bards. Clerics and Druids have more "extra" features so would receive more spells than Bards, Sorcerers, and Warlocks, but not quite as many as Wizards. Even clerics and druids have an spell known edge over Bards, and a significant on over Sorcerers and Warlocks at higher levels. I would call those percentages "quite a bit", yes.
Then 90% of the time there should be no delay. Most of the remaining 10% should also be no delay as they should know their spells and be thinking about what to switch out while watches are being taken and other things being described. The length of time it takes me to tell a DM I made changes is a few seconds on average.
The delay is still there. That is the annoying part. The time is wasted wondering what spells to swap out, and the vast majority of the time there are very minor or no changes made. It takes several minutes since the players discuss "What spells should my Wizard or Cleric take today?" If the changes were significant or actually played a role in the game I would be more understanding of it, but frankly that isn't what happens at our table anyway.
Edit: I'm also not sure why you gimped Sorcerers that badly. Warlocks are a spell light class. Sorcerers aren't and you removed 1/3 of their spells.
I'm not sure what you're talking about, I didn't change Sorcerers at all. They cap out at 15 known spells in the PHB.
As to the other comments, like any changes at our table we will discuss it and test it on a trial basis. If it works we keep it and if not, we go back to the RAW. Some of our changes have worked out great and we've kept them (like using average damage). I appreciate the warnings but all those have been considered, which is why I am not asking about possible pitfalls, but other options that would quicken the process of selecting prepared spells or remove it altogether.
Like one idea I had last night was maybe limited the spell swapping. Something like one spell per short rest, three over a long rest, or some combination not to exceed three?