Here's a scenario that illustrates my concerns:
The party includes a 10th-level sorcerer and a 10th-level wizard. They defeat the monsters in a dungeon, and come across a stone door with an impression of scepter in it. Some Intelligence checks let them determine that they probably need to place a scepter of that shape into the door to open it. (Did they miss it in the dungeon? Was it not in the dungeon? Irrelevent!) So they set up night for the camp. That morning, the sorcerer switches out cone of cold for creation. He creates a scepter (stone, metal, whatever, they have coins and there are rocks around), which they use to open the door.
Deeper in the dungeon, they come across some sort of fiend bound in a summoning circle. It appears to speak a language, but not the same language that the party does. The wizard ritual casts comprehend languages. Now she can figure out what the fiend is saying, but can't communicate back well enough to for the party to be confident they won't be getting themselves into trouble (they don't read the same language, so she can't just write things down for the fiend to read). So they find another place to rest, and the sorcerer switches out haste for tongues. Now they go and have a conversation to figure out what's going on. The fiend claims to have a way to help them progress further in the dungeon (by dispelling a magical protection that seemed resistant to the party's dispel magic attempts). They party says, "we'll think about it" and retreats back a few rooms.
They settle in for a few days. The sorcerer restores his haste. Since they think this fiend is resistant or immune to cold, his player rifles through the PHB and Xanathar's Guide to Everything, and grabs synaptic static. He normally likes his 4th-level polymorph, but for this fight, he conjures himself up a banishment spell instead. That only took 3 days, but the rest of the party is getting restless, so they return. They release the fiend, and when things go poorly...banishment!
Continuing through the dungeon, see invisibility and enlarge/reduce also get temporarily switched in for the sorcerer, to good effect.
With the information they acquire at the end of the dungeon, they head out in pursuit of a white dragon. The sorcerer spends the 10 days travel time redesigning his spell list for the mission...protection from energy check! ...earthbind check! ...
I'm going to make the rather modest proposal that this is stepping on the toes of the wizard.
The biggest problem is that the sorcerer has the mid-adventure ability to swap in spells from his entire spell list whereas the wizard is limited to his personal spellbook.
Thanks. This example goes a long way to help illustrate the underlying problem you see coming. That helps me, perhaps others, consider "is this anything like an issue we see in play?" and from that get a feel for how that is or is not a concern we share.
Different POV
So, first off - in my games if there is no time pressure or other kind of event pressure (not as someone said doomsday clock necessarily) then problems like doors eith scepter locks are considered terrain and delays, they are not expected to be things that stop the PCs. After all, with no time pressure or threats making long rest on-demand an issue, there are many ways around.
Second, as early as 3rd level you have other ways to fashion "keys" (depends on specifics of scenario as to which). Its not a serious "meantbyo stop" but a "meant to delay ir drain".
Third, a 10th level sorc has like 11 known spells divided over 5 levels of spells The examples here show two of those known shifted off from attsck spells to these keys. So, not only have we lost the spell slots to get past these but we have also cost two of our 11 "spells we wanted on hand" for any or all other challenges that same day after we open the door or talk with etc. Etc. The "cost" is doubled if you will because with only 11 spells on tap, 10 then 9 - we have likely created gaps in our "action options." (With wizards and clerics, we can shuffle many at once, so is we swap out one spell for "key" it can be from any level and we can shuffle others to cover the gaps.
Basically, only being able to swap one when we need an opportunity key, not being able to swap different levels, only 11 spells in our action set ypto begin with... this is actually pretty big opportunity cost for a long rest fix. (Funny part is - it's likely the cleric, druid or wizard is swapping around to cover the gaps we create- getting a cold spell, getting a haste ready etc.)
.
It's not the issue of dialing in one key - most problems worth anything have more than one way thru - but what you are then down after.
So, really, these examples of cases that your concerns are derived from, make me see how more limiting this option is than I had thought - in comparison to the options others have had for years. I mean, it's not like clerics, druids, wizards etc have not be the swap guys with big lists to draw on since 5e csmd out yet these have not been problems in cases like this.
I mean, 10th level wiz likely has a minimum of 24 spells in the spellbook that they could choose free - without any scribing additions.
So, really this isn't the sorc bringing a new option into play, but bringing his own flavor of the feature the others have been using but with more and longer limitations after the fact.
Thanks.
Even though I am more enamored of a meta-magic and invocation swap option, I am less concerned about the spell swap option if it does occur.