Do you see anything particularly game-breaking here?
Depends on the type of campaign game you wanted to run.
My question is what is the problem this new rule is trying to solve? What's the reason for not wanting to use spell slots to cast things like
Fly or
Invisibility? Is it because the PCs are casting so many combat spells that they do not have slots left over out of combat to use them for utility function? Or is it purely a narrative problem wherein you think that magical people should just be able to use magic wherever/whenever without anything to restrict them (or more to the point to change the restriction from "spell slots" or "mana" into "time" instead?)
If the situation is such that you would just like to see in your campaign powerful utility magic take a while to cast... then sure, making all utility spells into rituals would do that-- especially if you were to cut down on the number of spell slots caster got. Like for instance if you switched all of your casters over to the Warlock mechanics where they might only get 2 spells per battle (which would be used most likely on combat magic)... they could then cast any other spells they know as rituals out of combat. That probably wouldn't be that big of an issue and it would certainly change how magic is run for this particular campaign.
But of course, if you didn't want to cut down the number of spell slots a caster got from their normal chart *and* gave them unlimited use of utility via turning all spells into rituals... that would certainly also change the game style. You would see more high-level spells used strictly for combat since they wouldn't need to "save" any potential slots for utility later on. That would most likely require the DM to up the power level of enemies as more spell damage would get cast than in a normal campaign. But again, if that's the style of game you wished to play, the DM could certainly work around that.
So really... to ask whether anything would be "game-breaking" is completely contingent on the problem you are trying to solve and the style of game you wished to play. This ritual rule would be a wonderful fix for certain ones, and a massive issue for others. Figure out what style you want and the answer becomes rather clear rather quickly.