I couldn't agree more. One of the things that I absolutely loved about 4e was that it pushed some of the world altering magic to a higher level or into rituals. This made the game so much more...organic and flexible in how one wanted to approach the world building. Of course, there was plenty of short term/range magical wahoo at the tactical level (what with the nightcrawler-esque bamfs of warlocks and High Elves and whatnot), but that was much easier to deal with most of the time.
OTOH not having Fly in the game in any meaningful sense made it not feel very ...exciting. Hey, I can cast "overland flight" using a ritual in real life, it's called Expedia. Siloing good spells and effects like Fly, usable for time periods > 5 minutes per day, into a ritual bucket, made it feel like a cut-scene and the casting time was ridiculous. It was a mechanical hack to avoid having to design powers that worked in 3 dimensions, or balance pillars of play in an organic way. I don't siloing is roganic at all, it's a wall preventing all sorts of neat things from happening in the game. If you need ten minutes to get your party to fly out of the exploding volcano, the DM has no choice but to alter the flow of the lava to suit the game rules, or just kill you. That
sucks. 4e Rituals suck donkey juice and ruined the magical feeling of the game for me. If I want to be stuck to the ground, and I can play a game called Warriors and Warriors, not Wizards and Warriors. You cannot balance fly with not flying, except by duration and x/day. 4e nerfed it waaaaay too much, and the rituals were like a slap in the face to proper D&D experience.
Not once in any other edition with DMs have I seen the fly spell ruin an adventure. It's called design the game with 3d in mind, accordingly. That's just one example. There are just so many times when we wanted to do something that was only available in a ritual during combat in the heat of the moment, to react to an in-game event, but oh no, the duration silo'ed that possibility out of existence. It was a lazy hack to remove magic from exciting, real-time situations. Ten minute casting times is good for a cut-scene, but personally as a player, whether it's a videogame or an RPG, I hate cutscenes where I can't control my character. While a ritual is being cast, there is no possibility of combat, narratively (imagine a DM being a dick and dropping orcs on you in the midst of your ritual that you just spent 500gp on...wah waaaah, sucks to be you).
The problem that siloing solves is not worth killing the entire flavour of the game over. It's just another implicit way they forced combat / non-combat time to be strictly separate. THAT would be a dealbreaker for me. The fact that spells now have ritual versions attached to them as an option, is the best way they could do it. But making stuff ritual only was one of the worst design fails of 4e and an utterly detestable atrocity.