Nerfing cantrips is fairly common, it seems.
It doesn't seem like that to me, but go on...
A standard response seems to take these down from the at will state to something that involves using each X number of times based on some combination of ability score modifier, number of cantrips knowns and level per long rest. The impact of this means that a primary caster will have to manage their resource economy in similar ways to martial only builds.
Quite the opposite, it would tend to increase the difference between caster and non-caster builds and how they are played and manage their respective resources.
Those that would change the rest periods from short is an hour and long is eight hours to the variant of short equals an evening and long equals a rest day during a week, usually in some kind of sanctuary, also change the resource management of casters. Many who use this technique also change the pace of play, but if that isn't done casters are again reduced in power relative to non-casters.
Changing the duration of short or long rests should be fairly transparent. The vast majority (I can't think of an exception) of PC abilities are either at-will (quite a few, if you include /everything/ PCs can do, however universal or trivial), short-rest recharge (relatively few, really), or long-rest recharge (the majority of really meaningful ones). Changing a short rest from an hour to 8 hours sleep, and a long rest from 8 hrs with 12 hours between long rests to a full day of R&R in a (relatively) safe environment, with a week between long rests, actually has no impact at all on the relative power of the classes - iff the DM paces his campaign such that the number of encounters and the number of short rest between long rests averages about the same as it does under the normal assumptions (which the DMG, IIRC, states as 6-8 med-hard encounters between long rests, and at least implies (IMHO) that there would also be 2 short rests, on average, between long ones).
But what if you want a lower magic world than doing both of these options? How would you do that while still allowing magic?
If you want a lower magic world, you need to do 2 things:
1) cut down on the number/power of magic items you give out. this is easy, it's entirely within the normal realm of DMing, you can even reduce the number of items all the way down to 0, since items are /not/ assumed in D&D's level progression the way they were in prior eds.
2) ban full-caster or 'primary caster' builds. That means EK, AT, Pally, Ranger OK - full Sorc, Wiz, Cleric, Druid, Bard, Warlock, banned or restricted. You could make primary caster classes into de-facto PRCs. For instance, you could take Wizard 1 if you've raised your INT by at least 1 since first level and have Expertise in Arcana, and have acquired a spellbook during your adventures (since they're too rare to buy). If you want to make magic less overt, you could trim spell lists.
What if all non-cantrip spell magic (spells, invocations, ki) was rituals? Is that still DnD5? Could it be fun?
Would another option be reducing the number of spells used per long rest? Is it still DnD5? Is it still fun?
Fun is subjective. D&D 5e is very open about being just a starting point for the DM. Yes, you could do either or both those things and it's still very much in the spirit of 5e. You can do whatever you want, up to and including re-writing the 5e rules until you are not only figuratively, but actually, running it out of 1e, 2e, or BECMI rulebooks, and you'd still be running D&D in the spirit of 5e.
If you had a world where there was no spellcasting, but still some of the other spell-like abilities thereby changing the flavor of the world entirely are you playing the wrong game?
It'd just be a matter of banning most classes and sub-classes. You'd still have barbarians, monks, champions, battlemasters, thieves and assassins. Lack of healing would be a serious issue, and you'd want to be careful with encounter design and pacing, but you could make it work, and, 5e, by it's own philosophy, is never the wrong game.

(Still, there're games that do magic /that/ low significantly better and/or more easily - like 4e or Iron Heroes or 7 Seas or even FATE or - well, almost any game that's not D&D - but, that doesn't mean you /shouldn't/ go ahead and run it in 5e.)
Is there enough pay-off with such heavy house rules that it can be more than a gimmick?
Like whether it's fun, that's an answer each group would have to answer for itself.
Unfortunately, the PHB outright states on page 8 that magic is necessary in an adventuring party and a third of the book is dedicated to spellcasting, so I think you'd be losing a lot there. Every class has some spellcasting options, monsters have resistances requiring enchanted weaponry, and a ton of monsters are spellcasters. It's so ingrained in the system that, at least from my little experience with 5E, I don't know how you could mechanically change that.
Not that any of that is false - a lot of it may even be more or less true of most versions of D&D - but it's not quite that daunting an undertaking. Blanket banning primary casters or all caters or all builds that use magic is excising the majority of PC options, but it's not hard to explain or implement. Tailoring encounters to the abilities and resources of the resulting party means fewer and/or weaker encounters (because HD don't allow much healing), but simply not selecting monsters that need magic to defeat (or ignoring the traits that make it so) is not that cumbersome compared to tailoring encounters to any other party, or trying to keep everyone in party of mixed full-/half-/non-casters interested and relevant - it could even prove to be easier.