Definitely a question worth asking.
My inclination is to approach every situation and ask what tools a person with no magical ability might have at their disposal. That should almost always form the basis, and mundane improvement should show a clear sense of continuity (thematically and mechanically) with those most basic abilities. I think here I mean about the same thing as GHamster does with "superlative."
Magic, then, normally does one of two things:
1) Causes the ability to operate in a way that is essentially discontinuous with what is established. I'd emphasize this distinction with mechanics in many cases.
2) Operates in a way that is continuous with mundane abilities, but alters the tradeoffs in the fast, good, cheap (choose 2) spectrum. In many cases there would not be a need to emphasize the distinction with additional mechanics, since the different tradeoffs and the means required to achieve them will often do so implicitly.
The idea is to avoid making magic just the stuff that happens when mundane means no longer suffice, but to let both operate in conjunction. I'd use the two points above as a sort of touchstone, with a lexicon of common mechanics to help cue the player. Monte talked about "dice tricks" last week, and I think these can perfectly function in this role.
Here are a few examples.
1) Healing. The notion of magical healing as accelerated or aided natural healing seems natural to me. Therefore let mundane healing (at least of hit points) be good and cheap, and really integrate the heal check. Mundane healing might also be fast and cheap for things like first aid and traumatic injuries. Magic is then a tool which meets different needs, and it can build on top of the normal healing rules. Perhaps cure light wounds heals hit points equivalent to a night of rest (roughly the amount of time it would take for a "light wound" to go from having an impact to being something a person can basically fight past). It might even require a check, which would allow using a "magic skill" to achieve its basic function, but the heal skill to achieve what the healer could normally do in less time. In other words, an application of guideline 2. However, I'd not allow mundane healing to do things where the body itself has been destroyed (A healer might reasonably make a salve to return sight to the blind, but only as long as the eyes are intact, for example). Magic that achieves this more closely follows guideline 1, and perhaps should have some mechanical distinction as well.
2) Sneaking. Magic that just gives a bonus to sneak checks doesn't really do much for me. Perhaps the cloak of elvenkind (and similar magics that interact directly with abilities requiring mundane checks) causes enemies to roll 2d20 and use the lower check. Everyone with such a cloak is better at sneaking, but it prevents dedicated sneakers from stacking bonuses in such a way that they can make themselves immune to perception checks, etc. Using 2d20 as a cue could also help DMs trying to decide how to represent certain kinds of circumstantial modifiers to checks already being made. For example, creative use of ghost sound or other cantrips. Things like Rings of Invisibility break continuity with regular sneaking, and the fact that one simply cannot be seen is exactly the mechanical break necessary to represent it well.
Likewise, mundane uses of sneaking could have its own set of common mechanical tools. Things like skill tricks come to mind. Perhaps, for example, a skilled sneaker could force a target to reroll a check but lower the effect of his own sneaking check by a few points for the rest of the scene (or whatever). A skilled sneaker might eventually not only have a higher bonus, but would effectively have some "sneaking hp" that represents the ability to eke out of some tight situations. And again, this allows them to represent their superiority without making it possible to sneak past everyone due to a ridiculously inflated skill bonus. And perhaps a mundane skill trick for expert spotters is the chance to spot truly invisible creatures from very subtle details. Most DMs I think give their players a chance to "spot" invisible creatures from certain environmental clues (i.e. when it's snowing) but the skill trick might be to make such a thing something that is expected rather than bargained for for that PC.
3) Weapons. 5e has a golden opportunity to step back from +1 to +5 weapons as necessarily magical, and move toward mundane but excellent craftsmanship that lets skilled wielders get the most of their skill. The philosophy of many wuxia and wuxia-inspired movies toward weapons (e.g. Crouching Tiger, Hidden Dragon and Kill Bill) as powerful but not exactly supernatural is close to what I'd like to see, and reading the rest of the thread it's clear I'm not alone. Iconic things like flaming are easy, but perhaps there are both mundane and magical versions of an ability that gives a sword blinding speed. Or perhaps only swords that have the mundane "speed" property can actually be magically enhanced with the magical version, or maybe there is some synergy from having both. I admit I don't yet have a clear vision of how this might look.
I think the best dividing line is something like this: if we can imagine a mundane way to do something, than that's probably the way it is usually done. Different settings will have different sliding scales here. If we want D&D to cover a diverse set of fantasy subgenres, it won't hurt to make ways for many abilities to be either magical or mundane. I wouldn't want to make them entirely interchangeable, because when there are no distinctions in function or access even saying there is a distinction strikes me as meaningless. (I realize that is hardly a universal inclination, and I don't feel the need to get worked up about it.)
