One problem is that without magic, characters are simply not capable of doing things that fit the genre -- whether you're talking about Tolkien, Arthurian myths, Conan, or the Three Mustkateers. Issues:
(1) Problem: Combat in D&D assumes a fair amount of magic. Armor Class is totally independent of skill/level, making it difficult to simulate any combat other than medieval knights (and even that doesn't work well).
Possible Solution: Give some sort of class-based Defense (like WOT or Star Wars).
(2) Problem: Hit points are supposed to represent luck, ability, etc. as much as the ability to take physical damage. Obviously, they don't. Hit Point recovery IS tied to the idea of physical health. Without magical healing, your PCs will constantly be "resting" to recover (slowly). That's not only NOT genre, it's BORING.
Possible Solution: Perhaps try a vitality/wound points system. Or maybe Ken Hood's Grim & Gritty Hit Point system. Or Mutant's and Mastermind's Damage Save system. Lot's of choices....
(3) Problem: PCs have relatively few abilities in D&D compared to characters from myth and fiction. Magic supplies most of these powers in D&D.
Possible Solution: Give more skill points (check Star Wars, WOT, etc.)! Maybe give more feats to allow more "unusual" stunts. Or look at Four-Color to Fantasy or Mutant's and Mastermind's as ways to allow fantastic abilities without magic.
(4) Problem: Related to 3 (above), many PC classes have magical powers. It would be difficult to make them workable without.
Possible Solution: Drop the classes. Look at Swashbuckling Adventures (a low-magic game -- possible to make "no magic", I think). D20 Modern classes could work well. Or Mutants & Masterminds has a point-build system that could work.
Conclusion:
You can run an effective D20 "no magic" game, I think, if you carefully pick the options you want. Running a "no magic" D&D game, on the other hand, would entail so many changes that I can't imagine the effort would be worth the pay-off.