I'm using "magic" as short-form for "clerical, divine, or arcane magic, or other supernatural power". To me, you can substitute cleric for wizard in the entire argument and get the same results.
To be honest, it's a little bit of a tautology. What is magic? Magic is anything that allows you to break the normal physical rules.
The tautology matters, though, because it goes beyond "Cleric" and "Wizard."
Gandalf? Gandalf was the archangel Michael. He was also inspired by Odin, who was, you know, a god.
Merlin? Merlin was an antichrist.
Circe? Circe wasn't just a witch, she was also a minor goddess of magic.
When we look at "wizards" that D&D takes after, what we see is a direct connection to the divine.
My problem is when we then exclude the non-wizards from this. What of Hercules or Gilgamesh or Cu Chulain? The common response is: "Well, they're related to the gods." Well, yes, but so are the wizards!
If you have wizards and clerics in your game, you are already stating that it is alright for a Playable Character to be inherently supernatural. The only question now is: are all PCs supernatural, or are some not? If the answer is B, then the game you want is Ars Magica. If the answer is A, then it is not a question of verisimilitude, but rather of the ability or lack thereof of non-wizards to be supernatural. If anything, what destroys verisimilitude is the idea that "only wizards are allowed to be supernatural." That doesn't fit at all!