The real world never had to deal with powerful mages deciding that they wanted to destroy civilization every few hundred years and setting back progress. At best you have socio-economic collapses every once in a while that slow things down, but not to the level of your average fantasy world.
We just had disease, war and the inherent incompetence of empires doing that.
Remember: we don't have the tens of thousands of years of what we'd call 'civilization' fantasy worlds have due to spec-fic writers having no sense of scale. When 50,000 years you become, look as good you will not.
Look at the damage we have done with 1000 years of technology.I mean, fantasy worlds have all that too. Plus the occasional civilization destroying apocalypse that does even more than those things to boot.
Imagine if we could break natural laws with magic.