That's my point: if you have to make a pretty big rewrite/homebrew, then I don't feel the game is particularly suited for the genre. For example, if you have to ban or change half the classes that use magic, then the game a poor fit. Especially since that has knock-on effects to species, feats, monsters, etc. Now I will agree that you could make a damn fine d20 5e-compatible S&S game where you create an appropriate spell system that emulates the low-and-dangeous magic of S&S, but that's not D&D anymore. That's its own thing.
S&S in D&D looks like Primeval Thule: full casters, a variety of species (including dwarves, elves, and halflings), and semi-common magic, but dressed up in loincloths, cultists, elder gods, slavers, and the like. It being D&D takes precedent over its being faithful to S&S. Otherwise, you are building a Conan RPG and calling it D&D for name recognition.