Why do swords and spells tend to be more popular than spaceships and lasers? Is D&D (and Pathifinder) so popular because it's fantasy? Or is fantasy so popular because D&D did that first? Or some other reason?
First, I don't think you discount the "fantasy is popular because D&D is popular" thing. If you eliminate D&D/Pathfinder from the historical bestseller lists, you find that those lists tend to be filled with a constant fluctuation of different genres. This suggests that there is no stronger bias towards fantasy than any other escapist genre.
Second, why do escapist genres -- science fiction, fantasy, etc. -- tend to dominate RPGs? Some of it is just a matter of mutual geek interests. Some of it is that escapism suits RPGs well. Some of it is the fact that they create permissive creative environments in which "anything is possible".
Third, why does D&D continue to dominate? Well, a large part of that is probably a persistence of market dominance and the power of network externalities in driving RPG brands. But I'll also toss out the hypothesis that classical fantasy is, in fact, advantageous for introducing new players and GMs to roleplaying games.
I say this for a couple of reasons:
1. Low-level fantasy limits transportation and communication. This limits the scope of what a new player or GM needs to be "on top of" for a given session. Compare that to a game of
Vampire: The Masquerade where the GM has to grapple with an entire internet of information being ubiquitously available to the PCs. Or to
Eclipse Phase where players can spontaneously decide to transport their consciousness across the entire solar system if the whim takes them.
2. Historical settings achieve a similar limitation of scope, but bring a different stricture in the sense of "historical accuracy". Fantasy doesn't have that problem: GMs and players are free to make up pretty much anything they want and it'll fly just fine.
So classical fantasy -- compared to historical, contemporary, or sci-fi games -- has an inherently limited scope which makes it easier for new players to grasp.