There is a difference between a "power fantasy", which has negative connotations, and a mere "fantasy."
(In that it is an expression of imagination)
I think the problem here is that "power fantasy" is being used by its advocates without the negative connotation, but this isn't recognized by those disagreeing with them, nor are the advocates recognizing that it probably isn't the best term for what they're describing.
We play RPGs to live out a fantasy construct. It would be hard to argue that people create and play characters in a total, abstract, detached vacuum with none of their own personal quirks being worked in. I like playing faithful characters because I lack faith, I like playing old and wise characters because I am young and foolish. Or I like playing young and foolish characters because it is funny, and being funny is an enjoyable fantasy. And so on.
That's just regular fantasy. "Power fantasy", to me, suggests that the primary goal of it isn't the vicarious experience of a fantasy world, character, action, etc, but raw, unchallenged power. A power fantasist is not interested in being John McClane, Frodo, or even Sherlock Holmes. McClane and Frodo are fantastic because of what they did, not necessarily who they were. Sherlock Holmes was a deeply flawed, manic-depressive heroin addict and plot device.
A power fantasy is more like playing Steven Seagal; if you watch one of his movies, he's almost never even hit. I think he gets punched once in the sequel to the one where he's on an aircraft carrier, the one on the train. On the set he'd deny that he'd ever be unable to block a move- and all of his moves are instantly, typically completely, successful. This means that if you're looking for conflict or a challenged protagonist, you won't really find it in a Seagal movie; he's basically just a walking neck-snapping machine. But if you're just looking for the vicarious experience of power, he's only a rental away.
To me, "power fantasy" suggests a pretty damn dull experience. Power fantasy types want to be Elminster, the unkillable super-wizard. They're not interested in Harry Dresden, who has plenty of power but still goes through hell in every single novel.
Normal "fantasy" types might want power, but only if it lets them live out a more well-rounded experience- Dresden's power is a character trait that allows him to experience fantasy conflicts and face fantasy decisions (do I employ black magic or not? etc), just as John McClane's fantasy-level badassitude allows him to survive, but doesn't give him a guarantee of victory. The movie is well done enough that we forget the protagonist is going to win, and the path he takes towards victory is harsh enough we still feel he's "earned" it.
That's the crucial difference. "Power fantasy" wants a guarantee of power and doesn't want to pass through a gauntlet to get it. No price paid, no consequences, no nothing. Power is the exclusive goal.
That is not, I think, what people suggesting "power fantasy" as the basic template really want, but it is, I think, what the term is generally accepted to mean. I think it would be better to say that most D&D players want a different kind of fantasy- say, a "Heroic Fantasy" or "Fantasy Challenge" or something like that. Only a relatively smaller fraction of your exceedingly boring player base wants nothing but power to vicariously smite the guy who took their favorite parking spot.