I agree with the OP.
I'd like to add another reason to "why fantasy?"
Fantasy supports power growth through increasing acquisition of powerful magic items and abilities... from the a low-level village where a +1 sword is a notable heirloom, up to the iconic +5 flaming longsword and teleportation twice a day.
Most non-fantasy setting types (stone age, wild west, sci-fi) do not support such a breadth of "external to the character" options. A better gun in an American West setting does more damage, but if it also lets you teleport across the room or sets buildings on fire, you're stepping into magic item/fantasy territory.
Sufficiently advanced technology can be similar to magic (I know where the quote comes from, but given his life choices involving minors I'm not going to mention him), but there's usually not a lot of room for technology change on an individual level in sci fi. Star Trek covers a couple of hundred year (+-), but most of the technology changes are on the ship level (stealth, better teleportation, faster engines). At the level of the individual player (one of many characters aboard a ship), the only improvements you could hope for is getting better phasers, personal shields, and maybe a couple of other gadgets - a far cry from the dozens-to-hundreds of options available in D&D.
There's room for more "individual gadget/magic item" diversity in science fiction if you make the technology uneven, have relics from previous times, and give some people special psionic powers... but then you're doing something more like Star Wars, which is basically fantasy in spaaaaace.
The feedback loop of increasing personal power through leveling up, items, etc. is one of the attractions and reward mechanisms in D&D, and fantasy settings simply support it better.