I'll add my confusion to the issue. I don't understand which things make it "in" as fantasy in your world and which don't and why. I understand what I think are some examples of the lines you wish to draw (knights on horses = fantasy, knights on schwins = not fantasy), but I don't understand how those lines are valuable, useful, or even sensible. There's no useful distinction, as far as fantasy genre or game is concerned, between a knight on a horse, a knight on a unicycle, a jedi riding a giant lizard, a wizard pedaling a bike through the streets of modern London, a soldier on a chocobo, snakes on a plane, or a ninja on a skateboard. I can tell you think that one is better than the other and should be supported more, and that you think that D&D not only should cater to that, but that it would be denying it's very nature by not cating to that, and that not catering to that would be economically unwise.
Marketing trends would disagree vehemently with you. They would say that D&D should absolutely abandon Tolkien to the age of musty old academics and embrace, say, the ninja on the skateboard. This is the essence of the OP, in which dwarves don't sell novels -- mythologically authentic gnomes don't sell games. Now a tickster-race that is vaguely inspired by mythology and allows me to play a certain kind of character archetype (tricky wizard? crackpot inventor? sagacious sprite?) sells games. If 4e came out and was all about ninjas on dinosaurs and played entire adventures on your Xbox, perhaps online, and you could buy tradable and collectable characters online, it would be the hottest product this side of Pokemon. If D&D wants to make the big bucks, they should punt out Merlin and embrace Sabrina The Teenage Witch, because more people know Salem the Talking Cat than know what the hell a d20 is.
D&D, then, is working against it's own internal restrictions -- what it has defined itself to be. It doesn't WANT to be ninjas and dinosaurs on your Xbox, and most of the audience now (niche as they are) don't want it to be that, either.
What D&D wants to be is a game where you can pretend to be a fantastic hero for a few hours, beat up some monsters, gain power, and repeat, telling some sort of story as you go.
And none of THAT requires or even suggests that D&D has to or would even benefit from preserving anything that does not serve this purpose. Note that there is very little genre stricture there, because D&D has not ever had any kind of solid genre stricture, nor has it ever really seemed to want it. This is a good thing -- as the needs of it's audience changes, it can change, too. The needs of it's original audience was to emulate early fantasy, pulp, and the heroes they were (mostly) reading about in books about faraway places and unusual worlds and the myths and legends that inspire these books (or not). The needs of the audience have, in part, significantly shifted -- now they want to emulate Indiana Jones and Cloud Strife and Harry Potter and Naruto Uzamaki and He-Man, and they want to do it with friends and cool customization and button-pressing ease-of-use.
The world has changed since Gygax and Arneson and Blackmoor and Chainmail. The audience has changed. The game itself has changed (and will continue to do so). There are still those old fantasy novels mildewing in your mother's basement, but D&D has to come OUT of your mother's basement into the bright and loud light of pixels and spikey hair and allegorical boy-wizards in modern London. The old sources don't need to go away entirely, but they do need to be pushed to the background, because they aren't why new people are picking up D&D. They will always be one of the influences, but they have no sacred place in canon because there can be no sacred canon.
Dwarves might not sell D&D novels because dwarves aren't cool right now (for whatever reasons). Dwarves have no value. I know at least a half-dozen people who would drop $40 on a 260-page Naruto hardback sourcebook THIS VERY WEEKEND who wouldn't bother with finding the money for a 260-page "Inspired by actual Irish legends!" hardback sourcebook. Because actual Irish legends have no value (or at least, significantly less value).
You can make the artificial distinction between magic and science as much as you'd like, but the truth is that the majority of the buying public sees nothing whatsoever fundamentally flawed with elemtental druids who watch TV, or leiterhosen-wearing, Meg-Ryan-Looking sports celebreties who save a world in which dark wizards wear leather straps, or paladins in spaceships, or merlin being a powerful psychic and the lady of the lake being a powerful wizard in the same world. They have a fantasy world where cars with internal combustion engines can be enchanted to fly and where hidden ninja villages watch TV and eat instant noodles. Their sword-wielding knights can ride into furious combat on a motorcycle and they feel no great need to re-write the entire laws of physics for their setting just to do it. Magic can be a science (I cast a spell and this 747 flies!) and science can accomplish magic (It's ALIVE!) side by side and even together on the same day of the week. Magic can also be ofuda-wielding monks next to cowl-wearing wizards adjascent to tiny sprites all doing different things with different magic, too.
There simply IS NO PROBLEM WITH THIS in the minds of the people with money to burn. And D&D would be smart to follow suit. I'm not the biggest fan of the warforged, but I'd be a marked fool to say that they don't belong as the focus of what D&D is about at the moment. Because playing character archetypes such as "questioning my origins" and "mighty brute with a muddled past" has a LOT of value for the game.
D&D's greatest successes have been built on allowing others to take the works of the fantastic that inspire them and make them their own, combining them, twisting them,and turning them into something new in the process. It doesn't matter if it's from Wierd Tales, Homer, Arthur's Knights, Pre-Islamic Arabic Poetry, Star Wars, Indiana Jones, Bram Stoker, the latest issue of Shonen Jump, or just an idea that occured when you saw D&D had a rainbow of dragon colors. D&D does it's best not only when it allows people to OWN these archetypes for themselves, but when it allows them to mingle, twist, and digest these archetypes to re-interpret, re-imagine and re-birth (as well as simply replay) them.
Part of that involves turning Frankenstien not just into one definate thing, but into a whole host of them. There's Flesh Golems. There's Warforged. There's zombies and the undead. There's "mad alchemist's experiments." They are all entirely fantasy and entirely what D&D is good at, what D&D can sell you that World of Warcraft cannot -- a world where you can send Merlin up against the cthonian horror from beyond time and space, and have him team up with Bruce Lee and Sherlock Homes to do it if you want, or simply send Merlin up against a corrupt army of orcs and goblins lead by the vicious Faerie Queen. And maybe because you saw Dawn of the Dead last night, she also controls zombies. And maybe there's a dragon, and maybe the minotaur attacks you. Plus, you join up with Arthur and Lancelot and...some priest? And you save Gwenivere from a rampaging terrasque.
In some ways, Frankenstien is a great metaphor for D&D, but I'll go with a cooking one, something like gumbo. Gumbo doesn't taste much like anything it's composed of. D&D doesn't taste much like romantic literature or Homeric epics or Tolkein or Star Wars. You can pick bits out here and there, and some less-identifiable bits, too. But it's all mixed into a pot and stewed together. You can pick out the parts you don't like, but they're gonna sell the gumbot that's popular this season -- and if a lot of pork is in, you'll get a lot of pork, and if you don't like pork, you'll have to pull it out. There's no One True Gumbo (though I'm sure I will get called on this.
), there's no archetypal Gumbo. There's no archetypal D&D genre, there's no one true D&D flavor, even D&D's own kind of flavor is more the spice that blends everything together.
Genre adherence does not sell games. Genre flexibility certainly does. And a cooperative science/magic atmosphere will sell more games than one in which the laws are more rigidly defined.
