Well, for that, mostly, you can point the finger at the Tolkien estate. Since any D&D legit rendition of a Halfling, with hairy-topped feet, country gentleman, or even (probably) shadow-lurking rogue, is going to be seen (and potentially sued) as an infringement on the copyright and intellectual property.
Gnomes...eh, they get used sometimes. The D&D cartoon of the 80's, for example, used little, bearded, gnomish people (though I don't know if they were ever identified as gnomes, but they were definitely not the D&D -or even tolkien- imagination of dwarves. I don't think elves, even, ever had so much as an appearance in the cartoon. There was a sprite... once, I think). The aforementioned fantasy film classic, Willow, had a very gnomish portrayal of some of the "Pech" people. I think the Wizard of Oz's "Munchkins" always struck me as having a very gnomish feel to them (though at the time of the book's writing, or filming for that matter, D&D was some decades away). "Time Bandits," the Smurfs, the little folk in the film "Legend" had a very Gnome feel to them (I don't recall if they were ever identified as "dwarf" or "gnome" in the film). Various incarnations of Snow White (particularly those newer "Huntsmen" movies) have had very gnome-like renditions of the "Dwarf" characters, I think.
Have there been many RECENT media appearances by small folk in fantasy works? Have there been distinctly "D&D"or "Pathfinder" projects, 3-4-5e materials? No. None of that. Mostly because the fantasy oxygen of the past couple of decades was entirely consumed between LotR (and blech Hobbit trilogy) and Harry Potter. Then Hollywood fell into a Superhero-Comic Books black hole (not that I'm complaining. Love them!) That "Merlin" tv show was doing ok for a minute. Some other television or Netflix things that are more knights/kings/castles and battlefields -"Vikings," "Black Sails," "The Outlaw King," and the like, that either go for a "historic" type of fantasy from our actual Earth, or if in a fantasy world, as "real" a world as possible. So, there hasn't really been anything D&D or even "magic land/medieval medium to high-fantasy" for a while. "The Witcher," and "Game of Thrones" were probably the most D&Dish: one heavily uses "sorcery is real" and one was centered around dragons and undead. Apparently, there's something now call "Cursed" in Netflix that's doing pretty well (I'll have to check it out.), another retelling/reimagining/umpteenth version of Arthurian legend. Given it's Arthurian I wouldn't expect to see gnomes or halflings in there...but certain faerie folks, maybe. I have no idea how "sorcery/magic is a real thing" the series goes.
Long story long, it's all the Tolkien Estates' fault.
