And yes, I realize that tieflings and dragonborn are more popular than halfings. I think a lot of it has to do with newer generations of D&D players being more influenced by video games than they are fantasy literature, for better or worse.
Absolutely and demonstrably incorrect.
People with demonic heritage and dragon-people are far more common in modern fantasy literature than halflings (albeit largely because of the people with demonic heritage - dragon-people are rare-ish, though dragons who can turn into people are not uncommon). It's true whether we go with after 1995, after 2005, or after 2015.
Halflings or similar are
more common in
computer games and less common in fantasy literature. So it's the reverse. I can demonstrate this with examples, because I play vast numbers of fantasy video games AND read tons and tons of fantasy lit.
In the way I'm using them, tieflings and dragonborn are more "gonzo" than halflings because they diverge further from realism and traditional fantasy.
That's a extremely silly way to use gonzo imho. D&D is one of the most gonzo RPGs in existence in terms of "going beyond reality". It's extreme - with gods granting spells and walking the world (and you can go visit them for a chat at higher levels), world-changing or reality-changing spells, dozens or hundreds of sapient races, the Underdark, dragons being rife, people walking away from 1000ft falls, people coming back to life at the drop of a hat and so on. If Exalted is a 10 on gonzo scale, D&D is an 8 (Greyhawk certainly is - something like Taladas might only be a 6/7, but Planescape is a 9 and Spelljammer a straight 10). Something like most LotR games is a 4. A lot of other fantasy RPGs, like, say, Blue Rose, are down in the 3-5 range.
There's nothing "realistic" about halflings. They're no more or less "realistic" than the other races. As for "traditional fantasy", you're going to need to define what you mean by that. Do you mean Tolkienian fantasy? That's hardly "traditional fantasy". Or do you mean "computer game fantasy", or "1980s tabletop RPG fantasy"? Those are the only places where halflings were/are more common than tielfing-types or dragonborn-types.
Assuming we get some kind of new PHB in 2024, I highly doubt they'll cut halflings, tieflings, or dragonborn. My guess is that they'll include orcs as a PC race, and adjust the existing ones to include current ideas and preferences.
I agree that this is highly likely.