I'm missing the part where typical fantasy worlds have populations in the region of 7 billion participating in their economies. Think high end art dealing in the Renaissance, not the late 20th century or early 21st.
The population interested in such extreme luxury goods wouldn't be sufficient to support shops.
Renaissance worlds did not have magic or magic items. Apples and oranges. Once you add in magic, all bets are off.
It's quite simple. Real world mass production, population, world economies, historic decision making, none of this matters when making plausible game decisions.
What matters is human behavior (both of players sitting at the table, and of their roleplaying of their PCs/NPCs). That is the yardstick by which one measures what is plausible in a fantasy world.
Let's talk behaviors like greed. Some (many) people are greedy. They will do anything to make a buck. Including buying, stealing, and selling magic items. It's normal human behavior. Any DM who says "PCs cannot buy or sell magic items in my world" is creating a bizarro world that goes against type. Some players don't really understand such a campaign. They'll play in the game, but it won't make a lot of sense. In order to placate the fact that it doesn't make sense, a DM might say "magic items are rare and special, that's the reason they cannot be bought or sold". On the surface, this might satisfy some players or at least allow them to ignore the incongruity. But for other players, it still sounds like total BS. The very fact that they are rare and special means that they should be in even MORE demand. Supply and demand. Economics 101.
Now, your point appears solely to be about magic item shops. You think "high end art dealing during the Renaissance" where people were poor and only the very wealthy could afford these types of goods.
And that's an ok point, but it fails some basic tests. Like the fact that healing potions are in the PHB with a price. It's expected that they can be purchased in a shop by most players, just like rope. A price ($50 GP) which is WAY outside what poor people can afford. Wouldn't poor people raid shops that sell potions of healing? It's the "doctor in a can" that doesn't even exist in the real world. Poor NPCs would clamor for that.
How about plate mail? 1500 GP. 3 times the cost of an uncommon magic item.
Where does one buy this in a fantasy world if the $500 GP magic items cannot be purchased?
How is is ok to have a merchant to buy plate mail from, but not one to buy a magic item from?
The point is, your POV is fine for your game. Don't have magic item shops. Don't have plate mail shops. Don't have shops with healing potions in them. That's ok.
But, your point is less plausible than one that does have ways for PCs to acquire plate mail, healing potions, scrolls, and many other magic items. Your POV is contrived. If a magic fantasy world really existed, especially one where NPCs can craft magic items, there would be ways to acquire these types of items and a very common one, at least for lower end items, would be shops. Everybody would know that in order to buy magic stuff, one has to go to the Emporium in Waterdeep.
These shops would not be in villages and even not in most towns. But every adventurer who finds gold coins and spends them means that more and more money is going back into circulation and hence, the market is there. Other adventurers, young noblemen with fat purses, traveling merchants who no longer want to fear bandits and monsters.
Supply and demand and common human behavior.
Some players want to create a PC that will build a fortress or temple or home base. I can understand that. But, my PCs are not there to do that. I am playing a game of adventure and exploring and combat, not one of building a home. The enjoyment for me is not in roleplaying politics or pedestrian interactions. Rather, it is the mystery of going to other places, and doing other things. To me, collecting gold is not a means to an end of building a building. That is SOOOOO boring sounding to me as a player. Yeah, it's fine if the team comes up with a home base and we all chip in, but I'll let some other player come up with the floor plans. I want to spend my money on things that help my PC to explore, not for things to decorate a house.
You might prefer differently, and that's ok. But the plausibility of magic items shops is dependent on the expectations of everyone at the table, not just the DM. IMO.