I don't have issue with them wearing non-metal armours, but yes, for consistent world building reasons non-metal armours that offer equal protection than metal armours cannot be particularly common if you want classic medieval fantasy setting where metal armour is widely used. So such high-tier non-metal medium armour must be somewhat rare, perhaps even magical. And that's is perfectly fine. Seriously, many classes don't get any armour at all, the druids are not unfairly punished.
"many classes" being the Monk (who has an ability specifically to account for lacking armor) the Wizard (one of the most powerful classes in the game, with two subclasses that give them increased AC and has a long-term AC bump and a 1 off AC spell bump [mage armor and shield]) and the Sorcerer (who is identical to the wizard, and has a subclass that gives them a permanet AC buff)
Huh, I wonder if that indicates something...
But, moving on... so convenient isn't it? For consistent world-building reasons non-metal armors that make you safer against spells like heat metal or shocking grasp, that don't require heavy mining, and that can be made from the hides and shells of commonly and naturally occurring creautres in a fantasy world, must be rare. Because we have to all play in a classic medieval fantasy setting.
And, if we don't? If say we play in a world like.... the Forgotten Realms that is clearly full of fantastical elements and anachronisms all over the place. Like hand crossbows. Which are in every town and city all over the place. Or maybe spyglasses, which again are pretty easily found in any city, but were barely invented by the end of the medieval period. Advanced glassmaking to have things like hourglasses and the ubiquitous glass flasks that are every.
But, we don't want to have a city selling armor made form monster parts. I mean, it isn't like the medieval people had a long tradition of using all the parts of the animals they killed, leading to things like scrimshaw. I mean, that would be unrealistic, and as the DM, you can't let DnD be unrealistic just to benefit the players, like having a universal coinage that everyone follows exactly. It just has to be how the rules work. Can't let one persons preference dictate how the world works... well, unless they are the DM, and then they don't have a choice, a classic setting has to be run a certain way no matter what.