In one of my personal 'now shut up' responses to 'but my ~versimilitude~!' was that since adamantine and mithral exist in large enough quantities that doors are being made of the stuff, it's likely they're trace minerals that enter every living creature via the food chain, thus making all living things in the world lighter and more resilient than Earthbound versions.
Who needs iron supplements. That's why the local peddler sells adamantium and mithril supplements - for those that just don't get enough in their diet!
That would also be a fun mystery adventure with a nice mythical twist. The local baron's peasants are sick/dying. Turns out they're all suffering from mithril deficiency because long ago the baron figured out a way to siphon it from the water/food supply to augment his armory (and it's just now catching up with the peasants).