It's only controlled by the dual-wielder feat if you're using feats. It's one of the main reasons why we're not using them in my game. I think it's the same with that combat casting feat that's otherwise-required for an arcane fighter to use a shield or two weapons - if you're using feats, then the feat is necessary, but if you aren't using feats then it's within the realm of interpretation that anyone can just do it anyway.
By that logic everyone can move an extra 10' and not take opportunity attacks if they hit someone because you don't have the Mobile feat and it's within the "realm of interpretation." Not trying to be snarky but if you are leaving out the feats it means you only have the base rules and how they are written/interpreted, leaving out the feats doesn't change what any of the base rules say/how they function. Your paladin can't cast VS spells with his sword+shield, your EK can't cast VS, or VSM spells with his either, stab the sword into the dirt and get to casting, then draw it again. (flavorful dropping) And your 12th level Fighter can't throw 3 javelins in one round.
To be clear what you do at your own table is completely your business, I am merely stressing that this is a house-rule and not how the system works RAW or RAI.
Personally at my table we've been dropping, it makes sense as frantic as combat gets that you don't have time to safely sheathe a weapon and draw another. Most of the time in combat one wouldn't even switch weapons without a lot of time to do it, not wanting to leave openings for attack, and it being a desperate action, dropping seems appropriate. Remember you've only got 6 seconds a round, and you're also moving 30'+ making up to 9 attacks (average of 2-3) or casting an entire spell, that's a lot in 6 seconds already.
I do wish drawing was free and sheathing cost an action, drawing can be done with much less care and from a mechanics standpoint would be much more usable for range on martial characters, as is they are screwed. This comes from a 15/2 fighter barbarian, my only ranged for 90% of the game has been handaxes, not the best option, but that lets the ranger and monk on my team really shine, I can tell them to shoot people with Commander's Strike and I often do(they get colossus slayer again!).
As for shields as divine symbols, I don't dm much but I would allow shields as druidic/arcane foci too. it would be as expensive as those foci normally are, but there's no reason your shield can't be made of magical wood or have a crystal in it near your hand. I would probably even allow one shield to be 2 types of foci, like a shield that is divine/arcane.
@
slaughterj(I would quote you but I'm too lazy this morning)
Your shield hand doesn't have a 'free hand' so you can't hold a sword in it, even temporarily. Most combat shields strap your arm in at the forearm and have a sizable handle for your hand, there isn't room to hold a sword in the same hand.
Also I would
never put a bladed weapon in my armpit/ under my arm, especially in combat. o.o
As to "dropping his sword and picking it up again" I usually flavor it as sticking it tip-down in the dirt.