Imagine the PHB is blank on the subjects of hand use, object interaction and spell components.
Wow, surprisingly easy. And liberating, thanks.
Now it's your task, Oofta, to write up a set of rules that result in characters doing much the same things the PHB allows them to, yet avoids loopholes such as the ones we have been talking about.
Free Hand: some actions will require that you have a hand free, while some items must be held or 'wielded' to be of use. Most characters will have only two hands, so you have to decide what you're doing with your hands. You make that decision at the start of your turn, swapping out the items, any, in either or both hands if called for. Hands not used to hold/wield (and thus in any way benefit from) items, are 'free' for actions requiring one.
Interacting with attended items: Any item you have on your person is assumed to be either held or sheathed/in some other container/tucked into a belt/etc. Such items are 'attended' and you can swap attended items at the start of your turn. Pulling out an attended item at any other time requires a free hand and an action, on your turn, or reaction, if it is not your turn. You can also voluntarily drop an item at any time on your turn, or right before taking a reaction, which gives you a free hand and makes the item unattended. You cannot generally interact with items attended by other creatures unless you first grapple them. Once grappling, you can attempt to wrest a stowed item, the procedure is the same as for escaping the grapple, which you must also then do separately.
Interacting with unattended items: An item not held or carried by a creature is unattended. Most some items rest on a surface, typically the ground or perhaps hanging from the wall, ceiling, branches of a tree (like the Golden Fleece), etc - though it's not impossible, the fantastic D&D world, that you might find an unattended item floating in the air. You can pick up an unattended item you're adjacent to at the start of your turn, when you swap your attended items, the item you pick up is held and becomes attended (you can swap it for another attended item at the start of your next turn or use an action to stow it). During your turn, if you have a free hand, you can pick up an adjacent unattended item as an action, making it held & attended and the hand no longer free. You can use your reaction to catch an item tossed to you, or to pick up an item that becomes unattended adjacent to you, as with using an action, you must have a hand free, the item becomes held & attended, and the hand is no longer free. If you try to snatch up an item that is tossed past you, that the DM rules was tossed badly, or that another creature is trying to snatch, the DM may call for a check.
Spell Components: Spell components fall into 4 categories, Somatic (gestures/movement), Verbal (words and other sounds), Material (supplies expended when the spell is cast) and Foci (items used but not consumed in the spellcasting). You must have all the components required by a spell ready to use, including having any focus or material in hand - typically, you'll swap the focus/material into your hands at the start of your turn. Since many spells that use foci may use the same focus, they are particularly convenient, as you can decide which of them to cast later in your turn. Some spells will state exceptions, for instance, that the material component need only be on your person, or that the focus can also be a weapon, or that the verbal component need not be intelligible speech.
Somatic Components: Unless otherwise stated, somatic components require a free hand, so a bound or grappling caster cannot generally use them. Somatic components must be precise relative to the caster and the local gravity plane, so casting while moving (spells should be cast before or after taking your move, not during it), riding a mount or vehicle, climbing, flying or crawling can all be difficult. Similarly, the Blinded, Frightened, Grappled, Prone or Restrained caster will find it difficult or impossible to use Somatic components. The DM may call for a Concentration or DEX check to use Somatic components in difficult circumstances. A spell that has somatic components provokes an attack of opportunity if cast in melee.
Verbal Components: Unless otherwise stated, verbal components are intelligible words that must be clearly spoken by the caster, loudly enough to be clearly heard in the general area of the spell's effect. Being gagged, out of breath, or holding your breath - including the Deafened, Frightened, and Poisoned conditions, having taken the Dash action the previous turn or moving while casting, or any level of Exhaustion - make it difficult or impossible to use verbal components. The DM may allow a Concentration check to cast a spell with verbal components in less than ideal conditions.
Material Components: Unless otherwise stated, material components must be held to be used and are consumed (vanish, erased from the character's sheet, and unrecoverable) when the spell is completed. Depending on the spell description, the caster may need to hold materials in one hand and make gestures with the other, make gestures with the materials, or use both hands to manipulate the materials. Any environmental condition that makes handling small/delicate items difficult can prevent or hinder the use of material components. Being immersed in water, for instance, would spoil many materials. High winds could blow powders, partchment, and the like from the casters grasp, and so forth. The DM can call for Concentration or DEX check to keep hold of materials in difficult circumstances.
Foci: Unless otherwise stated, a focus must be held to be used. When a spell calls for both material components and a focus, one hand is needed for each. Somatic components can generally be done with the hand holding the focus, though the spell description may give exceptions. Foci are generally solid, easily grasped items, not unlike weapons in their practicality, and a focus rarely makes a spell more difficult to cast. If, for some reason, the focus is adversely affected by the environment, the DM may call for a Concentration or other check to use it successfully.
There's some classic feel for ya'.

Not so much simplicity, though, but that's how I role...