Here's my attempt:
Each round (from the start of your turn to the start of your next turn) each hand can only do one thing.
The things you can do are:
- Attack with a weapon. Some weapons require two hands.
- Wear a shield.
- Cast a spell with S components.
- Cast a spell with M components. This includes anything that says "you must present your holy symbol".
- Exception: The same hand can be used for a spell with both S and M components.
- Activate a magic item. E.g drink a potion, activate a wand.
- Hold or Use an item (which includes picking something up). E.g. hold a lantern, pick up a dropped sword, open a door, throw an oil flask, scatter caltrops. Some items require two hands.
- Make or continue a grapple.
- Break a grapple.
If you have two hands, you can pick at most two items from the list to do this round.
Some features will modify this. For example.
• Holy Symbols can be mounted on a shield. In this case, the same hand can "Wear a shield" and "Cast a spell with M components".
• If you have the War Caster feat then the same hand can "Wear a shield" and "Cast a spell with S components" and can "Attack with a weapon" and "Cast a spell with S components".
Upon consideration, I think this can be refined slightly. Say that each round, each hand can hold one item or be free, and certain actions require hands to be either holding certain items or free.
To attack with a weapon, you need to be holding a weapon. To attack with a weapon with two hands, you need to be holding a weapon with one hand and the other hand to be free.
To cast a spell with S components, you need one hand to be either free or holding an appropriate spell focus. (Properly decorated shields count as spell focuses for divine casters.)
To cast a spell with M components, you need one hand to be holding material components or an appropriate spell focus.
To use an item in your possession, you need to be holding the item. Some items may also require a free hand.
To use an item in the environment (opening a door, pulling a lever), you need one hand to be free. Some items may require two free hands.
To begin a grapple, you need one hand to be free. Your hand is then grappling the other creature and is neither holding an item nor free.
To continue a grapple, one hand needs to be grappling the creature.
To break a grapple, one hand needs to be grappling the creature or free.
Etc.
I think this approach lets you do an end-run around some of the modifications and exceptions you were beginning to spell out. For instance, as you can see, you just have to say that a cleric's shield counts as a spell focus and then it follows automatically that the hand holding a shield fulfills the requirements for casting a spell. It also lets you be a little more flexible with free hands, and makes having a free hand more valuable (which I think is desirable). You can open a door and then cast a somatic spell or swing a greatsword, for example.