I honestly think most of the optional rules in 5E should just be outright deleted or re-written from the ground-up, rather than "expanded upon", but maybe re-written counts? Like, let's go through the DMG:
1) Proficiency Dice - delete. Waste of everyone's time that makes the game work less well.
Before this gets argued, let me just lay something out: I think the DMG should set a good example to DMs about optional rules and house rules.
That means the DMG should not include rules that are just "for the lolz", which don't make the game better, nor change it to some other distinct playstyle, that just mess with the mechanics "just becuz". And Proficiency Dice are 100% "just becuz". There's no good mechanical case for them in 5E.
2) Skill Variants - Expand.
I know what I just said, but if there was one area to expand on, and make some mini-rules sets with examples for, this would be it. This section suffers badly from a lack of examples.
3) Hero Points - Fold into Inspiration or just delete.
4) Honor - For god's sake delete, this is borderline racist, esp. with the orientalist art.
5) Sanity - This is literally a terrible mechanic which achieves almost exactly the opposite of the stated goal and a great example of how rushed the DMG was and it's kind of not great conceptually ("seeing weird stuff makes you nuts" leads to a lot of problematic tropes). Replace with some kind of Stress mechanic or something.
6) Fear/Horror - Expand or delete. Pick one.
7) Healing options - ALL the healing options through the DMG and so on need to be unified in a single place, and both re-written and expanded and on top of that, they need to be explained in a lot more detail re: consequences etc. Essentially Expand but you need to re-write first! The "healing surges" they have here are an insult to the concept - I wrote a version that made way more sense and worked more like the concept in like 20 minute a few months back (can dig up if wanted).
8) Rest variants - Unify with healing options, they shouldn't be separated. Also completely re-write both. And/or add in two "middle ground" options.
9) Firearms - Delete. I know some will disagree. I don't care. Guns that aren't the most very basic arquebuses and the like work really badly in 5E, and cause all sort of "hmmmmm" situations unless they're solely for a "gunslinger"-type class.
10) Explosives - Expand. Perverse? Me? But I genuinely think explosives, as well as Greek Fire and Acid can work really well in D&D, as consumables rather than regularly-used weapons. I'd like to see this done a lot better, and Greek Fire/Acid made into something that scales, potentially.
11) Alien Tech - Delete. Leave it for a sourcebook/adventure. Save the space.
12) Plot Points - Expand and formalize. But don't use use the existing waffle, which is barely "rules" at all.
13) Initiative variants - Fine except delete Speed Factor. That's a huge amount of text for an optional rule I've never even HEARD of being used, even from you guys who LOVE the most obscure option rules possible. Ditch it. There's a limited page count people.
14) Climb onto a bigger creature - EXPAAAAAAND!!! Come on! We could have something awesome here. DO IT! Either that or delete it from the DMG and do it as proper optional rules in a later book.
15) Disarm - Should not be an optional rule, and should not be terrible.
16) Mark - I'm a 4E fan but delete this. It's just clumsy and falls totally between two stools. A sad, pointless relic of a modular 5E that never materialized and that 1D&D won't be.
17) Overrun/Shove Aside/Tumble - De-optionalize this stuff. Put it in the PHB. It doesn't significantly complicate combat and does add options.
18) Hitting cover - It's fine.
19) Cleaving - It's fine but tempted to say de-optionalize or delete so it can become a class Feature or even a Warrior Feature.
20) Injuries - Hmmm, like delete and re-write with something that is both expanded and much more compatible with D&D's rules. It's trash. I know people who use injury rules in their game. Literally none of them use these rules nor variants of them.
21) Massive damage - De-optionalize.
22) Morale - Replace with the approach that World Without Number takes, which works on a much broader variety of creatures, and actually work into the system. So Expand, I guess?
That's the main optional rules section so enough for now.
More that I think should be expanded than I remembered, I have to admit. A lot of these are so bad I'd forgotten them entirely - but the basic concept needs rules. It just doesn't need these terrible rules.