Personally, I struggled for years to bend 3e into the shape I desired. 8 in fact. 8 years of actively fighting against the system, trying to develop rules tech that was workable. As to 5e I run a game for 4 months, attempting to make it work for Apocalypse World style play, What I found is that in every moment of play I had to decide when a roll was necessary, what the possible suite of outcomes was and struggled against the resource paradigm, specific spell effects, how the game pretty much demands representing NPCs with the same sort of fundamental model as PCs. In both cases it was a tremendously frustrating experience and when I sought help on these boards during the 3e era I was told I needed to change my play priorities, that it would never work, etc.
That sounds painful. I've followed a somewhat thorny path, but not as frustrating and wearying as that must have been for you. I wanted to share the solutions I have been using for a couple of the problems you identify.
I had to decide when a roll was necessary, what the possible suite of outcomes was
The first part of this I eventually solved by adopting a principle that a) a roll is called when the rules indicate, and b) "uncertain" has a concrete meaning at the table, which is (only) anything that looks worse than +5 nett. The latter means for instance, that a talented or skilled PC doing something easy doesn't indicate a check (DM shall not call for one.)
The second part I solved by putting very strong emphasis on what the player describes doing. Barring actual magical mind control, players say what their characters do. Consequences, according to DMG 237, are known up front, so our conversation establishes them (or if there are none, the rules do not indicate a check.) I'm not confident this will work for pick-up groups, though.
the game pretty much demands representing NPCs with the same sort of fundamental model as PCs
I ran hard into this because my campaigns involve less murdering designated monsters and more interacting with NPCs. What I frequently wanted was a mechanically lighter-weight version of characters. That problem actually drove me to write a note on the interaction between complicated mechanics and extemporising (I have a feeling that I shared it only on an RPG discord I'm in.)
During the lockdowns when I started using a VTT in earnest, I discovered lightweight NPCs scattered among my sources. For example, just looking now alphabetically, I see Abjurer, Acolyte, Apprentice Wizard, Archdruid, Archer, Archmage, Assassin, Bandit, Bandit Captain, Bard, Berserker (various types), Captain (various), Commoner, Conjurer, etc. In play, I find that these light-weight versions serve well and are quick for me to modify using the VTT tools. Not much help at the table, perhaps. On my wishlist for a supplement is a book of characters! There are a few works in that direction on DMs Guild.
If you'd like to share any rules or principles you drafted over those fraughtful years, I'd love to see those. Equally, if you'd prefer not, then given what you have described I can readily appreciate why.