Not that the OP here is bad, but I read those points and don't think there's a lot of room for compromise based on what I think would be a usable game system.
Compromises are often things that both parties can agree to hate equally.
I'm not inclined to optimism, myself, but this is an interesting challenge:
- It needs to have a clear sense of what characters can do, what they cannot do, and what they can be expected to to be able to do all the way from level one to whatever level the game ends at. Those abilities should be roughly balanced between classes. Not exactly balanced but the same ballpark is necessary. Characters cannot be playing wildly different games at any point.
-Characters defined in open-ended fashion so I can make any character I want, and have it be reasonably functional and simple.
'Reasonably functional and simple' dovetails well with 'characters cannot be playing wildly different games at any point.' If we're willing to accept that functionality is important, balance can be a part of that, and simplicity aids balance, as well - There we have basis for class designs and more detailed 'build' rules that could fill both your requirements. Taken to the extreme, an 'open ended' character-creation system /couldn't/ be class based, classes just aren't that open-ended - and I'm sure 5e couldn't be taken to that extreme (nor would either of you want that). So the open-endedness would have to come from non-class options (races, kits, themes, PrCs, backgrounds, or whatever) and from multi-classing of some sort. Allowing swapping in/out/around abilities from multiple sources like that would require that any ability that can be swapped out for another be /balanced/ with that other ability, which, if done well, would also deliver on that 'clear sense of what characters can do at all levels.'
- Monsters must be easy to create and the math behind them must be standardized. Also there should be a clear idea of what monsters should be able to do at certain difficulties. When designing my own monsters I must be able to say "The party is level X therefore it's appropriate that I give the monster the ability to do Y to them." I should not especially have to consider the party's composition or their specific abilities when designing or appropriating monsters.
-Monsters created with the same level of detail as all characters, that routinely have mechanicallty interesting, game-breaking, and expectation-defying abilities that exist outside of any concept of making a balanced encounter fot a party.
"Routinely expectation-defying" seems a little contradictory to me.
There is some room for compromise, here. For instance, you don't object to monster design being independent of party composition. Easy-to-create and 'detailed' aren't mutually exclusive, as long as the level of detail isn't fixed by mechanical necessity, but left to the DM. A formulaic 'base' for monster design for a given role/level wouldn't get in the way of an exception-based approach to monster abilities that'd make it easy for DMs to throw in 'surprising' monsters, even to the point of all their monsters being 'surprising,' every time.
- The game must be a fun and interesting game without the addition of any roleplaying whatsoever. I know it sounds like an odd requirement but the engine must be able to run before we start talking about building a car around it. Roleplaying should make the game even more fun. We shouldn't be having fun roleplaying in spite of the game.
-A customizable exoskeleton of a system that is built around the roleplaying of the individual group. Characters that reflect the players' intentions. System rules that reflect the DM's style. Options for everyone.
I actually see a lot of potential for reconciling these two views. A functional (fun/interesting/balanced) system makes a fine base-line for customization, as well as being playable in it's own right. Whether you're adding mechanics or adding RP, it doesn't hurt to be adding to something that already works.
