For example a new class might causes compatibility with the Player stuff and Setting stuff from before.
But a making a monster a higher level threat might make incompatible with everything but the DM books.
There's a difference between compatibility and having balance issues. Being compatible, to me, means that they speak the same language (though possibly with different dialects). Balance is a different issue. For example, I can use a beholder from the 3.0 Monster Manual in Pathfinder (1e) and it will work pretty much seamlessly. It will still have the same AC, attacks, eye rays, and so on. I'll need to add Combat Maneuver Bonus and Combat Maneuver Defense, but that's like ten seconds of work. It would not be the same as a beholder built from the ground up for PF1, because PF1 makes some changes to how monsters are built (e.g. they get more feats), but it'll work.
If I instead try to use a beholder from 2e in Pathfinder, it will NOT work. The 2e beholder has AC 0 – worse than a gelatinous cube? And there's no attack bonus for anything, just this weird THAC0 thing. And it doesn't have any Fortitude, Reflex, or Will saves listed either, nor any skills. 2e stats make no sense in Pathfinder, so they are not compatible.
Adding a class generally doesn't impact compatibility, as evidenced by how 3.5 went from 11 classes to maybe 50 in the same edition. I would have no problem running a party with a psychic warrior, a scout, a dragon shaman, and a beguiler through the Sunless Citadel. I don't know if they'd do well, but the game would
work.
YES!!!!!! Otherwise, the game will stagnate and devolve. The game needs to evolve, 5(20).x1.1(24) is not the way to go. Mild updates, just rearranging the layout etc is not what the doctor ordered IMO. I'm sure I'll get lambasted for this, but D&D needs to move away from the d20 system and classes need to go.
Classless D&D is likely never going to happen. Last time they tried even a modest change to the class system we had 4e, and we saw how that worked out. If you want a classless fantasy game, there are plenty to satiate your desire.