I personally think you could get rid of it, but there would be a few implications to consider.
1. If you replace saves with something like a static defense that the source has to roll against (basically another attack type), you'll want to understand how to resolve it when multiple targets are involved. For example, with core rules a dragon breaths fire and "hits" everyone. The DM asks for Dex saves to see if anyone takes half damage, resulting in the DM only rolling damage, and each player rolling once for their own chance at partially avoiding it. It's easy to setup and easy to resolve.
Now if you change that to something like a static defense, you need to decide if the source (dragon?) should roll once and apply it to everyone, or if it needs to roll against each target separately. If you roll once and apply it to everyone, you have a very swingy mechanic where a low or high roll will often result in an all-or-nothing effect, possibly doing an unexpected amount of damage. If you roll against each target separately, you have potentially slowed combat down quite a bit just to more fairly determine the outcome of the effect.
2. Class and game mechanics that interact with saving throws would need to be adjusted to make sure they don't lose value. Things like "evasion" abilities would be easiest enough to just have a hit act like a miss (half damage), and a miss do no damage, but other things like the luck feat, halfling luck, buffs and abilities that grant bonuses to saving throws, and inspiration that a player can normally use would all need to be addressed. Suddenly, it's no longer quite as simple as just removing saves from the game.
3. It generally is more fun for players if they are the ones rolling dice, when possible. I know lots of people simply don't enjoy spells like Sacred Flame because fate is out of their hands. It's not satisfying to hear "he passes the save" and it's barely just a relief to hear "he fails the save." Even knowing that the mechanics are solid and having an option to damage something like a high-AC opponent by targeting a weak save is available, the spells are just not considered fun by many people. The proposal to remove saves would, ironically, result in a similar dissatisfaction because although it's not fun to force a save on a monster, it *is* fun to make a save that you might succeed in (or buff or interact with). Be careful about taking that sort of fun away.
XXXXX Final Thoughts XXXXX
I don't like the 5e save system, so I totally get the desire to question it. I just think that the flaw is with the implementation rather than the concept. They shouldn't have forced homogenized parity between abilities and saving throws, but instead should have used something closer to 3/3.5/4e saves. The broad categories of a physical defense, quick defense, and mental defense are perfect for what the system needs, and is very intuitive for figuring out what save should be used. I'd suggest starting with that kind of change rather than monkeying with save and attack mechanics entirely.