The surprise rules just don't seem robust enough to work when things start getting a bit complicated.
Well, they don't differentiate between various situations. Perhaps we'll come up with houserules which then become popular among DMs. What's the most useful TRPG rules, governing surprise, that you've seen or played?
There is a military saying: "Surprise is an event which happens in the mind of a commander."
A distinction I would find useful:
(A) Vrax is picking berries in a forest. Vrax hears the twang of a bow. Vrax has a split second to take a Reaction, such as drop prone or cast Shield. (Maybe there's an arrow moving towards Vrax; maybe an archer is shooting at something else; all Vrax knows, in that moment, is that SOMETHING went "twang".) Vrax doesn't get an action such as Dash or Dodge, nor does she have time to ready a weapon or raise her shield. If she was indeed the target of an arrow, then the archer makes Attack Roll, with Advantage, against Vrax's AC, which includes Vrax's DEX modifier.
After that is resolved, roll Initiative.
(B) Vrax is picking berries in a forest. Vrax *doesn't* hear the twang of a bow. Maybe the bow was fired from 300' away, and Vrax didn't hear it; or maybe the attacker silently threw a javelin; or maybe the attacker creeps up from behind, with a silver hammer; in any case, Vrax is still happily picking berries at the moment the attack arrives. In that situation, the attacker rolls with Advantage, and Vrax does NOT get a Reaction. In my personal houserules, Vrax's AC is calculated as if her DEX were 0. Vrax's first clue that she's in combat, is either "ow, something hit me!" or "hey, something went right past me, was that an arrow, or maybe a javelin?".
(C) Vrax is moving carefully through the forest, shield raised in one hand, her trusty +3 rock in the other. Bandits have spotted her, and are waiting for her to get closer; she has not spotted the bandits. When she's 20' away, two of them jump out from behind trees, charge her, and attack with swords, while the third bandit casts Magic Missile.
In this situation, the bandits should get SOME benefit from surprise, because they spotted her and chose the moment to attack, rather than vice versa. But this is a MUCH smaller case of surprise than A or B. Maybe the bandits automatically have Initiative for this round, and they engage her and attack, but then she gets her normal action, possible bonus action, interact with object, etc.; and she can use Reaction to cast Counterspell against the magic missile, or Cutting Words against the bandit's sword attack. After all that, then they roll Initiative and start a new round.