If it must be done (flanking) then this is how I would do it:
1) Flanking provides a +1 bonus to hit, as long as your flanking ally is not incapacitated, and you can see your enemy. +1 to hit is a
huge deal in 5E, and is a larger relative bonus that 3.X's +2 to hit.
As a bonus this fixed +1 doesn't devalue spells, class features and other abilities that grant advantage to hit (Barbarian Reckless attack, Wolf Barbarians, Pack tactics on monsters, Shove action, certain spells, certain Battlemaster manoeuvres, Samurai, Aim manoeuvre, Hiding, Vengeance Paladins etc) and it doesn't overvalue feats like Elven Accuracy.
2) Movement, flanking and OA rules need to be brought into line with 3.5. Namely:
- All movement out of a threatened square provokes AoO before that movement (unless you first withdraw, or take a 5' step [see below], or have some other ability like the Mobile feat or Swashbucklers Fancy Footwork, Relentless Avenger from the Vengeance paladin, and some monster abilities which allow you to move without provoking AoO's).
- PAM feat no longer provides a free attack when a creature enters your threatened area, instead it now provides advantage on attacks of opportunity against creatures that are not adjacent to you who move out of a square you threaten, to a square that is adjacent to you.
- The second benefit of the Sentinel feat changes to: Creatures within your reach provoke opportunity attacks when they leave a threatened square, even if they took the Disengage action or took a 5' step.
- Insert a new action entitled '5' Step' which can be taken as a Bonus action by all creatures.
- Have spellcasting with a casting time of one Action trigger OA's. Bonus action and reaction casting (Shield, counterspell, quickened spells etc) are not affected and those spells do not trigger OA's.
- Mage Slayer's 1st bullet point now states: When a creature you can see starts casting a spell, you can use your reaction to move up to 5' towards that creature as long as you threaten that creature at the end of that movement. You may then make a single melee weapon attack against that creature as part of the same reaction.
- Warcaster no longer allows spellcasting in place of an AoO. Instead it now lets you cast spells while threatened without AoO (in addition to its other benefit of advantage on Con saves to maintain concentration).
- d) Sneak attack now only triggers when you have advantage, or you flank your target and don't also have disadvantage.
Rogues remain the kings of manoeuvrability thanks to bonus action withdrawing letting them position themselves with ease. However now that they need to flank to get sneak attack (instead of just having a friendly ally next to them) they will have some tougher decisions to make about battlefield positioning. Ranged Rogues get weaker (but this is countered by the new Aim action in Tashas) and just like before, they can always Hide for advantage (and sneak attack) as needed.
Spell casters now have as much to fear as do Martials from being threatened and flanked by multiple creatures (3 creatures properly positioned around a caster ensures an AoO, and being stuck near a creature creature with reach, all but ensures one), but also have an easy way to avoid it (bonus action 5' step and cast) unless overwhelmed.
A +1 to hit wont overwhelm PCs when flanked by monsters like advantage would. It's still a bonus worth chasing, but it's not that out of proportion that a flanked PC is doomed like they would be if monsters got easy advantage.
If I were to implement it, the above is how I would do it.