I actually have a better idea now that I realised that there is an ability in Pathfinder 2E that I can let my design be inspired by.
"Every potential target within the affected area that can see you and which is ready for combat rolls a will save. If they fail this save they become immune to it for the next 24 hours and get to chose:
- They move next to you and you get a free attack against them
- They become frightened for <some duration>."
While this will obviously not be enough for a lot of people, this takes away the compulsion complaint entirely because now it is up to the GM to decide the reaction and the GM is not forced to make a
particular choice.
This transforms the ability into the result of a failed save becoming a forced choice between several non-attractive options.
Mechanically this is no different from having a functional 3.5 style grapple and grappling an NPC in battle. The character would be forced to chose between breaking free or trying to fight while grappled, which is identical to a non-magical ability forcing a choice between two bad options. Because that's what it is.