An alternative design would be to pick up the speaker's audio, then play it inverted at your own location, cancelling out his voice so that he can talk as much as he wants without you hearing him, whilst you can still hear everyone else.
Basically, the MMO /ignore command for the real world.