Yes other people need to try to hit him and NOT hit the grappler in the process. That is not easier IMO, in most cases it is actually more difficult.
The attack roll is an abstraction that your attack hits the person you are targeting and does not hit anyone or anything else.
Let's talk about...