However players that are cool with it on other people but not on themselves are missing the point I think. What is good for the goose is good for the gander.
There is not one single "the point" here.
The issues we are talking about are not generally born of cold logic and rational thought. Human psychology and neurology are not symmetric in this manner - we generally think about our selves and others differently (as in, literally using different areas of the brain). It therefore is not strange to see such asymmetry in our fears and reactions.
If you, as a GM decide that you want to impose symmetry, that's fine. But let us be clear - if you are the GM, removing charms and such from your arsenal doesn't really impede your ability to challenge the players, even if they still have them.