Drat. You beat me to the flaw in the "It is balanced because the NPCs can do it too," argument.
With regard so nerfing something, I do the same thing in my campaign. I will usually grant the PCs some compensation for the capability I took away. Usually I will give my PCs two weeks notice of...