The issue I find with 'attacking' in invisibility is that it has some interesting edge cases. What about the rogue that goes invis, sneaks up, lights some explosives and leaves them in the enemy camp? What if he just threw the grenade without delay? What about putting people to sleep with a gas, what if that was mustard gas instead? Putting poison in a drink, as opposed to splashing a contact poison in their face? Lobbing a snake at somebody as opposed to throwing a stone? Is it the physicality of the action? If so, why would pulling a crossbow trigger end it.
<edit> That is not to say that I really find this stuff an issue as the DM, it is just interesting. In my games magic is always wild and unpredictable, so my players don't expect any great consistency about magical rulings. Sometimes I even openly just roll a dice or flip a coin. 90% of the time it just works if it is a subtle trick, fails if it is a way of doing a normal or repeatable 'attack' without consequence.