Hyp said:
A fireball on your own party is a spell that deals damage, and is thus an attack per p171, and will thus break invisibility.
I don't disagree with you in theory. As the RAW states: "The spell ends if the subject attacks any creature." And if they'd left it at that, it'd almost make sense. But the RAW continues: "For purposes of this spell, an attack includes any spell targeting a foe or whose area or effect includes a foe. (Exactly who is a foe depends on the invisible character’s perceptions.)"
According to a strict (if mindless) interpretation of the RAW, you can cast fireballs at your allies because an attack is a spell cast at a foe. You can also cast an unlimited number of fireballs and remain invisibile, as long as you don't cast them at anyone.
My point is that it should be the act of casting Fireball that causes invisibility to fail, not who (if anyone) ends up in the area of effect. Basically, take out those two sentences in the RAW and replace them with, "The spell ends if the subject
makes an attack."
Now, "All spells that opponents resist with saving throws, that deal damage, or that otherwise harm or hamper subjects are attacks" would go along way towards clearing up the confusion about Invisibility, except that even here, there are logical inconsistencies. Detect Thoughts is resisted by a saving throw (on the third round) and so constitutes "an attack". This was why I chose it as an example, because it was a Will Save Divination, as opposed to Detect Evil, which doesn't have a save. According to the RAW, Detect Thoughts is an attack, but Detect Evil isn't. But Detect Evil is a spell with an area, and so would fail if a foe was in the area of effect. (Here are two spells that do practically the same thing-- get information from a subject-- but have two different effects on the Invisibility spell.)
All of which leads to more confusion. An invisible caster could cast Detect Thoughts on his party (presumably), at least for the first two rounds and then its up to the GM if the party can waive their saves so that the spell won't fail. Likewise, a caster could Detect Evil on his own party and remain invisible. But if a foe walked up into the area of effect, the caster would be in violation of the Invisibility spell and become visible, even though the character wasn't doing anything different from round to round. Likewise, and invisible person could cast Detect Evil and walk all over a city. No saves would be triggered, he doesn't see any of his foes, and so he'd remain invisible.
Even with the Invisibility spell description and the further explanation of what constitutes an attack, there still seems like there's a lot of grey area here. And my personal waystick of a rule is not if it makes sense after you read the spell, then look up something else in the PHB, and then follow the rules to their logical (if absurd) conclusion. The rule should make sense in the first five seconds, because that's really all the GM has time for when running a game. (This goes back to Vonnegut's rule that anyone who can't explain what they do to a 7-year old is a fraud.)
Edit: It occurs to me we are drifting far afield of the OT, and a discussion about how to fix Invisibility would be more appropriately located in the House Rules section. If I get the energy (which is unlikely) I'll post a new topic in that forum dealing with this facet of the discussion. But I'll stop flogging this dead horse in here.