The 3e/3.5e
Invisibility rules are just plain stupid... Sorry, but that's really the way that I've felt since 3e, and 3.5e made it worse, instead of better... In 1e (and, I believe, 2e)
Invisibility gave a -4 to hit. Now, it makes you unattackable (by normal means) until you Spot (succeeding by 30 or more, IIRC), or Listen (succeeding by 20+). It's just too good for a low-level spell...
If you succeed in a Spot, you know that something's out there, but not where... Now THAT needs some explaining! If you succeed in the Spot roll (with the penalty for
Invisibility), then you see a footprint appear, see something splash through a puddle, etc., and should have the location! Also, if you know the general direction, then you should be able to attack with missile fire, hitting anything along the missile's path (with high enough "To-Hit" rolls, and the miss chance). But, I don't write the rules, so here's what you can do, as things stand:
1) Immobilize the Opponents - Pit traps,
Webs, snares,
Entangle (if outdoors),
Alarm spells over doorways, etc. Once immobilized, use the
Detect Magic, and spend the three rounds.
2) Choose Your Battleground - The
Alarm idea, above, a closed door which they must open to get to you (and you will see opening), etc., all will help. A board across a pit or chasm (which bends down when they step on it) would also work. Fighting them in water will show them up ("See those leg-shaped holes in the water? THAT'S THEM!"). Dust where footprints appear, ash, loose, dry sand, mud, etc. If in a dungeon, spread something across the floor, or set up an ambush in an area filled with rubble, which will shift when stepped upon. If you know that they're about, use fire/acid to deny them certain areas, thus forcing them to come at you through your chosen "Kill Zone".
3) Pepper'em! - Hit'em with a bag of flour. They'll still be invisible, but the flour won't! It may also dazzle them, and/or cause them to sneeze. It will also coat the floor, revealing tracks. Pepper works even better, if you have some. Baking soda, or almost any light powder will work. Once you find'em, pepper'em with missile fire and area attacks. Grenade-like missiles, which splash, are, indeed, your best bet, here. Anything within 10' gets splashed, and a "splash" in midair is ONE OF THEM!
4) The Sacrificial Lamb Method -
Invisibility lasts until they attack. It is good for scouting, spying, backstabbing, and first-strike, but not so good for combat. Even if they can use it multiple times, it will take an action to do so, and you or your allies can attack while they re-up it. You may be able to draw them out by offering them a lone victim, whom they will then attack, and appear... Various forms of illusion work best...
Improved Invisibility is a bit harder to fight against.
Additionally, once they attack, hit or miss, they are in combat, and combat is a DC:0 Listen check. Beat that by 20 (IIRC), and you know where they are. Anyone hit knows which square they are located in, for that round, unless they can move-hit-move. You can also "sweep" any two five foot squares/round to "check" for invisible opponents therein.
5) Damage is a Good Thing! - Once you hit one, they bleed. Blood drips, and gives away their position. It takes at least an action to bandage up a wound. Also, if your archer can hit them, the arrows remain visible, and that locates them. After that, the
Invisibility really doesn't matter, much (unless they can remove the arrows).
Of course,
Detect Invisible works wonders!
6) Deny Them the Ground - Similar to 2, above, but can be used anywhere. Spread caltrops, broken pottery, broken glass, marbles, clay balls, sling bullets, and/or whatever else you have handy that is dangerous, all about the floor in areas that you wish to protect. Crossing them may well injure the opponents, and WILL cause noticable disturbances in the stuff laid out, if you have enough to cover the entire area. The mables, balls, or bullets will require about a DC:15 Balance check to walk over (like crossing slick ice), or can be kicked out of the way (with appropriate noise, also revealing their locations).
Rotsa ruck, Raggy!