A lot of the good stuff has already been covered, but here are some I haven't seen yet.
This is only against fireball specifically, but shooting flak into its path should have a chance to make it detonate early and safely away from your ship. Have several crews manning small ballistae/catapults loaded with baskets of junk. They ready an action to fire when they see the "glowing, pea-sized bead" of an incoming fireball and shoot into it's path. There's a chance they'll miss it's incoming path, or the flak could spread out too far to actually interfere with the fireball- but you might just get something in it's way to make it explode early. (It's similar to punching off chaff or flares to attract missiles.)
Take the flak idea a bit further and it's not too hard to imagine alchemically treated flak that hangs in the air for a while and is attracted to the pre-fireball bead. That way you could put up an anti-fireball screen, forcing your attackers to get in closer where your mundane missile weapons can hit them.
Of course, this gets into the "magical arms race" somebody mentioned above, which may not be the kind of thing you want in your game. But I tend to think that if magic exists in a world, then non-spellcasters would HAVE to have non-spell defenses to rely on. Without some level of "magical arms race" the mundane folk don't stand a chance.
If you don't mind the arms race however, here are a few more ideas...
--- Don't create the fog around your ship- create it around theirs! The Obscuring Mist spell has already been mentioned several times (and for good reason) but the problem with that spell is it's puny range- just 20 ft. Sure it blocks line of sight, but anyone with half a rank in spellcraft could figure out to aim 20 feet past the center of the cloud you just made. Smoke sticks tied to arrows on the other hand could be shot into the enemy ship to give this same problem to them. Make the smoke sticks extra long so they last a few rounds. Put enough in their hull and it blocks their line-of-sight completely but you just have to adjust your aim a bit.
-- Speaking of sticking stuff on the enemy ship. Make extra large tanglefoot bags and fill them with caltrops too. Lob them onto the enemy ship and they've got a painful, sticky hazard that's really hard to clean up! Sure they have lots of rum to deal with the goo, but do they
really want to make their ship MORE flammable in a battle?
-- And speaking of lobbing things onto the enemy- nothing says 'I love you' to a smoke-blinded enemy captain like splashing a sack full of alchemist's fire flasks onto his deck.
-- And why stop with lobbing an alchemist's shop of goodies at your enemies? Why not explore the caves that must surely be around your villainous pirate lair and start gathering all those really fun slimes and molds! Load your catapults with stone casks of green slime. Put on a parka or two and gather some brown mold to fling on your attackers.
Ok, enough alchemy madness and chemical warfare. Here are a few tricks you can use with low level spells (1-2 level). The key here, I think, is to focus on med and long range spells. There are darn few of them at these levels but a few interesting ones stand out.
1st level spells
--
Hold Portal It's medium range, so if your enemy is a hundred feet or so away you can "lock" the main door to go below decks. Or maybe there's a single door to the ammunition storage. Sure it's just 5 more to the break DC but that delay could be significant.
--
Sleep, also medium range. The uses of this on the probably low hit die crew are obvious. But what about that guy in the crow's nest that's above the effects of your smoke stick arrows and calling out ranges to his buddies? Knock him out and their blind again. And maybe he'll fall to the main deck as a bonus- I doubt they'll get another volunteer to climb up there.
--
Magic Missile also medium range and also obvious in it's application.
-- Here's a fun one,
Animate Rope. No lack of rope on an attacking ship. This could cause all kinds of problems.
2nd level
Here's where the real fun starts. I'll skip the easy ones (acid arrow, fog cloud, flaming sphere, web) and go to the less obvious spells.
--
Glitterdust, always a good spell, but its medium range means you can shut down an enemy ballista crew or two with blindness.
--
Daze Monster, make someone up to 6HD lose their next action. Could be very affective against say, a helmsman, if cast at the right time.
--
Hypnotic Pattern, take out the helmsman and everyone around him.
--
Scare, or just make them all jump overboard!
--
Pyrotechnics, a nice 'finale' for this post and kind of the big gun for this level and situation. Long range and a HUGE area effect. Blind everyone within 120 feet of any lantern on their ship! While everyone is blind- pepper the deck with smoke arrows, gooey caltrops and brown mold. FIGHT OVER!!
