Granted, the last pirate game I ran was back in 3.5E, using the Stormwrack rules for ships, and some Heroes of Battle ideas.
a) what do other players do while the captain farts around with sailing the ship in combat?
Sailors will probably be divided into groups trimming and adjusting sails to help the ship maneuver. At its simplest, this could be an Aid Another check to add to the captain's piloting check.
Marine snipers make be seated among the masts taking potshots at the enemy ship with rifles. When the time comes for boarding action, these individuals may stay at their posts or join the fighting at the deck level.
Repair crews will be attempting to minimize damage to the ship. They may be putting out fires caused by enemy shots, replacing or shoring timbers splintered by cannonballs, tending to wounded crew or otherwise trying to keep the ship in one piece. You will likely need an officer in charge of the crews to prioritize the work and repair.
Cannon crews, will of course, be manning the cannons. You'll also need additional crew to run powder and shot to the guns and relay commands up and down the line. You will also need a weapon officer to time the cannon volleys (volleys tend to be more devastating than "fire at will").
Marines will be on the wait for boarding actions with pistol, boarding axe, grapples and sabers. They may have other duties with the sailors or repair crew until it becomes apparent that a boarding action will take place.
b) on ships that have like 40 canons on them how do you measure/run combat?
For one thing, remember that cannons tend to be broken between both sides of the ship, so a 40-cannon ship likely only has 20 cannons to a side. Also, shots tend to be fired in "broadside", where every cannon unleashes at once, rather than in a staggered fashion.
Likewise, you might want to consider breaking shots up into volleys of 5 cannons per volley. Make one attack roll at +3 to hit that deals 2x the normal cannon damage (this assumes 2 cannon attacks being made, with 3 cannons performing an "aid another" action, split between the two cannons).
c) Ok so a galley is 104 squares big...holy crap I need a big grid to do this on :/
The vehicles section was lacking on ship to ship combat.
Yep, investing in a battlemat is a good idea. Also, since you're probably dealing with a lot of folks at once, you might want to make 1 square = 10' feet (instead of 5). This allows you to "group" attackers into squads of 4 or 5 (I prefer 5; quarters tend to be a bit cramped and it's a bit easier math - you might want to represent them with a die, with the number on top indicating how many figures are in the squad).
Like the cannon volleys, you might want to group attacks, one attack roll per squad. Each time a squad hits, don't deal damage, just assume the opposing squad loses a man* (as if all the attacks were concentrated against one individual) [if you get a critical, remove a number of individuals from the squad equal to the critical multiplier]. To represent the lost effectiveness of a squad as it loses members, assign a -1 penalty to the squad's AC per member it has lost.
Of course, in the case of PCs, and enemies directly attacking/affecting the PCs, you'll probably want to hash this out with the normal rules, though consider having NPC's use the Aid Another action in combat to make their attacks more effective against lone, higher-level PCs.
*
If you have time to do some prep beforehand, you might want to calculate the average damage the entire squad does and figure out how many of those hits it would take to kill one enemy figure. For example, a squad of five Fighter 1 w/ longsword (1d8+2) would deal an average 32 points of damage if all attacks hit. If they were fighting a squad of trolls (63 hp average), they'd kill one every 2 hits. If the squad could kill an opponent with less than everyone in the group hitting (say fighting Rogue 1's with 5 hp each), you might want to make it that they kill an additional opponent if the beat the target's AC by so much (if they could kill 2 foes if everything hit, then beating the opponent's AC by 5 should deal an extra kill; if they could kill 3 foes if everything hit, beating the opponent's AC by 3 should deal an extra kill; if they could kill 4 foes if everything hit, beating the opponent's AC by 2 should deal an extra kill, if they could kill a foe with every hit, beating the opponent's AC by 1 should deal an extra kill).
