Are you fixed on using the d20 system? If so, I would suggest:
Broadsides!! (Living Imagination) - very, very well researched and detailed book. Bit too detailed sometimes and the mechanics can drag.
Corsair - I haven't used these rules, but, I've heard very good things about them.
Seas of Blood (Mongoose) - Decent rules.
I had 2 issues with the d20 rules (at least as far as D&D was concerned) when trying this. First, ship combat usually features a larger number of combatants than D&D works well with but not enough to break out mass combat rules. Two sailing ships means probably about 50-100 combatants (give or take) and that's a difficult number to work with.
I'd honestly probably take a look at some of the Savage Worlds stuff for this if I did it again. Their small scale mass combat rules should do the trick.
The second issue, again, as far as D&D is concerned, is the magic system. It totally dominates naval campaigns. Whether it's a Lyre of Building to make your ship impervious to damage, or wands of Extended Fireball to let you nail enemy ships at long catapult ranges, the magic system can really be a pain here.
I'm not familiar with Nexus d20, so, this might not be an issue.