I've played in several of them. My preference, by far, was
Star Wars:Saga Edition.
The ship is the combatant. The ship's weapons and functions are controlled by the characters. The ship has the hit points, movement speed, weapon attack values, etc.. The characters modify the ships systems with their own skills (or lack of same). This works very well and "feels" right. It also gives each PC something to do during the combat so it does not all devolve into just one PC doing something. They PCs are ALL making rolls each round.
From a RPG perspective --
SW:Saga Edition is the best "base model" to use to add modifications to, imo. As 4E used SW:SE as its own base, you will find a lot of the assumptions between the systems will work for you. They will work well for Pathfinder, too.
Now, going a little deeper into your line of thinking, if I may be so bold - you appear to be thinking along the lines of rules for naval warfare in the Age of Sail -- or at least --
Sail and Sandals (slaves at some oars, too).
If you are inclined to have ranged combat between vessels via spells, cannon or large weapons, inevitably, facing is going to come into your consideration in terms of how fast the ship can move, turn and so forth.
You will wonder whether hexes make more sense than squares and if that will make it too "wargamey".
Hexes are
not superior for simulating Sail and Sandals/Age of Sail. They are, in fact, inferior and more limiting than squares.
Steal a page from an innovative design which appears in an old SPI magazine game. It is called
Fighting Sail.
Fighting Sail used squares, not hexes, in a SPI wargame! This was nearly heretical at the time, until Joe Balkoski seized upon the idea that the square, if you use the diagonal points in the corners to represent a separate / facing direction, gives you eight points to work with, not four. Eight is MUCH better than the six directions that hexes provide as it then allows you to match the 8 primary points of the compass (NSEW + NE, SE, SW, NW) using a square grid. This little touch adds greatly to the "naval" feel of the game.
For minis, please remember that there are a BOAT LOAD of simple ship minis from the
Pirates of the Spanish Main constructible strategy game that can be picked up for a
ridiculously small amount of money. A few dollars of this on eBay and any gamer will have more ship minis then they can EVER use. Touch up the edges of the white plastic card by quickly running a brown or black Sharpie along each edge and they look
much better than the shots below. I've tried this "Sharpie trick" I borrowed from papercraft modeling and it is FRIKKIN AWESOME.
End Result: (1) Assume PotSM minis; (2) 8 points of the compass for movement directions on a square grid battlemat; (3) PCs to modify ship's abilities when fighting the ship-as-a-ship and you are GOLDEN.
When the matter progresses to the point where ships are alongside and its time for a boarding action -- you revert back to the default combat rules of the game (or you switch to a mass combat system if there are a LOT of combatants). Either way -- that should not prove any sort of a problem for you.