Making ship on ship battles exciting

Hello guys. I am the DM of a group of 4 and recently I led the campaign into a world with a lot of islands with the intent of making things like ship to ship battles, giant sea creature attacks on the ship and other similar things. Problem is to my (albeit limited) knowledge there isn't a an easy way to do that with the game's mechanics. I really wanted some pirates of the Caribbean style naval battles. But it hardly is very entertaining when all they can do is fire the cannons with some hit die and tell me where in the enemy ship they are aiming. It's hardly interesting gameplay but I'm sure it can be a lot better and I'm just doing it wrong.

Has someone done this successfully? Either from reading something I haven't or just from creating his own mechanics for naval battles? I would love some advice.

I would probably find a way to steal the Ashen Stars approach to starship combat, or use a system that had ship combat well-designed for it.

For D&D... wait for the upcoming nautical book?

I recently had a sequence in my game where the party was on a sand sailing ship and on their way to the City of the Dead their ship was attacked by various undead creatures culminating in them being chased by a giant bone serpent. Because the goal was to break through the blockade and not truly fight it I ran this as a 4e style skill challenge. It was my first time running a skill challenge of any kind so I wasn't sure how it was going to work out but the players really enjoyed it in the end. I don't think it would be too difficult to abstract any kind of ship encounter into a skill challenge. Instead of going back and forth making attack rolls you can direct the scene to be more like an action movie sequence.

If you want to go this route I'd really recommend Matt Colville's video on the subject. It's available on Youtube. He does a good job explaining what a skill challenge is, what they're good for and most importantly he gives some good pieces of advice on how to keep them from going stale. I instituted the additional rule that a single character cannot repeat the same kind of roll once they have attempted it, which keeps the character with the highest relevant modifier from soloing the entire thing. Another important thing I did that Matt hints at but doesn't explicitly spell out is to change the situation as it goes on. As my players were succeeding at things I described new dangers to their ship that had to be dealt with. Forcing the players to be reactive to these complications really brought the players into the situation and fostered their creativity instead of just devolving into "roll dice until we either succeed or fail."

Examples from my game:
The wind blows the ship onto a collision course with a jagged rock, so one character made a vehicle tool roll to make a hard turn to avoid it.
An ally ship making the run with them takes a heavy blow and is set to crash into the player's ship. A character made a persuasion check to give the ally captain just the moral boost he needed to correct course.
The bone serpent emerges from the dunes and sends boulders into the air heading right for the PCs. A character made an attack roll to destroy them in the air.

We've had some success trying out Captains and Cannons from the DMs Guild. It has some nice mechanics that incorporate environmental effects and combine character and ship actions during the course of combat to make it more than just "fire at will". Additional action types allow non-ranged PCs to contribute to encounters in a meaningful way. There are also a few interesting mini-adventures included with the rules that help you learn the ropes, so to speak.

I'm also eagerly awaiting the upcoming "official" nautical book...

One idea I’ve been playing around with is the idea of making an adventure toyetic – that is to say, it encourages my players to well, play with it. I try to look at my adventure setpieces like playsets when I was a kid. I never had it, but look at Castle Grayskull – it’s got that lockable drawbridge that only Skeletor and He-Man’s swords combined can open (pro-tip, so can Mer-man’s), a laser canon, a throne, computer station, weapons rack. To get back to gaming, it’s got stuff that encourages PCs to do stuff.

Moving back to those ship battles, the PCs go to the cannons because that’s where the play value is. My approach would be to make the ships more toyetic. Maybe there’s a spyglass that someone can use to try to find weakpoints on the enemy ships, or they stray into a patch of ocean with tons of reefs and someone needs to take the helm. Or one of the enemy cannons knocks a hole in the side that needs to be fixed, or a fire starts somewhere. Or getting into the fantastical, maybe the ship isn’t powered by wind, but by magic – and maybe a wizard or sorcerer might be able to give the engine more power.

Another approach is to create complications – throwing in a storm in the middle of a naval battle, or a giant sea monster shows up to cause trouble.

You could try making the actual ship battle a narrative background element while the PCs themselves do something important in the battle. Maybe they have to board the enemy ship and get something while the cannonballs rain down and enemy sailors battle them in the lower decks.

Sounds simple enough. After all there's two ways to disable a ship:
Shoot the out of it, or disable the crew.

Make the ships the map, but don't forget to give them AC and HP like other objects. That way players can choose to board the opposition's ship, shoot the opposition's crew, or just shoot the other ship, be it with cannon or spell.

Here's a thing I was working on, although I've put this project on hold until the upcoming nautical book comes out. Pirates & Privateers

The main gist of it is:
1) Each ship stat block shows the two basic actions you can take in ship combat -- piloting the ship, and commanding the cannon crews to fire.
2) The "Naval Combat" section has a variety of extra (optional) actions that you might take in combat to increase the ship's capabilities in various ways, or resolve common combat actions.
3) Cannon layout and ship movement capabilities encourage fights to happen at relatively close quarters, which allows PCs with ranged attacks to participate directly.
4) Boarding actions!
5) Usually, the goal is not to sink the enemy ship, but to capture or evade it.

You could use the weather to indicate if the water is calm or not, causing the sea to be normal or difficult terrain. A failed piloting check could mean the guns cannot be fired as the shot would hit the waves.