I've gone fishing. I caught 2 sharks. It was hard.
First topic: how do you know they're going to spend a lot of time on the boat? What makes you think they'll want to go fishing? Be prepared to let them NOT go fishing, but also be prepared to make fishing be enticing (so that they'll use your new fishing rules).
Second Topic:
Fishing has to do with having the right bait, in the right spot, at the right time. You can resolve that with some die rolls to simulate that some times the fish are biting, some times they're not. A bite does not mean you got a fish, only that you have a chance to reel him in.
So, let's say the PC must wait N units of time until the next "bite", where N is deterimined by rolling 1d20. and the units of time might be rounds, minutes or hours (for slow spots) (perhaps 1d3 1=rounds, 2=minutes, 3=hours).
Now, you the GM need to know what's on the hook, they player won't really know until they pull him up. And the reeling is the real game. So, make yourself a big table of fish, and each one should have a roll to choose the size (because size matters). For each fish, it could be really small (young), or really big, or more likely, somewhere in between.
The size of the fish is what really determines how hard it is. Bigger fish fight harder, and are more likely to break the line.
So now, you know there's a bite, and what's on the line (more importantly, how big it is).
Reeling in a fish in deep water takes awhile. My 2 sharks were the biggest on the boat (longest was about 3-4'). I didn't have a clock, but it was probably 15-20 minutes each.
I recommend making it 3 stages (dice rolls/skill checks). The first is to set the hook. Probably the easiest stage make it the easy DC + size of fish modifier. Next, reeling it up. This is harder, so use a higher DC + size of fish modifier. Last, is getting it the last bit of distance and on the boat. This is the hardest. I found that the fish action is stronger, because the line distance and pole distance is so short. Similar formulat, except a higher DC.
The size of fish modifier, is something you'll have to work out. Let's try 10, 13, 15 DC for the base DCs of the 3 stages (assuming that for small fish, a person can reasonably catch the fish).
1' : +0 DC
2' : +3 DC
3' : + 6 DC
4' : + 9 DC
5' : + 12 DC
I'm making this up, but the pattern should be obvious. Biig fish = harder to catch.
This should turn each bite into a mini-game, and be sort of "realistic"
When I went fishing in the gulf, it was a 80 person fishing boat. Fish I remember being caught were: dolfin, red snapper, dogfish (thrown back), shark
dolfin is NOT dolphin. The former is a fish, the latter is a mammal.
To entice your players, some ideas:
a reward for catching a particular size of fish
a tall tale about a particularly hard to catch legendary fish (like the Simpson's episode)
fishmongers offering to buy fish from folks at docks (basically, we're travelling on a boat, might as well do some fishing and earn some money).
a rare fish breed that has some medical/magical property that's valuable to the PCs or some NPC.
Be sure to test out the fishing rule BEFORE making your players do it. You don't want to find that I invented an impossible to catch design, or just as bad, something so easy they catch every fish in the sea.
One last anecdote, when I went Catfish fishing in Nebraska a good many years back, we put in plain sticks with heavy line into the banks, with a young blue-gill on a hook as live bait. We checked the lines twice a day.
Each day, we went to a pond, and caught these baby blue gills (3" long maybe?) up to the limit (40 per person) for bait. These things were so easy to catch, we just dangled the hook in at the water service by the bank of the shallow pond and WATCHED the blue gill bite, and we'd pull it up and unhook and plop him in the bucket. So in the RIGHT conditions, it can be VERY quick to catch a lot of fish.
