I suggest musket 2 actions to reload, pistol 1 action to reload (but make the pistol a Light weapon), and increase musket damage to 2d8. A musket ball was a massive chunk of lead you definitely did not want to get hit by, but definitely not a practical skirmish weapon beyond the first shot! To be kind to high level warriors I'd let them use 1 attack for a reload action, so a Fighter-20 could potentially shoot a musket twice in 6 seconds!
Edit: And if you want heavy crossbows resembling historical ones they should be at least as slow as the musket, and do a lot more than 1d10 damage, likewise - I use 2d10 at short range (unlike a musket ball x-bow bolt effectiveness drops off fast with range). The 5e stats make them more like (slightly less) light crossbows.