Ok, then I would max it out at 1d8 personally, but 1d10 if you want to mirror Monk, at 5th, 11th, 17th, making it d4 - d6 - d8 - d10.
For me it would be about keeping it balanced with other Fighter builds:
With the Thrown Weapon Fighting Style, you get +2 to damage, just like Dueling. With 4...