I agree that if you're playing giants you should still be bigger than everyone else.
But for some reason I think this would combine well with a prehistoric setting like Planegea. Giants + lower tech feels thematically right.
Edit to add: also, I think you'd need to do a lot of homebrewing (or...