Nefermandias
Hero
Then do what Christopher Perkins did with SKT: Bring the giants in earlier and watch your players naughty word themselves while one or two of them gets their skulls smashed because they have understimated the threat.I think one (possibly minor) issue might be that the entry level hill giants are really just beefed up ogres. Then the next level of giants are stone giants, which I really quite like, but being dour and neutral is basically their thing, so they are not very exciting.
By the time you can deal with frost giants in groups, you're already playing in the big league, where the overall number of PCs is getting more scarce.
I think basically, giants might just be too big and powerful. Just like dragons and beholders. Look cool on paper, but impractical as opponents in most games.
You know, it's good to have the PCs on the run once in a while. People coming from the later editions have this weird idea that everything on their path should be a balanced encounter supposed to be won by rolling initiative and hitting things with their weapons and fancy feats and spells.