IME you have to pump Str (for heavy weapons) to make the -5 with GWM work effectively. That's less Dex or Con - which are vital for AC and Concentrating on those spells that allow you get Advantage on your attacks. Like a rogue, Dex based is the way to go, I think.
PAM is great for getting...