Sure, it's not ideal. But I'd point your wrath elsewhere, namely at the barbarian's agility class feature. I think Hide armor expertise is certainly a fine feat for most builds without that class feature.
People speak of barbarians the whole time because they have this weird combo; but shamans and warlock and even invokers might benefit from this feat too. For them it's either almost a requirement to be able to play a particular supposedly standard build in the first place, or merely a optional extra (and for non-primals, there's a multiclass cost too).
If Hide armor expertise didn't stack with barbarian agility there wouldn't be much of a problem. In fact, even without hide armor expertise the whirling barbarian build (dex secondary) will exposes the AC problems that barbarian agility can cause (and theoretically, the Ref problems too - except that FRW defenses are so systematically unbalanced, you hardly notice).