I've been pointing out the negative quirks of adv/dis since the beginning of the playtest, but most people have been simply too happy for its simplicity, that they refuse to see them.
Yeah, you're not the only one. Perhaps it's because the "honeymoon" isn't over yet, or because people don't run into the issue very often, or they're just too pleased with the overall system, I dunno. But yeah--people (including yours truly) were pointing this out back in the playtest, and not terribly many people even reacted. A lot of those that did gave the Standard 5e Response: "You're the DM, you fix it!" (even when the listener...isn't actually the DM.)
IMO the non-stacking isn't even the biggest deal with advantage, but rather the fact that (dis)advantage is also a condition that triggers a lot of special abilities, unlike the 3e vanilla +2/-2 modifier that you could always apply to represent various circumstances. Now if you over-use (dis)advantage freely, you have to be prepared that the PCs (and the monsters) will too often be allowed or otherwise prevented to use their special abilities and features.
I think they're both significant concerns. Advantage/disadvantage, probably more than anything except
maybe Proficiency, is the "core mechanic" that the whole engine is built around...and it's quixotic to the core. Most of the time, both of them are supposed to be
relatively rare/unusual things, and it's clear that certain features are balanced around that being the case. But then you have other features that grant it all the time--class features, items, a few spells--on top of it being the default "bennie" handed out for good roleplay,
and it being the intended replacement for the old "DM's best friend" aka the +/-2 situational modifier, and clearly some things (e.g. the Barbarian with Reckless Attack) are built expecting you to benefit from it on a regular basis.
So much--too much, I'd argue--rides on a single, non-stacking (and non-extendable), eliminable mechanic.