You thought right. For me, it was the magic system; years of being a tabletop RPG player means that I can't help but dissect any new magic system I come across in fiction in terms of game-ification, since "if PCs could use this, how would they abuse it?" is, to me, the most stringent level of verisimilitude there is. And unsurprisingly, the Harry Potter magic system doesn't clear that bar.
I mean, if a "secret keeper" can prevent all outside knowledge and magical access to whatever secret is being kept, then why doesn't someone just make a secret about, oh I don't know, horcruxes? Or killing curses? Or freaking Death Eaters? "Hey Lucius, what's that organization we all used to belong to back in the day?" "No idea, MacNair. Someone's keeping it a secret."
Or that "taboo" casually introduced in the last book, which functions that whenever anyone anywhere says a particular word, it alerts the caster to their identity and location, casually shattering all concealment magic? Just make a taboo for the word "the" and you'll know where everyone is all the time! Are you really telling me no one thought of doing this in a way that would weed out potential Death Eaters?
Those are just off the top of my head. The entire damn thing ran on narrative fiat, and in that regard it sucked.