Is the 5E system unbalanced?
Short Answer: Yes, it's D&D. ;P
So, I was thinking about what I could do wrong, because my players also mentioned that fights aren't fun anymore.
It's really not that odd for straightforward fights to get less interesting at higher level. They've been playing their characters for a while, and 11th is kinda a tipping point. A lot of folks have reported that the AP in question seems to get easier at high level, too. So it may not be anything to worry about.
1.) Magic Items:
I might have handed some items to my players that are too strong. E.g. Dwarven Plate (+2), Shield +1, Armor +1, Wings of Flying ... I heard that +armor items would be game breaking, because the 5E system is not able to compensate them. Is that true?
BA 'breaks' easily enough, sure, it's not that each + AC item is problematic, it's that the same character got two of 'em, and one of 'em was plate.
2.) Spells:
To be honest, I have never spend much time to spells. It's not that I am never casting, but most of the damage I deal with my monsters is through melee/ranged damage. I always felt a bit overtaxed by the richness of spells.
If your party is using spells to devastating effect, countering them now and then with an enemy caster could help, sure.
And, yes, you do have to be quite familiar with the rules (or willing to run roughshod over them - or both) to run 5e.
I always use "hard encounters", but it still seems not enough. Might that also be a problem?
Encounter guidelines are notoriously less than dependable. A 'hard' fight will only begin to be 'hard' if the party faces at least 5 other 'hard' fights that day.
I would be very interested in tipps and answers.
Since your 'mobs' are being blown up, beef up encounters with badder monsters instead of more of the same.
Also, since it's something of a sandbox, are your players resting more often than expected? (Do you squeeze in the prescribed 6-8 medium-hard (if not deadly) encounters between long rests? Is the SorLock maybe taking multiple short rests?) If so, they could be rolling over encounters with an excess of resources. If the barbarian can rage every encounter, for instance, that's a red flag.
There's a whole debate on how to fix frequent-resting problems.
In other words WotC shouldn't get away with not taking responsibility for making a game that cannot handle it's constituent parts.
They do not say "don't use feats and magic items unless you're okay with all our adventures becoming way too easy" on the label. In fact, they say next to nothing, hoping that apologists like you will deflect and shift the blame away.
They do say that feats and MCing are optional. That should be a clue. If you weren't happy with what they did to 3.x, maybe you shouldn't use 'em. If you're completely new, maybe you shouldn't turn on all the options out the gate. Seems like a modest level of caution in the presentation.
Many DMs fall into this "trap". The blame for this should squarely rest in the lap of the designers.
I admit it's facile to present a product as a sort of DiY 'kit' and take no responsibility for how it's used - if it were a potentially dangerous activity it could even attract lawsuits - but D&D /is/ mostly targeted at long-time & returning fans who know the ins & outs of running an RPG, and even if they're not ready for the issues 5e has as presented (or with options turned on or whatever), they'll figure it out pretty quickly.
The community tends to pride itself on being relatively intelligent, a sort of gaming 'elite' who can handle a hobby that's simply not for everyone.