1.) Magic Items:
I might have handed some items to my players that are too strong. E.g. Dwarven Plate (+2), Shield +1, Armor +1, Wings of Flying ... I heard that +armor items would be game breaking, because the 5E system is not able to compensate them. Is that true?
Yes.
5e does not assume magic items. So they are a bonus that outright makes characters stronger. As you can see with the cleric.
Numerical bonuses to AC are the big one, because it means all monsters cannot hit and you need to rely on other methods of damage (i.e. spells).
2.) Spells:
To be honest, I have never spend much time to spells. It's not that I am never casting, but most of the damage I deal with my monsters is through melee/ranged damage. I always felt a bit overtaxed by the richness of spells. Another problem is that most of the casters I use are almost one hit by the rogue/barbarian or sorcerer.
This is another hurdle. The big equaliser to AC is relying on spell saves.
Really, not knowing how spells work is always going to be problematic. Because you can't anticipate the spells your players will use and don't know how to counter them. And you thus don't know the best play to play and run a spellcaster character. Knowing what the best spell to use against certain characters, such as
dominating the barbarian or hitting the sorcerer or cleric with
feeblemind.
NPC spellcasters really need to hang back. Preferably behind tougher monsters. Having spells like
counterspell handy are also useful. They should also use the terrain, having barriers between them and the player characters, or traps.
But this is going to be a big hurdle. If you're mentally a rogue/fighter player, then as a DM it's going to be hard for you to challenge and counter the entire party. Not every DM has a tactical brain, which makes composing fights just that much harder.
3.) Wrong encounters:
Because the whole adventure is very "sandbox" I often need to adjust encounters. Therefore, I often use encouter builders like
Kobold Fight Club to fit the characters current level. I always use "hard encounters", but it still seems not enought. Might that also be a problem?
More enemies can work, but more enemies at once isn't always the best plan.
Waves of enemies can be more effective. As can reinforcements, possibly ones coming from a different direction.
Terrain is also a big equaliser. Cover and difficult terrain that slows down the PCs can be used to great effect. As can ambushes or traps.