The only thing that generally holds the spellcaster back is not having 100% clarity of what challenges are coming next. The ability of spellcasters to have a solution for so many in-game problems can be unbalancing, trivializing things that others might struggle to overcome. Really, how well a spellcaster can predict what upcoming challenges ahead can be a major factor in how well the group does.
While WotC's done decently in balancing effectiveness in combat at a single moment in time, when combining all three pillars spellcasters are off the chart.