You are vastly changing the action economy. It IS a problem, regardless if you realize it or not. Perhaps you/your table have nothing to compare it to. Or are buffing casters other ways like only running a few encounters a day.
Not being able to realize a problem does not mean there is no problem. This is a huge change to what a caster can do, either in removing concentration spells, or by removing casting actions while keeping them up. There definitively is a change in the relative power of the classes if casters can not cast while concentrating.