Especially if your caster is more controller than blaster.This would definitely tilt balance against casters and toward martials. There are lots of good feats for warriors, quite few for casters.
This is a funny sentence to me, since the vast majority of "balance" problems I've seen are due to multiclassing. Ban multiclassing, allow feats only with no ASIs, bask in the sudden variety of character development you see.
It's also possible to create a paladin and throw all of your stat boosts into Intelligence. The only balance in the game is a balance of opportunities, but once everyone has made their choices, it is no longer balanced.Since it is theoretically possible and legit by RAW, when using feats, to create a character who never takes an ASI but only feats - this by Definition cannot be unbalanced because RAW would be unbalanced then.
Has anyone considered banning ASIs and allowing only feats?

(Dungeons & Dragons)
Rulebook featuring "high magic" options, including a host of new spells.