Have some of all the restrictions, and most casters can benefit from several, if not all.
There is some weak, fairly reliable magic, that a caster can use more or less whenever they feel like it. It might miss (or be saved against), but it works about as well as sword or bow.
Then there is more powerful magic that has more limits--time, (real expense), danger, etc. Ideally, you'd have the option to do some trade-offs here. I'd say that a casting time of one round per level is too strong a scale--maybe one round per 3 levels will work. But if you also had some expense and danger options to blend with that one round per level, it could work. Cast faster, at cost and risk, or cast slower at less cost or risk. Those kinds of systems, if built moderately well, let each caster pick their preferred mix.
Then you might have a few classes that don't get such options, but in return get something else. Their casting is all one way or the other, but they are compensated.
Such a system provides plenty of obvious balance options for non casters. After all, a non caster can find some magical equipment that runs those same time, expense, and cost gauntlets. Meanwhile, the sword and bow still work.
BTW, another "time" option that is less intrusive in some ways than the 1 round per level is to make all spells require a move action to "ready" and then a standard action to cast. It takes a full round to cast a spell, some planning, but you don't get the immediate interrupts issues if the caster has to move. Basically, they can "ready" at some point, hold it for awhile, and then "cast" once in position. Presumably, concentration is possible to break in the meantime, but not nearly as likely as interrupting the actual casting.
For an even nastier variant, use move action to "cast" and then standard action to "target". Now, the spell is fully formed in the meantime, and interruptions can get dicy. That fireball spell is in your hand until you throw it!

On the plus side, personal protection spells (or any spell that doesn't require targeting) is as easy to cast as always. This appropriately makes defense easier than offense, while giving weapon users an edge. A wizard chased by a warrior can throw up defenses, but can't get much going offensively without some distance, help, or risk--or perhaps using an item.
With this system, you can make particularly powerful spells take an extra "move" or three to cast, but still release in one standard action. It only takes a round or two to get the full thing off, but the caster can't do anything else during that time.