That depends on the resources required to be decent at both magic and combat.
- If you have to buy basic stats (Strength, Dexterity, Intelligence, etc.) with a point-based system then it can be difficult to be any good at both magic and combat. Being poor at both is usually not much fun. Under such systems, I usually specialise, because I like accomplishing things within games.
- If stats are randomly generated, then there will be some characters who are good at both, as demonstrated with multi-classed characters in early versions of D&D. I've played both specialised and multi-classed characters in such systems, depending on the game and the setting.
Let's assume that the RPG permits being decent at both arms and magic. I'll agree that a game that forces incompetence on PCs doesn't sound like fun.
I honestly think that having freely-usable spells that are just better than skills are usually bad design. There should be something (other than prepared casting, since this isn't D&D) that makes it not always the best choice.
The point of using magic is to be able to do what one normally could not, right? That encompasses a lot of possibilities. The germane instances are spells that do more damage than weapons, that are more protective than armor, or that are more useful than skills, I guess. A reasonable limit or approach is that these spells are usable less frequently than the skills are. Or that the magical weapons, armor, or skills are inferior to the real ones, but easier to acquire (for the caster). So it's not about being better, but more accessible...
Obviously, yes, since being a martial or caster has always been voluntary, and it's always been a problem (even if not always to the same degree). The concepts that call for a martial character are too familiar/desirable to just be cast aside because they're inferior - thus the fighter, always the most popular class, generally among the worst classes, being a 'trap,' for instance.
Hardly a trap, if the player desires playing a fighter. If the player instead desires to do the most damage or have the most options for world interaction, then maybe caster would be a better choice.
In the OP game, someone wanting to play a fighter, but have the flexibility of a caster, can simply choose to learn a spell. If that solves the martial/caster problem, then I think the "martial/caster problem" is really the "WotC class design problem." (I, for one, will not tell WotC that it doesn't know how to design D&D classes. However, I will homebrew classes in a heartbeat.)
The balance issue with this hypothetical arrangement is that there appears to be a range of effectiveness outside of dishing damage in combat that is only accessible to casters, making them more versatile, as has always been the case in D&D. That said gap widens or narrows with the focus of a campaign only makes it worse.
(there also appear to be no non-damaging abilities other than spells, presumably just to simplify the example, perhaps they're equally accessible to all characters w/o any trade-off, or otherwise irrelevant?)
In the OP game, flexibility is a player choice. Does that make martial/caster a player problem, instead of a game problem?
If I understand the parenthetical correctly, yes, it's fair to assume that non-damaging abilities, what I'd call skills, are equally accessible to all characters. There's always a trade-off: a player chooses one ability at the expense of not choosing (presently) another. So, learning some smithing means waiting to learn some axe-throwing or teleport-casting.
There’s also reasons why 3.5Ed is my favorite iteration of D&D.
