You would have to ban the caster from using swords OR make the sword of slashball martial class only.
Or make it a class feature, was my point.
Must be Class X of Level Y to attune.
That'd be a class feature listed in the DMG.
A straight-up class feature would be, just, at a certain level you gain a magic item with a certain magical function.

No, other classes can't use it, unless they get the same feature. Simple. Bit abstract, but so's everything in D&D from hp on up.
And, TBH, I was not thinking of fireball at 5th level taking out an encounter full of orcs or something, but sleep at 1st level dropping, like 4 or 5 kobolds (an easy encounter) in one action, a feat a mundane fighter could accomplish at... 11th? 2 extra attacks, action surge, 6 total attacks (you can even afford to miss once or twice). 5th level with a build that finagles a bonus action attack, and a bit of luck.
Well, I mean, apart from the kobolds all needing to be within a 40' sphere centered within 90' of the caster (120' because the caster could move forward 30' and cast), and someone needing to finish them off in the next 10 rounds.
It's easier to push your point of view by using extremes and pretending there isn't a spectrum.
I mean, "mundane" and "supernatural" are extremes, and most fantasy-genre martial concepts probably fall between them?