It can work, but we can look back at the 3e Truenamer to see how difficult creating a skill-check based casting system can be in a normal D&D environment.More rituals? Maybe +1 on the check for each extra round you spend casting? Maybe only risk it if the situation is worth it? Maybe supercharge cantrips so that even if you get really unlucky and lock out all your spell levels, you still have good cantrips? Maybe play a Wizard if you want out-of-combat casting?
I don't know the exact answer, but I tend to not throw ideas out completely just because the first pass isn't perfect. (I know, I know....99% of the entire 1D&D discussion is exactly that so WTF am I doing here?)
And you may be right; it might just not work.
It's much easier to do if it's the only casting type available; but compared to the ease of use and reliability of standard D&D magic it's a hard sell.