For the "higher-lvl spells as incantations" idea, I'd go with the existing system for spell research (DMG, page 198), or a variant thereof, to learn the blasted things. Basically, researching whether or not a given spell-as-incantation would be viable would take 1 week per effective spell level of the incantation, require a Spellcraft check (DC 10 + effective spell level), and cost 1,000 gp per week of research.
If you want to limit the ability to research and cast incantations (unlike the default option in
Unearthed Arcana, which is, IIRC, not limited to only spellcasters), just tie the ability to research and use incantations into a feat (with prereqs of character level 6th and/or the ability to cast 3rd-lvl spells).
Your maximum number of incantations known could be equal to the number of times you've taken the aforementioned feat, or it could increase similarly to the
Psionic Talent feat (max of 1 incantation known the 1st time, +2 incantations the 2nd, +3 the 3rd, and etc.].
Since the default DCs of the checks required to perform incantations are rather high (especially considering the fact that skills top out at 9 ranks at 6th level), with every purchase of the feat you could toss in a free minor bonus (+1 or +2; the bonuses would stack) on checks required to perform incantations. Alternately, you could set the bonus equal to one step less than the number of times you've taken the feat (as the increasing option listed above; +1 bonus the 2nd time you take the feat, +3 total the 3rd, +6 total the 4th, and so on).
If you limit the feat so it can't be taken more than 5 times, you have the following (after taking the feat all 5 times):
- A maximum of 5 incantations known (or 15, if using the other option).
- A +5 bonus (or +10 for either of the other options) on all checks made to cast incantations.
Just my 2 cp. Apologies all around if this comes across as a ramble.