The funny thing about D&D's version of Vancian magic is that so many of the terms have gotten tied up into what a particular version did, that it is easy to lose sight of the underlying game and story reasons for having things the way they are in the first place. So let me try theory-crafting an 1E/3E/4E hybrid spell framework from scratch, and see where this leads us. I'm going to assume that this is mainly for wizards, just to keep it simple, though if it were to work it would have broader applications.
This hypothetical version is going to have wizards casting three types of spells:
- "Utility" - relatively minor spells that the wizard can do a lot of in a given 24 hours, with a lot of variety, but not terribly powerful.
- "I Win" - classic Vancian spells with big, quick effects, but in short supply, and definitely fire and forget.
- "Long Magic" - anything that doesn't fit either of the above, but is better represented by long casting, research, major components, enchanting, etc.
Note that nothing is said about combat versus non-combat above. You can have "utility" combat spells in this scheme. They just won't be very impressive. Now, the obvious way to get that is to have "utility" spells be at-will, "I Win" spells follow more the AD&D pattern, and "Long Magic" be rituals. But again, let's not get too caught up in mechanical terms.
For example, the purpose of "at will" in 4E is to avoid a lot of book-keeping with minor effects. Otherwise, you could get much the same balance effects by providing a whole bunch of Vancian spell slots expressly for such weaker magic, or using some form of power points for them, etc. Basically, these are things where a wizard isn't
likely to run dry, but might depending upon the exact scheme.
The critical piece is that they be separated from the other categories somehow. AD&D tries to do this with levels, but then so sharply curtails the amount of slots that the wizard does run out, sometimes quite rapidly. 3E corrects that problem, at the expense of letting the power get out of whack with too many slots avaialble for more powerful spells. Then 4E realizes they need separating, but largely removes the variety. Moreover, in this scheme the problem 4E has is that of its five divisions (at-will, encounter, daily, utility, and ritual), only ritual really maps to a category very well (for other reasons, granted).
There are probably 60 ways to handle this division. I'll sketch one as an example, but I'm not wedded to it:
- We'll call the categories "utility", "spell", and "ritual". I avoid "cantrip" for "utility" because magic in all categories will scale by level, and a high level cantrip might create the wrong impression. Power is relative to category and level.
- Wizards start with a fair number of slots, perhaps 3-4 1st level slots. Each time a new spell level is gained, the wizard gets 3-4 slots of that new level. The numbers for a given spell level increase fairly rapidly by level to whatever the cap is, probably around 6-8. Int does not modify this number in any way. (That is, a 5th level wizard might already be maxed out on 1st level slots, be about halfway on 2nd level slots, and just have picked up several 3rds.)
- However, there is a cap on the number of "spells" ("I win" effects) that a wizard can prepare at one time. This could be the original Vancian limit of 4-6, but for game purposes we might want it to be a bit higher. I'll guess that it should be about 1 per spell level. So let's be generous and say Int mod plus 1 per spell level obtained. So the 5th level caster can use 3+Int mod slots for "spells". Furthermore, we'll say no more than 2 per any given spell level higher than 1st. So a 5th level caster with Int 18 (+4) will have 7, which means that he can prepare 3/2/2 "spells". Look familiar?

- Rituals and Utility magic can now round out the remaining slots. The Next take on rituals is already promising. So mainly this is similiar, with the caveat that a "ritual" prepared in a slot already has a lot of the time, components, etc. invested in it ahead of time, and only needs to be released with a minor form of the ritual. It may take 1 hour to cast "alarm". But before you go into an adventure, you can prepare the ritual into a slot such that you now can do it the Next way on demand (some time, remaining focus or components). It doesn't get used up, though as an optional rule it might require some kind of check, and "burn out" if the checked is failed horribly. So this is the strategic level of magic, meaning really powerful stuff still doesn't cast immediately and takes considerable time and expense to get into a slot in the first place, and probably takes some static (i.e. large, not easily moved) resources to so prepare. Wizards tend to get really cranky when you ask them to swap these.

- Utility magic's different characteristic, in contrast, is that it limits what utility magic you have available, but you have a lot of it. It's easy to swap overnight, giving a spell book, even a traveling one. Once prepared, you can use it a lot. In fact, the magic is so easy to use, that when you prepare a utility spells, it comes with several uses, as "charges". Put magic missile in a 1st level slot, it operates at 1st level power, but you can cast it, say, 10 times before it runs out. (Obviously, there is a hook for a module here, with some groups making these functionally at-will, others using power point pools or even exotic limits.)
- It's implied but not required that there aren't that many "I win" spells. Historically, there aren't that many good ones, and these are the spells most in need of some thought put into balance. So Hussar's 10-12 per spell level is a good upper limit, at the very least for what a given wizard might know. Once utility and ritual magic is taken out, I doubt we could get to 12 on any but the first two or three spells levels, anyway. Meanwhile, utility magic can proliferate madly, for all we care. Rituals simply need to be well done or left out, and let the number take care of itself.
Again, this is merely a long example of the kind of division I'm encouraging, and the thinking behind it. There is no particular reason why "utility" magic has to be in slots, necessarily. I picked that to show that it need not be "at-will" to be effectively wide open in usage. It does need a framework that allows some variety in switching effects and/or limiting it's final usage, if only in a module. Here is obviously a place where a parallel (instead of replacement) power-point system might actually work.