You're talking about essentially a wizard with better HD, BAB, skills, and proficiencies, in exchange for no familiar, no bonus feats, and access to only three schools of magic or thereabouts. Of those, only the BAB advantage seems too much under the circumstances. I suppose, you could go with that but reduce the spells per day by 1 per spell level (imposed right when they would gain access to each spell level), and probably take away the medium armor proficiency too.
Thus, at 1st-level they'd get, for instance, a base of 0 spells per day of 1st-level, only receiving any bonus 1st-level spells for good Charisma (if any). Then at 3rd-level, they'd gain access to a base of 0 2nd-level spells per day, plus any bonus 2nd-level spells per day for good Charisma (if any). Etc. Same way that a bard has access to "0" spells per day of any given level when they would first gain access to it. They'd still be able to learn spells of a given level when they first gain access to it, but they wouldn't be able to cast them yet unless they had a high enough Charisma; otherwise they'd have to wait until the next level, when they'd gain a base of 1 spell per day of that level.