You could have the spellbook be treated as a focus for all Wizard spells. If the character doesn't have his spellbook in hand, he would have to make a spellcraft check to see if he can remember the spell well enough to cast it properly (DC 10 + 2x spell level, with 1 always failing). If this check fails, the spell fizzles and the spell slot is wasted. You could even have some more fun and say that if the check fails by 5 or more that the spell actually is badly screwed up and goes awry; the DM determining the effects.
It could backlash on the caster or do other weird stuff.
The wizard could still only cast spells he has learned (scribed into his spellbook). You could increase the DC of the spellcraft check if the Wizard doesn't regularly study his spellbook, to discourage characters from leaving their books safe in town and such.
This idea could be fun, but could be alot of bookeeping (pun intended), as the Wizard would have to keep track of which spells are in which books, and spend move actions to draw the proper book. The spellbook would also be vulnerable to damage, disarming or theft. But considering the immense versatility this would give him, I think these drawbacks are appropriate.
P.S. I have posted a more comprehensive explanation of this system on the house rules forum
http://www.enworld.org/showthread.php?t=201636
