I think the issue here is the flavour people have for a wizard's spellbook. Since flavour text can justify anything (see my sig), let me have a bash at what I think a spellbook is. That, I believe, will help explain why you can't cast a spell directly from a spellbook.
A spellbook is a how-to. It's a set of instructions that describes how the owner casts his spells. Magic is very complex, however, and also very personal. And by definition, it is arcane--in the wierd and bizarre sense of the word. Thus, the set of instructions does not consist simply of "move arm in manner A, while twiddling fingers in motion B", but requires words and descriptions for the manipulation of arcane energy that the 'common person' simply cannot even fathom. Thus, special materials are required as well as extensive descriptions.
However, each mage also approaches magic differently as well. While one mage might capture the magical energy, flow it down his arm, and thrust it forth from his finger as a magic missile, another might shape the magical force in front of him and push with his whole body to thrust the missiles at his target. Thus, when a wizard is forced to borrow another's spellbook (with a slightly different set of instructions) he needs to know enough about how magic works (represented by a Spellcraft check) to be able to adapt those instructions to his own use.
What spellbooks are not is a storehouse of magical energy. That comes from the wizard. You could not cast a spell directly from a spellbook any more than you could start a fire directly from a textbook that describes how to light a fire using sunlight and a magnifying glass. Just like you still need the magnifying glass and sunlight to start the fire, you need to have the magical energy stored inside you to be able to cast the spell.
A spellbook is a set of instructions, a how-to. It's a glorified textbook.