The Handy Haversack is odd, as it actually requires more effort to pull something out of it than out of a regular backpack!
"Interacting With Objects Around You
Here are a few examples of the sorts of things you can do in tandem* with your movement and action:
...
-withdraw a potion from your backpack" - PHB p.190
*In context, this means it doesn't take you an action to do, but you can only do it once per turn (if you do it twice, it requires an action).
"Heward's Handy Haversack
...
Placing an object in the haversack follows the normal rules for interacting with objects. Retrieving an item from the haversack requires you to use an action. When you reach into the haversack for a specific item, the item is always magically on top." - DMG p. 174
I really don't get that. I thought this item was supposed to be a replacement for a mundane backpack, but it's actually harder to use (even though the text implies is it supposed to be easier), and it isn't that much bigger. I'm not sure if the designers forgot what the PHB had said, or if they are treating the potion removing differently (ie, maybe they meant that to refer to a potion in an easily accessible side pocket, contrasted with taking a minute digging around for stuff inside of it.)
Using your Book of Shadows as a wizard spellbook is a DM call, but I'd definitely allow it, and I think the majority of DMs probably would also.