Based on the guidelines for spellcasting services which cite 10-50gp as a range for a first level spell, I would go with:
100gp for the empty spellbook + 30gp per spell level contained therein.
So if there are 3 1st level, 3 2nd level and 2 3rd level spells in it, that would be 3 + 6 + 6 = 15 spell levels worth of magic.
15 x 30 = 450
450 + 100 = 550.
They should not take/get less than 550gp total for the book...and other spellcasting NPCs in the game world are going to know that.
If you run "lower/rarer magic" world, use the 50gp per spell level. If magic is ubiquitous (or just cuz it makes the math easier), use the 10gp/level.
If you want to finely grain it and say that magic becomes more rare as it increases in level (as I personally enjoy), then make it:
10gp per spell level X spell level.
That is, a 1st level spell will be 10gp. Big woo. A 2nd level spell will be 20gp X 2 = 40gp ...just for 1 2nd level spell. A 3rd level spell would be worth, just on a scroll by itself, 30gp X 3 = 90gp ...and so on. Obviously, depending on the power level of the spells in the book, this could greatly enhance or diminish the overall cost, versus a single set rate per spell level.
In which case, our hypothetical book above with 3 1st, 3 2nd and 2 3rd would fetch a solid [10 x 3] 30 + [40 x 3] 120 + [90 x 2] 180 = 330 worth of spells/magic + 100 (book cost) or 430 total. [That seems low to me, so I'd probably be inclined to do 10gp @ 1st, (30gp x2) x2 @ 2nd, (50gp x3) x3 @ 3rd, etc... or something like that.]
Of course, then there is the consideration that a spellbook like that, with multiple 1-3rd level spells, could conceivably keep a mage busy for a long time (if not their entire NPC career/lifetime) and/or would supply an apprentice with spells from XP levels 1-5 just in that single volume. In which case the value goes up exponentially, even if the purchasing mage already has a spell or two in there [not that the character would be thinking in such metagame terms, obviously, but the value "in-world" would surely be recognized].
So, as a PC, I would start at at least double the calculation...and let the NPC purchaser haggle me down to no less than what the calculation actually is.
As the DM, I would explain all of this to the player. Give them the calculation you decide to use. Let them tally it up, search out a buyer, and try to get what they want for it.