It boils down to a series of very basic elements that some games expand on more than others. Note that I'm assuming a game with at least a bit of crunch to it, as the question is about books.
FOR THE PLAYER:
--- the very basics of the game in general i.e. what's an RPG, what are these funny-shaped dice, what actual items (dice, pen, paper, online access, companion books, whatever) other than this book are required in order to play this game, etc. 2 pages, max.
--- what a character can (and can't) be. This is where races, classes, roles, etc. get put.
--- how to make a character, i.e. the character generation bit. This includes equipping the character, so basic gear lists go here along with anything else needed to get a starting character up and running.
--- what you can do with that character now you've made it. This is where the combat-exploration-etc. rules go, along with spell and ability/feat write-ups
--- a clear and detailed written-out example of low-ish level play and how it would look
--- down-the-road options for later character development. Prestige classes, late-game options, feat-tree suggestions, how to change your character, etc.
--- oddball and strictly-optional stuff that doesn't fit anywhere else.
FOR THE GM:
--- notes on how to best run an RPG, ideally system-agnostic (there can never be too much of this!) and including notes on how to be descriptive in one's narration and how to use one's narration to reinforce the "feel" the game is going for
--- basic notes on how to build enough of a setting to allow play to proceed (and see below)
--- notes on how to make NPCs and where-how they might differ from PCs, along with clear fiction-based explanations of why these differences exist
--- nitty-gritty rules on how to run this particular game and keep it going. Moves and fronts, combat tables, random encounter charts, whatever nuts and bolts this game needs in order to be run, here they are. This fills most of the GM section.
--- in-game info the players wouldn't know but might eventually find out during play. This includes things like magic item properties-lists-pricing, notes on how some player-side abilities-spells-etc. might (or might not) interact, rulings to plug loopholes or exploits, and maybe write-ups on some monsters or foes (but see below)
--- GM-side options and-or guide as to how to tweak the system to suit different purposes.
NOT INCLUDED:
--- setting information. If the game is intended to use its own bespoke setting that's anything other than very bare-bones, that setting is going to need to be presented in enough detail to warrant its own separate book.
--- monster and-or enemy lists and write-ups. Again, this either needs its own book or could be combined with the setting book
Ideally, the player-side material is in one book and the GM-side material is in another.
