When I had 100+ books it was by genre, alphabetical (author) and their series were alphabetical too. Now I only have 10 novels by Glen Cook. Everything else is digital.
Physical RPG books have always been organized by the level of use. Commonly used books are at chest level, left to right, by level of activity. Everything else was by genre and alphabetical. Rare ones on the top shelves. That too was downsized in favour of PDFs. Now I have only six RPGs in book form: four fantasy, a modern, a cyberpunk and a sci-fi game. There are so few they are organized by genre only.