I'll pull some data from
http://www.librarything.com. This is what people have entered in for their "libraries"; varying definition of library and self-selection problems come about, but in theory, this is what's on book shelves today. This is also number of copies of individual books held. When editions are combined and are separated is sometimes wonky, and is basically out of my control, and I suspect there's at least one game that could hit the 200 copies limit if we merged editions. Numbers are for copies, not holders; if someone holds editions 1, 3, 5, and 7 of Call of Cthulhu, it will add 4 to number of copies. An * is when I combined LibraryThing works to form one count for several editions or the like.
I'm working from a semi-complete "library"/bibliography that was compiled in 2007. I checked other lists, and hopefully caught anything new that would cross the limit; I made sure to double check D&D 4 and Pathfinder (Corebook only hits 92.) The number of copies is all current.
Most-held RPG books:
* PHB 3.5 + Gift Set = 1000 copies
(This assumes that nobody lists both the Gift Set and the books in the Gift set in their library, a known incorrect assumption)
Player's Handbook: Core Rulebook I (3.5) 955 copies
Player's Handbook: Core Rulebook I (Dungeons & Dragons, Third Edition) 783 copies
Player's Handbook (1st edition) 682 copies
Player's Handbook (2nd Edition) 659 copies (includes revised edition copies)
* DMG 3.5 + Gift Set = 645 copies
Call of Cthulhu 625 copies (includes all editions)
Dungeon Master's Guide: Core Rulebook II (3.5) 601 copies
Dungeon Master's Guide (1st Edition) 574 copies
Dungeon Master's Guide: Core Rulebook II (3.0) 569 copies
* MM 3.5 + Gift Set = 564
Monster Manual: Core Rulebook III (3.0) 528 copies
Monster Manual: Core Rulebook III (3.5) 519 copies
Monster Manual (1st Edition) 516 copies
* Dungeons & Dragons Player's Handbook (4th Edition) + Dungeons and Dragons Core Rulebook Gift Set 512 copies
Dungeon Master's Guide (2nd Edition) 482 copies
Dungeons & Dragons Player's Handbook (4th Edition) 403 copies
* D&D 4 DMG + Gift set 374 copies
* Star Wars D20 core book (1st, Revised, Saga) 370 copies
Forgotten Realms Campaign Setting 363 copies
GURPS Basic Set (3rd Edition) 355 copies
* Dungeons & Dragons Monster Manual + Gift Set 354 copies
The World of Darkness (2004 book) 337 copies
Monster Manual II 318 copies
Deities & Demigods (1st Ed) 308 copies
Vampire : The Masquerade (3rd edition?) 304 copies
Fiend Folio (1st ed) 289 copies
Monster Manual II (1st ed) 285 copies
Unearthed Arcana (1st ed) 284 copies
Dungeons & Dragons Dungeon Master's Guide (4th ed) 265 copies
Manual of the Planes (3rd Ed) 260 copies
Monstrous Manual 258 copies
* Exalted (1st Ed + 2nd Ed) 250 copies
Dungeons & Dragons Monster Manual 4ed 245 copies
The Book of Nod (V:tM) 244 copies
Vampire: The Requiem 240 copies
Complete Warrior 238 copies
Eberron: Campaign Setting 235 copies
Book of Vile Darkness (3ed) 230 copies
Serenity Role Playing Game 229 copies
Complete Adventurer: A Guide to Skillful Characters of All Classes 227 copies
Sword and Fist: A Guidebook to Fighters and Monks 227 copies
Mage: The Awakening (2005 book) 223 copies
Vampire: The Masquerade (2nd Edition) 217 copies
Complete Arcane: A Player's Guide to Arcane Magic for all Classes 212 copies
Defenders of the Faith: A Guidebook to Clerics and Paladins 212 copies
Werewolf: The Apocalypse (both 1st and 2nd edition) 212 copies
Epic Level Handbook 209 copies
Player's Handbook II (3.5) 207 copies
Tome and Blood: A Guidebook to Wizards and Sorcerers 206 copies
Psionics Handbook (3.0) 203 copies
Complete Divine 201
TL;DR version: D&D 3 & 3.5 lead the list, followed by D&D 1 & 2. D&D 1, 3 & 3.5 are pretty much the only systems to have non-core books show up. Followed by Call of Cthulhu (arguably cheated through over-combination), D&D 4, Star Wars D20, GURPS 3, various WoD main books and the Serenity RPG.
Skipping further D&D books:
D20 Modern 188 copies
Mage: The Ascension (Revised Edition) 186
GURPS Basic Set: Characters (4th Edition) 184
Mage: The Ascension (2nd Edition) 169
GURPS Discworld 169
* Shadowrun 2nd Edition 168 (It's a bit of a mess in the system)
Call of Cthulhu D20 166
Darksword Adventures 163
* Rifts Original Edition + Ultimate Edition 163 copies
Middle Earth Role Playing (1st & 2nd edition) 162
Werewolf: the Forsaken 159
Unknown Armies 158
GURPS Basic Set: Campaigns (4th Edition) 156
Vampire: The Dark Ages (1st Edition) 154
GURPS Space (for 3rd Edition) 152
* Star Wars D6 RPG Corebook (multiple editions) 152
Vampire Players Guide (2nd Edition) 145
Cyberpunk 2020 (multiple editions) 144
Star Wars (Revised Edition d20) 144
Laws of the Night (Mind's Eye Theater) 143
Exalted (1st Edition) 142
Changeling: the Lost 140
Shadowrun, Fourth Edition 137
In Nomine 136
GURPS Compendium I: Character Creation 133
GURPS Magic (for 3rd Edition) 133
Amber 132
The Wheel of Time (d20) 131
Rifts 130
Wraith: The Oblivion (2nd Edition) 128
Nobilis (2nd Edition) 127
* Savage Worlds (- + Explorer's Edition) 127
Kindred of the East 126
Guide to the Camarilla 123
* Burning Wheel: The Character Builder (+set, etc.) 123
Werewolf: The Apocalypse (Revised Edition) 121
Guide to the Sabbat 121
Warhammer Fantasy Roleplay (1st Edition) 121
Buffy the Vampire Slayer 118
Changeling: The Dreaming (1st Edition) 117
GURPS Cyberpunk 115
Star Wars (1st Edition D20) 115
Street Samurai Catalog (Shadowrun) 114
The Book of Shadows: The Mage Players Guide 113
Mage: The Ascension (1st Edition) 113
GURPS Illuminati 113
Star Wars Roleplaying Game Core Rulebook, Saga Edition 111
Champions (Editions 1-4) 109
Teenage Mutant Ninja Turtles and Other Strangeness 108
Shadowrun (2nd Edition) 107
GURPS Supers 106
Exalted (2nd Edition) 104
Hunter: The Reckoning 104
Hero System 5th Edition 103
Mage: The Sorcerers Crusade 102
Big Eyes, Small Mouth (2nd Edition & Revised 2nd) 102
Feng Shui: Action Movie Roleplaying 101
Paranoia XP (6th Edition) 100
Trinity 100
Shadowrun (1st Edition) 100
If you combine all editions of the main book of Shadowrun, there's around 532 copies held. Outside of GURPS Traveller (83 copies), no Traveller book had more than 40 copies. The Deadlands main book had 70.
What do the copy numbers mean relatively? Harry Potter and the Philosopher's Stone is the most held book in the system, with 63k copies. All the HP books have over 50k, (* Fellowship of the Ring + the Lord of the Rings + Hobbit/LotR is 57k), then we get to the Da Vinci Code (44k copies), and the Hobbit (42k) (* + Hobbit/LotR = 46k). The Player's Handbook (3.5) that topped our list is the 6,470th most popular work; the books pulling up the end with 100 copies are in place 88,010.
Take with a lot of salt; combining different editions can make a huge difference, and there are a lot of games that could have been combined and represented with a * up there. The data itself is self-selected, and who knows what the bias is; maybe selecting for people who enter their book collection online samples for GURPS more than Rifts, for example. But I think the general stripes aren't too wrong.