Correct me if I am wrong, but RPG books generally cost less than hardcover nonfiction books, and take up less shelf space due to the peculiar form factor of gaming books. Nonfiction typically circulates only slowly, as compared to fiction. So, for price, collection size restrictions, and number of folks who use it, RPGs should be no worse off than hardcover nonfiction.
And somehow, libraries are always bringing in more of the hardcover nonfiction.
I will 100% accept the point that many librarians are not informed about RPGs, and so don't make them a priority in their buying choices. I'll point out the cost/size/circulation potential questions don't come up unless you consider buying the book first, and weigh it against other possible purchases.
Yes, libraries prefer hardcover books in general because they last longer in circulation. And, nonfiction books are important in any public library's collection, but regardless of their price (and note that many hardcover RPG rule books cost around 40-50 dollars as well), what really sets them apart is that RPGs serve the needs of a very small percentage of the library users (which is also divided into subgroups such as fantasy, horror and sci-fi fans) and nobody else, while a book about, say, gardening is a "better" acquisition in terms of general usefulness and circulation -- especially as libraries worldwide get less and less funds each year (again, this comes down to the numbers). In terms of cost and space (yet not in terms of circulation) indie RPGs are the best acquisitions for a library (usually they're relatively small and thin books that are cheap, and they also don't have any supplements, i.e. you get a complete RPG system for 20-25 dollars).
The actual size of the books is not as relevant as how much shelf space is *reserved* for the RPG collection (usually not much). For example, in my library I did collection development just a couple of weeks ago, and I got the permission to acquire a new RPG title for each one weeded from the collection, and this was partly due to lack of shelf space (partly because of the other reasons I listed). Naturally, the physical size of material *does* matter in terms of shelf meters (the measurement used here, i.e. how many items you can, in average, fit into the space of one meter), but usually it's a rough estimate that's used (i.e. "We can still fit five new items there, or three if they've over 500 pages").
IME popular RPG titles circulate around 10-15 times per year, but most around 3-5 at best (this is based on my own evalution work in the library I work at, and the numbers may be different in bigger or smaller libraries); yet, in general long-term circulation numbers as RPG material becomes outdated with new editions and design trends tends to result in zero circulation after a few years (note that apart from computer guides, most nonfiction books -- even with new editions coming out -- may still check out regularly and contain relevant information even after decades).
And, as I said, local needs vary a lot; for example, in a rural public library nonfiction books about fishing and hunting may be more popular than 'Harry Potter' or 'Twilight' series. Likewise, indie RPGs seem to be surprisingly popular here, while WFRP and WoD books mostly gather dust on the shelf (the local university has a very active RPG club with dozens of members, and they are mostly into indie systems at the moment).