I think it has a lot to do with changing publisher expectations of what the audience will buy.Now, I miss the Sci-fi/Fantasy books of the 70s/early 80s, where an writer could create a full world tell a full story with a much shorter page count. For example, Roger Zelazny's Jack of Shadows (happens to be on my bookshelf right beside me) is 236 pages.
I reread the Corwin books of Roger Zelazny's Amber series, and was struck by how short they were. Nowadays those five would be a single book.
Edit: There does seem to be an increasing number of shorter novels and novellas popping onto shelves. Not sure what's driving it exactly, but it's pretty cool that Murderbot is four novellas and a novel (so far). (Murderbot is AWESOME)