I agree with you. A good book is a good book, and pigeonholing something into a genre can be a trap. Genre is a tool for helping to find similar books on a bookshop shelf, it shouldn't be any more.
This isn't really true. Genre is much more than an organizing tool; it has a huge influence on style and content.
Genre fiction is almost always heavily plot-oriented, with more focus on action and setting. Science fiction further has a strong emphasis on scientific ideas which gives the prose a didactic tendency. Readers who prefer particular genres are expecting certain features, and that leads to genre fiction often feeling repetitive and derivative. Characters are often somewhat stock (so many orphans, for example).
Critics tend to look down on genre fiction because most of it is simply not that skilled. You get a lot more potboilers, and a lot of writing is done by authors who are more interested in exploring genre conventions or pushing the plot forward than crafting a gorgeous sentence. Look at almost any of the writers in this discussion: no one is going to mistake Isaac Asimov or Robert Heinlein for James Joyce or Toni Morrison any time soon.
What makes a book "good" is entirely contextual, so if you are in the mood for a page turner that you don't have to think too hard about, then a deeply introspective character study with challenging sentence structure and dense symbolism is probably not going to do it for you. That's me, a lot of the time: I teach language and literature all year, so on vacation I stay well clear of Booker Prize nominees. Instead, I look for well done genre fiction. Science fiction tickles different parts of my brain than those pedigreed novels do, as does horror, fantasy, historical fiction, travelogue, etc.
So I can read a book and recognize that it is not "good" in a literary studies sense - I wouldn't assign it for an oral commentary, for example, because it would be tough to find an extract with enough literary technique to unpack. And I can also think that it is a
great read and recommend it to all my friends. Like, I currently am
loving the Murderbot series by Martha Wells. Would I consider it great literature? No. Would I consider it great science fiction? Absolutely. Am I looking forward to the next novel more than any other book? 100%.
Edit: there are very few "pure" science fiction writers that would make critics' lists of great writers, and those ones are not being brought up in this discussion, except maybe William Gibson, and he would be marginal. If we are looking at writers of science fiction who are also acclaimed by literary critics, we are talking George Orwell, Margaret Atwood, William Burroughs, etc.