If we think of the "normal" tv show, if a show establishes a main protagonist that the show revolves around, and then you shift to now follow a different character, it can be jarring for people. Doesn't matter if its true to previous media, people get used to a certain thing and often don't like it when the show changes direction for a long while.
Imagine if half a season of supernatural focused on Bobby and we barely saw the brother's. Now some people would be fine with that (even think that was interesting)....but a lot of people would be thrown off by it. They come to the show to watch the brothers, not the other characters.
People watch the Witcher for...the Witcher, not the other characters.