I'm running CoS now.
I started with the adventure as written (AAW) but let things evolve based on the party's actions.
I don't see the need to remove anything. Let the party explore. One thing I love about CoS is that the number of locations and plot threads and the tarot-reading mechanic allow it to play very differently from one campaign to the next. Let the characters by their actions determine what is "left out." If they are completionists and want to follow up on every plot thread--well let them.
The one thing I would watch out for, however, is how quickly the players find the items/individuals in tarot reading. My players found everything fairly quickly, when they were still lower level, except for an item that they are certain is in Castle Ravenloft. This gives the feeling that all the other quests and locations are side quests that are you completing just to level up to take on Strahd. Luckily they are interesting on their own, but they work best when the party feels they are learning about Strahd and finding items and NPCs to help them. In hindsight, I might be tempted to have all items and NPCs place OUTSIDE of Castle Ravenloft. Just leave the location of the final showdown there.
Also, keep in mind that even if a goal of Tarokka reading is not in a location, it doesn't mean that location has no role in advancing the party's objectives.
For example, say you have no items in Werewolves den. There are still plot threads that could allow the party to turn the werewolves against Strahd and elicit their aid against him.
Barbarians are an interesting group that could just be random encounters. But they have so much potential. Aboriginal peoples who've thrown in with the Druids--and therefore--Strahd. Perhaps they only support Strahd because the Druids, who they have an affinity towards have been corrupted. Perhaps they follow him because he is the "strong man" of the area. But there are some seemingly throw-away plot hooks that can make them important allies. First, there is the special event at Yester Hill where the spirit of a Barbarian chieftain leads them to find the Blood Spear of Kavan. Second, there is the special event at Tsolenka Pass where the party encounters Sangzor ("Bloodhorn"). If the party has the Blood Spear and kills Blood Horn, and if they've already defeated the druids at Yester Hill and/or the Wizard of Wines, perhaps the Barbarians will change their allegiance to the party.
Now, storming Castle Ravenloft with a large number of Werewolves and/or Barbarians (or villagers, etc.) kinda kills the flavor of that awesome dungeon crawl. But a barbarian leader/hero and/or a werewolf leader, however, could join, giving the party a boost without turning it into a miniature war game.
Generally, once the party seems powerful enough (level 9 or 10 or perhaps lower levels with powerful allies), you can have them receive the invite to Castle Ravenloft. But if the party is having fun with the various side quests, you needn't force the end.