So, as we look at this, we should remember that it isn't just simple schadenfreude.
Generally speaking, the people we are talking about aren't just rejecting vaccines now, for example. They have broadly been resisting and eschewing any and all approaches to controlling the issue - masking, avoiding group gatherings, lockdowns, and so on. So, these people are not just victims. They are not just victims of their own bad choices. Many of us, and our families, and our economic situations, are victims of their choices. The entire pandemic would have been under better control if not for their recalcitrance. A great many of the nearly 700K deaths in the US were preventable, but for them.
Basically, in aggregate, they have been committing a kind of assault against the rest of us. Pardon us if we don't care much when they commit assault against themselves too.