No, privilege isn't a result of systemic racism, it's the state of not being racially oppressed and therefore not having to think about it. Privilege is the result of not being a victim of racism. Which is why it's a ridiculous argument outside of a narrow look at how institutional racism can persist. There it has some explanatory power in that it suggests that not being a victim inclines the not victim to not recognize racism towards others. It's just a 'you see what you're used to' argument.
As for it being the baseline, I'm worried that you're worried that not being a victim of racism should be the baseline. What baseline would you suggest, in place of that?