I reject the concept of non-victimhood being a privilege. I don't do this because I'm white, or male, or highly educated. I do that because it's abominable to me to suggest that you should feel guilty for not being a victim.
The real point of talking about privilege is not to make people feel guilt. I know that's what many people use it for, but that isn't the constructive use. Again, don't conflate use with the validity of the idea.
The real point is awareness. maybe you are aware. Many others are not. Even those who are intellectually aware can (and *do*) use the concept to maintain their awareness when going into an area where they are a bit blind. It is likewise a tool for those who are victims - they can become very, very angry that those with privilege "just don't get it". It is important to understand *why* they don't get it - it isn't really their fault, as they lack the context to be able to understand easily, and will reject the idea that they are in a superior position.
Yes, there are those who use it as a club, and don't fight ignorance constructively - that's an issue of people, not the theory.
It comes down to the difference between accountability and responsibility:
In ethics and governance, accountability is answerability, blameworthiness, liability, and the expectation of account-giving. This is where guilt comes in, in the "blameworthiness".
Responsibility may refer to: being in charge, being the owner of a task or event. This isn't about guilt, but about being the one who is supposed to take action.
In software production, for example, there's a "Business Owner", who is accountable for whether or not the software succeeds in its mission. It is the developer who is responsible for writing code. Or, if you will, consider Billy Joel's "We Didn't Start the Fire" - We didn't start the fire. It was always burning since the world's been turning. You are not accountable for being in a position of privilege - you were likely born there, and it isn't your fault. You are, however, responsible for using that position wisely, and being aware of it when speaking with those who are not in your position.
There is also the matter of the "Tone Argument". The person in the superior position really doesn't get to choose how the victim talks about their problems. Very specifically, the argument, "Your words/tone make me feel guilty, making *me* attacked, and now *I* am a victim..." is a non-starter, a misdirection that makes the discussion is about how the privileged feel, rather than about the racism/sexism or other problem.