That is with the supposition that he actually cares. I think the point is not what that means about the topic (in this case V), but rather that, like a light switch, he has two choices.
If he doesn't hate you, he lusts for you. Period. On or Off. According to V's theory, it has nothing to do...