You're right. Wrote that before my morning caffeine.
And how is it different to have this spell protect someone versus Cat's Grace or any other spell that enhances touch armor class?
Would you throw your players a bone when they roll an 18 and miss with a touch attack and it is because the BBEG is displaced or has a high touch AC? What happens when they roll a 4 and miss? Do you still tell them it is deflected away?
Where do you draw the line? As a DM, I think it is a lousy idea to always give your players the answer either directly or indirectly. If they ask what happened, then give them a Spot roll and if they make it, give them a hint. But just spoon feeding them the answer is not a good idea (which your "hitting the target perfectly and then being deflected away" is). How do you know that the Ray is not deflected at the point of origin?