Right on. Once the player isn't playing him, he's not the same character, at least not exactly.
Interestingly, I've done this exact thing to one (or two) of Nuclear Wookiee's characters, and neither of us has been too upset about it. I don't think I did anything that conflicted with the way *he* would have, but he wasn't there - so it didn't really matter, either.
If he doesn't like it, he can say it was "alternate universe" or a different character with a similar name and history and it wouldn't really matter to either of us.
There's *nothing I could do* to corrupt the way he chooses to imagine it, and there's *nothing he can do* to enforce his version in my mind. Accepting that is the key.