I'm not 100% sure that I understand what the OP is talking about. My role-playing experiences
usually have the sorts of character-driven play that
@Celebrim described in some detail, which I'm grateful for. I think
@innerdude might be getting at something a bit different, but I'm not sure how to place it.
The primary group I play with does a lot of the "method acting style" and sometimes that ends up with characters developing and evolving internally in ways that make sense in the ongoing campaign. I am usually
intentional in choosing which part of my personality to into a character I play, so that I will have something to strongly relate with and work with (or even work through in a therapeutic manner).
For example, in a very character-driven (by Celebrim's definition) 3.5e Ravenloft campaign, my character represented my "scared child" personality component. Over the course of the campaign, as he faced terrible Ravenlofty things and interacted with a group of other well-roleplayed characters, he was forced to address his "inner demons" (mechanically reflected by his class, a variant Wilder whose powers were occasionally Ravenloft-corrupted, but thematically addressed through dream sequences and just normal choices during play). He experienced/expressed new aspects of his fears and gradual sense of empowerment through the psionic powers I decided to have him acquire as he leveled up. He also met, developed a relationship with, and eventually married a complex NPC (who, now that I think about it, might easily be described as having a transformative character arc as described in the OP). By the end of the campaign he had overcome/processed much of his fear. Instead of being an Innocent (Ravenloft jargon) farm boy cursed with terrifying powers, somewhat reluctantly traveling around with this motley crew to help out his cousin in the fight against evil, he was a competent, if humble, hero aware of and in reasonable control of those abilities within himself (mechanically reflected by taking a level of Psion near the end of the campaign to pick up a few low-level powers that he didn't have to worry about random downsides using), and willing to step up and do what needed to be done because it was the right thing, not just out of filial piety. Even though his personality is quite a bit different than my own, I was able to take the couple of elements I related to and "get into his head", so that I could experience his journey as if I were him.
I think this does differ from what I'd experience with a book or movie.
Despite my connection with the character, I couldn't be as moved
by him, because I was the mover
of him.
I think this is
huge, and if no one has ever defined this principle before, I'm going to do it right now:
The author cannot also be the audience
If you write a story (say a book or short story), you will never get the same emotional effect as someone reading it might. You know what's going to happen. You have
control over it. You can have your villain massacre puppies and it isn't going to affect you the same way as it will the reader. More realistically, when you choose an ending out of a few possibilities, you might get a great sense of satisfaction out of making it work out according to your vision of how it should be, or feel a sense of dissatisfaction if it could've been better, but you will never feel the rage and anguish of those who are pissed off at you for "ending the story wrong".(1)
They got invested(2) with the characters and wanted a certain ending, and were devastated when that's not what happened. You might be sorry they felt that way, but you had a creative vision and think you did a really good job with it. While I remember those authorial betrayals (lol) better, the experience is similar with stories when you
really like the ending on an emotional level. For instance, I loved the final (2-part) episode of Star Trek: Voyager (other than the lack of post resolution denouement right at the end) because of Janeway's love for her friends and her conviction driving her to (in contrast with her normal choices) completely disregard all "the rules" and
literally go back in time to fix all the bad stuff that happened over the past (13?) or so years. There was just this exultant sense of
rightness to what she did. A strange, rebellious cosmic justice. And it didn't negate everything that had happened before, because all of the losses over the series were still there--just the losses in the many years between that episode and the previous one were resolved. I got that emotional connection you are talking about. If I had been the author I don't believe that would have been possible.
As far as the reason for this phenomenon, I'm not sure, but I suspect it has to do with there being interpersonal interaction when you are the audience consuming a story composed by someone else, whereas when you yourself are the author it is a solo experience that isn't as emotionally rich
in that way as the interpersonal one. The fact is, as much as you might be
creatively invested, even to the extent of strong emotion involving that creativity, as much as you might have a strong sense of delight, or disappointment with your work, you can't have the sort of
visceral emotional investment that the consumer has regarding the fate that you created for those characters.
So, although I'm still not 100% sure I've identified exactly what phenomenon
@innerdude is referring to (and in this type of conversation, I think exact identification is essential), based on the medium of role-playing, I would be surprised if that principle isn't at play here. As long as you are playing (ie, authoring) the character, you can't have the same experience as you could have as the audience of the performance.
Perhaps a way you
could get something like that is through the character-driven development of your fellow PCs.
I also think there are experiences you can get through playing a character that you can't get as well through either authoring a story
or being a consuming audience. The one I brought up is the therapeutic element of self-exploration. Your ability to guide that interaction dynamically through your character's choices when presented with an environment (by the GM/setting) makes it more effective than authoring a story in the traditional way, or than identifying with the experiences an author chose for some other character as you read/watch their story.
I get the idea that this therapeutic benefit is simply one manifestation of a larger beneficial effect (probably also themed around self-exploration) at work here--one in which role-playing is the best at providing--but I haven't really delved into exactly what that would be. Of course, now that's it's occurred to me I'm not going to be satisfied until I understand it.
Hope that was helpful!
(1) You might feel the rage and anguish of not being able to end the dang story, or not having it appreciated, etc, but that is all outside the story and different than audience investment.
(2) Apparently research indicates that binge watching/reading heightens this emotional investment