TLDR;
My experience points to a couple things: exposure and time/location constraints.
Firstly, many/most people don't even really know what goes on in a RPG. When I asked my gf if she was interested in trying out D&D, she really had no idea what it was, except that you dressed up like a wizard or an elf when you played
Now that she has 4 or 5 sessions under her, she has gotten into it with the best of them! She bought her own dice, PHB 1 & 2, and is getting attached to her character.
She tells her friends about her Changeling Bard, and they kind of get a glossy-eyed look; they have no idea what she is going on about.
Secondly, many of the group I used to play with now don't have the disposable time OR have moved away from the friends they used to play with... some have found other people to play with, but like any organizied group activity, its not always easy to get all the ducks in a row.
And, thirdly, I agree that many other leisure activities are easier to pick up and leave (CCG's, MMO's,vidya games,movies).
After starting to play RPG's again after a long hiatus (thank you 4e), I have really come to appreciate what a great return I get for my time and effort. My gf and I found something else we enjoy doing together. We met a neat couple who we are starting to do other things with. We are getting to be proactively creative, unlike most of the passive entertainment out there. We are gettting better at basic math and I am becoming a mentat from keeping track of combat conditions/bonuses (thank you 4e

).
But most of all we are having so much fun!