I'm an active M:TG player.. I'm going to the LGS tonight to play Pioneer.
There's a few misconceptions going on here.
Lorwyn has sort of gotten the same treatment as 4E D&D. It was largely panned at the time. For one thing, it came out late 2007, right at the dawn of a recession, and it also introduced the new card type "Planeswalkers" which, at the time was the big bugaboo that was going to "Kill Magic". It did not sell well.
However, fans have really turned on it. Between the charming theme, the wonderful art, and the simple mechanics like "Creature Types Matter" that tend to stand the test of time, it's a set a lot of folks now look back on fondly. Including people like myself, who didn't actually play then.
It depends on how you measure popularity. By sales, it's nowhere close.
Spiderman was a bad set. A couple of years ago, WOTC tried making "Mini-Sets" with just 50 cards, sold in 7 card booster packs. The first one they tried was "March of the Machines: Aftermath" and it was hated. Loathed, really. About a year following they tried mini-boosters again with an Assassin's Creed tie-in set, and it also didn't set the world on fire.
Like Parmandur mentions below, Magic sets start development roughly 2 years before they launch. Spiderman was also originally going to be one of these mini-sets, but it was far enough out that they were able to see the reception and try to change course. It was still a smaller than normal set, and felt really disjointed because of the change. A lot of the cards are just bad. It doesn't fully support 10 draft archetypes (Most magic sets are designed to be drafted with 10 themes, based around the 10 two color pairings.). They also tried to push a new, faster, simpler, pick-two draft format to go with it.
It was just like a perfect storm to be hated.
-It's a Universes Beyond set, so there's already a big swath of people who skip it just because it's a non-magic IP. (Most of the guys I play pioneer with skip these sets, and just buy the single cards for decks they play if needed. We all got together during the Spiderman Pre-Release and drafted someone's cube instead.)
-It was a poorly made set, so the people who don't care about the theme/setting and just draft didn't want to play it.
-It doesn't really have any banger, stand-out cards, so constructed players are skipping on it.
-It didn't have commander decks, so while it introduced a lot of potential new Legendary creatures to build commander decks around, it didn't have the introductory, road-map product to help capture people who are interested in Spiderman, but maybe new to magic.
My LGS is flush with spiderman packs they can't move.
MTG is actually the most profitable part of HASBRO. They are thrashing trying to figure out how to reinvigorate portions of it.. Namely the Standard Format, which used to be the primary way to play the game.
They don't seem to have issues raising prices. Packs have only gone up in the last couple of years. Standard boosters have gone from $4-5 up to $5-6, and all of the Universe Beyond sets have a premium, and tend to be more like $7-8. If it's popular it's even more. I saw Final Fantasy packs running for $15 at some stores when it first came out and no-one had stock.
In my experience, the "Young" MTG players I interact with tend to be in their early 20's. Don't see many kids and teens at FNM.
They've been teasing a K-Pop crossover for a few months now, and I think I recently saw an article about HASBRO securing some merchandising rights for Demon Hunters.
It would probably end up being a Secret Lair Drop of a few cards, rather than a whole set though.
Yeah, there are players who enjoy the story, but they are a small group. They've quit selling books, and instead post short-stories on their website for each set.
UB also has the bonus of reaching non-Magic players. I've met quite a few new people who have gotten into the game because they love Final Fantasy, and I'd imagine there's going to be another wave with Avatar, which just came out last week.
I think you're spot on for the demo, but you're a little off on the game itself. The most popular way to play magic now, is Commander, or EDH. It's a 4-player format, that's more casual, and games tend to last 45-90 minutes. My LGS does two Commander nights each week, each pulling 40+ players, whereas we get 8-12 for Pioneer and 4-8 for Modern on Fridays and 6-12 for Pauper on Saturdays.
