The notion that a human spends 1 year learning the sword and an elf spends 100 is kind of absurd.
If elves were so incompetent and so slow, they would have been wiped out centuries ago by wild animals and nature, let alone orc hordes, dragon attacks, etc.
How is this any more absurd than the idea that age gives you magic imagi-XP?

(Yay! I'm 70 years old! I can cast FIREBALL!)
They're not nessecarily incompetent. Have you ever imagined an elf swinging a sword the same way as a human has? To my mind, the elf would put the amount of effort and dedication into learning swordplay that a human may put into perfecting a lifetime of various skills. To them, it is a pinnacle of art, and to be waving around a blade like a human is distasteful and savage.
It takes time to turn swordsmanship into art. And elves turn EVERYTHING they do into an art. They're also rather unfocused in it -- those 100 years could be off-and-on swording while they're composing poetry, crying for months after a funeral, or spending days fletching a single arrow.
Elves are perfectionists. It takes TIME to make things look and feel that completely impressive. Anything less, to an elf, is hasty and sloppy.
They survive not because they're strong or fast, but because they're reclusive and secretive. They live alone in forests with nothing but elves and squirrels, consume only growing things, and generally stay away from danger and hardship and disease (since they don't last long in those situations, and there's no time to develop their skills). The womb of the elves was in peaceful forests that never knew the touch of axe or the blood of an orc. In addition, they have powerful friends -- fey, centaurs, dryads, nymphs. There's a lot of mighty creatures that like the elves. Powerful friends means that they can afford the time.
Also, there's nothing saying the first elves weren't born epic level and put in place all the bounds that guarded the budding race until it had enough powerful adventurers itself to take care of itself.
There is no reason to assume that elves HAVE to be more powerful than anything else.
-----------
Also, about the demographics/Time/XP issue...the rules assume that time does not give you XP. The rules DO assume that adventurers are common, and since adventurers all retire at different points, there are going to be adventurers that are NPC's that are of high level. These aren't commoners. These are people who raided dungeons back in the day, but lost a leg, so don't do much anymore (for instance), or decided just to make magic items for a while.
Any rule that gives people XP as they age is a house rule. And one that has no real place IMC (Happy birthday! You get a bonus feat!). Like Gez said, just because a fiend is a million years old doesn't mean it gets half a million levels of something because of y'know, bein' old and stuff.