The fastest way to get the Dragonrider class is 9 levels of Paladin. Because you can get a Dragonish mount (The Dragonelle) at level 9 with the dragon mount feat. This is particularly important if you're a Medium character like a human. Note that having a dragonish Paladin's Mount means a greatly reduced progression on Paladin's Mount abilities so it'll probably wind up weaker than a horse would anyway.
It's not that the prereqs are harsh, they're easy to get, but it's really hard to get a dragon cohort big enough to carry you.
The lowest ECL of a Medium Dragon 12 for a Young White Dragon. If you want a non-evil dragon you have to wait for a Brass Dragon at ECL 14. If you're a medium character the ECLs go up to 17 and 18 for Evil and Good Large Dragons, respectively. You can reduce ECL by 3 with the right feat but even so, given the way leadership works at best you can get that ridable dragon by 11th level unless you pick a race that's smaller than small. If you're medium and non-evil you'll probably need to reach near-epic levels to get a dragon you can ride without the Paladin's Mount trick.
I don't think the class is broken for straight-up combat because it's so hard to find a halfway decent dragon cohort (barring a purely role-played mount of course). However Dragons have a very powerful blindsense that renders ambushes, sneak attacks, and the like very difficult.