Dragons shouldn't be that big a deal, imo. They shouldn't be intelligent, or particularly powerful, they're just another monster. In the old medieval European legends they were beasts that guarded a horde of gold, they weren't the ultimate evil of the universe.
I love the European legends that you are trying to draw upon but as I recall in most of those tales the reason the shining knight had to go defeat it was because it was such a threat. It would kill all who came to battle it, except that one knight. To me that means resilient and powerful. Many of those tales tell of brilliant dragons who are outsmarted, that tells me intelligent. There are dumb (wyverns) or less powerful (drakes) dragons already. Keep those, separate them from true dragons and make true dragons more frightful again.
Oh, that also says nothing of non-european dragons. Even norse dragons don't fit. Asian dragons are either always a threat or remarkably tiny.
They're big reptiles, they can fly, they breathe fire, they hoard gold and have a taste for maidens. They represent greed, if you want to get fancy. They're tough, too tough for a level 1 party, but they're not epic level tough. By the time you're planehopping you've outgrown them, they're a terrestrial foe.
They aren't reptiles, at least not as they've been used in DnD, not for a long time. They should be tough, they shouldn't ALL be godly strong. I do agree, in varying amounts, to everything else you said here.
But I've never particularly liked dragons (especially the colour-coding), so my view shouldn't count for much. A lot of people really love em, I know.
I couldn't agree with THIS part more. In fact
Link.
Eh, it's just because it sounds good with the alliteration, it doesn't mean anything. How the heck does a dragon fit in a 10 ft corridor anyway? Dungeons and dragons don't reallly mix.
In most campaigns I've run or played in, the dragons live in their own rooms within the dungeon. The tunnels leading to them they simply do not travel.
I have run that the dragon carved out the tunnels by his traveling (burrowing) through them. That is one explanation. You can squeeze into a hole half your size - so 10 foot corridors.. 20 (space) wide dragon.
Often other holes into the central chamber exist, such as from a skyward access, or burrowed holes, or magma chambers, or subterranean waters, etc.
Firstly, I agree that dragons have never been done perfectly, but that some earlier versions are better than the (core) 4e dragons.
With regards to spellcasting, I think most dragons should have certain spells at-will instead of using a spell system like Vancian or sorcerer. Some old dragons could be wizards or sorcerers on top of that and have more usual spellcasting ability. I'm not sure how it should be implemented exactly, but maybe a short spell list for each color and all those would be available at-will.
I think a part of this has to do with the options available to creators in every edition. Especially in 3.5 - if they are already strong, can fly and have a breath weapon. They are not going to start sneak attacking like a rogue. That leaves spellcasting in one form or another. I can not really speak to dragons in other editions as I have not played them but I can imagine similar limitations exist for nearly all issues encountered.