You think so...
Hm...
OK, let's take the standard elite array (since we're talking about a PC): 15, 14, 13, 12, 10 and 8.
Next, we make our character 80 years old. And, by my average chart in the OP, this character is probably 7th level (though we all know that a PC at age 80 is probably approaching level 20 if not there already).
Next, we improve stats based on level. We get 1 pt at 4th and improve all at 6th. So, I'll improve my stats to: 16, 15, 14, 13, 11, 10.
I'll make him a Fighter, because that's easy.
STR 16
DEX 14
CON 15
INT 13
WIS 10
CHR 11
Now, we apply ageing modifiers from the PHB: -6 to physicals; +3 to mentals. Our 80 year old ex-fighter looks like this...
STR 10
DEX 8
CON 9
INT 16
WIS 13
CHR 14
I don't know about you...but that's still a pretty strong character at age 80. Don't forget he's got INT skill points to make him better at skill checks than he ever was before. In many cases, he can more than compensate for the Stat bonus loss by boosting a particular skill with skill points and feats like Skill Focus. This 80 year old could actually be better at Climbing mountains than he was at his prime at age 25!
Still think that the d20 system handles age appropriately?
So old age and skill beat youth and fitness?
Color me surprised.
My grandfather served in the Army, jumped from planes, shot at soldiers in two theaters of war. He came home and had screwed up his legs... A once pretty solid ballplayer took to the Trades. He kept up with his shooting until the age of 70, and had an encyclopedic knowledge of 'fixing things' from years of it. Could he lift out an A/C assembly or screw around with a rusty bolt? Nope. But he had the skill to tell the big dumb strong kids (and then grandkids, and would have kept it up not doubt with the great-grandkids) how to twist that wrench, or tote that load.
Skill points aren't retroactive, so you're not gaining an overly large amount of them overtime. The old fighter is slower, weaker, and more prone to illness, but has the mental acuity to get himself there until dementia or similar ailments plague him.
Sadly, my grandfather started failing his saves around 80, and I miss him dearly for the fact that he proved age stats well. He was always a bit of a loudmouth (wonder where I got that from?) and a bit of a joker, so not everyone 'got' him. As he mellowed out from his hotheaded days (wrapping up when I was around six or seven) he traded his cycles and Strohs for a nice comfortable living fixing broken electronics and diagnostics equipment and tending his garden. He stayed in shape but age gets us all, but he didn't stop learning until the dice started falling poorly.
When we finally had that last goodbye in the bed he told me all about his crazy life. Things I cannot even dare to repeat lest someone were to read this who knows my handle and it got back to the family

. Let us say he led a rich, fighteresque history, and definitely stood the tests of age well. I witnessed him beat a mugger within an inch of his life, throw teenagers by the wayside to help my cousin get out of a hectic idiocy-induced stupor called 'first lust', and kill a crazy new neighbor's insane Chows. He had reported the dogs to the cops after they killed a puppy of one of our neighbors, several farm animals, etc.
A 75 year old man, with an aluminum cane and a pocket knife, proceeded to beat down, stab, and generally open-a-can on two full-sized male bloodthirsty dogs coming after his woman.
I can also squickly attest to his, ahem, constitution scores, and the fact that more than one nurse 40 years his junior proceeded to talk him up while we would take him in for his treatments before his lucidity left him.
The guy may have been built Heroic. Hell, in my eyes he rolled straight 18s. But I'm telling you that I've seen some really tough old dudes and he's just my personal basis. My SO is a nurse and deals with geriatrics as their specialty. I've heard of men his age or older:
a.) Going out for long-ranging hunts in the Smokies, alone, for days and weeks on end,
b.) Climbing up on roofs and shoveling a foot and a half of wet snow without breaking a sweat,
c.) Fend off assailants with various weaponry,
d.) Charm themselves into all sorts of interesting situations.
When I was unemployed for a year after taking a hiatus to tend to the family after his passing I heard even more stories. I also began going to see various individuals, soldiers from WW2 and Korea. I once had the chance before my grandfather's passing to meet Frank Buckles (the 'last survivor' of WW1 in the US) when I was very young, and really have a lot of respect for the old geezers who allowed me to live in the lifestyle I do.
Old fighters are rare, but when they live their big GM in the sky sure as hell gives them excellent Point Buy
Slainte,
-Loonook.