I'm not entirely convinced that stat modifications are really the best way to represent aging. I wonder if it might make more sense for older character to do something mechanically similar to choosing additional backgrounds to represent their additional experience. Or, perhaps, choosing skills to gain expertise in. That's really what has happened. You've had time to burn through a lot of downtime activities.
I'm not sure how best to represent the effects on the body of aging, or from just being out of shape and out of practice. Flat penalties are simple, but they don't seem like the most accurate solution. I would almost go for something like atrophied class abilities. In fantasy stories, we often see things like old fighters who are still very deadly, but they can't do everything they used to. Or old wizards who can cast many spells, but no longer have a grasp on their most potent spells. Maybe you keep your proficiency bonus and your hit dice, but everything else starts to fade after decades of disuse. You're still a 13th level Fighter, but you've only got the class abilities through level 5.
At the venerable stage where you're so old that you need dedicated care, I kind of feel like that would best be represented with steadily progressing permanent exhaustion.