At the school where I teach (and run the D&D club), the cutoff age seems to be 8th grade (that's about age 13 for you non-Yanks out there). Younger than that, and the number juggling and bookkeeping seems to overwhelm them.
I started at age 12, but I'd already been doing tabletop mini wargaming for 3 years before that.