Of course, noone is actually going to be unable to cast their top-end spells after being hit by Touch of Idiocy.
Imagine, if you will, that you started life with a 17 in your spellcasting stat. With Gray Elves or Point Buy it can be higher - but we'll assume 17. Now you are 4th level. Your Spellcasting stat is now an 18 - you lose 6 poins of it and you can still cast 2nd level spells - your highest.
Move on down the line to 6th level. You can afford a Headband of Spellcasting Stat +2 - and would thus be foolish to not have one. Your Spellcasting Stat is 20, so the maximum penalty brings you to 14 - which is enough to cast 4th level spells except that you can't actually cast those because you aren't high enough level.
At first level your spellcasting stat starts out about 6 points higher or more than the minimum needed to cast your highest level spell. Dedicated spellcasters get spellcasting stat at a rate higher than 1 per two levels, so Touch of Idiocy never catches up to the point where it could actually shut down spell levels.
-Frank