5% of the time is rather excessive.
An automatic fail on the roll of a 1 makes sense, yes, but having to roll when your skill already beats the DC is a bit excessive. Maybe houseruling that a player must have a skill of at least 1.5 times the DC before they can skip the roll altogether would work, or just ruling it for those skills where it would make sense.
My concern when asking this question was more that me and my friends are playing the adventure Keep on the Shadowfell, just for the purposes of learning the game. Thankfully most of us were quick studies. Next session, I'm taking over as DM and also running my character at the same time (but lets not change the subject) and we're about to go into area 3, where there are planks that break under the weight of heavy armour. Two of our players wear heavy armour, me being the third, but I already know about the planks, and I was hoping to use the athletics check as a cue to the other two that walking on the planks is a bad idea, because I know these guys: they're not going to think of that on their own, and the pit they'll fall into has guard drakes. I know it's only a 5% chance, and it doesn't make much of a difference, but it'll be the first time any of us have had to do a long jump and I'd like to iron it out now, rather than when someone actually rolls a 1.