Those above said it as well, but I'll reiterate it in a way that I like (in case you're still unsure, or don't agree). From a game mechanics rule, it's that way because that's the way that all other similar mechanics work. The higher your level, the harder it is to spot or track or know about or whatever. Skill checks scale like that. However, so do your 1/2 level bonuses so basically as long as what you're doing is the same level as you, the difficulty doesn't change.
Does that mean an incredibly skilled athlete will appear to be the biggest clutz ever unless you're high enough level to tell that his tripping, eye pokes and self-inflicted wounds are really the signs of immense skill? Likewise someone with a negative dex mod while drunk with no athletics skill will be the most graceful and skillful person around and be first pick for teams?
No, it's backwards. It's 4E, don't try to make sense of it because there isn't any, just follow the rules.