The text absolutely does limit ability checks to such situations!
Your lack of understanding of what the sentence refers to is the issue.
Here's the full text (with the relevant part in boldface): "An ability check tests a character's or monster's innate talent and training in an effort to overcome a challenge." That emphasized phrase limits when a creature's talent and training are measured by an ability check.
The 'effort to overcome a challenge' is
not being consciously taken by the creature! The 'effort' is being made by the creature's 'innate talent and training'.
Only "in an effort to overcome a challenge" is the result of an ability check applicable to the outcome of the situation in question. The corresponding, also true statement is that an ability check does not test a character's or monster's innate talent and training in the absence of an effort to overcome a challenge.
The 'test' is of the creatures 'innate talent and training', which exist whether or not the creature is consciously trying to use them.
A creature's senses
are constantly making an effort to notice threats! They don't stop trying when you sleep! They don't switch off! Even closing your eyes fails to keep out all visual stimuli; you can still see the difference between brightness and darkness, and the change from dark to bright can indeed wake you up!
A Perception check represents a creature's innate talent (how sharp its senses are) and training (if the 'effort' to notice threats has moved from the conscious to the sub-conscious mind) with regards to reacting to stimuli.
The correct 'corresponding statement' would be that 'an ability check does
not test things that are
not part of a creature's innate talent and training'.
So when considering whether to call for an ability check to determine if a sleeping creature notices an approaching attacker, the DM should take into account whether the sleeping creature is making an effort to do so. Considering that the last action declared by the creature's player was for the creature to go to sleep, an action that is actually the opposite of staying alert and aware of one's surroundings, the answer is that no ability check is warranted in making that determination.
The answer is that the creature's SENSES
are 'making an effort' to notice an approaching attacker! It's those senses, that innate talent and training, that is being tested, not the creature itself.
You even quoted the part which says so! It doesn't say "An ability check tests a
character or monster", it says "An ability check tests a character's or monster's
innate talent and training".