Actually I probably don't know. I don't memorize every racial ability, I don't memorize every character sheet, and I have no problem with a player reminding me that they have a particular ability.
Sure, but I, as DM, don't memorize ever element of every player, I often have 5+ at my table. If the elf wants to "remind" me about his ability by saying "my elf senses give me a good feeling about finding a secret door in this room" or something to that effect, I will then actively keep it in mind.
You realise that you forcing your player to ask to sense secret doors all the time is what would consume more game time right? If you know he has the ability then you only need to roll once when you know there is a secret door he might find. That's hardly slowing the game down.
Olaf nails it, to me. This isn't that hard. But if you are going to place the burden on the player, then I think it is reasonable for the player to remind you whenever the Elf is in position to see a secret door, not just when he has (in your opinion) a good reason to believe a secret door is there.
This doesn't have to consume huge amounts of game time. If the Elf player notes when we leave the room that, as always, the Elf is alert foir secret doors in the passages they traverse, and comments in each and every room that, in the course of searching the room, the Elf will circle the exterior within 5' of the walls, where he will have a chance to notice any secret doors, then you have your reminder and he has his ability. If this is becoming frustrating tio the GM, the answer is to remember that elves can see secret doors without conscious effort, and tell the player you are aware of the ability and rolling the checks. Of course, then you have to do so.
Personally, if I knew my DM wasn't checking to see if I spotted secret doors, I'd remind him (politely) of my ability. If he told me that it wasn't something he was going to automatically check for and said that I'd have to ask to use it, I'd start to start to constantly search for secret doors.
As another player in the group, I might start adding reminders myself. "I wonder if the Elf sees any hidden doors". "If only we had a character who could detect secret doors (or evil, or magic, or what have you for various characters).
By that logic we could say that asking the player to take any action slows the game down. I care less about speed and more about the DM burden, speed only becomes an issue when the player starts becoming absurd about it.
At what point is it "absurd"? The elf has the constant ability to notice secret doors. Either that means you remember that ability and roll for it, or the player is justified in reminding you of this ability whereever it may be relevant. That means everwhere there are walls, not just when a fleeing opponent has mysteriously vanished in a dead end.
It's not a matter of forgetfulness. It's a matter of priorities. If there's a "secret door" that is of any value to the party, I will set a DC appropriate to how hard it is to locate after having said something like "You have chased the goblins into a dead end, there is seemingly no way to get out." And then let the players decide how they want to investigate further. I had one half-giant with an oversized hammer whose solution to hidden doors was the "hit it" method.
So do I have to TELL you I am using my Listen skill in case there is an invisible assassin skulking in our midst, or do I get an automatic roll for a passive ability? Because the Elf's ability to see a secret door is just as passive as the human ability to hear someone moving about, or spot something shiny in a haystack. "I use my Spot skill - I rolled a 5". "OK, you see that there is a town 30 yards ahead. Good thing you rolled - you would have walked right into the town walls." If my character can read Draconic, then I shouold be able to read it without switching my skill on. If he has knowledge of the local area, he should not have to chant a mantra to activate his memory that the person we just met is the local blacksmith, or the daughter of the mayor.
Ideally, the GM would remember those abilities, but the world is not ideal. Given this, I would expect players not to be annoyed they must sometimes remind the GM, especially of abilities that don't come up all that often. But I would also expect the GM to acknowledge that those reminders are reasonable, nit "imposing a burden" or "wasting game time". And I would also expect the GM to attempt to make amends where an ability is overlooked, not say "well you just didn't think about whether she might be the mayor's daughter for the past few hours".
I expect my players to be reasonable folk. As such, if you feel it is reasonable to suspect a secret door within your elven proximity, by all means, roll for it. I'm not a big fan of asking the DM to be able to do stuff a character can do, but I'm even less of a fan of doing it for them. I'd much rather my player take the inititive and say "My elven senses are tingling, this hallway feels odd, I'm making a secret door check." To which I'd go "okay", check the map for secret doors near them, and then see how they roll.
I expect my GM to be reasonable too. As such, if my character has a opassive ability, I expect it to work passively, whether by the GM remembering it or by him accepting that constant reminders are made necessary if he finds it too great a burden to remember. It's not practical to expect the GM to ask "does anyone have the ability to detect secret doors" whenever we pass one. It is practical for him to ask whether anyone reads Draconic, or what languages everyone can read, or what everyone's local knowledge skills are. Most GM's I've gamed with recognize when certain abilities are likely to come up near term, and get that info from character sheets in advance. Some track a lot more info, and others don't track much. Those that track little accept that there will be a lot more reminders in the game session.