Ok, I guess if it's a known irrelevant that makes sense but it's hard (at least for me it is) to know what is irrelevant to your players in the moment... I've had players pick up on something that was a minor detail and run with it for some purpose I wouldn't have fathomed and honestly if it's something I improv'd I just come clean and tell them I don't remember it... but I hate doing that and it tends to make my players feel as if I'm just pulling things out of thin air in the moment (which of course I am) and that's not a playstyle they tend to enjoy, especially once it's revealed to them.
Oh I've had my players take a throwaway comment and decide to make the entire session about that thing. It happens. In a case like that, the improvised detail becomes a more relevant detail. In a case like that, I tend to remember what it was that I said. Or if I don't, the players have since they are the ones who latched onto it.
I'm also open about forgetting details like that....I'll lean on my players to remind me of things, and they have no problem like that. I think it's a case of making up details on the fly more than making up everything on the fly. They're fine with the first, but can sometimes be a bit wary of the second.
All that said you could just be a better improviser than many, one of the things that gets looked over in these types of discussion around playstyle is that different GM's have different strengths and weaknesses. For some improvising the world may be a strong skill they wield and track with ease but for others it may be a weakness. For me personally coming up with stuff on the fly isn't an issue but tracking it all is a headache for me so I tend to rely on a moderately fleshed out world with smaller doses of improvisation at the adventure level.
I tend to outline ahead of time, but I keep it very loose. I have story ideas, and I have some campaign notes written down, but it's not a lot. My notes that I tend to use for any given session usually consist of a list of bullet points of things I expect to happen, or that are likely given my players and their characters. I try to predict and account for the most obvious courses of action, but there's never any way to fully predict what these maniacs may do, so I do wind up having to improvise a bit.
However, I find my loose outline to actually be a better tool to help improvise than a ton of specific details. I think it's good to have information to lean on when the players go off in a direction you did not expect, but I think having that info be broadly defined allows you to adapt things easily. So the thieves guild members stats that I came up with can instead become orc raiders when the PCs abandon investigating the thieves and go off in the hills to chase orcs. That kind of thing.
OAN: I will readily admit that the difficulty with tracking improv'd things may also arise from the fact that I and my group tend to enjoy a little scotch and often our fair share of beer when gaming...
If I did that, then the entire game would be improvised, and quite sloppily!