I agree, but I still think it is good to have a sense of how it could impact agency. Introducing example: I have prepared a cool ranger character. I have also prepared a tavern, and know the ranger sits in a dark corner of the common room. How do I describe the scene when the PCs enter the tavern in an optimal way from a player agency perspective?
If I mention the ranger in my first brief overview he sticks out as a sore thumb, hence likely providing a strong incentive for the players to engage with that. This is limiting player agency as it strongly points toward one spesific path of interest, and is particularly problematic as "realistically" the ranger might be where he is thinking it lowers his chances of being noticed.
On the other hand if you do not mention the ranger, you deprive the players of valuable information to make a decission. It might be that they would like to interact with the ranger, but not being aware of its existence they do not know that would be an option. Hence this approach also limits player agency in a way.
Delegating the decission to the dice by for instance doing a perception check also do not help with regard to player agency, it just provides more "agency" to the dice.
Describing everyone in the crowded tavern in a neutral maner like a "spot the hidden object" game is also a possibility, but it is impractical (and to most players boring) if you to describe everything in excruciating detail all the time - so the players would understand this is a hidden object game where something interesting is to be found, and again be lead to spend more time trying to find the interesting thing than they would otherwise have done.
I think an awareness of such interactions can be important. To call it an inherent "conflict" between player agency and prep might be a bit problematicly inflammatory language, but I do not think it is a completely unreasonable term to describe this phenomenom.