1) There is no way anyone is going to be able to perfectly and accurately model the entire agricultural schema of every random town on the off chance one of their players is a Food Historian.
No, but not having tropical plants growing in a non-tropical region is a pretty basic start. Add in the types of trade networks we saw in the Late Middle Ages, combined with magic and the majority of things are handled.
2) If you do try to be 100% accurate for some reason, what about the architecture? What about the dialect? What kind of disease do they have? What other minutia are you going to bury yourself in in your unstoppable hatred for halflings?
I keep telling people, I don't hate halflings. I think their lore is poorly supported. That's it. I have no hatred for them.
As for the rest, I do dabble in architecture sometimes. Nothing helps highlight an alien fantasy world better than people not building the same types of houses people are used to, or changing chairs. Dialect would be cool, but I have no voice skill so that is out. Diseases I've toyed with, but they are so easily eradicated with the preponderance of healing magic that generally they don't show up unless someone has made a magic plague.
3) What about flora and fauna that doesn't exist on Earth? Or mutations of those that do? Our species punched the noble wolf into teacup poodles and random grass into corn before we knew what a gene was. How can people with magic and alchemy not make some crap that exists outside of the scope of making a setting 'Earth, only elves'?
And hell, the thing about corn? The native Mexicans didn't know they were using technology to make it edible; its was just a coincidence of their water supply and cookware. Why couldn't those rice growing halflings just happen to have stumbled upon a method of growing water-sipping rice?
Could have. I think that is a fascinating thing to explore. In fact, I've spent years trying to gather a collection of magical flora to add into my games. Fauna less so because it is already pretty full.
But, there is a large difference between me thinking up the hows and whys of a race of people making a grain similar to rice but using less water, and me saying "Boom, desert Rice, because fantasy"
One is treating the world like a real place, with rules that can be understood and followed. The other is the equivalent of throwing balloons full of paint at a wall. I was being presented with "It's fantasy, no one cares." Which is ludicrous beyond belief, considering this is the fandom that created the Draconomicon, that had in depth sketches and organ descriptions for ten different types of dragons, how their eggs worked, their flight, their breath weapons, ect.
4) Improv leads to the best worldbuilding. That's why there are no elephants on the World of Ere. There are mammoths, there are phiomia, but there are no elephants.
Improv can be a lot of fun, but it doesn't mean it leads to the best results. It might for you, but I enjoy world-building too much to just make it up as I go along without exploring it.