When LotR was written, it was rare to acknowledge that your story was set in a totally invented world, even if it obviously was. The claim that Middle-Earth is a "prehistory" of the real world is the sort of fig leaf used to excuse such invention. I don't think we need hold Tolkien to any particularly high standard of plausibility here.
But it's fun to try and reconcile these things. So, let's suppose that Middle-Earth really is a pre-historical version of real Earth. (Note: Throughout the following, I am speaking from the point of view that I see Tolkien espousing in his work. I do not agree with this point of view, particularly as regards technological progress! But Middle-Earth is a world where moral forces influence physical reality, so we can't incorporate ME into the history of real Earth without accepting the nature of those forces as Tolkien described them.)
1. The technology shown at the end of the Third Age is far beyond that of the real-world Bronze Age, which in Tolkien's chronology came afterward. We see highly developed steel arms and armor, full-sized cavalry horses, etc. Therefore, the technology of LotR must somehow have been lost.
2. The history of Arda involves a general theme of decline. The world of the Third Age is diminished from the Second, the world of the Second Age is diminished from the First, etc. This follows the idea put forth in "Morgoth's Ring," that the corruption of Morgoth was a poison steadily working away at all of Arda, even after Morgoth himself was cast out.
3. In the real world, technology tends toward an "industrial" mode (standardized, hierarchically organized, overseen by elites and closely joined with the interests of the state) when a society becomes large and stable enough to make this practical. See for example China and ancient Rome.
4. In LotR, industrial-style production is consistently linked to the powers of evil: Sauron and the fallen Saruman. So we may assume that what we regard as technological progress is in fact a continued decline, Morgoth's corruption reaching toward its final ascendancy. (I think Tolkien actually said this at some point, though I don't have a reference off the top of my head.)
5. This means the technology seen in LotR is not the same as that of the medieval age, despite the obvious resemblance. It is purer, less tainted by Morgoth's influence. I would guess the swords, armor, and other artifacts were far superior in quality, manufactured by artisans of skill unmatched in latter days, who could and did devote a great deal of time -- perhaps years -- to each item. Since this would drastically slow down the pace of manufacture, the only way for a large stock of such things to exist is if they were preserved and maintained for a very long time. This is consistent with the reverence for heirlooms shown throughout the cultures of Middle-Earth, apart from Mordor and Isengard.
6. We may also note the extraordinary timespans involved in Tolkien's chronology, where it's common for societies to exist more or less unchanged for thousands of years, in stark contrast to the history we know. This, too, seems an indication of the purer, less-corrupt world of the Third Age: Preservation is easier, reverence for tradition is greater, and rapid "progress" is not as desired.
7. This still leaves the question of what happened to the relics of the Third Age. Why have we not unearthed the ruins of Minas Tirith or the foundries of Isengard? The answer may involve a catastrophe akin to the War of Wrath or the downfall of Numenor, which reshaped much of the world; after all, the geography of Middle-Earth doesn't match anywhere in the real Earth, so something drastic must have happened in the interim. It's also possible that Morgoth's minions at some point made a deliberate effort to obliterate Gondor from history, to wipe away for good all memory of Numenor and its children -- an effort which was thwarted once again by hobbits in the form of the Red Book, a copy of which survived and found its way down through the ages to a certain professor at Oxford.