What part of remembers OTHER lives did you miss there? Also, they taught the humans magic. They would have been around to see them become the magic empire, and they would have talked about it with other elves. Those memories would be all over the place.
Why do they need to know that to know...