Agreed. And while representing memory with a higher Int score would be perfectly logical, it would also be...boring. I'd much rather see a cleverly designed racial ability that specifically applied to memory.
And the same with Dexterity: three races (elves, halflings, aarakocra) all have their agility (grace) represented as +2 Dexterity. Which is fine (sorta) but leaves us with no differentiation. They are all dextrous in exactly the same way? I think it would be more fun to have active special abilities that bring their uniquness to life, and that get intentionally used in specific situations, instead of just a passive +1 applied to all Dexterity rolls, all the time.