A lot of good answers to this so far, but to give a macrosociological perspective on it:
Our world has consistently grown more and more interconnected as globalization sweeps across the... err, globe. There are many, many, ramifications to this, some positive, some negative, many where the jury's still out. One unequivocal positive of this, however, is our ability as a species to connect with regular people from all walks of life across cultures, nations, religions, experiences, etc; which has allowed more people the opportunity to grow in their abilities to imagine the other complexly. The more different types of people we get to meet and share stories with the more we come to realize that they are not good people or bad people but just people. In a world where significant political movements are growing increasingly reliant on painting entire groups of "others" with broad, largely negative, brush, stories that perpetuate these (provably false) narratives begin to chafe more and more against our actual lived experiences connecting with these very same "others".
At the same time, fantasy literature has grown a long ways from Tolkien's orcs. Seminal series from the 90's such as Wheel of Time and A Song of Ice and Fire, for instance, began subverting our expectations of others, from human cultures like the Aiel, Seanchan, or Dothraki, to non-humans like the Ogier. Warcraft 3 in 2002 turned everything we thought we knew about the orcs in that series on its head, and just a few years later Eberron brought a very different type of orc and goblin to D&D. With the global pressures noted above, it was only a matter of time for the rest of D&D to catch up. Honestly it took longer than I expected.
Tl;Dr: Changes in the global socio-political landscape began necessitating more complex narratives involving sentient, free-willed mortal beings to better align with consumer values.