Why?
Sales. Because D&D is no longer profitable in its current state.
It takes a lot of time and money to make a new edition. Development costs are high, since you're paying your best people to work on a new game for several years.
And it's a risk since no edition change has a 100% conversion rate. You risk losing more people than are currently buying.
So it's not something the business does on a whim.
And it's certainly not something done because the designers want to do something new and scratch their game design itch. Or because they think they could improve things.
Dirty secret: no game is ever perfect. Perfect balance is an illusion. Every game will have warts and flaws and proud nails. 6e will have giant flaws that grate on the DM & players after a while. So will 7e. And despite decades of greater game design experience and wisdom, I can guarantee that there will be problems with 8th Edition. The designers of 8e will look at their creation six months after launch and say "oh mannnn, I wish I could go back and fix _____."
Recuse making a game is a creative endeavour. Artists look at their art and see the flaws. It took Da Vinci four years to make the Mona Lisa and he reworked it several times and still wasn't happy with the final product.
When?
2017 was the best year D&D has ever had. And 2018 is shaping up to match that, if not beat it.
Let's go hypothetical and say this is the apex. And things decline rather than plateauing, doing so at a rate comparable to its growth. Even then, it should be able to go another 4 years before things get bad enough to warrant a new edition. Because even falling to 2014 levels would mean D&D is kicking all the ass. Sales wise.
That means we can expect 5e to be successful well until 2022. Easily. And likely far longer, since it'd have to drop well below 2014 sales levels to be a failure. It'd have to decline for several more years before things get bad enough to risk a rebook.
But let's consider what happens if sales just spike downward after this year. People stop buying.
They work a couple years ahead, so they couldn't just stop production. And they'd want to wait a little to see if the drop corrects itself or was temporary. Plus, it takes a three to four years to make a new edition. If they start work on that a couple years into the downturn—which is the earliest they would likely do so—that would be 2023.
But... their first reaction would not be to make a new edition. Their first reaction would be some new initiates to boost sales and reinvigorate the edition. So they'd try and reverse a downturn by 2020 and see if that works or fails by 2021. This would delay 6e to 2024.
Again, that's the earliest.