Would it have been better if D&D Next was a completely new game that the designers created from scratch?
Better is too subjective a term. I think some people would like it more, it would be more of a gamble though, look at how they screwed the pooch on 4e; people get upset when you change the things they love about the game. Right now 4e is their market, do they really want to risk losing those people, like they lost the 3.5ers. I mean when 3.5 was at it's height of popularity, they pulled the rug out from under it. Imagine when WoW was at the height of it's popularity if they decided to completely change it... bad business decision? Yeah.
With 4e, they were trying to cash in on the WoW market. With 5e, they are trying to keep the people that like 4e, and bring back those that liked the earlier editions that have mostly moved on to Pathfinder or other games. No matter what you do, you aren't going to please everyone. But, I do think it is better to lead than to chase.
And, I don't think the corporate drones they have at WotC and Hasbro are smart enough to come up with something completely new and different and wonderful. If they could have made a WoW and made billions of dollars, don't you think they would have? Disclaimer: I have never played WoW, I tried once and a rat killed me in a forest before I could do anything, so I gave up on it. It isn't my cup of tea, I am too slow for twitch games. There are millions of players of it though, so it is obviously very successful and profitable.
Pen and paper gaming is a truly unique hobby that combines social and tactical elements, it is not something that you can plug in a formula and just get oodles of money. Even if you hit on the perfect recipe, how many players could you attract? A million? More?
It's a tough industry, to succeed in it, I think you need to love the game more than money. What is it they said, "Do what you love and the money will follow"? I think that applies to RPGs more than most industries. Anyway, that is my 2 cents.