I would have guessed Traveller too (based mostly on Dragon and seeing books at hobby stores - D&D was the only one at local book and toy stores). Granted a very, very, very distant second. What do you think the second biggest RPG was in the late 70s early 80s?
Great question! I really think this deserves its own thread, given that we would be discussing anecdotal experiences with a lack of sales figures.
But I would set it up with a few parameters; I don't think it is helpful to think of a single period of the late 70s to the early 80s.
In the light most favorable to the original statement, let's start the "late 70s" as being 1977, when Traveller was introduced, and the "early 80s" to be 1986.
In that period, I would say that there are three periods.
1977-1979 The beginning.
1980-1982 The expansion.
1983-1985 The pre-crash boom.
The next thing you'd have to do is to decide what "D&D" is. If you count all of the D&Ds as one thing (including OD&D, AD&D, B/X, and variant rules that could be incorporated including Arduin) you'd have one giant market comprising the vast majority of play; but if you begin to sub-divide it, then you'd begin to occupy almost all the spots on the "leaderboard."
Then, you'd have to worry about hybrid and homebrew, especially the farther back you go 1974-1979 was the primary time for homebrew and DIY games, not to mention hybrid wargame/RPGs.
With that in mind, I vaguely recall the late 70s having a number of briefly popular fad licensed RPGs, and there was a lot more interest in Chivalry & Sorcery as a variant to D&D than Traveller (which was not available where I was). The vast amount of coverage and play in "the beginning" went to Gamma World, which was promoted by TSR and readily available at local stores in the familiar box.
By the expansion, we are looking at early CoC and Champions as well as additional TSR offerings (Top Secret, Star Frontiers etc.) and the Star Trek rpgs and MERP.
The pre-crash boom brought Palladium, Paranoia, and a plethora of RPGs that don't start with P that I don't feel like listing, because there's a lot. I'm not even getting into the Steve Jackson games products, which dot the years listed and had decent distribution.
I don't consider my personal experience to be controlling; is it possible that, in one year (1977, for example), that if you removed all TSR products, Traveller was the second-best selling RPG? Sure, maybe, who knows? I wouldn't assert that as fact.
(I could add the appearances at major conventions as well, but conventions back then were much less .... major ... than they are today)