It makes sense to me - I know all about Appendix N but having a read a few of the books and stories, sometimes actually finding those inspirations in the actual setting is hard unless one looks at the Vancian magic system, or just the pure existence of a thief class or a barbarian class, etc.
And let us not forget how
weird a lot those appendix N books were. It was less that there wasn't powerful magic in Conan's world, but the protagonist was not a weilder of magic and magic was rare and powerful societies with magic/advanced science we ancient fallen civilizations whose ruins Conan explored. Fafhrd and the Grey Mouser encountered powerful magic, it is just that they were generally the unfortunate targets who used bravery, recklessness, charm, and guile to come out ahead, or at least escape from, their encounters. Don't get me started with the Dying Earth setting, which was by no means low magic. It depends which book you were reading. The protagonist for one series in the setting may be a clever rogue, but in other books you had a cabal of powerful mages flying off to other worlds and even to the end of the universe.
And those weird influences were on display in Gary's adventures. Crashed spaceships, D&D versions of Alice in Wonderland, and all manner of weird encounters in gonzo dungeons. People often write about how Gary didn't like wizards and put in all manner of ways to destroy a party's magic items, cause them to lose levels, and make it very long and hard process to get to high levels. As well as his focus on humans, putting restrictions on demi-human characters. Some seem to take this to mean that his vision of D&D was low magic and more gritty realism. But looking at his adventures, it obviously was not. I think, at least early on, had a vision of characters against powerful, weird, mysterious forces that they had to be clever to best or avoid.
Yet, even that is not the full picture. Keep in mind that many of the spells in D&D were created by players in his game. Reading Greyhawk and seeing it peppered with names and accomplishments of the PCs from his home game, gives a hint to some of the high-powered, gonzo, shenanigans players got into in the games he ran.
I often feel that when talking about Gary you have to split him into Gary the TSR businessman and Gary, the guy who loves to play games. He has written on many occasions that he ignored many of the rules in his home game. All those weather-generation tables in Greyhawk? He didn't use them. He published tools that DMs might find useful and which TSR could sell and make money from. So he had to be a sales person and lean into how useful and even necessary they were (to the point of getting pretty offensively wrong-bad-fun in some of his letters and articles in Dragon Magazine), but he ran his own game how, in the early days at least--and again later in life--he expected everyone would. Take the rules you like, ignore those you don't, and make the game your own.