D&D amalgamates different but similar concepts with bards. Similar concepts existed in cultures all over the world, and the bard role in Western Europe that we normally look at is vastly over the span of a couple of thousand years. Differences over time and location are juxtaposed into that amalgamation in some cases.
Imagery required instruments to be included in the pictures otherwise the image wouldn't be clear it was the bard, and beyond that the roguish wandering minstrel has been around in stories for some time; that was folded in to the bard class. It was rather prominent in 2e, which was little like the 1e version or any other version because it only had mage spells. That version typified the roguish minstrel wanderer who simply picked up magic along the way the most and continues to be a typical archetype with which players are familiar.
Music during battle was used historically during fighting to inspire troops, decrease morale in enemies, and convey orders in battle (communication vs miscommunication over poorly heard verbal commands). It's older than old as a concept and is included in the Old Testament's Book of Joshua against Jericho, documented by the Romans facing the Celts during the Roman invasion, used by Roman and Greek armies early in recorded history, and Scottish bagpipes were even outlawed by the British for a while in 1746 due to their effectiveness.
The image of the heroic warrior-poet in battle was romanticized by French poets and that's the image that should be conveyed when fighting in battle. It could be someone using an instrument on the sideline instead of a weapon or it could be a 6'4" 240 lb warrior singing a battle hymn while swinging a claymore to inspire those around him. Both fit that style. It's not dancing and prancing unless someone chooses to portray it that way and an instrument isn't needed. An instrument wasn't needed in 2e, 3.x, 4e, or 5e no matter how many images there are or people pushing the concept. That's always been a choice with oratory options also existing.
I think the better question is why a person would visualize something goofy instead of something heroic, but I think many of the images that included playing the lute did come across more on the goofy side as you mentioned.
The inspiration, magic, some combat training, healing, etc can all be traced back to folklore, legends, and history. It's not a D&D invention like some might think, and it's not the comical version unless that's how a person chooses to portray a bard.
