If the bard is not built around music, than what will? Music and musicians are an extremely important part of folktales, magical traditions, rituals, and the entire fantasy genre. The equation of magic and music is an extremely common and popular idea. It is something that deserves its own niche in D&D.
Besides, what can the bard possibly be other than a musician and entertainer? Any other definition would be based far too much on previous implementations of the Bard in D&D (which have always been bad) rather than something that has any kind of wider recognition. Put simply, if it doesn't pass the 'Shakespeare test" (since Shakespeare is The Bard), then it is not a good use of the term. Besides, every Bard I have ever seen in a videogame is either a singer or plays a musical instrument.
Another important argument is based on powers and game mechanics. The identity of a Bard as a musician would be far more useful and interesting for defining powers and class abilities than just about any other possible idea for a Bard. It is important to be able to depend on something like "affects everyone who hears the Bard". Even mixing in visual forms of performing like dancing would lead to problems with overly complicated mechanics (everyone must see you if you are a dancing Bard or hear you if you are singing bard). It is far better to keep things simple and focused. Non-performing forms of art would be even worse in this regard, since it is absolutely impossible that magical forms of music and painting can be both be represented properly with the same set of mechanics and powers.
As a whole, the more the 4E Bard resembles the Bards from Final Fantasy Tactics and Song Magic (particularly Blue Song Magic) from Ar tonelico, the happier I will be.