Don't worry, you're not at all alone in this sentiment.
I have somewhere a book about art history written in the late 19th century, I'll see if I can dig it out later. It's written very much from the perspective that art can be objectively rated in quality, and classical sculpture was seen as since sort of pinnacle of aesthetic form.
The author, however, was struggling to deal with the, then recent, realisation that classical sculpture had originally been painted in what to his eyes, was garish colour. The white, alabaster aesthetic was that of the Renaissance, not the ancient world. How to reconcile with the worshipful view of classical culture?
The rather weak attempt he falls back on is to ascribe it to light. While in the climate of Northern Europe things obviously look better in pure white, the light in the southern Mediterranean is quite different, so the colour there probably works better than it would in the British Museum!