Interestingly, Albert Einstein is known to have said this (1933):
"The supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible, without having to surrender the adequate representation of a single datum of experience."
The musician, Roger Sessions, a composer, is thought to paraphrase this statement above by saying (1950):
"I also remember a remark of Albert Einstein, which certainly applies to music. He said, in effect, that everything should be as simple as it can be but not simpler!"
Then this in turn later paraphrases variously, including:
Newspaper columnist, Sydney J Harris (1964): "In every field of inquiry, it is true that all things should be made as simple as possible – but no simpler."
Yale Book of Quotations (1972): "Everything should be made as simple as possible, but not simpler."
Ultimately, the quotes are attributed to Einstein, albeit apparently via a chain of paraphrases.