Would you say the same thing about, say, strength? That a 3 strength wouldn't be noticeable? The monster with the lowest intelligence that I can think of that can still speak is an ogre with a 5 intelligence. Intelligence might not be visible, but I think it should make a difference.
Whether it as an impact at the table is up to the table and what they enjoy.
That's an illustrative example. These debates often end up comparing mental challenges to "picking up a weight", but those two things aren't even remotely comparable.
Here's a scenario:
Alfred and Betty both attempt an Athletics test, and they both succeed.
They attempt a different athletics test, and only Betty succeeds.
Does Betty have a higher strength than Alfred? If so, how much stronger?
There's not enough information to really answer that, is there? It could be that the first test was really easy, or maybe it was hard and they both got lucky with the dice roll. Maybe the 2nd test was medium, and Alfred got lucky but Betty rolled a 1.
On the other hand, if we knew that the first test was to pick up a 10 pound rock, and the second test was to pick up a 50 pound rock, we would start to have some information. I would probably guess that Alfred has a pretty low Strength score since he couldn't even pick up 50 pounds, but we still wouldn't have a lot of information about Betty. At least as strong as is required to pick up 50 pounds.
The problem with Int, Wis, and Cha, however, is that there is no unit of measure to express the difficulty of individual tasks. Take a dungeon puzzle where you have to realize that a message is an anagram. How many "mental pounds" does that weigh? In the real world, it could very well be that the brain surgeon fails to see it (because her brain is busy doing numerology on the letters and noticing that when you square the numbers and take the modulus of a certain prime number it
almost produces the fibonacci sequence) until the coal miner announces, "Oye, that actually spells KILROY WAS HERE."