This is the "stochastic parrot" theory: AI Unplugged: AI Want a Cracker?
To your point, it's hard to "prove" sentience, and as I frequently point out, if it can fool one side into treating the other like it's sentient, does it matter? Philosophers argue it does.
That said, this isn't really...