Reply to thread

Message

<blockquote data-quote="ichabod" data-source="post: 9292875" data-attributes="member: 1257">The problem here is that they don't have access to the training data. This is mentioned in the article. So how can they say that it's not in the training data if they don't have the training data?I did not find the description of their method very rigorous. They have created a layer of abstraction removed from the actual processing to make their case. They use that abstraction layer to assume that LLM already has understanding. Then they show that this understanding is growing faster than the error rate is supposed to drop according to statistical models. Again, note that they don't have the actual change in the error rate, they are using a statistical proxy. It's not clear to me how dependent these models are on the particular LLMs, and how much bias could be introduced by using the wrong model.I think methods are rather weak, and full of a lot of assumptions. I am not convinced at all.</blockquote>

[QUOTE="ichabod, post: 9292875, member: 1257"] The problem here is that they don't have access to the training data. This is mentioned in the article. So how can they say that it's not in the training data if they don't have the training data? I did not find the description of their method very rigorous. They have created a layer of abstraction removed from the actual processing to make their case. They use that abstraction layer to assume that LLM already has understanding. Then they show that this understanding is growing faster than the error rate is supposed to drop according to statistical models. Again, note that they don't have the actual change in the error rate, they are using a statistical proxy. It's not clear to me how dependent these models are on the particular LLMs, and how much bias could be introduced by using the wrong model. I think methods are rather weak, and full of a lot of assumptions. I am not convinced at all. [/QUOTE]

Verification