AI/LLMs Navier-Stokes "Solved" by AI

[trappedslider]

Let us also add the fact that no matter what some people aren't likely to believe whatever OpenAI says unless it's to confirm what they already believe.

 

[The Firebird]

The questions I have:

• Who is Noam Brown?

Noam is a prolific AI researcher working for openAI.

• How did OpenAI learn about Buckmaster and the NS results? High level Math can be a small leaky community so I could see some basics getting around.

Iirc Buckmaster approached them, cagily, hinting he had some results which turned out to be shy of the full NS solution. It sounds like OpenAI figured it was the solution and went for it.

• Do we have the technical details of exactly how OpenAI set up their agents and prompted them?

I don't believe so.

• What all did they check to ensure it was highly unlikely? Who performed those checks? What additional checks can be confirmed?

I assume, but do not know, they were checking the details of training for their unreleased model that solved it. For example, it could have been the case that the model, unknown to the researchers, had recently been trained on codex chats. This turned out not to have happened.

 

[FrogReaver]

Let us also add the fact that no matter what some people aren't likely to believe whatever OpenAI says unless it's to confirm what they already believe.

And some people are likely to believe whatever OpenAI says. I don’t quite see the relevance of bringing up either kind of person here. The question before us is about the provenance and verification of the Navier-Stokes result and possibly about what that means, not about who trusts or distrusts OpenAI.

 

[Justice and Rule]

The timeline that makes sense to me is:

0) Noam Brown reports they have tried Millenium problems without success, but haven't throw much inference at them (Aug 28)
1) OpenAI learns that Buckmaster has some exciting NS results. Reasoning that problems which one LLM can do, multiple can do (which has been shown via previous mathematics results), they start throwing a lot of inference at NS to get a result.
2) Their run succeeds in ~88 hours or so, and they report positive results to Buckmaster. That prompts Buckmaster to question them about the training leakage.
3) They consider it highly unlikely for such an event to occur, but it takes time to verify, so they hedge a bit shy of the full statement.
4) After having the time to look into, they categorically deny it.

I don't think OpenAI comes off well in that story, because it is clearly true they made a big rush to scoop Buckmaster, even in the absence of using data from his chats. And I think the point from the intern is very much worth listening to.

That said it is also distinct from straight reading the chats.

I apologize, but that feels way cleaner and more sanitized than reality typically is? Everything here seems to give OpenAI the benefit of the doubt, and I'm not really convinced given their history that they deserve it. The technology can be good, but that doesn't mean we should start ignoring the track record of the people who use it and typically do so in quite unethical forms (as has been pointed out on this forum before).

That OpenAI (Or, I suppose, Sebastian Bubeck, if we want to be specific here) said that this was just something they put in the problem and had "very little human input" (something that I don't think any statement by either the man or the company has actually refuted) is perhaps the biggest red flag for me, because why lie about the method in a private conversation if you aren't trying to hide something? To me, that's to try and cut off questions as to the actual method of solution, and I wonder if Buckmaster hadn't challenged them how much we would have actually learned about it.

Along similar lines, I think Buckmaster is absolutely right to question the prompt that they gave to solve it. When he says that is an esoteric prompt not easily reached, I find that rather compelling and OpenAI hasn't really given us any look at how they got to that prompt in the first place: they have mathematicians, but none that specialize in the area that N-S inhabits, and none that had done on any work on it before. If the machine came up with it, we should be able to see the logs and such, right? Or have some insight into how they came up with it? I dunno, maybe I'm asking too much, but I feel like they should be able to better address this concern and they just sort of move past it.

There's more in Buckmaster's statement, but the overall response from OpenAI has been to brush off the accusations and to say that they investigated themselves and found nothing. And that's generally not convincing for any organization to say, but doubly so for one that feels like it has been operating past the limits of ethicality for a while and whose best-case scenario is that they were only guilty of trying brute force a problem to claim credit before someone else could.

Again, this is not an attack on the usage of the techology: Buckmaster is rather eloquent in his defense of the usage, though he also makes a good point of how most of this stuff doesn't come out in useable fashion (He remarks that the Euler solution is very much "slop", something that needs to be understood and refined by the people who properly understand the math to actually make use of it), and that is a whole other conversation to have on releasing these sorts of solutions without really understanding how we got there. But again: the technology has proper uses, I just don't really see why we should expect OpenAI to operate in a fully ethical way on this when past behavior would indicate that they typically don't.

Let us also add the fact that no matter what some people aren't likely to believe whatever OpenAI says unless it's to confirm what they already believe.

I think any statement by OpenAI (and most AI companies, honestly) shouldn't be taken at face value without ample proof. They're just really prone to omitting key facts (or costs, when it comes to saying when they'll be making a profit in the future) or fabulize (Hey, AGI is here, right? Right?!). It doesn't help that Bubeck has kind of done that sort of thing before:

This is stuff that they are known to do. They build hype, because the whole money machine runs on that right now and anything that threatens that hype needs to be dealt with or dismissed. So it's not that I'll never believe them, I just don't believe their statements without outside confirmation or solid proof that goes beyond just their word.

And that's the bigger problem here: this is, at it's heart, still a "he said, she said" sort of case, where we have to take the people at their word and their integrity. I don't know much about Buckmaster, but I've not seen a lot of people say he was untrustworthy. The same can't really be said of OpenAI (or, again, Bubeck specifically; it's wild that his former intern literally made a twitter account to say "Yeah, this tracks", latter confirming his own identity on his website as well as through his colleagues). I am much more willing to buy Buckmaster's version of how conversations went and his logic (especially given his expertise in the area) versus OpenAI's version (which, again, doesn't refute them initially telling lies to Buckmaster about how they came about a solution).

 

[trappedslider]

And some people are likely to believe whatever OpenAI says. I don’t quite see the relevance of bringing up either kind of person here. The question before us is about the provenance and verification of the Navier-Stokes result and possibly about what that means, not about who trusts or distrusts OpenAI.

In between a rock and a hard place comes to mind.

EDIT: Self-inflicted to be sure.

 

[Stalker0]

And that's the bigger problem here: this is, at it's heart, still a "he said, she said" sort of case, where we have to take the people at their word and their integrity. I don't know much about Buckmaster, but I've not seen a lot of people say he was untrustworthy. The same can't really be said of OpenAI (or, again, Bubeck specifically; it's wild that his former intern literally made a twitter account to say "Yeah, this tracks", latter confirming his own identity on his website as well as through his colleagues). I am much more willing to buy Buckmaster's version of how conversations went and his logic (especially given his expertise in the area) versus OpenAI's version (which, again, doesn't refute them initially telling lies to Buckmaster about how they came about a solution).

And in such cases, at least with US criminal courts, we lean on the concept of “innocent until proven guilty”

 

[The Firebird]

I apologize, but that feels way cleaner and more sanitized than reality typically is? Everything here seems to give OpenAI the benefit of the doubt, and I'm not really convinced given their history that they deserve it. The technology can be good, but that doesn't mean we should start ignoring the track record of the people who use it and typically do so in quite unethical forms (as has been pointed out on this forum before).

That OpenAI (Or, I suppose, Sebastian Bubeck, if we want to be specific here) said that this was just something they put in the problem and had "very little human input" (something that I don't think any statement by either the man or the company has actually refuted) is perhaps the biggest red flag for me, because why lie about the method in a private conversation if you aren't trying to hide something? To me, that's to try and cut off questions as to the actual method of solution, and I wonder if Buckmaster hadn't challenged them how much we would have actually learned about it.

Along similar lines, I think Buckmaster is absolutely right to question the prompt that they gave to solve it. When he says that is an esoteric prompt not easily reached, I find that rather compelling and OpenAI hasn't really given us any look at how they got to that prompt in the first place: they have mathematicians, but none that specialize in the area that N-S inhabits, and none that had done on any work on it before. If the machine came up with it, we should be able to see the logs and such, right? Or have some insight into how they came up with it? I dunno, maybe I'm asking too much, but I feel like they should be able to better address this concern and they just sort of move past it.

There's more in Buckmaster's statement, but the overall response from OpenAI has been to brush off the accusations and to say that they investigated themselves and found nothing. And that's generally not convincing for any organization to say, but doubly so for one that feels like it has been operating past the limits of ethicality for a while and whose best-case scenario is that they were only guilty of trying brute force a problem to claim credit before someone else could.

Again, this is not an attack on the usage of the techology: Buckmaster is rather eloquent in his defense of the usage, though he also makes a good point of how most of this stuff doesn't come out in useable fashion (He remarks that the Euler solution is very much "slop", something that needs to be understood and refined by the people who properly understand the math to actually make use of it), and that is a whole other conversation to have on releasing these sorts of solutions without really understanding how we got there. But again: the technology has proper uses, I just don't really see why we should expect OpenAI to operate in a fully ethical way on this when past behavior would indicate that they typically don't.



I think any statement by OpenAI (and most AI companies, honestly) shouldn't be taken at face value without ample proof. They're just really prone to omitting key facts (or costs, when it comes to saying when they'll be making a profit in the future) or fabulize (Hey, AGI is here, right? Right?!). It doesn't help that Bubeck has kind of done that sort of thing before:

View attachment 447967

This is stuff that they are known to do. They build hype, because the whole money machine runs on that right now and anything that threatens that hype needs to be dealt with or dismissed. So it's not that I'll never believe them, I just don't believe their statements without outside confirmation or solid proof that goes beyond just their word.

And that's the bigger problem here: this is, at it's heart, still a "he said, she said" sort of case, where we have to take the people at their word and their integrity. I don't know much about Buckmaster, but I've not seen a lot of people say he was untrustworthy. The same can't really be said of OpenAI (or, again, Bubeck specifically; it's wild that his former intern literally made a twitter account to say "Yeah, this tracks", latter confirming his own identity on his website as well as through his colleagues). I am much more willing to buy Buckmaster's version of how conversations went and his logic (especially given his expertise in the area) versus OpenAI's version (which, again, doesn't refute them initially telling lies to Buckmaster about how they came about a solution).

Yeah, I think these are well-articulated reasons to be distrustful of OpenAI. That said, for me it isn't enough to get from those reasons to the belief that they are lying about reading the prompts. I could be wrong, but that in particular feels to me like a step further, and the pace of their response makes sense for me.

And gosh, I hesitate to bring this up, because I think it is unfair to Buckmaster. But it is worth noting that Kasparov's response on losing to Deep Blue was to accuse IBM of cheating. There is a sort of precedent, resonance here.

 

[Umbran]

Let us also add the fact that no matter what some people aren't likely to believe whatever OpenAI says unless it's to confirm what they already believe.

This is a general human behavior, not specific to one side of a question - sometimes, humans reject information that contradicts our preferred narratives. This is apt to happen to people because they are people, not due to their position on a point of discussion.

So, please, be aware that you're making an implication by omission - that supporters of OpenAI, or generative AI more broadly, are not subject to this behavior - and that this implication is pretty patently false.

 

[Umbran]

And gosh, I hesitate to bring this up, because I think it is unfair to Buckmaster. But it is worth noting that Kasparov's response on losing to Deep Blue was to accuse IBM of cheating. There is a sort of precedent, resonance here.

It is unfair to Buckmaster, specifically because he isn't trying to take the credit. He's already stated he feels Luis Martínez-Zoroa is the one who deserves the Fields Medal for the solution.

 

[The Firebird]

It is unfair to Buckmaster, specifically because he isn't trying to take the credit. He's already stated he feels Luis Martínez-Zoroa is the one who deserves the Fields Medal for the solution.

Perhaps you missed my previous post on the subject. Please, either place me on two way ignore or stop responding to my posts. (Or help me place you on my ignore list, which I understand is not possible?)