Judge decides case based on AI-hallucinated case law

[Ryujin]

To be fair, wikipedia is a lot better source than many others because of how rigorous its standards are. The sun’s not hot unless you can cite primary sources. It’s got its problems, of course it does, but it’s far more reliable than any LLM.

And it has been around for much longer, and has gone through the problems I mentioned. The more "citation needed" entries, the less you trust an article. At least many of them list their cites at the bottom, giving you an actual source that you can reference. LLMs, when they cite sources, seem to often make those up as well.

 

[jian]

-

I’m afraid that’s basically nonsense, as far as I can tell. MTHFR mutations probably do affect B vitamin metabolism but they don’t have any established symptoms and pretty much no primary care doctor (or even a clinical geneticist) would think to test for them since they’d have no idea what to do with the results. The evidence just isn’t there for clinical presentation or treatment.

 

[The Firebird]

I’m afraid that’s basically nonsense, as far as I can tell. MTHFR mutations probably do affect B vitamin metabolism but they don’t have any established symptoms and pretty much no primary care doctor (or even a clinical geneticist) would think to test for them since they’d have no idea what to do with the results. The evidence just isn’t there for clinical presentation or treatment.

Yeah I wonder if a lot of these "chronic cases resolved by LLM" can be attributed to placebo effects

 

[Dannyalcatraz]

When I read that people are being bullied by companies legal claims because they can't afford a lawyer or renounce their rights because they can't afford to enter the judicial system to get their rights enforced, I am all in favour of any tool diminishing the cost of the lawyer...

When I took Oil & Gas back in the 1990s, our prof pointed out there’s clauses present in the boilerplate of the majority of Texas O&G royalties contracts that are poorly drafted and/or legally unenforceable (in his opinion), but have never been challenged in court.

Part of this is because most people (and institutions) don’t understand those clauses- including the O&G companies themselves- and can’t afford the services of those who can enlighten them.

The result? By his estimation, some of these clauses that are multi-billion dollar time bombs waiting to explode in the faces of Big Oil companies.

 

[Dannyalcatraz]

The whole job of the lawyer isn't to check precedent.

The whole job? No.

But it’s a HUGE & FUNDAMENTAL part of the job, big enough that “Shephedizing” is taught in Year 1, and it’s one of the reasons why those using ChatGPT et alia as a crutch are getting sanctioned.

And when you take a CLE (continuing legal education) course, the presenters will have done the shepherdizing for you when they’re presenting key changes in the case law.

 

[billd91]

Wikipedia is unreliable largely because of its crowdsourced nature and there are cases of bad actors actively crapping in articles. It's still just a starting point, as any LLM should be, rather than a true source.

For being crowd-sourced, Wikipedia is shockingly reliable these days. A number of studies show Wikipedia holds its own compared to a variety of other encyclopedias with its most significant lag being in general readability. While it remains potentially subject to vandalism, it has achieved a measure of authority through its citation policies, even when cumbersome, where a more traditional encyclopedia publisher would be relying on expert staff and reputation to establish its authority.

By comparison, I’m not sure what kind of authority a LLM can show or demonstrate, particularly if subject to hallucinations.

 

[Umbran]

I know some companies claim a higher success rate than physicians.

So, it is extremely important to know that LLMs/generative AI is not the only stuff under the banner "AI".

That matters when we talk about success claims.Just because one tool had a good success rate, doesn't mean the LLMs you have access to will.

My point wasn't that Wikipedia is 100% reliable. But it is clearly reliable enough to be useful.

It is important to fully qualify that. It is useful.... for what? Under what circumstances? On what topic?

As noted upthread, and in language most of us can understand from gaming - what are the consequences for failure?

 

[The Firebird]

So, it is extremely important to know that LLMs/generative AI is not the only stuff under the banner "AI".

That matters when we talk about success claims.Just because one tool had a good success rate, doesn't mean the LLMs you have access to will.

Yes, agreed.

This was the publication I had in mind when I wrote that. While promising it is important to note it seems like a setup job in which the physicians were designed to perform poorly to emphasize the success rate of the model.

That said, there have been other results in the literature that show good success rates for off-the-shelf LLMs. That isn't an endorsement of 'prompt and hope', because they can get things, major things, wrong. But it does show they are a useful tool.

It is important to fully qualify that. It is useful.... for what? Under what circumstances? On what topic?

As noted upthread, and in language most of us can understand from gaming - what are the consequences for failure?

I choose wikipedia because I imagine most readers have used it, and understand the cases where it is useful despite reliability concerns. The challenge with LLMs is for everyone to understand where the fail cases are and how to avoid them.

 

[Clint_L]

But did the court reach the wrong conclusion?

 

[doctorbadwolf]

It’s not just law that’s hallucinating. It’s medical staff, scientists, engineers- you know, those topics that takes years of study and seconds to sabotage with one prompt.

My whole company is at a crossroads on this stuff. Quality control on AI output has become the new demand. Only problem is quality control requires years of real world experience, and how are today’s graduates going to get that experience if we have AI do the grunt work? It’s a conundrum with no clear path forward.

The very clear path forward is to ban generative ai and let people grow in experience by actually doing things.