Judge decides case based on AI-hallucinated case law

[Evaniel]

This is the "Geek Talk & Media" sub-forum.
1) genAI is geek stuff.
2) We have law geeks here, too.

The NuTSR thread attests to this many, many times over.

 

[Lidgar]

It’s not just law that’s hallucinating. It’s medical staff, scientists, engineers- you know, those topics that takes years of study and seconds to sabotage with one prompt.

My whole company is at a crossroads on this stuff. Quality control on AI output has become the new demand. Only problem is quality control requires years of real world experience, and how are today’s graduates going to get that experience if we have AI do the grunt work? It’s a conundrum with no clear path forward.

 

[Jfdlsjfd]

Humans lawyers have tried to mention non-existant cases forever. I can't count the number of time when I read "Cases abound..." and wondered if I'd get even one actually referenced. Providing creative interpretation of what the actual ruling was is also quite common. Inventing false case out of the blue must be rare -- professionals wouldn't like to be caught or at least have proofreading procedures. I guess the practice is probably more problematic in common law systems.

General-purpose LLM hallucinating is a boon for law journal publishers, who are testing dedicated AI actually trained on their database to offer a service. In the past 3 month, I have been offered to test 3 major publisher's offering, so they are all trying to get to the market first. I am guessing it's a tool lawyers offices are probably willing to invest quite a lot in. The amount of money that could be saved with this is tremendous, once it is trained on a correct database and told not to guess and provide source (or references are checked automatically in a database by another model before being sent to the customer).

 

[Kaodi]

No, dude, you do not, in fact, have to take crystal meth to do your job.

 

[Jfdlsjfd]

No, dude, you do not, in fact, have to take crystal meth to do your job.

A technology that multiplies one's output, even if failing sometimes, will be terribly appealing to most. Especially in countries where lawyers are expensive. When I read that people are being bullied by companies legal claims because they can't afford a lawyer or renounce their rights because they can't afford to enter the judicial system to get their rights enforced, I am all in favour of any tool diminishing the cost of the lawyer... Sure, the quirks need to be ironed out, as with any tool. This process can happen as the tool is being adopted. Many people died in car crashes before we had safety belt and airbags, yet many people ditched their riding horses before the generalization of airbags.

 

[Parmandur]

That's...astounding.

 

[BookTenTiger]

It’s not just law that’s hallucinating. It’s medical staff, scientists, engineers- you know, those topics that takes years of study and seconds to sabotage with one prompt.

My whole company is at a crossroads on this stuff. Quality control on AI output has become the new demand. Only problem is quality control requires years of real world experience, and how are today’s graduates going to get that experience if we have AI do the grunt work? It’s a conundrum with no clear path forward.

It feels like to me that the people most qualified to do quality control would be the people who are actually qualified to do the work that the AI is purporting to replace!

"We've fired you as a teacher, but we would like you to go through these lesson plans and make sure they will be effective at teaching children."

 

[jian]

It’s not just law that’s hallucinating. It’s medical staff, scientists, engineers- you know, those topics that takes years of study and seconds to sabotage with one prompt.

My whole company is at a crossroads on this stuff. Quality control on AI output has become the new demand. Only problem is quality control requires years of real world experience, and how are today’s graduates going to get that experience if we have AI do the grunt work? It’s a conundrum with no clear path forward.

Yeah. As a doctor I’ve had plenty of time for Dr Google (patients looking up symptoms and conditions on Google) because I certainly don’t blame them for seeking more information or clarification. But now a small but significant chunk of my job involves explaining to patients that Dr Google hallucinates 20-40% of the time.

As for the issue around training, experience and education - we already know from studies that students who use genAI more to research and write essays develop cognitive debt, so they can barely remember anything from the essays they didn’t write and have much less useful skill and knowledge. Apply that to all fields and you’ll have millions of technically qualified people who can’t do what they’re supposed to be trained to do.

 

[Ryujin]

A technology that multiplies one's output, even if failing sometimes, will be terribly appealing to most. Especially in countries where lawyers are expensive. When I read that people are being bullied by companies legal claims because they can't afford a lawyer or renounce their rights because they can't afford to enter the judicial system to get their rights enforced, I am all in favour of any tool diminishing the cost of the lawyer... Sure, the quirks need to be ironed out, as with any tool. This process can happen as the tool is being adopted. Many people died in car crashes before we had safety belt and airbags, yet many people ditched their riding horses before the generalization of airbags.

You only use something that is 80% effective in cases where being wrong 20% of the time doesn't hurt you (pulling the numbers out of my butt, but I believe I've heard similar in articles). In law, being wrong can result in incarceration or, at the very least, just plain losing. In medicine being wrong can result in the ultimate price being paid. In finance, ruin.

 

[Jfdlsjfd]

You only use something that is 80% effective in cases where being wrong 20% of the time doesn't hurt you (pulling the numbers out of my butt, but I believe I've heard similar in articles). In law, being wrong can result in incarceration or, at the very least, just plain losing. In medicine being wrong can result in the ultimate price being paid. In finance, ruin.

The whole job of the lawyer isn't to check precedent. A tool that goes fast enough for situation where a small "quality insurance check" is enough to detect the 20% failure situation and save a lot of time for the all the remaining cases is a sound business proposal. Plus, that guy was probably using a general-purpose LLM and found a lazy judge, so the whole error wasn't detected until after the trial, whereas dedicated agents using some AI technology plus automated verification on a database can get to a low enough error rate to make it a professional tool to increase lawyer's productivity (or replace assistants).

On a wider point of view, I also think that between "dying with 100% certainty from lack of surgery because I can't afford a surgeon that only fails 5% of the time" and "being treated by an AI surgeon that fails 20% of the time", I'd choose to go with the AI surgeon. I am happy not to have to make this choice. [Number also pulled out of my butt of course].

A tougher question would be "You're innocent. You can't afford a renowned lawyer. You can either accept a plea bargaining to get 10 years in jail or hire an AI-using lawyer that is half as expansive because he's more productive even if there is a risk he doesn't detect the AI is making him doing a bad decision. You'll either be proven innocent or get a 20 years sentence. You don't know the exact percent of cases the AI-using lawyer is losing due to using AI. You also don't know the exact number of case the renowned lawyer is losing due to giving your case to be handled by a junior recruit, but you suspect it might be lower, and you can't afford the renowned lawyer anyway. What do you do?"