Judge decides case based on AI-hallucinated case law

[FrogReaver]

That's why I mentioned that this is an entire profession. It is kind of like asking, "How do you do surgery?" You want someone who actually does surgery to answer it, and the answer is in no way short and simple.

Okay. So no relevant additions to the discussion. We are at an impasse. My suggestion is that there is no great way to test ai outside of a live environment. Your suggestion is that there must be a good way but that you cannot comment on what it might be due to lack of expertise. There’s nothing more to say here.

 

[Jfdlsjfd]

I am not against AI. But if you're going to use it as a tool, then it needs to be considered a product. With everything that it entails.

I agree with that. And a product isn't sold in isolation. It's marketed to a target, with appropriate expectation. Many products sold to the general public aren't fit for professional use, and the reverse is also true.

In the case of ChatGPT, the target isn't "law firms". I haven't seen the "ChatGPT for business" contrat, but I am pretty sure they mention quite explicitely the limitations of it. Professional editors like LexisNexis are selling AI tools (supposedly) fit for this use case. The error is using a general purpose tool, designed to chat, to do business-critical tasks and not checking the result afterwards.

 

[FrogReaver]

And what level of quality you want to deem the program appropriate. The level you'd expect for a program designed to guide your nuclear missiles to destination will be higher than a program that will calculate your taxes, which will be higher than a program that you write a letter with. ChatGPT is a program that is not advertised to deal with legal case. It is a chat program. People alarmed at hallucination seems to have never encountered a bug in a commercial product.

I am pretty sure some Q&A was applied to the tool before it was sold. Simply, they felt that the level of usability was appropriate for what they market it as (a chatting program).

Sure and I 100% agree but also pre live testing only ever goes so far.

 

[Ryujin]

Okay. So no relevant additions to the discussion. We are at an impasse. My suggestion is that there is no great way to test ai outside of a live environment. Your suggestion is that there must be a good way but that you cannot comment on what it might be due to lack of expertise. There’s nothing more to say here.

There's a very real reason why the phrase "test it in production" is an ongoing joke, in IT circles.

 

[Jfdlsjfd]

As a side note, here is a ChatGPT transcript I just had:

If I am a qualified attorney, can you draft filings for me?

---

What I can do for you as a qualified attorney:​

• Draft or refine legal documents based on your instructions (briefs, motions, contracts, demand letters, memos, etc.).
• Format and structure filings according to your jurisdiction’s requirements (if you tell me the rules or upload templates).
• Review or summarize case law, statutes, or legal arguments (if you provide the text and jurisdiction).
• Generate multiple versions or argument strategies for the same filing.
• Help with tone, clarity, and persuasiveness.

What you remain responsible for:​

• Legal accuracy: You must check all facts, citations, and interpretations.
• Jurisdictional compliance: I don’t have access to local court rules or precedents unless you provide them.
• Ethics and confidentiality: While this platform is designed with privacy in mind, sensitive or privileged data should be used with discretion.
• Final legal judgment: I don’t "understand" the law like a human expert; I model patterns based on texts.

I am pretty sure it's enough, as a warning, for a qualified attorney. Perhaps a stronger warning should be used?

 

[Ryujin]

As a side note, here is a ChatGPT transcript I just had:

If I am a qualified attorney, can you draft filings for me?

---

What I can do for you as a qualified attorney:​

• Draft or refine legal documents based on your instructions (briefs, motions, contracts, demand letters, memos, etc.).
• Format and structure filings according to your jurisdiction’s requirements (if you tell me the rules or upload templates).
• Review or summarize case law, statutes, or legal arguments (if you provide the text and jurisdiction).
• Generate multiple versions or argument strategies for the same filing.
• Help with tone, clarity, and persuasiveness.

---

What you remain responsible for:​

• Legal accuracy: You must check all facts, citations, and interpretations.
• Jurisdictional compliance: I don’t have access to real-time local court rules or precedents unless you provide them.
• Ethics and confidentiality: While this platform is designed with privacy in mind, sensitive or privileged data should be used with discretion.
• Final legal judgment: I don’t "understand" the law like a human expert; I model patterns based on texts.

I wonder what the result would be from ChatGPT-Law.

 

[Jfdlsjfd]

I wonder what the result would be from ChatGPT-Law.

There is a strong chance a commercial AI targetted at law films would say the exact same thing: "YOU are solely responsible for the accuracy of information and we decline any responsabilities, as the tool is still prone to make errors." I am pretty sure the software editors don't want to assume any contractual liability. They might provide easy links to a reputable database to ease the fact-checking by the user, though.

Edit: I checked what LexisNexis says about its AI assistant...

C'est l'une des choses qui différencie Lexis+ AI. Lorsque vous l’utilisez, vous obtenez des résultats avec des sources citées afin que vous puissiez revenir en arrière et vérifier les lois et la jurisprudence pour garantir une précision complète.

They market the fact that any legal fact they provide will be sourced, so the use can check the laws and case to guarantee optimal precision. They don't seem to exonerate the user from checking the content.

 

[Snarf Zagyg]

I agree with that. And a product isn't sold in isolation. It's marketed to a target, with appropriate expectation. Many products sold to the general public aren't fit for professional use, and the reverse is also true.

In the case of ChatGPT, the target isn't "law firms". I haven't seen the "ChatGPT for business" contrat, but I am pretty sure they mention quite explicitely the limitations of it. Professional editors like LexisNexis are selling AI tools (supposedly) fit for this use case. The error is using a general purpose tool, designed to chat, to do business-critical tasks and not checking the result afterwards.

Yep. Which is why if the product is for the general public, is shouldn't produce material that could be confused with professional material.

In other words, if you're going to release something that is used by the general public, think of the most stupid person you know. Now, do you want them to use this in place of an attorney? A doctor? An architect?

People seem to be confusing the cool abilities to do things themselves using AI in a responsible manner with the great damage that can be done with a tool that appears to allow people to do things right some percentage of the time, but does so disastrously wrong some other percentage of the time.

In other words, there is a difference between an AI diagnostic engine used by medical professionals, and a person at home using a random AI to self-diagnose with that person's inputs and knowledge. Or, for that matter, a lot of other uses. If AI is going to use the general populace to train itself to be better, then it should expect a lot of lawsuits when it goes poorly for the beta testers.

 

[Jfdlsjfd]

Yep. Which is why if the product is for the general public, is shouldn't produce material that could be confused with professional material.

In other words, if you're going to release something that is used by the general public, think of the most stupid person you know. Now, do you want them to use this in place of an attorney? A doctor? An architect?

In my experience, when you chat about specific subjects (law, healthcare, diet), it is quite proactive in sending the user to a lawyer, doctor, nutrition expert... with explicit warnings.

Outside of outright refusing to answer questions in that domain, they can't really do more.

 

[Snarf Zagyg]

In my experience, when you chat about specific subjects (law, healthcare, food), it is quite proactive in sending the user to a lawyer, doctor, nutrition expert... with explicit warnings.

Then the company that makes the tool has nothing to worry about. It's an open question how "some other tool" seems to generate headlines.

By the way, while failure to warn is a cause of action, there's also the issue of product design defect- it's not sufficient to simply warn people if your product is doing something dangerous. It's an open and interesting question what a jury might think about the defense of "it's a black box, and we don't know if it can be less dangerous," although the fact that they can warn about certain topics certainly seems to indicate that certain topics could be flagged and not provided for misuse at all.