Judge decides case based on AI-hallucinated case law

[FrogReaver]

I remember when people used to joke that WebMD would tell you that you've got cancer or are dying...

You mean it doesn’t still do that

 

[trappedslider]

Aaaaand it looks like he’s finally getting his wish.

so i did a test just now:

Is there a jewish conspiracy in Hollywood?


The idea of a "Jewish conspiracy" in Hollywood is a longstanding trope rooted in anti-Semitic stereotypes, often alleging that Jewish people control the entertainment industry to manipulate culture or power.
Let's break it down:
Historically, Jewish immigrants like those who founded major studios (e.g., Warner Bros., Paramount, MGM) played a significant role in building Hollywood in the early 20th century. This was due to their entrepreneurial drive and the industry's openness to newcomers, not a coordinated "conspiracy." By the 1920s, Jewish studio heads were prominent, but their influence was driven by market competition, not collusion. Today, Hollywood's leadership is diverse, and while some Jewish executives exist, the industry is far from monolithic—spanning global corporations, independent creators, and varied cultural backgrounds.

The "conspiracy" narrative often stems from cherry-picked anecdotes or outdated stereotypes, ignoring the complex, decentralized nature of modern Hollywood. For example, the Motion Picture Association represents a global industry with thousands of stakeholders, not a single ethnic or religious group. Claims of a "Jewish agenda" typically lack evidence and rely on vague generalizations, often amplified by historical anti-Semitic texts like The Protocols of the Elders of Zion, a debunked forgery.

On the flip side, some argue that cultural biases in media—any media—can reflect the backgrounds of its creators, Jewish or otherwise. This is less about conspiracy and more about human nature: people tell stories from their perspectives. But Hollywood’s output is too vast and varied to be controlled by any one group. Blockbusters, indie films, and streaming platforms reflect market demands and creative trends, not a unified agenda.No hard data supports a "Jewish conspiracy." Anti-Semitic tropes often exaggerate Jewish influence while ignoring the industry’s broader dynamics. If you’re looking for specific claims or posts on X about this, I can dig into those for you. Want me to check?

 

[Snarf Zagyg]

I've been occupied lately, but I hope that this helps a little. I read the opinion in order to understand a few things. I hadn't paid attention after the first two pages of comments, so I apologize if this was covered elsewhere, but a few procedural issues need to be brought up.

1. I don't mean to absolve the judge, but the original order wasn't the judge's fault, technically. This was a state court decision. The way a lot of state court proceedings work is this- both side submit filings (a motion and a response, for example) and there is a hearing. The judge reads the filings and listens to the arguments, and decides at the hearing who "won." The winning side writes the order based on the hearing and submits the order to chambers for the judge's signature. This might sound weird, but that's common practice in a lot of state courts for most hearings. Also, most state court judges, especially in areas like family law (this was family law) don't have clerks.

2. So what happened is that some attorney likely submitted a filing (with hallucinated authority) and then argued to the court at a hearing and won the hearing. The winning attorney then submitted the order to the Court that the Court signed ... based on the same hallucinated authority.

3. This order was appealed. This is where it gets fun! Because the appellate court was like, "Not only were the cases cited in the order hallucinated, but the cases you cited to us in your appellate brief? Those were hallucinated, too."

4. Next procedural issue- the appellate court was limited in its review because there was no transcript of the hearing to determine exactly what happened. And limited in the sanctions it could impose on the attorney (a $2500 judgment against the attorney). I don't know enough about the specific issues, but I wish that they could have at least required the attorney to provide the order to the Georgia Bar- perhaps the trial court will do that on remand.


So here's the thing. If you're an attorney, you know (or should know, given the widespread publicity) that using AI can, and will, cause these issues. You are an officer of the Court- you do, in fact, have an obligation to provide correct authority to the Court (and to correct authority when it is incorrect, withdrawn, etc.). That's a bedrock principle of the law.

If you are using AI in this manner, not only should you face severe sanctions for the use of it, but I will bet that your billing records are also suspect- how can you bill for drafting something that you can't be bothered to even check? And so on.

For those stating that this is "just a tool," well ... sure. Kind of. But attorneys have to sign filings with the Court- and that means that if an AI wrote your work, you are signing your name to work that you don't understand. On the broader scale, as more people use AI and see it work well ... some percentage of the time ... without understanding that it works disastrously that other percentage of the time? If you don't warn people, that's not a user problem. That's a defective tool.

 

[Umbran]

The next question is how do you test them?

Quality Assurance is an entire profession - several, really, across various domains. You hire a bunch of them and let them do their jobs.

 

[Jfdlsjfd]

1. I don't mean to absolve the judge, but the original order wasn't the judge's fault, technically. This was a state court decision. The way a lot of state court proceedings work is this- both side submit filings (a motion and a response, for example) and there is a hearing. The judge reads the filings and listens to the arguments, and decides at the hearing who "won." The winning side writes the order based on the hearing and submits the order to chambers for the judge's signature. This might sound weird, but that's common practice in a lot of state courts for most hearings. Also, most state court judges, especially in areas like family law (this was family law) don't have clerks.

Thank you for this enlightening precision. In my juridiction, it's the judge's work to write the decision, and each word is carefully weighted, but it might depend on place and importance.

2. So what happened is that some attorney likely submitted a filing (with hallucinated authority) and then argued to the court at a hearing and won the hearing. The winning attorney then submitted the order to the Court that the Court signed ... based on the same hallucinated authority.

And the judge has no business checking that the order is correct? Just the dispositive of the order?

For those stating that this is "just a tool," well ... sure. Kind of. But attorneys have to sign filings with the Court- and that means that if an AI wrote your work, you are signing your name to work that you don't understand. On the broader scale, as more people use AI and see it work well ... some percentage of the time ... without understanding that it works disastrously that other percentage of the time? If you don't warn people, that's not a user problem. That's a defective tool.

ChatGPT (at least) warns people to double-check everything. Which shouldn't even be needed, as it's self evident that one should double-check anything it signs before submitting them to a court. The same applies to the work of an intern or junior collaborator, let alone a computer tool designed to "chat" with you [and to which the same level of trust should be granted as with any random people you chat with in a bar].

 

[FrogReaver]

Quality Assurance is an entire profession - several, really, across various domains. You hire a bunch of them and let them do their jobs.

Obviously. That doesn’t answer the meat of the question of how you do that in the legal domain. Nor of what standards denote a pass vs a fail. Etc. The broader point is sometimes testing can only be achieved in live environments as opposed to testing environments.

 

[Snarf Zagyg]

Thank you for this enlightening precision. In my juridiction, it's the judge's work to write the decision, and each word is carefully weighted, but it might depend on place and importance.

In most state courts that I am familiar with, there are a lot of hearings. Depending on the issue, the parties may submit an order (an agreed-to order) that the judge signs. Or the judge might write the order on dispositive motions (MSJs, MTDs, certain other issues). But a lot of state courts, in a lot of matters, has the hearing and the lets the winning side write the order based on the hearing.

Now, practice is that the order is based on the finding of the judge at the hearing (ore tenus). But the judges will depend on the parties to submit the accurate order based on the hearing. If a party won based on incorrect law, then it would follow that the other party will actually allow the submission of that order to reflect that for the appeal. Capiche?

And the judge has no business checking that the order is correct? Just the dispositive of the order?

Judges can do whatever they want, of course. But for purposes of expedience, they have the attorneys write the order and sign it.

ChatGPT (at least) warns people to double-check everything. Which shouldn't even be needed, as it's self evident that one should double-check anything it signs before submitting them to a court. The same applies to the work of an intern or junior collaborator.

And I will note again that maybe there needs to be a much more prominent disclaimer since this kind of thing keeps happening in multiple areas that aren't checked as closely as the law. Businesses have a financial incentive to make their product look reliable.

In a past life, I used to work on a DOD-related project that was pretty high-tech for the time. It achieved over 98% reliability. Trouble was, it had to do with weapon targeting. Never got to to the field.

I am not against AI. But if you're going to use it as a tool, then it needs to be considered a product. With everything that it entails.

 

[Umbran]

Obviously. That doesn’t answer the meat of the question of how you do that in the legal domain. Nor of what standards denote a pass vs a fail. Etc.

That's why I mentioned that this is an entire profession. It is kind of like asking, "How do you do surgery?" You want someone who actually does surgery to answer it, and the answer is in no way short and simple.

 

[Jfdlsjfd]

Obviously. That doesn’t answer the meat of the question of how you do that in the legal domain. Nor of what standards denote a pass vs a fail. Etc. The broader point is sometimes testing can only be achieved in live environments as opposed to testing environments.

And what level of quality you want to deem the program appropriate. The level you'd expect for a program designed to guide your nuclear missiles to destination will be higher than a program that will calculate your taxes, which will be higher than a program that you write a letter with. ChatGPT is a program that is not advertised to deal with legal case. It is a chat program. People alarmed at hallucination seems to have never encountered a bug in a commercial product.

I am pretty sure some Q&A was applied to the tool before it was sold. Simply, they felt that the level of usability was appropriate for what they market it as (a chatting program).

 

[Ryujin]

Quality Assurance is an entire profession - several, really, across various domains. You hire a bunch of them and let them do their jobs.

As an aside, I have a friend who used to be a SDET (Software Development Engineer in Test) for various large software companies, that you definitely would have heard of. Companies have been laying off their SDETs over the last 5 years and, instead, having developers test and certify their own work. I think that many here can understand that when looking at huge text blocks of your own work, you tend to become blind to your own mistakes. Have you ever read something like a work of fiction, before a Proofer has gotten to it? I've proof read a few books, for author friends, and the pre proof works certainly had issues, even after several run throughs by the authors.