Have you ever had that sinking feeling right after you hit "send" on a hugely important email? That split-second panic where you wonder, "Did I attach the right file? Did I spell the client's name right?"
Now, imagine that feeling, but multiply it by about a million.
That's probably what it felt like over at Sullivan & Cromwell, one of the most prestigious law firms on Wall Street. They recently had to send an apology letter to a federal judge because a court filing they submitted was riddled with errors, including completely made-up legal citations.
And who was the culprit? Not a tired junior associate or a careless paralegal. It was their artificial intelligence tool. The AI, in a now-infamous tech flaw, had what are called "hallucinations." It just invented things that looked real but were completely fake.
This story is more than just a bit of embarrassing legal gossip. For anyone running a business or working in a professional field today—and that includes us in the insurance world—this is a massive, flashing red warning light. It begs the question: When our shiny new AI tools mess up, who's on the hook? And more importantly, is our insurance ready for it?
So, What Exactly Happened Here?
Let's break it down, because the details are pretty wild. In a letter dated April 18, a top partner at the firm had to explain to a judge that they’d relied on an AI tool for legal research. This tool, instead of pulling up real, verifiable case law, simply fabricated some.
Think of it like asking your GPS for directions to a new restaurant, and instead of taking you there, it confidently guides you to an empty field and announces, "You have arrived." The GPS was supposed to know, right?
This isn't the first time this has happened in the legal world, but seeing it come from a powerhouse firm like Sullivan & Cromwell really makes you sit up and pay attention. It shows that no one is immune to the risks of this new technology. They had to own the mistake, apologize profusely, and correct the record. It’s a huge blow to a firm’s reputation, not to mention the potential for serious legal and financial consequences.
AI "Hallucinations": The Ghost in the Machine
The term "hallucination" sounds a bit dramatic, but it's honestly the perfect word for it. These large language models, the tech behind tools like ChatGPT, are designed to predict the next most likely word in a sentence. They are incredible at sounding confident, human, and authoritative.
The problem is, they don't actually know things. They don't have a consciousness or a fact-checker in their digital brain.
So when you ask it for something it can't find, instead of saying "I don't know," it often does what it does best: it predicts what a correct answer should look like. It creates a perfectly formatted, official-sounding legal citation that just… doesn't exist. It’s a ghost. A phantom piece of data that looks solid until you try to touch it.
And this is where the real risk for businesses comes in. We’re all being encouraged to use AI to be more efficient, to draft reports, to analyze data, to write code. But what happens when that efficiency comes at the cost of accuracy?
Let's Talk Insurance: Is This Covered by Your E&O Policy?
This is the million-dollar question for every professional. If you’re an architect, an accountant, a consultant, or an insurance agent, you have Errors & Omissions (E&O) insurance. It’s your safety net for when you, a human, make a professional mistake that costs a client money.
But what happens when the mistake wasn't made by you, but by the AI you were using?
Here’s the thing: your E&O policy might not be so clear on this. Most policies were written long before anyone was asking an AI to co-author a legal brief. The core of an E&O claim is a negligent act, error, or omission in the performance of your professional services.
So, the insurance company is going to ask some tough questions:
- Was using the AI itself a negligent act? Did you do your due diligence? Did you have a human review the work?
- Is the output of an AI considered part of your "professional service"? Or is the AI considered just a tool, like a calculator or a word processor?
- Does your policy have any specific exclusions for technology-related failures or data errors?
Honestly, this is brand new territory. We're seeing carriers start to grapple with this, but there isn't a simple, one-size-fits-all answer yet. My gut feeling is that insurers will argue that the ultimate responsibility lies with the professional who signs off on the work. You can't just blame your software. You are the licensed expert, the one the client hired. The AI is your tool, and you are responsible for how you use it.
The Blame Game: Who's Ultimately Liable?
When an AI-driven mistake causes a financial loss, you can bet there will be a lot of finger-pointing.
- The Professional/The Firm: This is the first and most obvious stop. Sullivan & Cromwell took the blame directly. They were the ones who filed the document. For most E&O claims, this is where the liability will land.
- The AI Software Company: Could the firm turn around and sue the company that made the AI for providing a faulty product? Absolutely. This will likely become a whole new area of litigation. But winning that case could be tough, especially when most software comes with lengthy terms of service agreements that limit their liability.
- The End Client: Let's not forget them. They could suffer the biggest loss from a botched court case or a flawed financial projection. They will sue the professional they hired, not the software they used.
This creates a messy, complicated chain of liability that the insurance industry is working hard to figure out. It touches on E&O, cyber liability (if there's a data component), and even tech liability for the software developers.
How to Protect Your Business in the Age of AI
Look, we can't put the genie back in the bottle. AI is here to stay, and it offers incredible benefits. But as this legal drama shows, we have to be smart about it. We have to be the humans in the loop.
So what can you do?
- Create an AI Policy: Don't just let your team run wild. Establish clear guidelines on which AI tools are approved, what they can be used for, and—most importantly—that all AI-generated work must be verified by a human expert.
- Train Your People: Make sure everyone understands the limitations of these tools. Teach them to be skeptical and to treat AI output as a first draft, not a final product.
- Talk to Your Insurance Broker: This is critical. Have an open conversation with your agent or broker about how you're using AI. Ask them to review your E&O and cyber policies. You might need to look for specific coverage endorsements or even a new type of policy as they become available.
The story of the law firm and its hallucinating AI is a cautionary tale, but it's not a reason to panic. It's a reason to be prepared. Technology will always outpace the rules and regulations that govern it. Our job is to bridge that gap with common sense, solid processes, and a strong insurance safety net that's ready for the future, whatever it looks like.
