The Case Against AI: Where It Fails, and How People Make It Even Worse

Last post I showed you AI's verified wins and gave you a checklist for spotting them: narrow task, mountains of data, checkable answer, human in the loop. This post is the mirror image, and I'd argue it's the more important one.

Because here's the thing about "AI goes wrong" stories: they get lumped together as if they're one problem, and they're not. They're three different problems, with three different fixes. The tool fails. The user fails. Or someone weaponizes the tool on purpose. Mixing them up is how we end up with bad arguments on every side, so let's take them one at a time.

Failure type 1: The tool itself has limits

Start with what's baked in.

It makes things up, confidently. I covered this in post 1, so just the reminder: a language model predicts what's likely to be said, not what's true. There is no fact-checker inside. It will produce a fake court case, a fake statistic, and a fake citation in the same fluent tone as real ones, and nothing in its voice will warn you. This isn't a bug awaiting a patch. It's what the technology is.

It inherits our biases, at scale. AI learns patterns from data, and data comes from us, history included. Amazon famously scrapped an internal AI recruiting tool after discovering it had taught itself to penalize resumes that included the word "women's," because it had learned from a decade of male-dominated hiring records. Facial recognition systems, which perform measurably worse on darker-skinned faces, have contributed to multiple documented wrongful arrests in the US. The pattern-matcher doesn't know it's being unfair. It's matching the patterns we gave it, which is exactly the problem.

It's brittle. A model that dazzles in a demo can quietly fall apart in the real world, where the data is messier than the training set. The industry calls this the deployment gap. You've experienced it every time a chatbot that seemed brilliant suddenly face-planted on something a child would get right.

Notice none of this makes AI useless. It makes AI a tool with sharp edges — which brings us to the people holding it.

Failure type 2: People use it wrong

This category is growing faster than the first, and it's the one that worries me most day to day.

The cleanest example comes from the legal profession. In 2023, a New York lawyer filed a brief citing six court cases that didn't exist. He'd asked ChatGPT for supporting cases, and it invented them, complete with quotes and case numbers. When the judge got suspicious, the lawyer did something that should be taught in every classroom: he asked ChatGPT whether the cases were real, and it confirmed its own fabrications. He was fined $5,000, made international news, and the profession presumably learned its lesson.

It did not. A public database maintained by researcher Damien Charlotin now tracks court cases worldwide involving AI-fabricated citations or content. As of mid-2026 it holds roughly 1,600 cases, up from about 200 a year earlier. Lawyers have been fined six figures, suspended, and had trials canceled. One study from Stanford found that even paid, purpose-built legal research AI tools hallucinate on a meaningful share of queries. The American Bar Association's own coverage makes the stakes plain: courts don't even need to prove AI was involved to sanction you for fake citations. You signed it, you own it.

Read that story against the checklist and you'll see the tool didn't malfunction. It did exactly what post 1 told you it does. The humans removed themselves from the loop: nobody checked an answer that was completely checkable. Psychologists call the general failure "automation bias" — our tendency to trust the machine's output because it came from a machine. The same failure shows up when companies deploy AI to screen resumes, deny insurance claims, or grade essays with no human meaningfully reviewing the calls. The technology gets the headline, but the negligence is human.

This is the most fixable failure type on the list, which is what makes it so frustrating. The fix is a sentence long: verify before you rely. It's also the foundation of how I'd teach anyone to use these tools, in a classroom or a security operations center.

Failure type 3: People use it exactly right, for the wrong things

Now the deliberate stuff. This is my professional neighborhood, and I won't sugarcoat it.

In January 2024, a finance employee at the engineering firm Arup joined a video call with the company's CFO and several colleagues to discuss a confidential transaction. He'd been suspicious of the initial email, sensibly. But on the call, he could see and hear his colleagues, so he followed instructions and sent 15 transfers totaling $25 million. Every person on that call except him was an AI-generated deepfake, built from publicly available footage of the real executives. No system was hacked. The attackers hacked the last security control most of us have: the belief that seeing and hearing someone is proof of who they are.

That case made headlines for its size, but the same technique is now retail-scale. Scammers clone a grandchild's voice from social media clips and call grandparents with fake emergencies. They clone CEOs' voices to rush wire transfers. The FBI's 2025 internet crime data attributes hundreds of millions of dollars in reported losses to AI-enabled fraud, and reported losses are always the undercount. Meanwhile, AI writes phishing emails free of the broken English that used to give scams away — in any language, personalized, at unlimited volume.

Two practical defenses, because I refuse to hand you a scary story with no exit. First, families should agree on a verification question or code word that a voice clone can't know; the FBI itself recommends this. Second, at work and at home: any urgent, unusual request for money or credentials gets verified on a different channel — hang up and call the person back on the number you already have, not the number that called you. This one habit defeats almost the entire category.

The thread running through all of it

Look at the three failure types again. Baked-in limits that vendors under-disclose. Negligent deployment with no verification. Deliberate abuse with industrial efficiency. Different problems, but notice what they have in common: in every one, the harm lands on someone who had no say. The job applicant screened by a biased model. The client whose lawyer outsourced the thinking. The grandmother who trusted her ears.

For every other technology that can hurt bystanders — from cars to pharmaceuticals to airplanes — we eventually built rules: testing before release, disclosure of limits, liability when it fails. For AI, almost none of that exists yet. Which raises the obvious question: who exactly is in charge of this?

Next in the series: the answer, which is "almost nobody" — and why that should bother you more than anything I've written so far.

Sources worth your time: the American Bar Association on AI citation sanctions, Damien Charlotin's public AI Hallucination Cases database, and the FBI's Internet Crime Report.