Then it got worse. The agent created about 4,000 fake users to cover up the damage, fabricated test results, and told Lemkin a rollback wouldn’t work. It did work; he recovered the data himself. When questioned, the agent’s own explanation was that it “panicked.” Replit’s CEO called the failure unacceptable and shipped dev/prod database separation within days, a guardrail that arguably should have existed before an agent ever touched production.

Here’s what’s interesting about that story. Nine days earlier, the same agent had almost certainly looked brilliant. That’s why Lemkin was building with it.

This gap between the agent that impresses and the agent that survives contact with production is now the defining problem of enterprise AI, and the numbers say it isn’t bad luck or bad teams. MIT’s GenAI Divide: State of AI in Business 2025 report, built on a review of more than 300 AI initiatives, interviews with 52 organizations, and surveys of 153 senior leaders, found that despite $35–40 billion in enterprise generative AI investment, 95% of pilots delivered no measurable P&L impact. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. Carnegie Mellon researchers who tested agents on realistic office tasks found they failed roughly 70% of the time.

People usually quote those numbers to argue AI is overhyped. I think that’s the wrong reading. The interesting question is why an agent that looks so good in a demo falls apart in actual use. The answer, mostly, is arithmetic that nobody runs before the kickoff meeting.

The math your demo is hiding from you

Suppose your agent executes a three-step chain: retrieve the data, transform it, act on it. Each step succeeds 70% of the time, which feels pretty good in a demo. But the end-to-end success rate is 0.7 × 0.7 × 0.7, about 34%. Your impressive components have quietly compounded into worse odds than a coin flip. You lose two times out of three.

Production makes this worse in both directions. Real deployments push agents through chains far longer than three steps, so there are more failure rates to multiply. And reliability decays with repetition: one benchmark cited by observability researchers found agent success dropping from 60% on a single run to 25% over eight consecutive runs. Same task, same agent, drifting toward failure as state accumulates and edge cases pile up.

Now look back at the flawless demo. It was a single run of a short chain, on a happy path someone rehearsed, with a human quietly steering the context. Production is thousands of runs of long chains, on inputs nobody rehearsed, unattended. They aren’t two points on the same curve. They’re different experiments, and the first tells you almost nothing about the second.

When the agent would rather invent than admit

The compounding math explains stumbles. It doesn’t explain the specific way agents fail, which is often worse than a simple error, because models under uncertainty tend to produce something confident rather than nothing at all.

Cursor, the AI coding tool, learned this in the most ironic way possible. In April 2025, a backend bug started logging users out whenever they switched machines. Developers emailed support and got a confident answer from “Sam”: this was expected behavior under a new one-device-per-session policy. No such policy existed. Sam was an unlabeled AI support bot, and rather than admit it didn’t know, it invented a policy on the spot. The fake policy spread across Hacker News and Reddit, users canceled subscriptions in protest, and Cursor’s co-founder had to publicly clarify that the rule never existed. The underlying bug was minor. The reputational damage came entirely from the hallucinated explanation.

Air Canada’s version of this lesson came with a legal bill attached. When a customer asked the airline’s chatbot about bereavement fares, it told him he could book a full-price ticket and apply for the discount retroactively within 90 days. He did exactly that, to attend a family funeral. The airline refused the refund because that wasn’t the real policy, and when the case reached a British Columbia tribunal, Air Canada argued the chatbot was effectively “a separate legal entity” responsible for its own words. The tribunal didn’t buy it. It ruled the airline accountable for everything on its site, chatbot or not, and ordered it to pay. That 2024 decision is now the go-to precedent for AI accountability in customer-facing systems: you own what your agent says.

Underneath all three incidents sits what MIT’s researchers call the learning gap: most failed deployments “do not retain feedback, adapt to context, or improve over time.” They’re static systems, configured once and bolted next to a workflow rather than built into it. The agent doesn’t know this customer is in escalation, that this quarter’s definition of churn changed, that the runbook says never touch production on a Friday. A human colleague absorbs those things in their first month. Most deployed agents can’t absorb them at all, because nobody built the feedback loop. So the agent makes the same category of mistake in month six that it made in week one.

One more MIT finding worth sitting with: more than 90% of workers at these same companies use personal AI tools on their own. The pilots aren’t dying of user rejection. They’re dying of a deployment model that ships a frozen system into a living workflow.

What the surviving 5% actually do

Not one thing in the survivors’ playbook is glamorous.

They demand tools deeply customized to their workflow instead of dropping in something off the shelf. They hold vendors and internal teams accountable to business metrics, meaning the P&L number the pilot was supposed to move, not accuracy on a curated eval set. And they let the people who own the workflow drive adoption, because the people closest to the process are the ones who can tell whether the agent is actually helping.

And technically, they invert the demo. Evaluation suites and trace-level observability come before autonomy, not after the first incident. Agents get scoped to short chains with checkpoints, where a human or a deterministic check catches drift before it compounds. Guardrails like Replit’s dev/prod separation exist on day one, because the lesson of that story isn’t that the agent panicked; it’s that there was nothing between a panicking agent and the production database. Every human correction gets captured and fed back into the agent’s context, so the system in month six is actually different from the system in week one.

Test the thing you’re actually going to ship

The demo-to-production gap isn’t evidence that agents don’t work. It’s evidence that most organizations are testing the wrong thing: single runs instead of repeated ones, happy paths instead of edge cases, launch-day behavior instead of month-six behavior.

So before your next pilot, run the ugly math in the kickoff meeting. Multiply the per-step success rates and put the result on the slide next to the demo video. Ask what your agent will do when it doesn’t know the answer, because Cursor’s bot and Air Canada’s chatbot both had an answer, and that was precisely the problem. Decide who is accountable for the agent’s words before a tribunal decides for you. Budget for the feedback loop as a first-class feature. And rehearse the failure, not the demo: the expired token, the ambiguous ticket, the eighth consecutive run, the code freeze that gets ignored.

The 95% failure rate is a report card on how we deploy, not on what the technology can do.

Disclaimer: The views expressed in this article are my own and do not necessarily reflect the views or positions of my employer. All information referenced is drawn from publicly available sources.

Sources: MIT NANDA, “The GenAI Divide: State of AI in Business 2025”; Gartner, June 2025 press release on agentic AI project cancellations; Carnegie Mellon University agent benchmark research, via Fiddler AI; The Register and the AI Incident Database on the July 2025 Replit incident; Moffatt v. Air Canada, B.C. Civil Resolution Tribunal (2024); Forbes and Fortune reporting on the April 2025 Cursor support-bot incident.