Eight of our ten AI pilots died. Here's what the two survivors had in common

I'm not going to dress this up as a strategy. For about fourteen months we threw things at the wall. Some of it was curiosity, some of it was a board that had read an article, and a decent chunk was me wanting to play with new toys on company time.

Ten pilots. Two are running in production today. Eight are dead, and only one of those deaths was dramatic.

What follows is the whole scoreboard, then the three things the survivors shared. Spoiler: none of the three are about model quality, and one of them is a bit embarrassing.

The scoreboard

Before anyone despairs at a 20% survival rate: the two survivors save us roughly £91,000 a year between them. The whole programme, failures included, paid back in under four months. Pilot portfolios are supposed to have a low hit rate. The mistake isn't failing — it's failing slowly and quietly.

Trait one: a single named human who wanted it to exist

Both survivors had an owner who had personally been annoyed by the problem. Invoice extraction: Priti in finance, who had been keying in supplier PDFs for six years. Support triage: Dev, who ran the inbox and could recite the top ten questions in his sleep.

Every dead pilot had a sponsor rather than an owner. A sponsor approves things. An owner notices when it breaks at 4pm on a Tuesday, and cares.

The internal chatbot (#3) is the clearest case. Everybody thought it was a great idea. Nobody owned the documents it read from. Within four months it was confidently quoting a 2021 process that had been superseded twice, and people stopped trusting it — which is a one-way door. Marla made exactly this point in the RAG thread: retrieval needs someone owning document hygiene forever.

Trait two: the work already existed as a written-down process

Invoice extraction had a documented approvals flow. Support triage had a categorisation scheme in the helpdesk. Both had rules a person had already written down and argued about.

The failures were mostly attempts to automate things that lived in people's heads. Predictive maintenance (#5) died because "the engineers just know when it sounds wrong" turned out to be literally true and completely unavailable as training data. We spent £6,300 discovering that our sensor logs had a five-year gap nobody had mentioned.

"If you can't write the rule down, you can't automate it — you can only pretend to." — something Priti said in a review meeting that I've quoted about forty times since.

Trait three (the embarrassing one): the survivors were the boring ones

Invoice extraction and email triage were the two least interesting projects on the list. No demo value. Nothing to show the board. Both were suggested by the people doing the work, not by anyone in a strategy meeting.

Meanwhile the shift-rota optimiser (#8) had a beautiful interface, genuinely clever constraint solving, and was killed in a single meeting because we hadn't spoken to the union before building a thing that rearranged people's weekends. That one's entirely on me. It's also the reason our current process requires a named "who does this affect and have we asked them?" section before any pilot gets budget.

Three failure modes worth naming

The demo trap

A pilot that impresses in a meeting is optimised for meetings. Ours died at the point where someone had to use it on a Wednesday afternoon with real, messy inputs. Test on your worst data on day one, not your best data on day one and your worst data in month three.

The tiny-win trap

Warehouse photo QA (#9) worked flawlessly. It saved 20 minutes a week. It also required someone to maintain a camera rig, a model endpoint and an alerting rule. The maintenance cost more attention than the saving was worth, so we switched it off — and I'd count that as a good decision, not a failure.

Our rule now: if the annual time saved is less than 40 hours, don't automate it. Write it on a checklist and move on.

The "nobody asked" trap

Rota optimiser, CV screening, board-pack summariser. Three pilots built for people who hadn't asked and, in two cases, actively didn't want them. Cost: about £5,000 and a fair bit of goodwill.

What our pilot template looks like now

1. Who is annoyed by this? Named person. If the answer is "the business", stop.
2. Where's the written process? Link it. If it doesn't exist, that's the first project.
3. Who does this affect, and have we asked them? Added after the rota incident.
4. What does "good" look like, numerically? Baseline measured before we build anything.
5. Decision date. Maximum eight weeks out. In the calendar at kickoff.
6. What's the annual hours saved if it works? Under 40, we don't start.
7. Spend cap and kill switch. Both in place before the first live run. See Gwen's budget write-up for why.

Further reading

• Our AI use policy guide — the governance side of the same problem
• The handoff loops thread — how the technical failures actually look
• NIST AI Risk Management Framework — the "Map" function is a decent sanity check on step 3 above

---

Replies (52) — highlights