Most AI projects fail after launch, once real work arrives. The model is rarely the cause. A demo shows the machine can do the task. A deployment has to show the business can live with the machine doing it.
MIT NANDA's 2025 report, The GenAI Divide, found that about 95 percent of generative AI pilots produced no measurable profit and loss impact. This note covers the work around the model, which is often left unspecified until the budget is spent.
The difference between a demo and a deployment
| Where | In the demo | In production |
|---|---|---|
| Inputs | A few examples someone chose | Whatever arrives on Monday, including the malformed ones |
| Being wrong | Costs a laugh in the room | Posts an invoice, denies a claim, emails a customer |
| Permissions | A sandbox, nothing real attached | Your mail, your ERP, your customer records |
| Cost | A small token bill | Metered spend with no natural stopping point |
| Who is watching | Everyone, it is a demo | Nobody, it is Tuesday |
| Accountability | The person presenting | Undefined, until something goes wrong |
The three questions a demo never answers
- /01
What is this allowed to touch? A demo runs in a sandbox. A deployment runs against real systems, and in most pilots nobody ever wrote its permissions down.
- /02
What happens when it is wrong? Every model is wrong sometimes. The question is whether a mistake is caught, traced to a person and cheap, or silent and expensive.
- /03
What does it cost when nobody is watching? Metered APIs plus freedom to act is an open cheque. Most pilots have no limit at all, and find out what one would have been worth in the first bad week.
Why the pilot looks so good
A pilot shows what the system can do under easy conditions, and everyone involved wants it to work. The inputs are chosen. The reviewer is the person who built it. The edge cases are noted and set aside for later. None of that is dishonest. It is not evidence about production, and it is routinely mistaken for evidence about production.
The failure then arrives slowly. The system works for two weeks. Then an unusual input appears, quality slips, trust drops, and people take the work back. Six months later there is a line in the budget and no one can point to what it bought.
What the projects that survive do differently
- /01
They write down scope before they write the prompt. What the agent may read, what it may act on, and what it must never touch.
- /02
They decide what happens when confidence is low, and the answer is to stop and ask a named approver rather than to guess.
- /03
They cap spend in money, per run and per day. The system enforces the cap, so nobody has to watch a chart.
- /04
They record every run so someone outside the team can read it months later. The first audit will ask about the period nobody recorded.
- /05
They keep operating it after launch. Automations rarely break on day one. They get worse over time, and someone has to own that.
None of this is unusual. It is the ordinary discipline of putting anything into production, applied to a part that can give different answers to the same input. The reason it gets skipped is that the demo was so convincing.
GatehouseGatehouse, Surehand's control plane, enforces the scope, the hold, the spend limit and the record while the agent runs. Surehand runs each deployment after launch. See Gatehouse
Keep reading
What is AI governance?
Six controls that decide what an AI system may do, on whose authority, at what cost, and with what evidence. A practical guide rather than a policy template.
How to monitor AI agents in production
Four signals worth watching: quality drift, hold rate, spend per run and latency. Uptime alone does not show whether the answers are still good.
