The demo that never shipped

A few years into this work, I stopped getting excited by demos. Not because they’re fake. The good ones are real. An engineer wires up an agent on a Friday, it handles a genuinely hard task on the first try, and everyone in the room can see the future. The screenshots go in the deck. Someone starts drawing a roadmap.

Then it’s October, the agent is switched off, and nobody wants to talk about why.

What happened is simple. The demo answered one question: can this thing succeed? Production asks a harder one. Can it fail without anyone noticing? Those are not the same project, and the second one is where most of the real engineering lives. It’s also the part the demo never shows you.

The part nobody puts in the estimate

People assume the distance between a demo and a deployment is polish. A little more testing, tidy up the prompts, ship it. It isn’t. It’s a different kind of work, and most of it is unglamorous.

What does the agent do when the model returns confident nonsense? When Salesforce rate-limits you at 9am on the morning of a board meeting? When a CSM overrides a decision and, six months later, someone has to reconstruct why it was made? None of that is in the demo. The demo only walks the happy path. That’s the entire point of a demo.

So that is the work I actually sell. The model is the cheap part now. Anyone can call an API. What I get paid for is everything wrapped around it: the schema checks, the fallbacks, the override surface, the one number on a dashboard that tells you it’s drifting. That is what lets a revenue team hand a real workflow to an agent and stop watching it.

Four questions I ask first

Before anything goes near production, I run the build through four questions. Miss any one of them and you have a demo. That is fine, as long as the room agrees that is what it is.

• What happens on the worst input, not the best one? “It probably handles it” is a demo answer. In production the behaviour on the bad path is designed and known, not hoped for.
• Who owns it when it’s wrong? Every automated workflow needs a name attached to it. “The agent did it” is not a sentence you can say to a customer.
• How would you find out it’s degrading? If the answer is a support ticket, you found out too late. The system has to watch itself.
• What does it touch that it has no business touching? Demos grab everything in reach. A real system pulls only the fields it needs, because every extra scope is one more thing you’ve signed up to defend.

I open with these instead of model choices or framework arguments, because they sort a project in about five minutes. They tell you whether you’re looking at something built for the stage or something you can put your name on. I only build the second kind. I’d rather have that conversation on day one than in October.