Delivery
Why AI features take five times longer than the demo suggested
The prototype took two days and was genuinely impressive. Then it took four months to ship. Here is where that time goes, why it is not anyone's fault, and how to plan for it.
Key takeaways
- The demo proves the capability exists. It does not touch the work that makes a capability shippable, which is most of the budget.
- Four things the prototype never had to handle: inputs you did not choose, cost per request, latency under load, and a defined behaviour for being wrong.
- Ask for the evaluation set before you approve the build. If nobody can say how quality will be measured, the timeline is fiction.
- Inference is a recurring per-use cost, so an AI feature can be successful and unaffordable at the same time. Set the ceiling before building.
- Plan for the model call to be a small fraction of the engineering. Everything around it is the project.
There is a reliable pattern in AI feature work, and if you have funded one you have probably lived it. The prototype takes two days and is genuinely impressive, impressive enough that it gets shown to the board. Then it takes four months to ship, and everybody is surprised and slightly suspicious.
Nothing has gone wrong. The demo simply proved the easy part, and the easy part is not where the money goes.
What the demo did not have to handle
A prototype runs on inputs its author chose, once, with no cost ceiling and no consequence for being wrong. Production removes all four of those at the same time.
| The demo | Production |
|---|---|
| Inputs someone picked | Whatever users type, including nonsense and abuse |
| Ran once | Thousands of times a day, at once |
| Cost irrelevant | A per-request bill against a real margin |
| Latency irrelevant | Users leave; timeouts cascade |
| Wrong was funny | Wrong is a support ticket, or a liability |
| Quality judged by looking at it | Quality has to be measured to be managed |
The engineering that closes each of those gaps is the four months.
Where the time actually goes
Measuring quality. Before anything can improve, you need a way to tell whether it has. That means assembling realistic examples with acceptable answers and running them on every change. Teams that skip this cannot distinguish an improvement from a regression, so they oscillate, fix a complaint, break something nobody was watching, find out six weeks later from a customer.
Getting the right information to the model. If the feature works over your own documents or records, this dominates everything else. Which material gets retrieved, how it is ranked, how fresh it is. Research on long contexts is unambiguous that supplying more material is not a fix, accuracy depends on where relevant information sits in the input, and burying the right answer among twenty near-misses measurably degrades results.
Deciding what happens when it is wrong. A product decision, not a technical one, and it has to be made explicitly. Does the system validate and retry? Show sources so the user can check? Flag uncertainty? Route to a person? Each is a different amount of work and a different user experience.
Controlling cost and latency. Instrumented, budgeted, and tuned, routing simple requests to cheaper models, caching what repeats, keeping the input tight.
Handling the provider being unavailable. It happens. What does your product do?
The cost model is genuinely different
Most software has near-zero marginal cost: the thousandth user costs roughly what the first did. AI features do not. You pay per request.
The consequences are worth internalising before you commit:
- A generous free tier can become a real expense
- Heavy users cost more than light users, which your pricing may not reflect
- Costs scale with success, which is a strange position to be in
- A single prompt change can move the bill noticeably
This is why "set a per-request ceiling" belongs in the requirements next to the functional ones. A feature can be excellent and unaffordable, and that is a worse outcome than one that never shipped.
Questions that get you a real timeline
Ask these before approving the build. They are all answerable in plain English:
- How will we measure whether the output is good, and can I see it? If the answer is "we'll look at it", the estimate is a guess.
- What is the cost per request, and what is the ceiling?
- What happens when it is wrong? What does the user see?
- What happens when the provider is down?
- What are we shipping to a small group first, and what would make us stop?
A team with real answers has thought about production. A team that has only shown you a demo has not started yet, which is fine, as long as the plan reflects it.
What a realistic plan looks like
- Week one: build the evaluation set and establish baseline quality
- Weeks two to six: build and iterate against it, with cost instrumented from the first deploy
- Weeks seven to nine: hardening, failure handling, latency, guardrails, the interface for wrongness
- Weeks ten onward: limited rollout, watching quality, cost, latency and the support queue together
The support queue is the underrated one. It surfaces failure modes your test set was never going to contain, which is precisely the category you most need to hear about.
The reframe worth taking away
The prototype was not a waste, and it was not a lie. It answered a genuinely valuable question: is this capability achievable at all?
It just did not answer the other one, can we run this for thousands of people, affordably, and handle it being wrong? That second question is the project. Budget it as such and these builds stop surprising anyone.
Frequently asked questions
Why do AI projects take so much longer than the prototype suggested?
Because the prototype ran on inputs its author chose, once, with no cost ceiling and no consequence for being wrong. Production removes all four of those at the same time. The engineering that closes each gap, evaluation, retrieval quality, failure handling, cost control, latency, is the actual project, and it typically dwarfs the model call the demo consisted of.
What should I ask before approving an AI feature build?
How will we measure whether the output is good, and can I see that measurement? What is the cost per request and what is the ceiling? What happens when the model is wrong, and what does the user see? What happens when the provider is down? A team with clear answers has thought about production. A team that has only shown you a demo has not yet started.
How much does an AI feature cost to run?
Unlike most software, cost scales with usage, you pay per request roughly in proportion to the amount of text involved. This means a successful feature costs more than an unsuccessful one, which is an unusual position. Set a per-request ceiling as a requirement before building and instrument it from the first deployment; discovering it in the first month's invoice is the common and avoidable path.
Can we just use a bigger model to fix quality problems?
Usually not, because most quality problems are not model problems. If the feature works over your own data, the dominant factor is whether the right information reached the model at all, retrieval quality, not raw capability. Research on long contexts also shows that simply supplying more material degrades accuracy, since attention across a long input is uneven. Upgrading the model is the expensive way to not fix a retrieval problem.
How do we know when an AI feature is ready to launch?
When quality is measured rather than asserted, cost per request is inside budget at expected volume, there is a defined and tested behaviour for wrong output and for provider outages, and it has run against real users at low volume without surprises. Shipping to a small cohort first is not caution, it is the only way to find the failure modes your test set did not contain.
References & further reading
- [1]
- [2]
- [3]
Related resources
Should your startup build AI features?
A decision guide for founders under pressure to add AI, how to tell a genuine use case from a defensive one, what it costs to run, and the failure modes that cost customers rather than money.
How long does software take to build?
Why software estimates are wrong in a consistent direction, what actually determines a timeline, and how to plan around uncertainty instead of pretending it away.
Is AI-generated code safe to ship?
Most of your product's code was probably written by a machine. What that changes about your risk, the questions to ask your engineering team, and the controls that actually matter.
Turn the trend into shipped software
I contract as a senior fullstack and data engineer, building the production systems your customers and your reporting depend on.