Strategy

Should your startup build AI features?

A decision guide for founders under pressure to add AI, how to tell a genuine use case from a defensive one, what it costs to run, and the failure modes that cost customers rather than money.

3 min readBy Mayur Kale

Key takeaways

  • Build an AI feature when the task is genuinely fuzzy, language, judgement, unstructured input. If a rule or a search index would do it, use those; they are cheaper and predictable.
  • The engineering cost is not the model call. It is evaluation, retrieval quality, failure handling and cost control, which typically dominate the build.
  • Inference is a recurring per-use cost, so unit economics need checking before you build, not after the first bill.
  • Design for being wrong. A feature that fails invisibly inside a workflow that assumed it was right is worse than no feature.
  • 'Our competitors have it' is a reason to investigate, not a reason to ship. Defensive AI features are the ones users notice and dislike.

Most founders asking this question are under pressure, from investors, from a competitor's launch, or from a general sense that not having AI is becoming a position that needs defending.

That pressure is real. It is also a poor basis for a product decision, and users are noticeably good at spotting features that exist for the pitch deck.

When AI is genuinely the right tool

The honest test is whether the task is fuzzy. Language, judgement, unstructured input, many valid answers.

Good candidates:

  • Summarising long or messy documents
  • Extracting structured data from unstructured input, emails, PDFs, transcripts
  • Answering questions over your own content
  • Drafting things a user then edits
  • Classifying cases where the rules have too many exceptions to enumerate

Poor candidates, where something cheaper is better:

  • Anything expressible as rules, use rules; they are faster, free and auditable
  • Exact lookup, use a database
  • Finding known items, use a search index
  • Calculation, use arithmetic
  • Anything where being wrong is unacceptable and unverifiable

What it actually costs to build

The prototype takes days. The shippable feature takes months. That gap is not incompetence, it is the work the prototype did not have to do:

  • Evaluation. A set of realistic inputs with acceptable outputs, run on every change. Without it you cannot distinguish an improvement from a regression, and you will make both.
  • Retrieval quality. If the feature works over your own data, this dominates output quality. Chunking, ranking and freshness are the actual engineering.
  • Failure handling. Validation, fallbacks, and a defined behaviour when the model is unavailable or wrong.
  • Cost and latency control. Instrumented per request, against a budget you set upfront.
  • Interface design for wrongness. Usually the difference between a feature users trust and one they abandon.

Plan for the model call to be a small fraction of the effort.

The cost model is different

Most software has near-zero marginal cost. AI features do not, you pay per request, roughly in proportion to how much text is involved.

This has real consequences:

  • A free tier can become genuinely expensive
  • Heavy users cost more than light ones, which your pricing may not reflect
  • Costs scale with success, which is an unusual position to be in
  • A prompt change can move your bill substantially

Work out the per-request cost before you build, and set a ceiling as a requirement alongside the functional ones. Then instrument it from the first deployment, because the alternative is finding out at the end of the month.

The failure mode that costs customers

Being wrong is a property of these systems, not a bug to be eliminated. The question is what the user experiences when it happens.

Acceptable, in rough order of effort:

  1. Structural validation, if the output should parse, reject what does not and retry
  2. Visible sources, users forgive a wrong answer they can check
  3. Confidence signals, flag the cases the system is unsure about
  4. Trivial correction, an edit button is worth more than a few points of accuracy
  5. Human approval, for anything consequential

Unacceptable: silent wrongness inside a workflow that assumed correctness. That is how an AI feature costs you customers rather than money.

A reasonable sequence

  1. Name the task. One specific job a user currently does manually and dislikes.
  2. Check the cheaper option. Would rules, search or better interface design solve it? If so, do that.
  3. Prototype in a week. Establish that the capability exists at all.
  4. Build the evaluation set before building the feature. Thirty realistic examples is enough to start.
  5. Set cost and latency ceilings. Alongside the functional requirements.
  6. Ship to a small cohort. Real usage finds failure modes your evaluation set will not.
  7. Measure quality, cost and the support queue together.

The honest answer

Build an AI feature when there is a fuzzy task your users currently do by hand and dislike, when you can measure whether the output is good, and when the unit economics work.

Do not build one because a competitor has one. That reasoning produces features that exist to be mentioned rather than used, and users can tell the difference immediately.

Frequently asked questions

How do I know if my product needs an AI feature?

Look for a task where the input is unstructured, the correct answer is a matter of judgement rather than rule, and users currently do it manually and dislike it. Summarising, extracting from messy documents, drafting, classifying ambiguous cases, answering questions over your own content. If the task can be expressed as rules or solved with a good search index, use those instead, they are cheaper, faster and predictable.

What does it cost to run an AI feature?

Unlike most software, cost scales with usage, you pay per request, roughly in proportion to the amount of text involved. This means unit economics need checking before you build. Work backwards from what a user or transaction is worth, set a per-request ceiling, and treat exceeding it as a failure. The usual levers are routing simple cases to smaller models, caching, and keeping the context tight.

What is the most common mistake startups make with AI features?

Assuming the prototype represents most of the work. The demo takes days; the shippable feature takes months, because evaluation, retrieval quality, failure handling, latency and cost control are the actual engineering. Teams that plan as though the model call was the project routinely miss dates by a wide margin.

Do we need our own model?

Almost certainly not. Training or fine-tuning a model is rarely the right first move, hosted models with good retrieval over your own data solve the overwhelming majority of product use cases at a fraction of the cost and complexity. Revisit only if you have a specific, measured quality gap that retrieval and prompting cannot close.

How do we handle the model being wrong in front of customers?

Decide it explicitly as a product question before building. Depending on stakes: validate the output structurally and reject what does not parse, show sources so the user can check, add a confidence signal, make correction trivial, or require human approval for consequential actions. The unacceptable option is silent wrongness inside a workflow that assumed correctness.

References & further reading

  1. [1]
  2. [2]
  3. [3]

Turn the trend into shipped software

I contract as a senior fullstack and data engineer, building the production systems your customers and your reporting depend on.

Explore services↗