Adding AI to a SaaS product that already works

Most teams asking how to add AI to an existing SaaS have already built the demo. A weekend with an API key produced something impressive, and the question is really: how does this become a feature customers rely on, without becoming a support problem?

The model is the easy part. What decides whether this works is which job you give it, and the plumbing around it that nobody demos.

Start with the job, not the technology

The strongest AI features do a job the product already does badly, slowly, or not at all. The weakest ones add a chat box to a product nobody wanted to talk to.

Three shapes work far more often than they fail, because in each one the user can tell immediately whether the output is right.

• Extraction. Pull structured data out of documents, emails or free text your users are already pasting in. The result is checkable at a glance.
• Drafting. Produce a first version — a reply, a summary, a description — that a human edits. Being 80% right is genuinely useful when editing is faster than writing.
• Search and Q&A over the customer's own data. This is retrieval-augmented generation, and it works because answers can cite the source they came from.

Where it goes in your architecture

The mistake that costs the most is calling a model API inline, inside the request, the way you would call your own database. Model latency is measured in seconds and varies by an order of magnitude between calls. Put that in the request path and you have rebuilt the concurrency problem that breaks MVPs under load.

The pattern that holds up: the request enqueues a job and returns immediately, a worker pool calls the model, and the result is delivered by polling or a websocket. That is the same shape as any other slow third-party integration — we wrote about picking the queue for it.

Two things follow from that, and both are easy to skip and expensive to retrofit:

1. Make the jobs idempotent. Model calls fail, time out, and get retried. A retry that charges the customer twice or sends two emails is worse than the original failure.
2. Cache aggressively. Identical inputs are common in practice, and inference is the one part of your stack that bills per request.

What it costs to run

Inference is a variable cost that scales with usage, which is unlike almost everything else in a SaaS cost base. Current per-token rates are published by the vendors — OpenAI and Anthropic both list them — and they fall regularly, so build the estimate from your own token counts rather than from a figure you read once.

The number that surprises teams is not the price per token, it is the token count. A RAG feature that stuffs ten retrieved documents into every prompt can use twenty times the input tokens of a naive implementation, for a marginal quality gain. Measure tokens per request before you launch, not after the first invoice.

Model your unit economics per customer, not in aggregate. One power user on an unlimited plan can cost more in inference than they pay you, and averages hide that until it is a pattern.

The parts that decide whether anyone trusts it

Show the source

Any answer drawn from the customer's data should link to the record it came from. Citations do more for trust than accuracy improvements do, because they let the user verify in one click instead of deciding whether to believe you.

Make it correctable

Give users a way to fix a wrong output and have the fix stick. A feature that is wrong and uncorrectable gets abandoned after two bad experiences; one that is wrong but easily corrected keeps its users.

Treat prompts as untrusted input

The moment user content reaches a prompt, prompt injection is in scope — and if your model can call tools or read other customers' data, so is everything downstream. The OWASP Top 10 for LLM Applications is the standard checklist, and NIST's AI Risk Management Framework is worth a read before you sell this into an enterprise.

Know what you are sending

If your customers are in the UK or EEA, sending their personal data to a model vendor is a processing decision with contractual consequences. The ICO's guidance on AI and data protection covers what you need in place. Enterprise buyers will ask; having the answer ready shortens the sale.

A sensible first release

1. Pick one job where a wrong answer is obvious to the user, and ship it to a subset of customers behind a flag.
2. Run the model out of the request path from day one. Retrofitting that later means rewriting the feature.
3. Log every prompt, response, token count and latency. You cannot debug or price what you did not record.
4. Put a correction path and a source link in version one, not version two.
5. Review real transcripts weekly for the first month. This is the only reliable way to find out what your users actually asked for.

That is roughly the shape of a three-to-four week build. If you want a second opinion on which job to give the model, or on what it will cost to run at your volumes, that is what our AI and LLM integration work covers.

Frequently asked questions

References

1. OWASP Top 10 for LLM Applications — OWASP
2. AI Risk Management Framework — NIST
3. Guidance on AI and data protection — ICO
4. API pricing — OpenAI
5. Pricing — Anthropic