RAG or fine-tuning: how to choose, and why it is usually RAG

The question usually arrives phrased as "should we fine-tune a model on our data?" — and the phrasing already contains the mistake. Fine-tuning is not how you teach a model facts. It is how you teach a model behaviour.

Get that distinction right and the decision mostly makes itself.

What each one actually does

Retrieval-augmented generation leaves the model alone and changes its input. At question time you search your own content, pull the relevant passages, and put them in the prompt. The model reasons over text it has just been handed.

Fine-tuning changes the model's weights by training it on example inputs and outputs. It adjusts how the model responds — tone, format, structure, domain vocabulary — far more reliably than it adjusts what the model knows.

The decision framework

Four questions settle it in almost every case:

1. Does the answer depend on data that changes — prices, policies, inventory, tickets, documents? → RAG. Retraining a model every time a price changes is not a strategy.
2. Do you need to cite sources, show provenance, or survive an audit? → RAG. A fine-tuned model cannot tell you where it got something; a retrieval pipeline can link to the paragraph.
3. Is the problem a rigid output format that prompting keeps getting wrong — strict JSON, a regulatory form, a specific extraction schema? → fine-tuning is a genuine fit.
4. Are you trying to cut cost or latency by getting a small model to imitate a large one on a narrow task? → fine-tuning, and this is one of its best uses.

In practice most enterprise use cases are question one or two, which is why RAG is the right starting point for the large majority of them.

What people get wrong about RAG

RAG being the usual answer does not make it the easy answer. The failures are rarely in the model:

• Chunking. Splitting documents on a fixed character count cuts sentences in half and destroys the meaning you were trying to retrieve. Chunk on structure — headings, sections, list items.
• Retrieval quality. Pure vector search misses exact terms like part numbers and error codes. Hybrid search — vectors plus keyword — fixes more real-world failures than any prompt change.
• Stale indexes. If your pipeline re-indexes nightly and your prices change hourly, you have built a confidently wrong answer machine.
• No evaluation. Without a set of questions with known-good answers, you cannot tell whether a change helped. Most RAG systems are tuned on vibes.

The model is almost never the bottleneck. The retrieval is.

What fine-tuning really costs

The training run is the cheap part. The costs that bite are the dataset and the lock-in.

You need enough high-quality labelled examples — usually hundreds to thousands — and someone has to produce and check them. Then, every time a better base model appears, your tuned model is a fork you have to re-do to benefit. With RAG you change one line and inherit the improvement.

The hybrid most teams land on

Production systems frequently end up using both, and the split is clean: retrieval for knowledge, fine-tuning for behaviour. Fine-tune a small open model so it reliably produces your output format and speaks your domain's vocabulary, then put it behind a retrieval pipeline that supplies the facts.

That gives you cheap, fast, consistently formatted answers grounded in data you can update without touching the model. But start with RAG alone. Add fine-tuning when you have measured a specific failure it would fix — not before.

Where to start

1. Write twenty real questions with known-correct answers. This is your eval set and it is the most valuable artefact you will build.
2. Build the dumbest possible RAG: chunk on headings, hybrid search, top five results in the prompt.
3. Measure. Where it fails, diagnose whether retrieval returned the right passage. Usually it did not.
4. Fix retrieval before touching the model. Then re-measure.

Most teams that believe they need fine-tuning discover, at step three, that they need better chunking.

Frequently asked questions