This started as a mildly heated conversation in our Wednesday call. Tobi said fine-tuning is "a trap for people who haven't tried harder at retrieval". I said that's an easy thing to believe when your documents don't contain 4,000 product codes.
Marla made us write it up properly instead of arguing. So here's the honest version, with the numbers from three real projects.
First, in plain words
RAG — retrieval-augmented generation — means: keep your knowledge in a searchable store, find the relevant bits at question time, and paste them into the prompt. The model doesn't "know" your business; it's handed the right pages each time.
Fine-tuning means: take a base model and train it further on your own examples so it internalises a style, a format, or a set of labels. It changes how the model behaves, not what facts it can look up.
The line I use with non-technical colleagues: RAG gives the model a filing cabinet. Fine-tuning sends it on a training course. You almost never want the training course to teach it facts that change weekly.
The question that settles it 80% of the time
Ask this: "Is the thing I want fixed a knowledge problem or a behaviour problem?"
- "It doesn't know our refund policy" → knowledge problem → RAG.
- "It knows the policy but writes six paragraphs when we need two lines in our house style" → behaviour problem → fine-tune (or, first, try a much better prompt with three examples).
- "It can't reliably output our internal 14-field JSON schema" → behaviour problem, but try structured outputs first. Genuinely, try that first.
Three real projects, three answers
Project A — internal policy assistant (RAG won, easily)
1,800 HR and finance documents, updated constantly. Fine-tuning would have needed retraining every time someone edited the expenses policy. We used a straightforward retrieval setup with chunked documents, hybrid keyword + vector search, and citations in every answer.
Cost: £340 to build (two days of Tobi's time), £62/month running. Accuracy on our 120-question eval set: 91%, and every answer links its source, which is what compliance actually cared about.
Project B — product code classifier (fine-tuning won, clearly)
Incoming supplier descriptions had to map to one of 4,100 internal SKU categories. Retrieval kept returning plausible-but-wrong neighbours. We fine-tuned a small model on 6,000 labelled historical examples.
Cost: £890 for data prep and training runs, £0.0004 per classification after. Accuracy went from 73% (prompted frontier model, expensive) to 94% (fine-tuned small model, cheap). Latency dropped from 2.4s to 310ms.
Project C — proposal drafting (both, in the end)
RAG for the facts (past proposals, pricing tables, case studies), light fine-tuning for the house voice. We resisted this for months because "both" felt like a cop-out. It wasn't. It was just correct.
Cost comparison, roughly
| Dimension | RAG | Fine-tuning |
|---|---|---|
| Time to first working version | 1–3 days | 1–3 weeks (mostly data prep) |
| Upfront cost (typical SME project) | £300–£1,500 | £800–£6,000 |
| Per-request cost | Higher — you're paying for retrieved context every time | Lower — shorter prompts, often a smaller model |
| Handles frequently changing facts | Yes, natively | No — needs retraining |
| Citations / auditability | Straightforward | Hard |
| Enforces tone and format | Partially, via prompt | Very well |
| Breaks when the base model updates | Rarely | Sometimes — you may need to redo it |
The order we'd try things in now
- Better prompt + 3–5 examples. Free. Fixes more than people expect. Do this before anything else.
- Structured outputs / JSON schema. Solves most "it won't follow the format" complaints.
- RAG. If the gap is knowledge, this is your answer. Spend your effort on chunking and search quality, not on the model choice.
- Fine-tune a small model. When you have thousands of consistent examples and a narrow, stable task. The cost win at volume is real.
- Both. When you've genuinely got a knowledge problem and a behaviour problem. It's allowed.
Crew verdict
Tobi's "fine-tuning is a trap" line is 80% right and he hates that I'm saying it that way. Most teams reaching for fine-tuning have a retrieval problem, a prompt problem, or a data-quality problem wearing a lab coat.
But if you're doing high-volume classification or extraction on a stable taxonomy, fine-tuning a small model is one of the biggest cost and latency wins available to you. Project B pays for itself every eleven days.
References people posted
- OpenAI fine-tuning guide
- Anthropic — search and retrieval patterns
- The original RAG paper (Lewis et al., 2020) — still worth twenty minutes
- Hugging Face training docs for the self-hosted route
Replies (112) — the sharp ones
I'll concede Project B. But I'd add: every fine-tune I've seen fail, failed on data quality, not on training. If your 6,000 examples were labelled by three people who disagreed with each other, you've just taught a model to be confidently inconsistent.
Nobody's mentioned the maintenance side. RAG needs someone owning document hygiene forever. We had an assistant confidently citing a 2019 policy for eight months because nobody deleted the old PDF. That's not a model problem, that's a filing problem — but it lands on the same people.
Underrated middle option: prompt caching. We cut our RAG per-request cost by about 60% just by caching the stable part of the context. Took an afternoon. Should honestly be step 2.5 in Priya's list.
Adding that to the list, you're right. Anyone reading this later: Hansi's point is the cheapest win in the whole thread.
Still stuck between the two?
Post your use case in the thread with three example inputs and what "good" looks like. Someone will usually spot within a day whether you've got a knowledge problem or a behaviour problem. It's the single most useful thing this forum does.
Ask the crew → Back to all threads