You want a language model to work with your information — your documentation, your house style, your policies. There are three ways, they cost wildly different amounts, and beginners routinely pick the most expensive one first.
Prompting
Change the words you send. No training, no infrastructure, instant to iterate.
The techniques that reliably help: state the role and the desired format explicitly; give two or three worked examples of what good output looks like (few-shot prompting); ask the model to work through its reasoning before answering; and put your most important instruction at the end, where it carries most weight.
Its limit is the context window. You cannot paste 400 pages of documentation into every request — and even where you can afford to, quality degrades when the relevant fact is buried in a wall of irrelevant text.
RAG — retrieval-augmented generation
Instead of giving the model everything, fetch only the relevant few passages and put those in the prompt. That is RAG, and it is the right answer for most “make it know our stuff” problems.
🧪 How RAG works, end to end
At question time:
1. Embed the user’s question.
2. Find the chunks whose vectors are closest, by cosine similarity.
3. Paste those chunks into the prompt along with the question.
4. The model answers using what it was given.
The model has learned nothing new. It has simply been handed the right page at the right moment.
The advantages are substantial: update a document and the answer changes immediately, with no retraining. You can cite sources, because you know which chunks were used. And it reduces made-up answers, because the facts are in front of the model rather than recalled from weights.
⚠️ Where RAG goes wrong
- Retrieval quality is the whole system. If the right chunk is not retrieved, the model cannot answer — and will often answer anyway, from its own weights, with no signal that it is guessing.
- Chunking matters more than people expect. Too small and you lose context; too large and you retrieve mostly noise. Overlapping chunks of a few hundred tokens is a common starting point.
- It cannot answer questions spanning the whole corpus. “What are the themes across all 500 documents?” is not a retrieval question — you will get whichever five chunks scored highest.
Fine-tuning
Continue training the model on your own examples, adjusting its weights. This is real training — backpropagation and gradient descent, as everywhere else on this site.
What it is genuinely good at: teaching form. A consistent house style, a specific output structure, a domain’s phrasing conventions, or reliable adherence to a format. Given a few hundred good examples, fine-tuning teaches these better than any prompt.
⚠️ Fine-tuning is a poor way to add facts
If your goal is “the model should know our current pricing”, you want RAG. If it is “the model should always reply in our support tone, in this JSON shape”, fine-tuning earns its cost.
Modern fine-tuning is cheaper than it used to be: LoRA trains a small number of additional parameters and freezes the rest, giving most of the benefit for a fraction of the compute and storage.
Choosing between them
| Your problem | Reach for |
|---|---|
| Output format or tone is wrong | Prompting, then fine-tuning if it must be perfect every time |
| It does not know our documents | RAG |
| Information changes frequently | RAG — fine-tuning would go stale immediately |
| We need citations | RAG — only retrieval knows the source |
| Specialist vocabulary it handles badly | Fine-tuning |
| Responses are too slow or costly | Fine-tune a smaller model on a larger one’s outputs |
They also combine: fine-tune for tone, and use RAG for facts. That pairing is common in production, and it works because the two are solving genuinely different problems.
🔑 Key Takeaways
- Prompting changes the input, RAG supplies documents, fine-tuning changes the weights.
- Try them in that order — cheapest and fastest first.
- RAG is the right answer for facts, changing information and citations.
- Fine-tuning teaches form and style, not facts.
- In RAG, retrieval quality is the system — if the wrong chunk comes back, nothing downstream can save it.
Further reading
Hugging Face’s fine-tuning guide covers the training loop and LoRA, and the original RAG paper sets out the retrieve-then-generate idea that every current implementation builds on.
Where to go next
- Embeddings explained — the machinery RAG retrieval runs on
- What is a transformer? — what you are adapting
- Bias in machine learning — worth reading before you deploy anything
- Start Here — the full learning path