The brilliant new hire problem
Imagine you hire the smartest person in the world. They've read half the internet, they write beautifully, and they never get tired.
On their first day, a customer asks: "What's your refund policy for annual plans?"
Your genius has never read your company wiki. So they do what smart people do when they don't know: they make up something that sounds right. Confidently. In perfect grammar.
That's a large language model without RAG. Smart, fluent, and occasionally inventing your refund policy.
The fix: an open-book exam
RAG turns the closed-book exam into an open-book one. For every question, you quickly find the right pages in your documents and say: "Answer using these."
It happens in two halves:
Once (and whenever docs change): you take your documents, cut them into small chunks, turn each chunk into an embedding (a list of numbers that captures its meaning), and store them in a vector database.
For every question: you turn the question into an embedding too, find the chunks with the closest meaning, and send the question plus those chunks to the model. The model reads them and answers, ideally citing which document it used.
"Embedding" sounds scary, but it's just a way to make "refund policy" and "how do I get my money back" land close together, even though they share no words.
Where it actually helps a business
- Customer support. Answers drawn from your real help center, with links. Agents get a first draft in seconds instead of searching five tabs.
- Internal knowledge. "How do I request a new laptop?" "Which client uses the old billing API?" New hires stop interrupting senior people every ten minutes. Senior people rejoice.
- Sales and proposals. Pull the right case studies, specs and past answers into a new proposal. I worked on the frontend of an AI product that drafts construction bids this way, and the time saved on each bid is the whole sales pitch.
- Contracts and compliance. "Which of our contracts have a termination clause under 30 days?" Answered with the clauses themselves, not a guess.
Why not just train the model on our data?
Fair question. Short answer: RAG is usually cheaper, faster and easier to trust.
| | RAG | Fine-tuning | | --- | --- | --- | | Docs changed today | Re-index, done | Train again | | Shows its sources | Yes, it can cite the chunks | Not really | | Respect user permissions | Filter what you retrieve | Hard, it's baked in | | Cost to start | Low | Higher |
Fine-tuning is great for teaching a style or a format. RAG is great for teaching facts that change. Most business questions are the second kind.
Where RAG goes wrong
RAG is not magic. It's a librarian, and librarians are only as good as the library.
- Garbage in, garbage out. If your wiki has three conflicting refund policies, the AI will find all three. Clean up the sources first.
- Bad chunking. Cut a document in the wrong place and the answer gets split across chunks. Keep chunks meaningful: a section, a FAQ entry, a clause.
- Ignoring permissions. If an intern can retrieve the CEO's salary doc, that's not an AI problem, that's a career-ending problem. Filter by access before retrieval.
- Never measuring. Build a list of 30 real questions with known answers and test against it every time you change something. Otherwise you're just vibe-checking.
- Trusting it blindly. RAG greatly reduces made-up answers, it doesn't make them impossible. Ask for citations and show them to users.
How I'd start tomorrow
If you're on AWS, managed options like Bedrock Knowledge Bases can handle the indexing and search part for you. Either way, the hard part isn't the AI. It's knowing which documents are true. Funny how that was always the hard part.