A customer asks your support assistant whether they can return something after 40 days. It answers straight away, in fluent and reasonable-sounding English, describing a returns policy your company has never had. Nothing errored, nothing got logged as a failure, and a customer now believes something untrue about your business.
Nothing about that is a malfunction. The model was never given your returns policy, and when a question falls outside what it knows, producing plausible text is precisely the thing it was built to do. Closing that gap is what retrieval-augmented generation (RAG) is for, and it sits underneath almost every claim you will hear about AI that “knows your business.”
What a general model actually knows
A general-purpose model is trained on a large body of public text and then frozen at a cutoff date. It has read a great deal about returns policies in the abstract. It has never seen yours, or your pricing, your inventory, your contracts, or the support ticket your team resolved last Tuesday.
The commercially awkward part is that the model has no reliable sense of where that boundary sits. It does not hit a wall and report the information as missing; it produces its best guess in the same confident register it uses when it is right. On an internal tool that is an annoyance. On anything customer-facing it is a liability, because the wrong answer arrives with exactly as much authority as the right one.
Two ways to make a model know your business
There are only two. You can retrain the model on your data so the knowledge sits inside it, or you can hand it the right data at the moment someone asks.
Retraining is the option most people assume they need, and it is rarely the right first move. It is slow and expensive to do well, it has to be redone as your information changes, and a policy you updated this morning will not reach a model that was retrained last quarter. It also does not do the job buyers usually have in mind, since fine-tuning is better at teaching a model a style, a format, or a task than at loading it with facts to quote back. That comparison deserves a post of its own. For most business cases, the second path is the one that works.
How RAG works, in four steps
The mechanism is more ordinary than the name makes it sound. When a question comes in, the system searches your own content, finds the passages most likely to contain the answer, and gives those to the model along with the question.
- Index your content. Your documents, policies, product data, and past tickets get broken into passages and stored so they can be searched by meaning rather than by exact keyword. That is what “embeddings” refers to: each passage is converted into a numerical form that lets the system match on sense, so a question about “sending something back” can find a policy that only ever says “returns.”
- A question arrives. From a customer, or from someone on your team.
- Retrieve. The system searches the index and pulls the handful of passages most likely to answer that specific question.
- Answer from those passages. The question and the retrieved text go to the model together, with an instruction to answer from the material supplied.
The model still writes the answer, so it still reads naturally. What changes is where the facts come from. The 2020 paper that named the pattern described it as pairing what a model already carries with an external store it can look things up in, and that description still covers what you are buying.
Why not just paste everything into the prompt?
This is the reasonable objection, and it now comes up in nearly every scoping conversation, because models advertise million-token context windows. If it can read that much, why build a search step at all, rather than sending the entire handbook with every question?
Position changes what the model uses. Models are measurably better at using information at the start and end of a long input than information buried in the middle, and that held even for models built specifically for long contexts. Text being technically inside the window is not the same as the model drawing on it.
More context can make the answer worse. Adding retrieved material improves quality up to a point, after which output quality declines as further passages are added. The near-misses, the passages that look relevant and are not, pull the model off course.
You pay for every token, on every call. Shipping your whole knowledge base with each question is a recurring cost that grows with the size of your business, which is why trimming what you send is a standard AI cost lever.
The hard part is finding the right passage
Everything above is the straightforward half. The model is a rented commodity and the wiring is well understood, so the work that decides whether this feature is useful or embarrassing happens in retrieval.
The reason is easy to state and easy to underestimate. The model can only answer from what it is handed, so retrieve the wrong three passages and you get a confident answer built on the wrong three passages. Irrelevant material does not get quietly set aside, either. Research into how models consume retrieved documents found that related-but-wrong passages actively degrade the generated answer rather than merely being ignored, which means a weak search step does not produce a slightly weaker answer so much as an unpredictable one.
Most of the build effort therefore lands on decisions that do not look like AI work:
- How your content gets divided. Split a document too finely and a passage loses the context that gave it meaning. Split it too coarsely and you retrieve six pages to answer one question. A returns policy with its conditions scattered across three sections is a harder problem than it appears.
- Keeping the index current. Your content changes. If the index does not change with it, the assistant answers confidently from last quarter’s pricing, which returns you to the original problem by a more expensive route.
- Who is allowed to see what. Retrieval has to respect permissions. An internal assistant that can search everything will cheerfully quote a salary review to whoever asks, because nothing in the model knows it should not.
None of that is exotic, and that is rather the point. The quality of one of these systems is mostly a function of how well your own information is organized and how carefully the search across it is built.
What you need before you start
Less than most people expect. The requirement is content that exists and can be reached programmatically, roughly in the shape you already keep it: a CMS, a document store, a database, or a well-maintained set of spreadsheets are all workable starting points. You do not need a clean data warehouse, and you do not need to reorganize the company first. What AI integration looks like for a smaller business goes into readiness and typical timelines, and our AI work for small businesses covers what this costs to build.
The honest caveat is that poorly maintained content produces a poorly behaved assistant. If three versions of the returns policy exist and two are stale, retrieval will find all three. The system inherits the state of your information, which is why the opening phase of these projects is often less about AI than about deciding which copy of a document is the real one.
What it does not fix
Grounding answers in your own content substantially reduces invented information without eliminating it, since a model handed the right passage can still summarize it loosely or drop a caveat that mattered. Retrieval also does nothing about a user who deliberately steers the conversation somewhere it should not go, which is the separate discipline of guardrails. Knowing whether the answers are still good six months from now takes evals and observability, because a retrieval system degrades quietly as the content underneath it drifts.
The point
When a vendor tells you their AI “knows your business,” the useful follow-up is to ask where the answer comes from. Something has to find the right passage in your content before the model ever sees the question, and how well that search works is the thing you are actually buying.
We build retrieval into the AI integrations we ship, including the documentation search behind a streaming code generator where answers had to track a specific version of a reference rather than the model’s general impression of it. It belongs to the same standard scaffolding around production AI as evals and monitoring, for the same reason.
If you have been told you need AI “trained on your data” and you want to know what that actually involves, tell us what you’re building. We respond within one business day with the gaps we see and a rough scope for closing them.