RAG: Build AI That Actually Knows Your Business Data

Retrieval-augmented generation (RAG) is a pattern that grounds a language model's responses in documents you control, fetching relevant context at query time before the model generates an answer. It solves the two hardest problems with raw LLMs in production: hallucination on domain-specific facts, and the hard cutoff of the model's training data. If your users are asking questions that require knowledge of your internal docs, product catalog, legal agreements, or anything else the model was never trained on, RAG is usually where you start.

The Problem RAG Actually Solves

A general-purpose LLM knows a lot, but it does not know your business. It does not know what your SLA says, what version of the API is current, or what your refund policy is. Worse, it will often confidently fabricate an answer that sounds plausible. Fine-tuning is one response to this, but it is expensive, slow to update, and still prone to hallucination on facts embedded deep in weights. RAG is cheaper, updatable in real time, and gives you a traceable source for every answer.

The retrieval step acts as a constraint: the model is told to answer only from the retrieved passages. This does not eliminate hallucination entirely, but it dramatically reduces it and gives you a clear audit trail when something goes wrong.

The Full RAG Pipeline

Ingestion

Everything starts with getting your documents into a form the retriever can work with. This typically means:

1. Loading — pull from PDFs, HTML, databases, APIs, Notion, Confluence, wherever your knowledge lives.
2. Cleaning — strip boilerplate, normalize whitespace, handle encoding. Garbage in, garbage out.
3. Chunking — split documents into pieces the retriever can return and the model can fit in context.
4. Embedding — convert each chunk to a dense vector that captures its semantic meaning.
5. Storing — write chunks and vectors to a vector store (Pinecone, Weaviate, pgvector, Qdrant, etc.).

Chunking: The Decision That Breaks Most RAG Systems

Chunking strategy has more impact on retrieval quality than almost any other choice. Too small and a chunk lacks enough context to be useful. Too large and you waste context window and dilute relevance scoring.

A reasonable starting point for most prose documents is 512 tokens with a 64-token overlap. The overlap ensures that sentences split at a boundary are still retrievable from either chunk. For structured content like code or tables, chunk by logical unit (function, table row) rather than token count.

A few heuristics I use in practice:

• Recursive character splitting works well for general prose. Split on paragraphs first, then sentences, then characters.
• Semantic chunking — splitting where embedding similarity drops — produces more coherent chunks but adds latency and cost to ingestion.
• Parent-document retrieval — embed small chunks for precision, but return their parent section to the model for context — is worth the extra plumbing when your documents have rich hierarchical structure.

Embeddings

The embedding model maps text to a fixed-dimension vector. The similarity between two vectors approximates the semantic similarity between two pieces of text. Use the same model for ingestion and retrieval — mixing models corrupts your index.

Current strong general-purpose options include OpenAI's text-embedding-3-large, Cohere's embed-v4, and open-weight models like nomic-embed-text or mxbai-embed-large if you need to keep data on-premise.

Retrieval and Reranking

Plain vector (dense) search finds semantically similar chunks. It misses exact keyword matches. Hybrid search combines dense retrieval with sparse BM25 keyword search and merges the result sets, typically with Reciprocal Rank Fusion (RRF). This is now the default approach for anything beyond a demo.

After retrieving the top-k candidates, a cross-encoder reranker (e.g., Cohere Rerank, cross-encoder/ms-marco-MiniLM) scores each candidate against the query with full attention, not just vector similarity. It is slower and more expensive, but it meaningfully improves precision. Run it on the top 20–50 vector results, then pass the top 3–5 to the model.

Prompt Assembly and Generation

The retrieved chunks are inserted into the prompt, usually as a context block, before the user's question. The model is instructed to answer from the context and to say when it cannot.

from openai import OpenAI
from your_retriever import retrieve  # returns list[str]

client = OpenAI()

def rag_query(user_question: str, k: int = 4) -> str:
    chunks = retrieve(user_question, top_k=k)
    context = "\n\n---\n\n".join(chunks)

    system_prompt = (
        "You are a helpful assistant. Answer the user's question using only "
        "the context below. If the answer is not in the context, say "
        "'I don't have that information.' Do not speculate.\n\n"
        f"Context:\n{context}"
    )

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_question},
        ],
        temperature=0.2,
    )
    return response.choices[0].message.content

Keep temperature low for factual retrieval tasks. Citation prompting — asking the model to reference which chunk supported each claim — helps both with accuracy and with debugging.

Hybrid Search in Practice

Here is a minimal example of merging dense and sparse results with RRF before reranking:

def reciprocal_rank_fusion(
    dense_hits: list[str],
    sparse_hits: list[str],
    k: int = 60,
) -> list[str]:
    scores: dict[str, float] = {}
    for rank, doc_id in enumerate(dense_hits):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(sparse_hits):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=lambda d: scores[d], reverse=True)

Run the merged list through your reranker before passing to the model.

Evaluating a RAG System

You cannot improve what you do not measure. The two metrics that matter most:

• Faithfulness — does the answer contain only claims supported by the retrieved context? This is the hallucination check. Tools like RAGAS or LLM-as-judge prompts can automate this at scale.
• Answer relevance — does the answer actually address the question? A faithful answer that misses the point is still a bad answer.

Beyond those two, track retrieval recall (did the right chunks land in the top-k?) separately from generation quality. When your answers degrade, you want to know whether the retriever or the model is the bottleneck. Log query, retrieved chunks, and final answer for every request. Without that trace, debugging is guesswork.

Build a golden dataset of 50–100 representative questions with known correct answers early. Run evals on every significant change to chunking strategy, embedding model, or retrieval parameters.

Common Failure Modes and Fixes

Bad chunking kills retrieval. If chunks cut across the natural boundaries of your content, the retriever will return fragments that lack the context needed to answer correctly. Fix: audit a random sample of your chunks. If they read like sentence fragments or orphaned bullets, revisit your splitter.

Lost-in-the-middle. Research has shown that language models tend to underuse information placed in the middle of a long context window, favoring content near the start and end. Fix: keep your context short (3–5 chunks), put the most relevant chunk first, or use a model with demonstrated long-context reliability.

Stale index. Your vector store is a snapshot. If source documents change and you do not re-index, the model answers from outdated context. Fix: treat your ingestion pipeline as a continuous process, not a one-time job. Track document hashes and re-embed on change. Add a last_updated metadata field and surface it to users when freshness matters.

Query-document mismatch. User questions are short and colloquial. Documents are long and formal. The embedding of "how do I cancel?" may not be close to the embedding of a paragraph titled "Subscription Termination Policy." Fix: use HyDE (Hypothetical Document Embeddings) — ask the model to generate a hypothetical answer, then embed that for retrieval — or use query expansion to generate multiple phrasings before retrieval.

Over-retrieval. Dumping 20 chunks into the context to "be safe" inflates cost, slows responses, and introduces noise that confuses the model. Fix: be deliberate about k. Start with 4, measure faithfulness and relevance, and only increase if retrieval recall is genuinely the bottleneck.

When RAG Is the Wrong Tool

RAG is not always the answer. Some cases where I reach for something else:

• The knowledge fits in the context window. If your entire knowledge base is a few dozen pages, just include it. No retrieval needed.
• The task requires multi-hop reasoning across many documents. RAG retrieves relevant passages; it does not reason across them. For complex analytical tasks, you may need an agent that can iteratively retrieve and synthesize.
• The data is highly structured. If users are querying a database, generate SQL or use a tool call. Do not embed your entire database and hope the retriever finds the right rows.
• Low-latency is critical. Retrieval adds a round trip. If you need sub-100ms responses, a fine-tuned model or a cached response layer may serve better.

Key Takeaways

• RAG grounds LLM outputs in documents you control, reducing hallucination on domain-specific and time-sensitive knowledge.
• Chunking strategy is the highest-leverage decision in your pipeline. Start at 512 tokens with overlap and audit the output.
• Hybrid search (dense + sparse) and cross-encoder reranking consistently outperform plain vector search. Use both in production.
• Measure faithfulness and answer relevance separately from retrieval recall so you know which layer to fix when quality drops.
• Watch for lost-in-the-middle, stale indexes, and query-document mismatch — these are the failure modes that bite teams after launch, not before.
• Know when to skip RAG: structured data, tiny knowledge bases, and multi-hop reasoning tasks each call for different patterns.