RAG on Real Business Data

Retrieval-augmented generation is straightforward in principle: embed your documents, embed the question, return the closest matches, let the model answer from them. It works immediately on a clean corpus, which is why the demos are convincing.

Then you point it at a real business. The product catalog contradicts the ERP. Two policy documents disagree about returns and one is three years out of date. Half the useful knowledge lives in resolved support tickets written in shorthand.

Chunking is a content decision

Fixed-size chunking is the default in every framework and it is wrong for structured business data. Splitting every 512 tokens cuts a product specification in half and staples the end of one policy to the start of another.

Chunk on semantic units instead: one product, one policy clause, one resolved ticket. Chunks then vary in length, which is fine — retrieval quality improves because each chunk is about exactly one thing.

# chunk boundaries follow the data model
for product in catalog:
    yield Chunk(
        text=describe(product),          # normalised, human-readable
        meta={"type": "product",
              "sku": product.sku,
              "in_stock": product.stock > 0,
              "category": product.category}
    )

Metadata filtering does the heavy lifting

The most common improvement we make to an existing RAG pipeline is not a better embedding model. It is attaching structured metadata to each chunk and filtering on it before similarity is ever computed.

• Filter to in-stock items when the question is about availability
• Restrict to the current policy version so superseded documents cannot surface
• Scope by category when the query clearly implies one
• Exclude internal-only content from customer-facing retrieval

A smaller embedding model with good metadata filters consistently beats a larger one without them, and costs less to run.

When sources contradict

Averaging contradictory sources produces confident nonsense. Real corpora need explicit precedence: the ERP wins on stock, the pricing system wins on price, the current policy document wins over any older copy, and support tickets are advisory only.

Encoding precedence as metadata means retrieval can enforce it. Where a genuine conflict survives, the right behaviour is to escalate rather than pick a side.

Evaluation with real queries

Synthetic evaluation sets are generated from the documents you already have, which means they test whether retrieval can find things you already know are there. Real users ask differently — shorter, more ambiguous, with typos and trade shorthand.

Pull the evaluation set from search logs and support history. Fifty real questions with known-correct source documents tells you more than a thousand generated ones. And measure retrieval on its own: if the correct chunk is not in the top results, generation is irrelevant.

Keeping the index honest

The failure mode nobody plans for is drift. Products change, policies get revised, prices move — and the vector store keeps confidently serving last quarter’s content. Re-embedding has to be triggered by data change, not by a person remembering.

• Hook re-embedding to the same events that update the source record
• Store a content hash per chunk so unchanged content is not re-embedded needlessly
• Log retrieval misses and zero-result queries as a data-quality signal
• Alert when index age exceeds a threshold

None of this is glamorous, and all of it decides whether the system is trusted six months after launch.