Choosing an Embedding Model & Vector Store for RAG

The first RAG demo works, the second one embarrasses you, and the instinct is always to reach for a bigger embedding model. That is almost never where the problem is. After enough of these builds, the ranking of what matters is clear and it puts the embedding model near the bottom.

Chunking is the real lever

If the answer is split across two chunks, no retriever will find it whole. If a chunk mixes three unrelated topics, its embedding is a muddy average that matches nothing well. Chunk on the document’s real structure — headings, sections, logical units — not on a fixed token count, and keep a little overlap so a sentence at a boundary is not orphaned.

• Respect structure: split on headings and paragraphs, not every 512 tokens blindly
• Keep chunks single-topic so the embedding means one thing
• Attach metadata — source, section, date — and filter on it before vector search
• Store enough context in each chunk that it is intelligible on its own

Picking an embedding model

Choose for your domain and your constraints, then stop optimising. A strong general open model is fine for most business text. If you self-host for privacy, a good local embedding model closes most of the gap to hosted ones. What actually forces the decision is dimensionality, language coverage and whether the data can leave your network — not a two-point difference on a leaderboard built from data unlike yours.

Benchmark on your own documents and your own questions. A public leaderboard measures someone else’s corpus, not yours.

The vector store: start simpler than you think

The usual honest answer is pgvector. If your data already lives in Postgres, you get vector search next to your relational data, one system to operate, and transactional consistency for free. That carries you a very long way — into the millions of vectors with sensible indexing.

• pgvector — one database, joins against your real tables, easiest to operate. The right default
• Qdrant / dedicated stores — reach for these at large scale, or when you need advanced filtering and payload features Postgres makes awkward
• Managed services — fastest to start, least to run yourself; you trade cost and data residency for that

Adopt a dedicated store when Postgres genuinely strains, not on the assumption that it will. Most projects never reach that point.

Re-ranking is the cheapest big win

Retrieve generously — say the top 20 by vector similarity — then run a cross-encoder re-ranker to reorder them and keep the best handful for the prompt. Vector search is a fast, fuzzy first pass; the re-ranker reads query and passage together and is far better at judging real relevance. Adding one usually beats upgrading the embedding model, and costs less to do.

# retrieve wide, then re-rank narrow
candidates = vector_search(query, k=20)              # fast, approximate
ranked     = reranker.score(query, candidates)       # slow, accurate
context    = top(ranked, n=5)                         # into the prompt

Measure retrieval on its own

When answers are wrong, you cannot tell whose fault it is unless you separate the stages. Build a small evaluation set of real questions with the passages that should be retrieved, and score retrieval by itself — did the right chunk make the top k? Most of the time the generation is fine and the retriever simply never handed it the right passage. Fix that and the model looks twice as smart.

The order of operations

Get chunking right, use a reasonable embedding model on pgvector, add a re-ranker, and measure retrieval independently. Only then does swapping the embedding model or the vector store become the thing worth arguing about — and by then it usually is not.