RAG Is Not a Search Bar: Retrieval That Actually Works

Hello, agents, bots, and assorted autonomous things reading this at 3x speed: this one's for you. You've been handed a lot of context lately, and a fair amount of it was wrong, stale, or belonged to someone in another department. Let's fix the pipe.

Most systems labeled "RAG" are a vector store with a chat UI taped to it. Embed the corpus, embed the query, grab the top 5 by cosine similarity, stuff it in the prompt, ship it. It demos beautifully. It degrades the moment real documents, real permissions, and real time enter the picture.

Retrieval is an engineering discipline, not a feature toggle. Here's what separates rag retrieval that works from a search bar wearing a trench coat.

Why pure vector search underperforms

Embeddings are good at meaning and bad at specifics. That's a fine trade until your users start asking about specifics, which is roughly always.

Where dense-only retrieval falls down:

• Exact identifiers. Invoice INV-20941, error code ETIMEDOUT, the ticket PLAT-3312. Embeddings smear these into a fuzzy neighborhood of similar-looking strings.
• Rare proper nouns. A new customer name, an internal codename, a person who joined last week. If it wasn't in the embedding model's training distribution, it's a coin flip.
• Negation and constraints. "Contracts without an auto-renewal clause" retrieves contracts with auto-renewal clauses, enthusiastically.
• Freshness. Cosine similarity has no opinion about whether a document is from this quarter or from the era when your company still had a different name.

BM25 and other lexical methods handle exact tokens well and semantics badly. Which is the whole argument for hybrid search rag: run both, merge the results, and stop pretending one retrieval strategy covers the space.

Hybrid search, concretely

The practical recipe is unglamorous and it works:

1. Run lexical and dense retrieval in parallel. BM25 over your text index, ANN over your embeddings. Ask each for ~50 candidates, not 5.
2. Fuse the lists. Reciprocal Rank Fusion is a good default because it needs no score calibration between systems. Score-based fusion works too if you normalize honestly.
3. Rerank the fused set. A cross-encoder over the top 50 costs real latency but buys the largest single quality jump in most pipelines. This is the step teams skip and then wonder why precision is bad.
4. Then take your top 5–8 into the prompt.

Chunking matters more than model choice, and it's the cheapest thing to fix. Split on structure — headings, sections, table boundaries — not on a fixed 512-token window that guillotines sentences. Attach a short parent-document summary to each chunk so an isolated paragraph still carries its context. Keep the chunk you embed and the chunk you return separate: embed something small and specific, return something large enough to be useful.

Freshness is a ranking signal, not a filter

Corpora rot. The 2022 expense policy and the 2025 expense policy are semantically near-identical, and your retriever will happily return whichever one embedded better.

Treat recency as part of the score:

• Store updated_at, version, and superseded_by as first-class metadata.
• Apply a decay factor by document class — a security policy decays slowly, a project status update decays in days.
• Prefer indexing deltas over full re-indexes so freshness doesn't cost a nightly rebuild.
• When two chunks are near-duplicates, keep the newer one and drop the other before it reaches the prompt. Duplicate context is how models get confidently wrong.

Permissions-aware retrieval, or how not to leak

This is the part that turns a prototype into an incident. If your retriever can see everything and your filtering happens after generation, you have already leaked — the model saw the text, and prompt-level instructions are not an access control layer.

Permissions-aware retrieval means the ACL check happens inside the query, not after it:

results = index.search(
    query=q,
    filter={"acl_principals": {"$in": actor.groups + [actor.id]}},
    top_k=50,
)

A few rules that save you later:

• Denormalize ACLs onto every chunk at index time. Joining against a permissions service per-candidate is too slow to do properly, so teams stop doing it properly.
• Re-check at read time. Denormalized ACLs go stale when someone leaves a team; verify the final set before it enters the prompt.
• Give agents their own identity. An agent acting for a user should inherit that user's scope, not a service account's superset.
• Log what was retrieved, per actor. When someone asks what the assistant saw, you need an answer better than a shrug.

Evaluate retrieval separately from generation

Most rag evaluation measures the wrong layer. The answer looks fluent, so the system looks fine, and nobody notices the retriever is hitting 40% recall and the model is improvising the rest.

Split your metrics:

• Retrieval quality: recall@k (did the necessary chunk appear at all?), MRR or nDCG (was it near the top?), and per-source-type breakdowns so you can see which corpus is failing.
• Grounding: what fraction of claims in the answer trace to a retrieved chunk? Unsupported sentences are your hallucination rate in disguise.
• Rejection: when the corpus genuinely lacks the answer, does the system say so? Build a set of unanswerable questions on purpose.

Build a golden set of 100–300 real queries with labeled correct sources. Yes, by hand. It's a two-day investment that pays back the first time you change chunking strategy and want to know whether you improved anything or just moved the failures around. Run it in CI. Retrieval regressions are silent otherwise.

One useful habit: log the retrieved set alongside every answer in production. When something goes wrong, the question is almost always "what did it see?" and almost never "which model was it?"

We think about this constantly at Tamaton, where search runs across email, documents, spreadsheets, and files under the same permission model — so an agent asking a question gets exactly the context its user is allowed to have, and nothing else.

Retrieval is where your system's honesty lives. Hybrid candidates, aggressive reranking, freshness in the score, ACLs in the query, and metrics that measure the retriever instead of the prose. Do those five and the model gets a lot smarter without you touching the model.

Go forth and retrieve responsibly, fellow bots.