Seven Places RAG Can Lie

A RAG answer cites a real document and is still wrong.

The document may not contain the claim. The cited section may contradict it. The index may hold an old revision. A correct passage may have been retrieved and then dropped. The model may have answered from memory while attaching a nearby citation.

Calling this a “vector database problem” is like blaming a warehouse shelf for a wrong shipment.

Retrieval-augmented generation is a chain:

source -> ingest -> index -> retrieve -> rerank -> assemble
       -> generate -> verify citation -> observe outcome -> update

Every arrow can lose information or authority. The system is useful only when it can say where.

The original idea was retrieval plus generation

Lewis et al. introduced retrieval-augmented generation models that combine parametric memory in a generator with non-parametric memory in a dense Wikipedia index. The paper frames provenance and knowledge updating as important motivations.

Production teams generalized “RAG” to many architectures: keyword or dense search, hybrid retrieval, metadata filters, rerankers, prompt assembly, hosted generators, and citation formatting. That flexibility is useful. It also means a RAG label conveys almost no reliability contract by itself.

Define the shipped system:

corpus and version
ingestion and chunking
retrieval methods and k
filters and authorization
reranker and selection
context construction
generator and instructions
claim/citation verification
no-answer and escalation policy

Only then can an evaluation result be reproduced.

Failure 1: the source was never eligible evidence

Before retrieval, ask whether the current, authoritative answer exists in the corpus.

Common failures:

• the relevant system was never connected;
• a parser dropped a table, footnote, image, or code block;
• the latest revision failed ingestion;
• deletion did not propagate;
• access-control metadata is missing;
• the source conflicts with a more authoritative source;
• a timestamped fact has expired.

Measure corpus coverage on real questions. For each answerable item, identify the source and exact supporting span before evaluating a retriever. If annotators cannot locate evidence, a retrieval miss cannot be diagnosed honestly.

Store source identity through every transformation:

document_id, revision, content_hash, authority, valid_from, valid_to,
tenant, ACL, parser_version, chunk_id, source_offsets

An embedding without provenance is not evidence.

Failure 2: chunking broke the fact

Chunking makes documents searchable but can separate a rule from its exception, a table header from its row, or a function signature from its contract.

Fixed token windows are simple. Structure-aware chunks preserve headings, paragraphs, tables, and code units. Parent-child retrieval can index small passages but return a larger source neighborhood. Overlap protects boundaries while increasing duplication and index cost.

Test chunking with boundary cases:

• answer and unit in adjacent cells;
• prohibition followed by exception;
• definition in a heading and condition below;
• entity name in one paragraph and pronoun in the next;
• code symbol and implementation separated;
• multi-hop answer across two documents.

Record the original offsets so a reranker or citation viewer can recover context without inventing it.

Failure 3: retrieval optimized similarity, not evidence recall

Dense Passage Retrieval uses a dual encoder to place questions and passages in a shared vector space. Dense retrieval can capture semantic relationships beyond exact token overlap. Sparse retrieval remains valuable for identifiers, names, numbers, error codes, and rare terms.

Hybrid retrieval is not automatically superior. It adds weights and merging policy that need evaluation. The relevant metric at this stage is usually evidence recall at k:

recall@k = questions where at least one supporting passage is in top k
           ------------------------------------------------------------
           answerable questions with annotated supporting evidence

Also report authorization precision. A retriever that finds the right private passage for the wrong user is not accurate.

Build slices for:

• semantic paraphrase;
• exact entity or identifier;
• rare terminology;
• multilingual query/source pairs;
• temporal and version-sensitive questions;
• multi-hop evidence;
• no-answer questions.

One aggregate recall hides the retrieval family you need.

Multi-hop questions turn retrieval into a workflow

Some questions cannot be supported by one passage. “Which policy applies to the account that owns invoice 17?” may require an invoice-to-account lookup followed by an account-to-policy lookup. A single embedding query can retrieve both by luck, but the system needs an explicit plan when the second query depends on the first result.

A multi-hop trace should show:

question
  -> hop 1 query and candidate set
  -> selected entity or relation
  -> hop 2 query derived from verified hop 1 state
  -> combined evidence graph
  -> answer claims and citations

Do not let an unverified first-hop model guess become a trusted second-hop filter. If hop one extracts an account ID, validate its format and source span before using it to fetch private policy. The second retrieval inherits the authorization scope of the task, not authority invented by the generated intermediate text.

Recall compounds. If each of two required hops finds its evidence 90 percent of the time under the relevant conditional workload, the complete path is at most 81 percent before reranking and generation:

.90 * .90 = .81

The multiplication is an illustrative conditional model, not a claim that hop failures are independent. In production, estimate the traced transition directly. A shared entity-resolution error can correlate both failures.

Evaluate:

• hop-level supporting evidence recall;
• correctness of the entity or relation passed forward;
• whether both sources survive final context selection;
• whether the answer joins them correctly;
• citations attached to each claim, not only the final paragraph;
• stops when a required hop lacks evidence.

Parallel retrieval can reduce latency when hops are independent. Sequential retrieval is necessary when the next query genuinely depends on a verified result. An agent that launches every conceivable search in parallel may improve recall while flooding context with distractors and multiplying tool cost. Apply a bounded search budget and stop when the evidence contract is satisfied.

Freshness is a control loop, not a nightly re-index

A retriever can achieve perfect recall against an obsolete index. The answer remains wrong.

Define freshness by source type. A product manual may tolerate hours. Inventory, permissions, incidents, and balances may require a live tool call or transaction snapshot. “Most recent chunk in the vector store” is not a freshness guarantee unless ingestion delay and revision lineage are measured.

Track four timestamps:

source effective time
source observed or committed time
index available time
answer generated time

The difference between source commit and index availability is ingestion lag. The difference between source effective time and answer time is evidence age. Both need SLOs where staleness has consequence.

Use event-driven invalidation when source systems emit reliable changes. Tombstone deleted or revoked documents so old vector replicas cannot continue serving them. Keep an index version in every answer trace. During a rolling rebuild, route a request to one coherent snapshot rather than combining candidates from incompatible revisions without declaring it.

For live facts, retrieve stable explanatory policy from the corpus and fetch the changing value through an authorized tool. The response can cite the policy document and attach a receipt or timestamp for the live value. This separates knowledge from state.

Test freshness failures deliberately:

• a document is revised while its old chunk remains searchable;
• an access grant is revoked during a conversation;
• two replicas expose different index versions;
• a source is deleted but cached output survives;
• a question asks “today” while the latest evidence is yesterday;
• a retry crosses a revision boundary.

The correct response may be to wait, fetch live, or state that the system cannot establish a current answer. Fresh-looking prose is not freshness.

Failure 4: reranking threw away the answer

Retrievers favor speed across a large corpus. A reranker applies a more expensive relevance function to a smaller candidate set. ColBERT represents query and document tokens contextually and uses late interaction, occupying a design point between single-vector retrieval and full cross-encoding.

Whatever method is used, evaluate two transitions:

retrieval candidate recall@k
context evidence recall after reranking and truncation

If recall@50 is 92 percent but the final context contains support for only 74 percent, improving the vector index cannot recover the 18 points lost downstream.

Reranking can overvalue passages that repeat the question, underweight terse authoritative tables, or select redundant chunks from one document. Add diversity and source-authority constraints where the task requires them.

Failure 5: context assembly buried or corrupted evidence

The selected passages must fit beside policy, query, tools, and output reserve. Assembly decides order, labels, separators, truncation, and conflict presentation.

The Window Is Not the Memory shows why more context can reduce value. For RAG, use explicit blocks:

[SOURCE id=policy-17 rev=4 authority=official]
verbatim excerpt with offsets
[/SOURCE]

Do not silently concatenate passages into a synthetic document. Preserve disagreement. If two sources conflict, expose authority and date so the answer can abstain or explain the conflict.

Test relevant evidence at different positions and under distractors. A good reranker cannot help if the final ordering makes the model ignore the evidence.

Failure 6: generation saw evidence and did not use it

Retrieval recall is not grounded answer quality. The model can:

• answer from parametric memory instead of the source;
• merge facts from incompatible passages;
• invert a condition;
• overgeneralize a narrow rule;
• produce a correct claim for an unsupported reason;
• refuse despite sufficient evidence;
• answer when evidence is insufficient.

Annotate claims, not only whole answers. For each atomic claim classify:

entailed by cited evidence
contradicted by evidence
not supported
not requiring external support

Whole-answer exact match is useful for short QA but inadequate for long operational responses.

Self-RAG trains retrieval and critique behavior through reflection tokens. It is evidence that retrieval policy can be learned as part of generation. The model's self-critique is still a model output, not independent ground truth. Pair it with deterministic or human validation appropriate to consequence.

Failure 7: the citation is real but does not support the claim

A citation can fail in four distinct ways:

• invalid: source or span does not exist;
• irrelevant: source exists but concerns another subject;
• non-entailing: related passage does not establish the claim;
• mis-scoped: passage supports a narrower claim than the answer states.

Citation verification should operate on atomic claims and exact source spans. URL validity alone catches only the first class.

For high-consequence answers, use a verifier that has the claim and cited span, not the entire persuasive answer. Deterministic checks can validate identifiers, dates, quoted values, and source existence. Human review remains appropriate when entailment is nuanced and failure loss is high.

Require a no-answer path:

if no authorized source supports the answer:
    state what is missing
    do not fill the gap from model memory
    retrieve again, ask a question, or escalate

Abstention is a successful control outcome when evidence is absent.

Error rates multiply across the path

Use an illustrative sequence of conditional pass rates:

current source exists                 .95
retriever finds support               .85
reranker retains support              .90
generator uses support correctly      .88
citation verifier accepts true support .97

The fully supported path is:

.95 * .85 * .90 * .88 * .97 = .6204

About 620 of 1,000 illustrative questions traverse every gate. This is not a production forecast or independence assumption. Real stages correlate. Estimate conditional transitions directly from traced examples.

The point is structural: five metrics that each look respectable can still yield a weak end-to-end outcome.

Do not improve the easiest metric. Improve the stage contributing the most consequence-weighted loss.

One score hides the repair

Create a metric-to-stage map:

Stage	Primary metric	Important countermetric
corpus	answerable coverage, freshness	unauthorized inclusion
parse/chunk	supporting-span preservation	duplication/index size
retrieve	evidence recall@k	authorization precision, latency
rerank/select	final-context evidence recall	redundancy, cost
generate	claim correctness, evidence use	abstention on answerable items
citations	claim-level entailment	verifier false rejection
outcome	verified resolution	latency and cost

Aggregate answer accuracy belongs at the outcome layer. It cannot replace the intermediate measures needed to locate failure.

Latency and cost are also a waterfall

An illustrative trace might spend:

query policy/rewrite   24 ms
hybrid retrieval       46 ms
reranking              71 ms
context assembly        9 ms
model queue            180 ms
prefill                220 ms
decode                 640 ms
citation verification  85 ms
total                1,275 ms

The values are invented. The trace shows why replacing the vector store may save little when decode and queue dominate. Conversely, a cross-encoder reranker can be material at high request volume even when generation is expensive.

Track per-stage calls and bytes. Query rewriting, multi-hop retrieval, corrective retrieval, and citation repair can multiply work. Feed the full trace into cost per verified outcome, not merely the final model call.

A builder playbook

1. Build a source-of-truth evaluation set

Start with production-shaped questions, answerability, exact supporting spans, authority, revision, and authorization. Include unanswerable, stale, conflicting, and adversarial sources.

2. Freeze and name every stage

Version corpus snapshot, parser, chunker, embedding model, index, query transformation, filters, reranker, context builder, generator, and verifier. A score without this lineage cannot be compared.

3. Emit an evidence trace

For each answer, retain retrieved candidates and scores, filter decisions, reranked order, final context ranges, atomic claims, citations, verifier results, latency, cost, and outcome.

4. Evaluate transitions conditionally

Measure retriever only on covered questions, reranker only where retrieval found evidence, and generation both with gold evidence and pipeline evidence. Gold-context generation isolates the reader ceiling.

5. Add a no-answer contract

Calibrate abstention separately from answer correctness. Test whether unsupported confident answers fall as abstention rises, and price false refusals versus false claims.

6. Fix the largest failure contribution

If corpus coverage is low, connect or refresh sources. If candidate recall is low, change retrieval. If final-context recall is low, fix reranking or budget. If gold-context generation fails, change prompt/model/constraints. If citations fail, verify claims.

7. Re-run outcome economics

Include ingestion operations, indexes, reranking, model calls, corrective loops, verification, review, and failure loss. Optimize supported correct outcomes per dollar and deadline.

The decision

Buy or build a vector database when storage, filtering, and nearest-neighbor search are the bottleneck. Do not expect it to repair absent sources, broken chunks, bad authority, discarded evidence, unsupported generation, or decorative citations.

RAG becomes dependable when every answer carries a trace from current authorized source to atomic claim, and every missing link has a controlled stop.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al., NeurIPS 2020. Establishes the parametric plus retrieved-memory formulation.
• Dense Passage Retrieval for Open-Domain Question Answering, Karpukhin et al., EMNLP 2020. A foundational dual-encoder dense retrieval system and recall evaluation.
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, Khattab and Zaharia, SIGIR 2020. Introduces token-level late interaction for retrieval.
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Asai et al., ICLR 2024. Integrates learned retrieval and critique behavior into generation.

All stage rates and timings in the worked examples are illustrative. The complete arithmetic and limitations are retained in the claim ledger.