Agentic RAG: Self-RAG, CRAG, and Multi-Hop Reasoning

Standard RAG is a single-shot pipeline: embed the query, retrieve top-k chunks, shove them into the prompt, generate. It works surprisingly well for simple factual lookups, but falls apart in three predictable ways:

• Irrelevant retrieval: the user's query doesn't semantically match the right chunks, especially when the answer requires traversing multiple documents.
• Silent hallucination on retrieval failure: if nothing relevant comes back, the model generates plausibly-sounding nonsense rather than admitting uncertainty.
• No iteration: one retrieval step is never enough for questions like "Who founded the company that acquired the startup where Alice worked before joining OpenAI?"

Agentic RAG addresses all three by treating retrieval as a dynamic decision the model controls, not a hardcoded preprocessing step. The model can retrieve zero, one, or many times; evaluate what it got; decide whether to search again; and produce a final answer only when it has enough signal.

Self-RAG (Asai et al., 2023, UW / Allen AI) fine-tunes a single LLM to emit special reflection tokens inline with its generation:

The model generates a [Retrieve] token mid-sentence when it predicts that continuing without grounding would be risky. At inference time, the controller watches for that token, fires the retriever, appends the top passage, and lets the model continue. The model then emits [ISREL] to judge the passage before incorporating it.

This turns a static generation loop into a self-supervised critique cycle. The fine-tuning data is synthesised using a teacher model that inserts reflection tokens into existing text, so you don't need manual annotation.

CRAG (Yan et al., 2024) adds a relevance evaluator that scores every retrieved document and triggers one of three actions:

1. Correct (score ≥ threshold): use the passage as-is.
2. Incorrect (score below floor): discard all retrieved docs, fire a web search instead.
3. Ambiguous (in between): use both the retrieved docs and a web search, then merge.

The evaluator is a lightweight classifier — often a cross-encoder or a prompted LLM judge — not the same model doing generation. This separation of concerns means you can swap in a cheap model for scoring.

A key CRAG trick is knowledge refinement: when a passage is kept, it's stripped down to only the sentences directly relevant to the query before going into the context window. This reduces noise and token cost simultaneously.

Heads up: CRAG's web-search fallback makes latency non-deterministic. Add a timeout and a max-retry cap in production or you'll burn tokens on adversarial queries.

Multi-hop questions require information from at least two documents where the bridge entity is only knowable after the first retrieval. Example: "What is the capital of the country where the IAEA headquarters is located?" — you need to retrieve "IAEA is in Vienna", then retrieve "Vienna is the capital of Austria", then conclude.

Three patterns handle this:

• Iterative retrieval: run retrieval → generate intermediate answer → use that answer as the next query. Simple but greedy; early errors compound.
• Sub-question decomposition: use an LLM to break the original question into atomic sub-questions, retrieve and answer each independently, then synthesise. IRCoT (Trivedi et al., 2022) interleaves chain-of-thought steps with retrieval to guide decomposition.
• Graph-based traversal: build a document-entity graph offline; each hop follows an edge (e.g. HippoRAG, 2024). Fast at inference but expensive to index.

For most production systems, sub-question decomposition with a small orchestrator model is the right starting point — it's auditable and requires no graph index.

The cleanest modern pattern is to expose retrieval as an LLM tool call. The model sees a function signature like search(query: str) -> list[Passage] and decides autonomously when to invoke it.

tools = [
    {
        "name": "search",
        "description": "Search the knowledge base. Call this when you need a fact you don't know.",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Precise search query"}
            },
            "required": ["query"]
        }
    }
]

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=2048,
    tools=tools,
    messages=[{"role": "user", "content": user_question}]
)

while response.stop_reason == "tool_use":
    tool_call = next(b for b in response.content if b.type == "tool_use")
    passages = retriever.search(tool_call.input["query"])
    messages.append({"role": "assistant", "content": response.content})
    messages.append({"role": "user", "content": [{"type": "tool_result", "tool_use_id": tool_call.id, "content": format_passages(passages)}]})
    response = client.messages.create(model="claude-sonnet-4-6", max_tokens=2048, tools=tools, messages=messages)

The loop runs until stop_reason == "end_turn". The model may call search zero times (if it already knows the answer) or five times (for a complex multi-hop question) — and that decision is implicit in its reasoning.

Retrieval quality is upstream of everything else. Two techniques reliably improve it:

HyDE (Hypothetical Document Embeddings): instead of embedding the raw query, ask the LLM to generate a hypothetical answer, then embed that. The embedding of a plausible answer is much closer to real answer documents in vector space than the embedding of a question. Works especially well when queries are short and documents are long.

hypothetical = llm.generate(f"Write a short document that answers: {query}")
query_vector = embedder.encode(hypothetical)
passages = index.search(query_vector, top_k=5)

Step-back prompting: prompt the model to first ask a more abstract "step-back" question, retrieve on that, then retrieve on the original. For "What laws governed Kepler at age 21?" the step-back is "What laws were in effect in 16th century Germany?" — which retrieves the right historical context the specific query would have missed.

Both techniques add one LLM call per retrieval round, so gate them behind a complexity classifier if latency matters.

Standard RAG benchmarks (EM, F1 on NQ) miss what makes agentic RAG hard. Use these instead:

• Retrieval precision@k: fraction of retrieved passages that are actually useful. Tells you if the model is retrieving junk.
• Answer faithfulness: does the generated answer contradict any retrieved passage? RAGAS (Es et al., 2023) automates this with an LLM judge.
• Answer relevance: does the final answer address the question, ignoring whether it's grounded? Separate from faithfulness.
• Hop efficiency: for multi-hop, how many retrievals did it take vs. minimum needed? An agent that always does 5 hops on a 2-hop question is wasting tokens.
• Abstention rate: when retrieval fails, does the model correctly say "I don't know"? Measure false positives (hallucinations) and false negatives (refusing good-faith questions).

The RAGAS library (pip install ragas) provides end-to-end evaluation with minimal setup. Run it on a 100-question sample before shipping to production — a week of silent hallucinations is much worse than an afternoon of evaluation.

The retrieval tool is only as good as the index behind it. Agentic patterns expose index weaknesses that naive RAG hides:

Chunk size matters more with multi-hop: small chunks (128 tokens) give precise retrieval but lose context needed to answer within the chunk. Large chunks (1024 tokens) keep context but reduce recall. The practical sweet spot is a hierarchical index: embed small chunks for retrieval, but return the surrounding parent chunk to the model.

# Parent-child chunking pattern
parent_chunks = split(document, size=1024, overlap=128)
for parent in parent_chunks:
    child_chunks = split(parent.text, size=256, overlap=32)
    for child in child_chunks:
        child.parent_id = parent.id
        index.add(child)  # embed and store children

def retrieve(query, top_k=5):
    child_hits = index.search(query, top_k=top_k)
    return [store.get_parent(h.parent_id) for h in child_hits]  # return parents

Metadata filtering: expose a filter parameter on the search tool so the model can restrict to date ranges, document types, or source IDs. This alone cuts hallucinations on time-sensitive queries by ~30% in practice.

Agentic RAG introduces new failure modes that single-pass RAG doesn't have:

Retrieval loops: the model keeps searching because each result is slightly off, never converging. Fix: cap max tool calls at 5-8 and return a soft failure message.

Over-retrieval on easy questions: the model retrieves when it doesn't need to, adding latency and cost. Fix: add a no_retrieval_needed confidence signal in the system prompt, or train/fine-tune a retrieval decision classifier.

Context window overflow on long chains: 10 hops × 5 passages × 500 tokens = 25k tokens of retrieved text before the answer is generated. Fix: summarise and compress intermediate results before feeding them back, or use a memory buffer that evicts old passages.

Bridge entity confusion in multi-hop: the model correctly retrieves both hops but fails to connect them because the entity name differs ("OpenAI" vs. "OpenAI LP"). Fix: entity normalisation at index time, or re-rank retrieved passages by overlap with previously retrieved passages rather than just query similarity.

Heads up: instrument every tool call with latency and token counts. Multi-hop agents without observability are impossible to debug.

A production-grade agentic RAG system has five components:

1. Orchestrator: runs the tool-call loop, enforces limits, handles timeouts.
2. Retriever: vector search + optional BM25 hybrid, with metadata filtering.
3. Reranker: cross-encoder that scores passage relevance after retrieval (colBERT or a prompted LLM judge).
4. Query planner (optional): decomposes complex questions into sub-questions before the first retrieval.
5. Answer synthesiser: generates the final answer citing specific passages.

For the orchestrator, LangGraph and LlamaIndex Workflows both provide graph-based loop control with explicit state. For a lighter approach, a plain while loop with a message list (as shown in the tool-call example above) is perfectly viable and easier to debug.

Start with retrieval-as-tool + a reranker. Add sub-question decomposition when you see multi-hop failures. Add CRAG-style fallback when retrieval precision is below ~0.6. Don't over-engineer upfront — each addition has a latency cost.