RAG Query Rewriting: HyDE, Multi-Query, Decomposition, and Step-Back

A RAG pipeline is only as good as its retrieved context. Most engineers spend days tuning chunk sizes and embedding models, then leave the query itself untouched — despite the query being the single biggest lever on retrieval quality.

The core mismatch: a user's question is short, conversational, and vague. A document chunk is dense, domain-specific, and written for a different purpose. Vector similarity between those two is often misleading. A query like "what's the latency budget?" sits far in embedding space from a spec doc that says "end-to-end round-trip must not exceed 200 ms at p99".

Query rewriting bridges this gap by transforming the raw query — using the LLM itself — before retrieval happens. The four main techniques are HyDE, multi-query expansion, query decomposition, and step-back prompting. Each attacks a different failure mode.

Proposed by Gao et al. (2022, Precise Zero-Shot Dense Retrieval without Relevance Labels), HyDE flips the retrieval problem. Instead of embedding the query and hunting for similar chunks, you ask the LLM to write a hypothetical answer to the query, then embed that answer as the retrieval key.

Why it works: a generated answer uses the same vocabulary, density, and structure as real corpus documents. The embedding of a 150-word hypothetical paragraph is far closer in vector space to a real 150-word chunk than the embedding of a 10-word question.

The hypothetical answer can be completely wrong on the facts — that doesn't matter. What matters is that it pulls retrieval into the right neighbourhood of the embedding space.

Heads up: HyDE adds one LLM call before retrieval. On cheap models (GPT-4o-mini, Haiku) this is usually worth it; on slow/expensive models, benchmark first.

Here is a minimal, dependency-light implementation. The only external calls are to your embedding model and your vector store.

from openai import OpenAI

client = OpenAI()

HYDE_PROMPT = """
Write a short, dense passage that would directly answer the following question.
Do not mention that it is hypothetical. Just write the passage.

Question: {query}
Passage:"""

def hyde_retrieve(query: str, vectorstore, k: int = 5) -> list:
    # Step 1: generate a hypothetical document
    hyp_doc = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": HYDE_PROMPT.format(query=query)}],
        max_tokens=256,
    ).choices[0].message.content

    # Step 2: embed the hypothetical document (not the query)
    embedding = client.embeddings.create(
        model="text-embedding-3-small", input=hyp_doc
    ).data[0].embedding

    # Step 3: retrieve using that embedding
    return vectorstore.similarity_search_by_vector(embedding, k=k)

The vectorstore here is any LangChain-compatible store (Chroma, Pinecone, pgvector). For LangChain users, HypotheticalDocumentEmbedder wraps this pattern directly.

A single query can miss chunks that use different terminology. Multi-query expansion asks the LLM to rewrite the original question into N semantically similar but lexically diverse variants, retrieves independently for each, then merges and deduplicates the results.

MULTI_QUERY_PROMPT = """
Generate {n} different phrasings of the following question.
Return one per line, no numbering.

Question: {query}"""

def multi_query_retrieve(query: str, vectorstore, n: int = 3, k: int = 4) -> list:
    variants_text = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": MULTI_QUERY_PROMPT.format(n=n, query=query)}],
    ).choices[0].message.content

    variants = [query] + [v.strip() for v in variants_text.splitlines() if v.strip()]
    seen, results = set(), []
    for v in variants:
        for doc in vectorstore.similarity_search(v, k=k):
            if doc.page_content not in seen:
                seen.add(doc.page_content)
                results.append(doc)
    return results

Typical N is 3-5. More variants yield higher recall but more noise — and more tokens for the generator. Reciprocal Rank Fusion (RRF) is a better merge strategy than simple dedup when ranking matters.

Multi-part questions frequently fail because no single chunk covers all of them. "Compare the memory model of Rust and Go, and explain how each handles data races" needs at least two distinct retrieval paths.

Decomposition breaks a complex query into atomic sub-queries, retrieves for each independently, then provides all retrieved context together to the final generator.

DECOMP_PROMPT = """
Break the following complex question into simple, self-contained sub-questions.
Return one per line.

Question: {query}"""

def decompose_retrieve(query: str, vectorstore, k: int = 3) -> list:
    raw = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": DECOMP_PROMPT.format(query=query)}],
    ).choices[0].message.content

    sub_questions = [q.strip() for q in raw.splitlines() if q.strip()]
    all_docs = []
    for sq in sub_questions:
        all_docs.extend(vectorstore.similarity_search(sq, k=k))
    return all_docs

A useful variant is step-wise decomposition: the answer to sub-question 1 is fed into the query for sub-question 2. This is slower but handles questions with causal chains. LangChain calls this a multi-step retrieval chain.

Step-back prompting (Zheng et al., Google DeepMind, 2023) handles questions that are too specific to match any single chunk. The idea: ask the LLM to generate a broader, more general question from which the specific question follows, retrieve on that abstracted version, then use both the abstracted context and the original query in the final prompt.

Example:

• Original: "What is the boiling point of ethanol at 500 hPa?"
• Step-back: "How does pressure affect the boiling point of liquids?"

The step-back query is far more likely to match a general chemistry section in your corpus than the hyper-specific original.

STEP_BACK_PROMPT = """
Given the specific question below, write a more general question that covers
the underlying principle needed to answer it.

Specific: {query}
General:"""

def step_back_retrieve(query: str, vectorstore, k: int = 5) -> list:
    general = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": STEP_BACK_PROMPT.format(query=query)}],
    ).choices[0].message.content.strip()

    # retrieve on both; the generator sees both sets
    return (
        vectorstore.similarity_search(general, k=k) +
        vectorstore.similarity_search(query, k=k // 2)
    )

None of these methods wins on all query types. A production system should route queries to the right rewriting strategy:

You can implement a lightweight classifier (even a prompted LLM call) that labels the incoming query and dispatches to the appropriate pipeline. Alternatively, run HyDE + multi-query in parallel for most queries and fall back to decomposition when the query contains conjunctions like "and", "also", "as well as".

Heads up: combining all four on every query is tempting but expensive. Profile latency and cost before going full ensemble.

Do not ship a rewriting strategy without an evaluation harness. Three metrics that matter:

1. Retrieval recall@k — for a labeled test set, what fraction of gold-standard chunks appear in the top-k results? This is your primary signal.

2. Context precision — of the retrieved chunks, what fraction are actually relevant? Precision-recall tradeoffs differ: multi-query increases recall but often hurts precision.

3. Answer faithfulness — using an LLM judge (e.g., ragas library), does the final generated answer contradict the retrieved context? Good retrieval is necessary but not sufficient.

# ragas evaluation example
pip install ragas

from ragas import evaluate
from ragas.metrics import context_recall, context_precision, faithfulness

result = evaluate(
    dataset=your_hf_dataset,  # question, contexts, answer, ground_truth
    metrics=[context_recall, context_precision, faithfulness],
)
print(result.to_pandas())

Run this on a held-out set of 50-200 realistic queries. A 5-10 point recall@5 lift is a meaningful improvement; anything below 3 points is probably noise.

HyDE hallucination drift: if the LLM generates a hallucinated hypothetical document that is confidently wrong, the embedding vector will lead retrieval away from the truth. Mitigation: keep the generation temperature low (0.2-0.4) and limit token length.

Multi-query token explosion: 5 variants × k=5 retrieved chunks = 25 chunks passed to the generator. That can overflow the context window. Use a reranker (Cohere Rerank, cross-encoder) to cut back to the top 5-8 most relevant before generation.

Decomposition over-splits: "Compare X and Y" decomposes fine, but "Why did X happen given Y?" can produce 6 sub-questions, most of which retrieve the same irrelevant chunks. Add a max-sub-questions cap and a dedup step.

Step-back over-generalises: stepping back from "Python 3.12 free-threaded GIL behavior" might produce "How does Python manage threads?" — losing all version-specific context. Keep the original query in the retrieval mix alongside the step-back query (as shown in the code above).

Latency stacks: each LLM call before retrieval adds 200-800 ms. Cache rewritten queries keyed on the raw query hash when the same questions recur.

Query rewriting is not free, and it is not always the right fix.

• Corpus-query mismatch is the wrong diagnosis: if retrieval is poor because chunks are too large, overlap too little, or the embedding model is domain-inappropriate, rewriting masks but does not fix the problem.
• Very short corpora: with fewer than ~500 chunks, exhaustive search (retrieve all, rank all) is often cheaper and more accurate than any rewriting strategy.
• Structured data: if the corpus is really a database or a table, the right tool is text-to-SQL or a hybrid sparse-dense retriever (BM25 + vector), not query rewriting.
• Latency-critical paths: real-time applications with a <150 ms SLA cannot afford an extra LLM round-trip. Use a smaller, cached rewriting model or skip rewriting and invest in better chunking and indexing instead.

Rewriting is a power tool. Reach for it when you have measured that retrieval quality is the bottleneck, and when your latency/cost budget has room for it.