RAG at Query Time - Rewriting, Multi-Query, Reranking and Self-Reflection

Part two was mostly things you build yourself. This part is the opposite: nearly every strategy here already exists in org.springframework.ai.rag.* as a component that slots into RetrievalAugmentationAdvisor.

That’s the asymmetry I flagged in part one, and it has a practical consequence. Query-time strategies are cheap to try and cheap to remove — they’re a bean, not a re-ingest. So this is where to experiment. It’s also where you pay forever, because every one of them adds work to every single request.

The pipeline you’re modifying

From the basic RAG post, the modular advisor:

RetrievalAugmentationAdvisor.builder()
        .documentRetriever(VectorStoreDocumentRetriever.builder()
                .vectorStore(vectorStore)
                .similarityThreshold(0.7)
                .topK(5)
                .build())
        .build();

Four seams hang off that builder: a query transformer and a query expander before retrieval, a document joiner to merge results, and a document post-processor after. Everything below fills one of them.

Query rewriting: fix the question first

Users ask badly. They paste three paragraphs of context around one actual question, or they write “what about the other one?” as a follow-up. Neither embeds into anything useful.

RewriteQueryTransformer runs the query through an LLM to produce something a vector store can work with:

RetrievalAugmentationAdvisor.builder()
        .queryTransformers(RewriteQueryTransformer.builder()
                .chatClientBuilder(chatClientBuilder)
                .targetSearchSystem("vector store")
                .build())
        .documentRetriever(retriever)
        .build();

targetSearchSystem matters more than it looks — rewriting for a vector store is a different job from rewriting for a web search engine, and the transformer takes it into account.

Two siblings worth knowing. CompressionQueryTransformer collapses a conversation history plus a follow-up into one standalone query, which is the correct fix for “what about the other one?” — it needs the earlier turns to resolve the pronoun. And TranslationQueryTransformer translates a query into the language your corpus is written in, which is the cheapest possible fix for a multilingual front end over an English knowledge base.

Cost: one extra LLM call before every retrieval. Use a small model; this is not hard work.

Multi-query: several angles at once

A single phrasing retrieves a single neighbourhood of vector space. If the user’s wording is unlucky, you miss.

MultiQueryExpander generates variations and searches all of them:

MultiQueryExpander expander = MultiQueryExpander.builder()
        .chatClientBuilder(chatClientBuilder)
        .numberOfQueries(3)
        .build();

Wire it in alongside a joiner, which is the piece that makes it usable:

RetrievalAugmentationAdvisor.builder()
        .queryExpander(expander)
        .documentRetriever(retriever)
        .documentJoiner(new ConcatenationDocumentJoiner())
        .build();

ConcatenationDocumentJoiner merges the result sets across every query, removes duplicates, and sorts by score descending. Without it you’d hand the model the same chunk three times, which wastes context and skews the model’s sense of what’s important.

This is the recall play, and it maps directly onto the argument from part one: with a large context window you care more about finding the right document than about finding few documents. Multi-query trades precision for recall deliberately.

Cost: one LLM call to expand, then N searches. The searches parallelise, so latency is roughly one extra model call plus the slowest query — but you are doing four times the database work, and if you’re paying per query on a hosted vector store, that shows up.

Reranking: the biggest query-time win

Vector similarity is an approximation. It’s fast because it compares two independently-computed embeddings, and it’s imprecise for exactly the same reason — neither vector was computed with the other in mind.

A cross-encoder reads the query and the document together and scores the pair. Far more accurate, far too slow to run over a whole corpus. So you use both: retrieve widely with vectors, then rerank the shortlist.

RetrievalAugmentationAdvisor.builder()
        .documentRetriever(VectorStoreDocumentRetriever.builder()
                .vectorStore(vectorStore)
                .topK(20)            // retrieve wide
                .build())
        .documentPostProcessors(rerank)   // then narrow to the best few
        .build();

Note topK(20) rather than 5. The common rule of thumb is to retrieve about four times what you intend to keep. Retrieving five and reranking five achieves nothing — reranking can only reorder what retrieval already found.

DocumentPostProcessor is an interface, so the model is yours to choose. cross-encoder/ms-marco-MiniLM-L-6-v2 is the standard small open cross-encoder; the hosted rerank APIs from Cohere and Jina are a call away and need no infrastructure.

Spring AI’s own documentation names the problems this solves: the lost-in-the-middle effect, context-length limits, and cutting noise. That first one is why reranking helps even when retrieval was already correct — getting the right chunk into position one measurably changes what the model does with it.

The evidence is strong. In Anthropic’s numbers from part two, reranking took contextual retrieval from a 49% failure reduction to 67%. It is the highest-value query-time change available, and unlike everything else here it costs no LLM call — just a small model inference over ~20 documents.

If you adopt one thing from this post, adopt this one.

Self-reflective RAG: grade, then try again

Retrieve, have the model grade whether the results actually answer the question, and if they don’t, rewrite the query and go again.

Spring AI has the grader already — RelevancyEvaluator from the evaluation API. The loop is a CallAdvisor, the pattern from the advisor chain:

public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
    for (int attempt = 0; attempt < maxAttempts; attempt++) {
        ChatClientResponse response = chain.nextCall(request);

        if (evaluator.evaluate(toEvaluationRequest(request, response)).isPass()) {
            return response;
        }
        request = withRefinedQuery(request, response);
    }
    return chain.nextCall(request);   // give up, answer anyway
}

The reference implementations grade on a 1–5 scale and retry below 3. Two details that matter in production: bound the loop — two attempts, not “until good” — and fail open, so that if the grader itself errors you proceed rather than hanging. An unbounded self-critique loop is a way to spend an unbounded amount of money on a question that was never going to converge.

Cost: two to three LLM calls per request, and by some margin the slowest option here. Genuinely worth it for research-style queries where being right matters more than being quick. Not for anything a user is watching a spinner for.

Agentic RAG: let the model choose

Everything so far is a fixed pipeline. Agentic RAG makes retrieval a set of tools and lets the model pick:

@Service
public class RetrievalTools {

    @Tool(description = "Semantic search over documentation chunks. Use for general factual questions.")
    public List<String> searchChunks(String query) { ... }

    @Tool(description = "Fetch a complete document by id. Use when the user needs full context, not a snippet.")
    public String getFullDocument(String documentId) { ... }
}

As in tool calling, the description is the prompt — it’s the only thing the model sees when deciding. Vague descriptions produce a model that picks badly and a pipeline that behaves unpredictably.

This buys flexibility, and it costs determinism. You can no longer say what your system will do with a given question; you can only say what it tends to do. My honest advice is the same as for agents generally: start with the fixed pipeline, and reach for tool selection only when you can point at queries that genuinely need different retrieval strategies. Most applications can’t.

The ladder

Every strategy adds calls. Stacked naively, they multiply:

Rewriting and multi-query and self-reflection is four to five model calls before the answer even starts generating. That’s several seconds and several times the token cost, on every request — which is where the cost engineering from the previous series stops being theoretical.

The ordering I’d actually recommend:

1. Reranking first. Best payoff, no LLM call, and it improves whatever retrieval you already have.
2. Query rewriting if your users write badly — and check whether they do before assuming it.
3. Multi-query if recall is the measured problem, and you can afford 4× the searches.
4. Self-reflection only for high-value, latency-tolerant queries.
5. Agentic only when the retrieval strategy genuinely needs to vary.

Measure after each. One at a time.

Conclusion

Query time is where Spring AI earns its abstraction. The transformers, the expander, the joiner and the post-processor are all first-class, all compose into one advisor, and all can be removed as easily as they were added. There’s no re-ingestion to undo.

Reranking is the one I’d fight for. It’s the only strategy here that adds no LLM call, it fixes the lost-in-the-middle problem directly, and it accounts for the jump from 49% to 67% in the only well-documented numbers anyone has published. Retrieve twenty, rerank to five, and stop tuning the embedding model.

Next, the last part: what to do when vectors are the wrong tool entirely — keyword search, knowledge graphs, and a table mapping symptoms to fixes.

More info: docs.spring.io/spring-ai/reference/api/retrieval-augmented-generation