Two of the five failure modes from part one are decided before a single query arrives: the chunk boundary that split an answer in half, and the fragment that means nothing on its own. No amount of query-time cleverness recovers from either. If the information isn’t in your index in a retrievable shape, it isn’t coming back.
This is also where Spring AI helps you least. It gives you a perfectly good ETL pipeline — readers, transformers, writers — and no opinions about what to put in it. Everything here is a DocumentTransformer you write or a decision you make. That’s not a criticism; the strategies are genuinely application-specific. But it means this part is more design than API.
Chunking is still the highest-leverage setting
I said in the basic RAG post that chunking affects answer quality more than model choice. A year of watching people tune the wrong things hasn’t changed my view.
The default is a token splitter, and it’s fine to start there:
List<Document> chunks = TokenTextSplitter.builder()
.withChunkSize(400)
.withMinChunkSizeChars(200)
.build()
.apply(documents);The problem is that a token count knows nothing about your document. It will happily cut between a heading and the section it introduces, or midway through a table.
Split on structure instead. Markdown headings, document sections, ticket boundaries, function definitions. A chunk that corresponds to a real unit of meaning beats any character count you can guess, because the thing you’re really optimising is “does this chunk answer a question on its own”.
The Python ecosystem has Docling’s HybridChunker for this, which is token-aware and structure-aware and merges small adjacent chunks rather than emitting useless fragments. There’s no direct Spring AI equivalent, so you either pre-process documents with a structural parser before they reach the pipeline, or write a splitter that understands your format. For Markdown and HTML that’s an afternoon; for arbitrary PDFs it isn’t, which is the honest reason the Python tooling is ahead here.
Two settings people get wrong:
- Overlap is not waste. Overlapping neighbouring chunks means a sentence that straddles a boundary appears in both, so no boundary can hide it. The cost is a slightly larger index.
- Chunk size is a bet about question shape. Large chunks blur into a topical average that matches nothing precisely; small ones retrieve a sentence without the context that makes it meaningful. Settle it against your golden dataset on a sample, not by arguing.
Settle chunking first, before anything else in this series. Everything downstream is embedded on top of it, and changing it later means re-embedding your entire corpus.
Contextual retrieval: the one with real numbers
This is the technique I’d adopt before any other, and it’s the only one in the series where I think the payoff clearly justifies the cost.
The problem is the fragment. A chunk reads:
This threshold defaults to 30 days and can be extended once.Which threshold? The document knew. The chunk doesn’t, and its embedding sits somewhere vague and unhelpful in vector space.
Anthropic’s fix is almost stupidly direct: before embedding, ask an LLM to write one or two sentences situating the chunk in its document, and prepend that. So the stored text becomes:
This chunk is from the Returns Policy section of the 2026 Customer Handbook,
describing the refund request window. This threshold defaults to 30 days and
can be extended once.Now it embeds where it belongs.
The measured results, from Anthropic’s own evaluation across codebases, fiction and scientific papers, using 1 − recall@20:
| Setup | Failure rate | Reduction |
|---|---|---|
| Baseline RAG | 5.7% | — |
| Contextual embeddings | 3.7% | 35% |
| Contextual embeddings + contextual BM25 | 2.9% | 49% |
| The above + reranking | 1.9% | 67% |
Those are large numbers by the standards of anything else in this series. The cost is one LLM call per chunk at ingestion, which Anthropic prices at $1.02 per million document tokens — a one-off, and cheap next to a 35% failure reduction.
In Spring AI it’s a transformer in the pipeline:
public class ContextualEnricher implements DocumentTransformer {
private final ChatClient chatClient;
private final String documentText;
@Override
public List<Document> apply(List<Document> chunks) {
return chunks.stream()
.map(chunk -> {
String context = chatClient.prompt()
.user(u -> u.text("""
Here is the document:
{document}
Here is a chunk from it:
{chunk}
Give a brief 1-2 sentence context explaining what this
chunk discusses in relation to the overall document.
Answer with the context only.
""")
.param("document", documentText)
.param("chunk", chunk.getText()))
.call()
.content();
return new Document(context + "\n\n" + chunk.getText(), chunk.getMetadata());
})
.toList();
}
}Three practical notes. Cap the document excerpt you pass in — a few thousand characters is plenty, and it stops a 300-page PDF blowing the context window on every chunk. Use your cheapest model; this is a summarisation task and a small model does it well. And use prompt caching — the document is identical across every chunk of that document, so putting it first in a stable prefix is exactly the case prompt caching exists for. That’s the difference between a manageable ingestion bill and a silly one.
Note also what the table shows: contextual BM25 contributes nearly as much as contextual embeddings. Keyword search is doing real work there. That’s part four.
Late chunking, and why I’m not recommending it
Late chunking is a genuinely elegant idea. Instead of splitting text and embedding each piece independently, you run the whole document through a long-context embedding model, then pool the resulting token embeddings into chunks. Every chunk vector is computed with the full document in attention, so it carries context without anyone writing a summary.
It solves the same problem as contextual retrieval, with no LLM calls at ingestion. It sounds strictly better.
Then you read the paper. The measured improvement over naive chunking is around 3.6% relative — roughly 1.9 points absolute. Real, reproducible, and an order of magnitude below contextual retrieval’s 35%.
There’s also a hard blocker for us: late chunking needs access to token-level embeddings inside the model so you can pool them yourself. Spring AI’s EmbeddingModel gives you a vector per input string. There’s no seam for this. You’d bypass Spring AI entirely and call a provider that exposes late chunking natively — Jina’s embedding API has a flag for it.
So my honest read: interesting, defensible if you’re already on an embedding provider that supports it, and not worth restructuring your pipeline for. The paper’s own framing suggests structure-aware chunking plus contextual retrieval gets you the same benefit by a more available route.
I’m including it because you’ll meet it in every advanced-RAG list, usually presented as equivalent in value to contextual retrieval. It isn’t.
Hierarchical retrieval: search small, return big
This one resolves a genuine tension. Small chunks retrieve precisely, because their embeddings are focused. Large chunks answer better, because they carry context. You want both.
Parent–child does exactly that: index the small chunks, but when one matches, return the larger parent section it belongs to.
There’s no built-in support, and you don’t need much. Store the relationship in metadata at ingestion:
child.getMetadata().put("parentId", parent.getId());
child.getMetadata().put("headingPath", "Returns Policy > Refund Window");Then wrap retrieval so it swaps children for parents:
public class ParentDocumentRetriever implements DocumentRetriever {
private final DocumentRetriever delegate;
private final ParentStore parents;
@Override
public List<Document> retrieve(Query query) {
return delegate.retrieve(query).stream()
.map(child -> parents.find(child.getMetadata().get("parentId")))
.flatMap(Optional::stream)
.distinct()
.toList();
}
}Because DocumentRetriever is the interface RetrievalAugmentationAdvisor consumes, this drops straight into the pipeline from the basic RAG post.
Worth it when your documents have real hierarchy — legal contracts, technical manuals, anything with numbered sections. Not worth it for a flat pile of support tickets. And note the deduplication in that snippet: three sibling chunks matching means one parent, not three copies of it.
Fine-tuned embeddings: last resort, not first
Train an embedding model on your own query-document pairs and it learns your domain’s vocabulary — that in your corpus “discharge” is a medical event rather than an electrical one. Reported gains run 5–10%, and a well-tuned small model can beat a larger generic one.
The costs are the problem. You need labelled query-document pairs, which mostly means you need traffic you’ve already annotated. You need a training and evaluation pipeline. And critically: vectors from different models are not comparable, so every retraining means re-embedding the entire corpus. That’s not a tweak, it’s a migration, and you’ll be doing it on a schedule forever.
Reach for it when you’ve done everything else and you have genuinely specialised vocabulary — medicine, law, chemistry, internal jargon that appears nowhere in the pretraining data. For most applications a strong general model plus contextual retrieval gets you further for a fraction of the effort.
What I’d actually do
In order:
- Fix chunking to follow document structure. Free, and everything else compounds on it.
- Add contextual retrieval. One cheap LLM call per chunk, one-off, with the largest verified payoff in this series. Use prompt caching.
- Add parent-child if your documents have real hierarchy.
- Skip late chunking unless your embedding provider hands it to you.
- Consider fine-tuned embeddings only with specialised vocabulary and labelled data.
And re-measure after each one against the golden dataset from part one. Ingestion changes are the expensive kind to undo.
Conclusion
The ingestion half is where the biggest single win lives, and it isn’t the sophisticated technique. It’s asking a cheap model to write two sentences of context per chunk — a 35% reduction in retrieval failures for about a dollar per million tokens, once.
It’s also where the marketing gap is widest. Late chunking is the more elegant idea and buys roughly a tenth as much. Fine-tuned embeddings sound like the serious-engineering answer and commit you to permanent re-embedding. The numbers are public in both cases, and they don’t match the enthusiasm.
Next: query time, where Spring AI stops making you build things and hands you most of the strategies as components.
More info: anthropic.com/news/contextual-retrieval