Spring AI RAG - Talking to Your Own Documents

Every team that adds AI to a product arrives at the same request within about a week: can it answer questions about our documents? The handbook, the API docs, the four years of support tickets.

The model can’t, on its own. It was trained on public text that stopped at some date, and it has never seen your refund policy. The fix is retrieval-augmented generation, and the good news is that the version that actually works is much less complicated than the diagrams suggest.

This is part four. Part three covered chat memory, and the cost problem there — resending everything on every turn — is exactly the problem retrieval solves at a bigger scale.

Start with the version that isn’t RAG

Before retrieval, there’s prompt stuffing: put the documents straight into the prompt.

String answer = chatClient.prompt()
        .system("Answer using only this policy document:\n\n" + policyText)
        .user(question)
        .call()
        .content();

This works, and for a small fixed document it’s the right answer. Don’t build a vector database to answer questions about a two-page policy.

It stops working for two reasons, both hard limits rather than matters of taste. Context windows are finite — you cannot stuff a 400-page handbook into a prompt. And you pay for every token on every request, so stuffing 50,000 tokens of context to answer a one-line question is expensive by a factor that gets embarrassing at volume.

RAG is the obvious response: don’t send everything, send the parts that matter.

The flow, in one paragraph

Split your documents into chunks. Convert each chunk to a vector that encodes its meaning. Store the vectors. When a question arrives, convert the question to a vector too, find the chunks whose vectors are nearest, and put just those into the prompt.

That’s the whole idea. Ingest, embed, store, retrieve, generate.

RAG with no infrastructure at all

You don’t need Docker to see this work. SimpleVectorStore is an in-memory implementation:

@Bean
VectorStore vectorStore(EmbeddingModel embeddingModel) {
    return SimpleVectorStore.builder(embeddingModel).build();
}

Add some documents:

List<Document> docs = List.of(
        new Document("Refunds must be requested within 30 days of delivery."),
        new Document("Express shipping is delivered within 2 business days."),
        new Document("Warranty covers manufacturing defects for 24 months."));

vectorStore.add(docs);

And search:

List<Document> results = vectorStore.similaritySearch(
        SearchRequest.builder().query("Can I send this back?").topK(2).build());

The refund document comes back first. Look closely at why that’s interesting: the question says “send this back”, the document says “refunds”, and they have not one word in common. Keyword search returns nothing here. Semantic search returns the right answer, because the vectors encode meaning rather than spelling.

Complete the loop and you have working RAG:

String context = results.stream()
        .map(Document::getText)
        .collect(joining("\n\n"));

String answer = chatClient.prompt()
        .system("Answer using ONLY the context below. If it isn't there, say you don't know.\n\n" + context)
        .user(question)
        .call()
        .content();

No database, no cloud account. SimpleVectorStore is not a production answer — it’s in memory, gone on restart, and does a brute-force scan — but it’s a perfectly good answer for a prototype, and it lets you understand the mechanism before a vector database hides it from you.

A real vector store

For production, swap the bean. Qdrant:

spring:
  ai:
    vectorstore:
      qdrant:
        host: localhost
        port: 6334
        collection-name: docs
        initialize-schema: true

Or pgvector, which is usually the right call if you already run Postgres — one less system to operate, backed up by your existing backups. The VectorStore interface doesn’t change, so this really is a configuration swap.

The ETL pipeline

Real documents aren’t new Document("..."). Spring AI models ingestion as read, transform, write:

List<Document> documents = new TikaDocumentReader(resource).read();
List<Document> chunks = new TokenTextSplitter().apply(documents);
vectorStore.write(chunks);

Tika handles PDF, Word, HTML and most things you’ll be handed. The shape is the same whatever the source format.

The pattern matters more than the classes, because it’s a pipeline and you can insert your own stage anywhere. Redact PII before embedding. Tag every chunk with the tenant it belongs to. Strip boilerplate headers. All of that is just another transformer — and doing it once at ingestion is far cheaper than doing it on every query.

@Service
public class IngestionService {

    private final VectorStore vectorStore;

    public void ingest(Resource resource) {
        List<Document> documents = new TikaDocumentReader(resource).read();

        List<Document> chunks = TokenTextSplitter.builder()
                .withChunkSize(400)
                .withMinChunkSizeChars(200)
                .build()
                .apply(documents);

        chunks.forEach(c -> c.getMetadata().put("source", resource.getFilename()));

        vectorStore.write(chunks);
    }
}

Chunking is the setting that decides answer quality

If your RAG gives bad answers, chunking is the first thing I’d look at — ahead of the model, ahead of the prompt.

Chunks too large and each one contains several topics, so its vector is a blurry average of all of them and matches nothing precisely. Chunks too small and you retrieve a sentence without the context that makes it meaningful.

Overlap matters too, and for a specific reason: a chunk boundary can land in the worst possible place, separating a definition from the thing it defines. Overlapping means the important sentence appears in both neighbouring chunks, so no boundary can hide it.

Better than tuning numbers, though: split on structure. Markdown headings, document sections, ticket boundaries. A chunk that corresponds to a real unit of meaning beats any character count you can guess.

RAG in one line

Once you understand the hand-rolled version, Spring AI packages it as an advisor:

ChatClient client = builder
        .defaultAdvisors(QuestionAnswerAdvisor.builder(vectorStore).build())
        .build();

String answer = client.prompt().user(question).call().content();

The advisor does the search and the prompt assembly. Configure it:

QuestionAnswerAdvisor.builder(vectorStore)
        .searchRequest(SearchRequest.builder()
                .topK(5)
                .similarityThreshold(0.7)
                .build())
        .build();

topK is how many chunks to retrieve. similarityThreshold is a floor below which results are discarded as noise, and it matters more than people expect. Without a threshold, a question with no good answer still returns your five least-bad chunks, and the model will cheerfully answer from irrelevant text. With one, you get nothing back and can say “I don’t know” honestly. 0.7 is a reasonable starting point.

When you need the seams

QuestionAnswerAdvisor is the convenient entry point. RetrievalAugmentationAdvisor is the modular one — same behaviour to begin with, but every stage is a component you can replace:

RetrievalAugmentationAdvisor.builder()
        .documentRetriever(VectorStoreDocumentRetriever.builder()
                .vectorStore(vectorStore)
                .similarityThreshold(0.7)
                .topK(5)
                .build())
        .build();

The seams that earn their keep:

Query transformation, before retrieval. Users ask badly. “What about the other one?” retrieves nothing useful. A rewriting transformer turns a vague follow-up into a standalone question first; a translating one handles queries in a different language from your corpus.
Post-retrieval processing. Rerank, compress or deduplicate what came back. Retrieval optimizes for vector similarity, which is not the same as usefulness, and a reranking pass over ten candidates often beats retrieving five directly.
Query augmentation. How retrieved context gets folded into the prompt — including what the model is told to do when nothing was retrieved.

Metadata filtering, which is a security control

This is the part I’d insist on before any multi-tenant deployment.

Tag at ingestion:

chunk.getMetadata().put("tenantId", tenantId);
chunk.getMetadata().put("classification", "internal");

Filter at query time:

SearchRequest.builder()
        .query(question)
        .topK(5)
        .filterExpression("tenantId == '%s'".formatted(currentTenant()))
        .build();

The filter is applied inside the vector store, so another tenant’s vectors are never even candidates. Three rules I’d treat as non-negotiable:

Derive the filter server-side from the authenticated principal, never from a request parameter. Same rule as the conversation id in part three, and the same consequence if you get it wrong.
Default to deny. An untagged document should be unreachable, not visible to everyone.
Test the negative case. Write the test that asserts tenant A cannot retrieve tenant B’s document, and run it in CI. A cross-tenant leak through a vector store is silent — nothing errors, the answer just quietly contains someone else’s data.

Citations

Include source metadata in the context and ask for citations:

String context = results.stream()
        .map(d -> "[%s p.%s] %s".formatted(
                d.getMetadata().get("source"),
                d.getMetadata().get("page_number"),
                d.getText()))
        .collect(joining("\n\n"));

Now every claim is traceable, and when an answer is wrong you can see why: was retrieval bad, or did the model misread a correct chunk? Those have completely different fixes, and without citations you can’t tell them apart.

Who pays for embeddings

Worth knowing before it’s a surprise. You pay to embed every chunk at ingestion, and to embed every query. Ingestion is a one-off per document but scales with corpus size; queries are forever.

Two consequences. Re-embedding your whole corpus because you changed the chunk size is a real cost, so settle chunking on a sample first. And if you change embedding model, you must re-embed everything — vectors from different models are not comparable, and mixing them produces retrieval that’s subtly, unfixably bad.

Conclusion

RAG is much simpler than its reputation. Chunk, embed, store, retrieve the nearest, put those in the prompt. Spring AI reduces the whole thing to an advisor, and you can build the entire pipeline on your laptop with SimpleVectorStore before committing to any infrastructure.

The parts that decide whether it works in production aren’t the model: chunking quality, a similarity threshold so you can honestly say “I don’t know”, metadata filters derived server-side, and citations so failures are diagnosable.

Next: tool calling — letting the model do things rather than just answer, which is where this stops being a search feature and starts being an application.

More info: docs.spring.io/spring-ai/reference/api/retrieval-augmented-generation