Spring AI Chat Memory - Making a Stateless Model Remember

The first genuinely surprising thing about building on an LLM is that it has no memory at all. Not a short memory — none. Ask it your name, tell it, ask again in the next request, and it has no idea.

That catches people out because ChatGPT feels like it remembers. It doesn’t; the product around it resends the conversation on every request. In part two I covered the advisor chain, and memory turns out to be the cleanest possible demonstration of why that abstraction exists.

Prove it forgets

Two calls, no memory:

curl "localhost:8080/chat?message=My+name+is+Rida"
curl "localhost:8080/chat?message=What+is+my+name"

The second response will be some version of “I don’t have access to that information.” Nothing is broken. The model is a pure function: messages in, text out. Every HTTP request to the provider is a blank slate.

So “remembering” is not a model feature you switch on. It’s something you do: keep the transcript, and resend it. Which sounds trivial until you ask where you keep it, how much of it you resend, and who pays for the tokens.

One advisor is the entire integration

@Bean
ChatClient chatClient(ChatClient.Builder builder, ChatMemory chatMemory) {
    return builder
            .defaultAdvisors(MessageChatMemoryAdvisor.builder(chatMemory).build())
            .build();
}

That’s it. Knowing the advisor shape from part two, you can picture exactly what it does: on the way in, load this conversation’s history and prepend it to the messages; on the way out, save the new user message and the model’s reply.

ChatMemory is the storage abstraction. The default implementation in 2.0 is MessageWindowChatMemory, which keeps a bounded window of recent messages and is backed by a ChatMemoryRepository.

The change that breaks every 1.x snippet

Run the two-call test now and it still fails. There’s a second piece: the conversation id, which tells the advisor whose history to load.

If you copy a Spring AI 1.x example you’ll find this:

// 1.x — no longer available
MessageChatMemoryAdvisor.builder(chatMemory)
        .conversationId("user-42")
        .build();

That’s gone in 2.0, and it deserved to go. The advisor is a singleton bean shared by every request — baking one conversation id into it never made sense for a real application. In 2.0 you pass it per call, which is the only correct place for per-request state:

@GetMapping("/chat")
public String chat(@RequestParam String message,
                   @RequestParam(defaultValue = "default") String conversationId) {
    return chatClient.prompt()
            .user(message)
            .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId))
            .call()
            .content();
}

Rerun the two calls with the same conversationId and the second one answers correctly. If .conversationId(...) doesn’t compile against 2.0, this is why, and moving it to the call site is the fix.

Where the conversation id must come from

One rule, and it’s a security rule rather than a style preference:

Never take the conversation id from a request parameter the client controls. It is an access key to somebody’s conversation history.

The example above uses a query parameter because it’s a demo. In anything real, derive it from the authenticated principal:

@GetMapping("/chat")
public String chat(@RequestParam String message, Principal principal) {
    String conversationId = principal.getName();

    return chatClient.prompt()
            .user(message)
            .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, conversationId))
            .call()
            .content();
}

A client-supplied id is an enumerable identifier for other people’s private conversations. Treat it exactly as you’d treat a user id in a URL.

Surviving a restart

In-memory storage works until you deploy. Then every conversation in flight is gone. For persistence, add the JDBC repository starter:

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-chat-memory-repository-jdbc</artifactId>
</dependency>

Point it at a datasource and JdbcChatMemoryRepository is auto-configured — Spring AI creates the schema and the advisor now reads and writes through your database. Wire the window explicitly when you want control over its size:

@Bean
ChatMemory chatMemory(ChatMemoryRepository repository) {
    return MessageWindowChatMemory.builder()
            .chatMemoryRepository(repository)
            .maxMessages(20)
            .build();
}

Nothing in the controller changes. That’s the payoff of memory being an advisor over a repository interface rather than something baked into the call.

Semantic recall, when the window isn’t enough

A window keeps the last N messages. For a long-running relationship — a support assistant that should recall a preference stated six weeks ago — recency is the wrong selector.

VectorStoreChatMemoryAdvisor stores the full history in a vector store and retrieves the messages semantically similar to the current question rather than the most recent ones. Mention shipping and it surfaces the exchange about your delivery address, however long ago that was.

The trade-off is real: you’re now paying for an embedding on every stored message and running a similarity search on every turn. Worth it for genuinely long-lived assistants, over-engineering for a checkout bot. Vector stores are the subject of the next part.

The cost nobody budgets for

Here is the thing that surprises teams on their first bill.

You are billed per token, and the transcript is resent in full, on every single turn. Turn one sends one message. Turn twenty sends twenty. The cost of a conversation doesn’t grow linearly with its length — it grows with the square of it.

A long support conversation can cost more in resent history than in anything the user actually asked. And beyond cost there’s a hard ceiling: context windows are finite, and a transcript that outgrows the window gets truncated, at which point the model silently loses the beginning of the conversation.

Three levers, in the order I’d reach for them:

Bound the window. maxMessages on MessageWindowChatMemory caps what gets resent. Start here; it’s one line and it solves most of the problem.
Summarize instead of replaying. Compress old turns into a running summary and keep recent turns verbatim. You keep the thread of the conversation at a fraction of the tokens.
Retrieve instead of resending. The VectorStoreChatMemoryAdvisor approach — pull back only the handful of past messages that are relevant now.

Log token usage per call from the start. The advisor from part two does it in five lines, and it turns “the bill went up” into a number you can point at.

The gotcha: tool results aren’t in memory

Save yourself an afternoon on this one.

When a model calls a tool, several messages get exchanged behind the scenes: the model requests the tool, your code runs it, the result goes back, the model answers. Those intermediate tool messages are not stored in chat memory.

Which produces a genuinely baffling bug:

Turn one: “What’s the weather in Paris?” The model calls a weather tool, gets 18 degrees, tells the user.
Turn two: “And what did you say it was?” The model may have no idea, because the tool call and its result weren’t kept. Only its final text reply was.

It’s a defensible design decision — tool exchanges can be enormous, and persisting a 500-row result set into every subsequent request would be ruinous. But it will surprise you.

Two ways to handle it. Make the final answer self-contained: instruct the model to state key facts in its reply — “The weather in Paris is 18 degrees” rather than “It’s 18 degrees” — so the fact lands in the transcript because the reply does. Or store what matters yourself, writing tool results into your own state and injecting them deliberately rather than hoping memory kept them.

Either way: don’t assume tool results persist. They don’t.

Choosing a strategy

Strategy	Survives restart	Cost per turn	Use when
In-memory window	No	Low	Development, short sessions, stateless demos
JDBC window	Yes	Low	The default for most applications
Vector store recall	Yes	Higher (embeddings + search)	Long-lived assistants with months of history
Summarized window	Yes	Low, with an extra model call	Long conversations on a tight token budget

Most applications want the JDBC window with a sensible maxMessages. Reach past it when you can point at the specific problem it solves.

Conclusion

Memory is a good early lesson in what building on LLMs is actually like. The model contributes nothing — you keep the transcript, you resend it, you pay for it every turn. Spring AI makes the mechanics almost free, and that’s exactly what makes it easy to forget there’s a quadratic cost curve underneath.

Three things to remember: the conversation id is passed per call in 2.0 and must never come from the client; bound your window before you need to; and tool results are not in the transcript.

Next: retrieval. Instead of stuffing everything the model might need into the prompt, we go and fetch only the parts that matter.

More info: docs.spring.io/spring-ai/reference/api/chat-memory