In part one I got a ChatClient running and switched it between OpenAI, Anthropic and a local model with a single property. That’s the setup. This part is the working vocabulary — the four things you’ll touch every day once the novelty wears off.
Two of them are ordinary and two are the reason Spring AI is worth using at all. I’ll go in that order.
Message roles are not decoration
Every chat model sees a list of messages, and each one carries a role. There are three that matter.
System is instruction: who the assistant is, what it may and may not do, what format to answer in. User is the request. Assistant is what the model said previously, which is how you replay a conversation.
The distinction is not cosmetic. Providers train models to weight system messages differently from user messages, and to treat them as more authoritative. Put your rules in a user message and you’ve made them negotiable — a later user message can argue with them. That’s the seed of prompt injection, which gets its own part later in this series.
In Spring AI:
String answer = chatClient.prompt()
.system("You are a support agent for an e-commerce API. Never invent order numbers.")
.user("Where is order 4417?")
.call()
.content();Set defaults once, on the builder
Repeating a system prompt at every call site is how it drifts. Set it on the builder instead:
this.chatClient = builder
.defaultSystem("You are a concise assistant. Answer in at most three sentences.")
.defaultOptions(ChatOptions.builder().maxTokens(500).build())
.defaultAdvisors(new TimingAdvisor())
.build();defaultSystem, defaultOptions, defaultAdvisors — all three apply to every call made through that client, and all three can be overridden per request. It’s the same mental model as default headers on a RestClient. Anything that should hold across your whole application belongs here, not scattered through controllers.
Prompts longer than a line don’t belong inline either. Externalize them:
@Value("classpath:/prompts/support-agent.st")
private Resource supportAgentPrompt;
String answer = chatClient.prompt()
.system(supportAgentPrompt)
.user(u -> u.text("Summarize order {orderId} for the customer.")
.param("orderId", orderId))
.call()
.content();Now the prompt is a reviewable file that shows up in diffs, which is exactly what you want the first time somebody changes one word and the output quality moves.
ChatOptions, and the trap that will cost you an afternoon
ChatOptions is where you set the model, token ceiling and sampling parameters — globally on the builder, or per request:
String answer = chatClient.prompt()
.user(message)
.options(ChatOptions.builder()
.model("gpt-4o-mini")
.maxTokens(1000)
.temperature(0.2)
.build())
.call()
.content();Temperature is the one everyone reaches for. Low values make output more deterministic, high values more varied. For extraction, classification and anything you’ll parse, keep it low. For copy, raise it.
Now the trap, and it’s a good one.
Current Claude models reject
temperatureoutright.claude-opus-5andclaude-sonnet-5removed the sampling parameters. Sendtemperature,top_portop_kand the API returns HTTP 400.
This is worse than it sounds, because of a version detail. Spring AI 1.x always sent a default temperature of 0.7, whether you set one or not. Spring AI 2.0 stopped doing that, so an unset value is now genuinely unset.
That change is correct, and it also means the 400 is entirely invisible in your code. You never wrote temperature anywhere. On 1.x it was being sent for you; on 2.0 it isn’t — and a large share of the Spring AI material currently online was written against 1.x, so following it will reintroduce the parameter and break Claude for reasons that look nothing like the cause.
Two practical consequences. Set options per provider profile, not globally, when the values aren’t portable. And treat “portable API” as meaning portable code, not portable parameters — the abstraction covers the shape of the call, not the union of every provider’s quirks.
Advisors: the interceptor chain around every call
This is the part I’d argue is the actual reason to adopt Spring AI.
An advisor wraps the model call. Everything before chain.nextCall(request) runs on the way in, everything after runs on the way out. If you’ve written a servlet filter, a Spring HandlerInterceptor, or around-advice in AOP, you already know this shape:
public class TimingAdvisor implements CallAdvisor {
private static final Logger log = LoggerFactory.getLogger(TimingAdvisor.class);
@Override
public String getName() {
return "timing";
}
@Override
public int getOrder() {
return 0;
}
@Override
public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
long start = System.nanoTime();
ChatClientResponse response = chain.nextCall(request);
long ms = (System.nanoTime() - start) / 1_000_000;
log.info("Model call took {} ms", ms);
return response;
}
}Three methods: a name, an order, and the one that does the work. Register it with .defaultAdvisors(new TimingAdvisor()) and every call in the application is timed, from one class, with no controller aware of it.
Make it useful by logging what you’re actually being billed for:
ChatClientResponse response = chain.nextCall(request);
Usage usage = response.chatResponse().getMetadata().getUsage();
log.info("Call took {} ms, {} prompt + {} completion tokens",
ms, usage.getPromptTokens(), usage.getCompletionTokens());That is the seed of real cost tracking, and it’s about fifteen lines.
Spring AI ships advisors for the things you’d otherwise build badly yourself — chat memory, retrieval, tool calling. In 2.0 the tool-calling loop itself is an advisor (ToolCallingAdvisor), which means other advisors can intercept it. That’s a genuinely useful piece of design and it shows up again when we get to guardrails.
Three things that make advisors powerful
Ordering. Lower getOrder runs earlier on the way in, and therefore later on the way out. If you want to log the fully assembled prompt, your logger has to run after the advisors that assemble it — a higher order number, not a lower one. Get this backwards and you’ll conclude memory is broken when your log line is just firing too early.
Request mutation. Requests are immutable, so you build a modified copy:
@Override
public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
ChatClientRequest tagged = request.mutate()
.context(Map.of("tenantId", TenantContext.current()))
.build();
return chain.nextCall(tagged);
}That context map travels down the chain and is readable by everything downstream — tenant id, user id, trace id, all without changing a method signature anywhere.
Not calling the model at all. chain.nextCall is mandatory only in the sense that skipping it means the request never reaches the model. Which is a feature:
@Override
public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
String question = request.prompt().getUserMessage().getText();
if (blocklist.matches(question)) {
return ChatClientResponse.builder()
.chatResponse(refusal("I can't help with that."))
.build();
}
return chain.nextCall(request);
}Short-circuit and return your own response and you’ve built a cache, a rate limiter, or an input guardrail. Same interface, three very different features.
One thing to plan for: the SPI has two halves. CallAdvisor handles blocking calls, StreamAdvisor handles reactive ones. If you stream in production, implement both from the start rather than discovering the gap later.
Structured output: stop parsing strings
Models return prose. Your application wants objects. The naive fix is asking for JSON and parsing the result, and it fails in all the obvious ways — a stray “Here’s the JSON you asked for:”, a markdown fence around it, a trailing comma.
Spring AI handles it:
record Invoice(
String invoiceNumber,
String vendor,
LocalDate issuedOn,
BigDecimal total) {}Invoice invoice = chatClient.prompt()
.user("Extract the invoice details from:\n" + rawText)
.call()
.entity(Invoice.class);.entity(Invoice.class) instead of .content(). That’s the entire change. LocalDate parsed, BigDecimal parsed, no JSON handling, no cleanup, no try/catch. Behind the scenes Spring AI generates a schema from your type, includes it in the request, and converts the response back.
For a List<T>, generics need a hand:
List<Invoice> invoices = chatClient.prompt()
.user("Extract every invoice from:\n" + rawText)
.call()
.entity(new ParameterizedTypeReference<List<Invoice>>() {});Two things make this much better in practice.
- Field names are prompts. The names go into the schema the model sees. Rename
totaltototalIncludingTaxand extraction accuracy improves. Descriptive naming isn’t only for your colleagues here. - Annotations are prompts too.
@JsonPropertyDescription("Total including tax, digits only, no currency symbol")lands in the schema. You’re prompt-engineering inside the type definition, which is a far better home for it than a string constant.
And one honest limit: structured output guarantees the shape, not the truth. You will get a well-formed Invoice. Whether total is the right number is a separate question. Validate anything financial, and be ready to retry — @Valid on the record plus a bounded retry around the call covers most of it.
Conclusion
The four pieces in this part are the ones you’ll reach for constantly. Roles decide what the model treats as authoritative. Builder defaults keep configuration in one place. Advisors are where cross-cutting behaviour belongs, and they’re the extension point that everything later in this series plugs into. Structured output ends string parsing.
If you take one operational thing away, make it the temperature trap: options are not portable even when your code is, so set them per provider and be suspicious of any Spring AI tutorial that predates 2.0.
Next: chat memory — why the model forgets everything between requests, and the surprisingly sharp edges in making it remember.
More info: docs.spring.io/spring-ai/reference/api/chatclient