Spring AI - One Codebase, Many Model Providers

Most Spring developers I talk to have the same reaction to the AI ecosystem: everything interesting seems to be written in Python, and the Java options look like thin HTTP wrappers. That was roughly true two years ago. It isn’t now.

This is the first part of a series about building AI features in Spring Boot properly — not just getting a response back from OpenAI, but the parts that decide whether the thing survives contact with production. I’m starting where the value is most obvious and least talked about: writing one codebase that runs against any model provider you like.

Everything here is Spring AI 2.0 on Spring Boot 4 and Java 21. That matters more than it usually does, and I’ll explain why near the end.

Why not just call the REST API?

It’s a fair question. Every provider ships an HTTP API. You have RestClient. You could be done in an afternoon.

You could, and for a single call you’d be right. The problem shows up on the second and third requirement:

Every provider has a different request shape. OpenAI wants messages, Anthropic wants messages plus a top-level system, Ollama wants something else again. That difference leaks into your service layer the moment you hand-roll it.
The interesting features are cross-cutting. Conversation memory, retrieval, tool calling, logging, guardrails — none of them belong inside your controller, and all of them need to wrap the model call.
Switching costs are real. Prices change, models get deprecated, a model that was best-in-class in March is mid-tier by September. If your provider choice is spread across twenty classes, you won’t switch even when you should.

Spring AI’s answer is a portable ChatClient plus an interceptor chain called advisors. The advisors are the subject of the next part. For now, portability.

The smallest possible Spring AI application

One dependency, one property, one controller. Here’s the whole thing:

@RestController
public class ChatController {

    private final ChatClient chatClient;

    public ChatController(ChatClient.Builder builder) {
        this.chatClient = builder
                .defaultSystem("You are a concise assistant helping a Spring developer learn Spring AI. Answer in at most three sentences.")
                .build();
    }

    @GetMapping("/chat")
    public String chat(@RequestParam String message) {
        return chatClient.prompt()
                .user(message)
                .call()
                .content();
    }
}

Look at what is not in that class. There is no OpenAiChatModel, no AnthropicChatModel, no API key, no provider name anywhere. You inject ChatClient.Builder — Spring AI auto-configures it — and you get a fluent API that reads like the thing it does: prompt, user message, call, content.

defaultSystem sets a system prompt once for every call made through this client. It’s the same idea as a RestClient default header.

The dependency is one starter:

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>

with the Spring AI BOM imported so you don’t manage versions by hand.

Running it without an API key

Before adding a provider that bills you, it’s worth knowing you don’t have to. Ollama runs real models on your laptop, and Spring AI treats it as a first-class provider rather than an afterthought.

docker run -d -p 11434:11434 --name ollama ollama/ollama
docker exec ollama ollama pull llama3.2

Swap the starter for spring-ai-starter-model-ollama, point it at the container, and the controller above does not change by one character:

spring:
  ai:
    ollama:
      base-url: http://localhost:11434
      chat:
        options:
          model: llama3.2
          temperature: 0.7

The first response will be slow — the model is being loaded into RAM. The second one won’t be. If you’re evaluating Spring AI at all, do it this way first; there is no reason to put a credit card down to find out whether the abstraction fits your codebase.

The part that actually matters: one property

Here’s the piece I think is undersold. Spring AI 2.0 auto-configures every model provider it finds on the classpath, and a single property decides which one is live.

So put all of them on the classpath at once:

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-anthropic</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-ollama</artifactId>
</dependency>

and select one:

spring:
  ai:
    model:
      chat: ollama        # openai | anthropic | ollama | none
      embedding: ollama

That’s the switch. Not a code change, not a pom.xml edit — a property. Which means it’s a Spring profile, which means it’s an environment variable, which means you can run the same artifact against a local model in development and a hosted one in production without a rebuild.

I put each provider in its own profile YAML and never touch the application code again.

Proving the switch is real

Claims like “provider-agnostic” are cheap. This endpoint makes it falsifiable — same code path as /chat, but it also reports who actually answered:

@GetMapping("/whoami")
public Map<String, String> whoAmI() {
    var response = chatClient.prompt()
            .user("In one short sentence, say which AI model you are.")
            .call()
            .chatResponse();

    return Map.of(
            "activeProfile", String.join(",", environment.getActiveProfiles()),
            "configuredProvider", environment.getProperty("spring.ai.model.chat", "(unset)"),
            "modelReportedByApi", response.getMetadata().getModel(),
            "answer", response.getResult().getOutput().getText());
}

Note the difference between .content() and .chatResponse(). content() gives you the string. chatResponse() gives you the whole envelope — model name, token usage, finish reason, all the metadata you’ll need later for cost tracking and observability. Reach for chatResponse() more often than feels necessary; the metadata is where the operational answers live.

Run it on two profiles and only the metadata changes:

SPRING_PROFILES_ACTIVE=ollama    ./mvnw spring-boot:run
SPRING_PROFILES_ACTIVE=anthropic ./mvnw spring-boot:run

Two provider constraints worth knowing before you commit

Portability is real but it is not total, and the two places it leaks are worth stating plainly.

Anthropic has no embeddings API. There is no Claude embedding model to call. If your app does retrieval — and by part four of this series it will — you need embeddings from somewhere else. In my setup the Anthropic profile deliberately falls back to Ollama for embeddings:

spring:
  ai:
    model:
      chat: anthropic
      embedding: ollama   # Anthropic ships no embedding model

That’s not a workaround, it’s the shape of the provider. Good news: because chat and embedding are selected independently, mixing them is a one-line configuration, not an architecture problem.

Current Claude models reject temperature. This one bites people. claude-opus-5 and claude-sonnet-5 removed the sampling parameters — send temperature, top_p or top_k and you get back an HTTP 400.

Spring AI 1.x always sent a default temperature of 0.7 whether you set one or not. Spring AI 2.0 stopped doing that, so an unset value is now genuinely unset. This is exactly the right change, and it also means most of the Spring AI tutorials currently indexed by Google were written against 1.x and will fail on current Claude models for reasons that look nothing like the actual cause.

So the Anthropic profile sets no temperature at all:

spring:
  ai:
    anthropic:
      api-key: ${ANTHROPIC_API_KEY:}
      chat:
        options:
          model: claude-opus-5
          max-tokens: 2048

max-tokens is set explicitly because 2.0 also dropped Anthropic’s old default. I’ll come back to ChatOptions and this whole class of cross-provider trap in the next part.

One more trick: any OpenAI-compatible endpoint

There is no spring-ai-starter-model-dmr for Docker Model Runner, and you don’t need one. DMR exposes an OpenAI-compatible API, so you reuse the OpenAI client and repoint its base URL:

spring:
  ai:
    model:
      chat: openai        # the OpenAI *client*, pointed somewhere else
    openai:
      base-url: http://localhost:12434/engines
      api-key: not-needed-for-local-dmr
      chat:
        options:
          model: ai/llama3.2

The same trick works for vLLM, LM Studio and LocalAI. Any server that speaks the OpenAI wire format is already a Spring AI provider — you just have to notice.

Streaming

Blocking until a long answer is fully generated is a bad experience, and the fix is one method:

@GetMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> stream(@RequestParam String message) {
    return chatClient.prompt()
            .user(message)
            .stream()
            .content();
}

.call() becomes .stream(), String becomes Flux<String>, and the browser gets server-sent events. Worth knowing early: streaming has its own half of several APIs later in the series — the advisor SPI, for instance, has both CallAdvisor and StreamAdvisor. If you plan to stream in production, wire it up early rather than retrofitting it.

Choosing a deployment

Having three options is only useful if you know when to pick which. Roughly how I decide:

	Local (Ollama)	Hosted API	Self-hosted GPU
Cost	Free	Per token	Fixed and high
Data leaves your network	No	Yes	No
Quality ceiling	Moderate	Highest	High
Ops burden	Minimal	None	Substantial
Best for	Development, tests, CI	Most production apps	Regulated data at volume

The pattern that works for most teams: local models in development and CI so nobody’s laptop burns budget and tests are free to run, hosted models in production, and the switch between them expressed as a property rather than a branch.

Conclusion

The single most valuable thing Spring AI gives you isn’t the fluent API — it’s that provider choice stops being an architectural commitment. One ChatClient, all providers on the classpath, one property to select. You can develop for free, test for free, and decide who bills you at deploy time.

Two things to carry into the rest of this series. First, use .chatResponse() rather than .content() when you might ever care about tokens or model metadata, because you will. Second, if you follow a Spring AI tutorial and get an unexplained 400, check whether it was written for 1.x — the defaults changed, and the change was correct.

Next up: message roles, ChatOptions, the advisor chain, and getting typed Java objects out of a model instead of parsing strings.

More info: docs.spring.io/spring-ai/reference