Class 7: Spring AI Integration
Duration: ~60 minutes | Difficulty: Advanced | Prerequisites: Class 4 completed; Ollama with a tool-calling model.
This lesson closes the framework track (Classes 5 to 7). It moves the callback, approval gate and run budget from Class 4 into Spring AI. Classes 8 to 12 write their code against the ChatClient that this lesson introduces, and what those lessons teach applies to all three frameworks.
The code here builds on Class 4. It uses the same generated catalog and the same GraphQlToolCallback, with its login, gate and budget, in a different framework. Every Java class that it calls comes from Classes 2 and 4, so you can come here straight from Class 4. Class 5 and Class 6 are the other two paths through the same material.
The repository has one line of branches, though. The branch before this lesson is agents_class_6, and it also contains the LangChain4j and LangGraph4j code. If you followed the whole track, keep working on the project you already have. Otherwise clone the agents_class_6 branch to start from the same place:
git clone --branch agents_class_6 https://github.com/graphqlguy/graphQLBeyondMCP.git
Code blocks are titled with the file they belong to, and the note in brackets says what to do with it. (new file) is a file to create. (edited, whole file) replaces a file you already have. Anything else names the one part to add or replace, as in (edited, the call method). Where a block shows part of a new file, the complete file follows in a collapsed block underneath.
The framework track ends with Spring AI. Our MCP course already builds on it, and the rest of this course assumes it. It is also the smallest integration of the three, because the GraphQlToolCallback from Class 2 already implements Spring AI's own ToolCallback interface. Class 5 needed a bridge class to translate that callback into LangChain4j's two objects. Class 6 needed a graph class around it. Here the same callbacks plug straight into the client.
By the end of this lesson, the concierge runs through Spring AI's ChatClient. You will know where the tool loop went in Spring AI 2.0, and what that means for the controls. Along the way we measure what a growing tool catalog costs and what the framework's tool search does about it. We also watch the client stream an answer token by token while tools run in the middle of the stream.
The Whole Integration
Here is the whole integration:
var spec = client.prompt()
.system(SYSTEM)
.user(turn)
// tools(Object...) replaces toolCallbacks(), deprecated in Spring AI 2.0
.tools(callbacks.toArray())
.options(OllamaChatOptions.builder().model(context.modelName()))
.advisors(MessageChatMemoryAdvisor.builder(memory).build())
.advisors(advisorParams -> advisorParams.param(ChatMemory.CONVERSATION_ID, conversationId));
Both advisors read the same ChatMemory.CONVERSATION_ID parameter, which the request above sets once. The memory advisor uses it to find the conversation. The tool-search advisor uses it to find the index for that session, so each conversation reads only its own tool catalog. The advisor checks on every request that the value is present. If the session id is missing, the request fails with context must contain a non-null value for 'chat_memory_conversation_id'. The message names the missing key, so the repair takes one edit.
Class 5 needed a schema translator and an executor wrapper, and Class 6 needed nodes and channels. This lesson skips both. ToolCallback is Spring AI's own tool contract, and our GraphQlToolCallback has implemented it since Class 2. The request spec's toolCallbacks(...) accepts the list as it is. That is why Class 2 built against a framework interface instead of inventing a neutral one. Borrowing a contract from one framework means writing a small translator for each of the others, as Classes 5 and 6 did. In its own framework, the contract works as it is.
The Command, Assembled
The rest of the course is written against this client, so here is the whole run in one place. It names the pieces from earlier lessons where they enter. Only the last six lines are new:
package com.graphqlguy.moviedb.agents.command;
import com.graphqlguy.moviedb.agents.auth.AuthSession;
import com.graphqlguy.moviedb.agents.safety.ApprovalGate;
import com.graphqlguy.moviedb.agents.safety.RunBudget;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.toolsearch.ToolSearchToolCallingAdvisor;
import org.springframework.ai.chat.client.advisor.ToolCallingAdvisor;
import org.springframework.ai.model.tool.DefaultToolCallingManager;
import org.springframework.ai.model.tool.ToolCallingManager;
import org.springframework.ai.chat.memory.ChatMemory;
import org.springframework.ai.chat.memory.MessageWindowChatMemory;
import org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor;
import org.springframework.ai.chat.metadata.Usage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.toolsearch.ToolIndex;
import org.springframework.ai.tool.toolsearch.index.lucene.LuceneToolIndex;
import org.springframework.ai.tool.toolsearch.index.regex.RegexToolIndex;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.stereotype.Component;
import java.util.List;
/**
* Class 7: the same concierge with Spring AI's ChatClient owning the loop. The
* callbacks work without a bridge, because they already implement Spring AI's own
* ToolCallback interface, which is what Class 2 built them against.
*/
@Component
public class SpringAgentCommand implements AgentCommand {
protected static final String SYSTEM = """
You are a concierge for the Movie Database. Use the tools to answer
and to act; never invent ids, and never claim an action succeeded
without a tool result proving it. Ids are numeric strings like "2";
a name is never an id. Before any write, first call the read tools
to obtain every id the write needs.""";
private final ObjectProvider<ChatModel> chatModels;
public SpringAgentCommand(ObjectProvider<ChatModel> chatModels) {
this.chatModels = chatModels;
}
@Override
public String name() {
return "spring-agent";
}
@Override
public int order() {
return 70;
}
@Override
public String summary() {
return "the ChatClient with the advisor-owned loop";
}
@Override
public String prompt() {
return "task";
}
@Override
public void run(List<String> args, CommandContext context) {
String task = String.join(" ", args);
ChatModel springModel = chatModels.getIfAvailable();
if (task.isBlank() || springModel == null) {
System.out.println(task.isBlank() ? "a task is needed"
: "No chat model configured.");
return;
}
String role = context.role();
AuthSession auth = context.login();
RunBudget budget = context.budget();
ApprovalGate gate = new ApprovalGate(context.console());
List<ToolCallback> callbacks = context.callbacks(context.tools(role), auth, gate, budget);
boolean toolSearch = context.env().getProperty("agents.tool-search", Boolean.class, false);
String indexType = context.env().getProperty("agents.tool-search-index", "regex");
System.out.println("framework : Spring AI (ChatClient, advisor-owned loop)");
System.out.println("role : " + role + " (" + callbacks.size() + " tools)");
System.out.println("tool search: " + (toolSearch
? "tool search on, " + indexType + " index, definitions sent on demand"
: "off, every definition sent on every call"));
System.out.println("task : " + task);
System.out.println();
// Memory is an advisor here, which is the framework being consistent with
// itself: the loop belongs to the advisor chain, so what the loop remembers
// belongs there too. One conversation id covers the whole exchange.
// The loop belongs to the advisor, so the model-call ceiling has to belong
// there too. A ToolCallingManager built with the run's own limits bounds how
// far the advisor will go: each further model call needs another tool call to
// justify it, so capping tool calls caps the loop.
ToolCallingManager boundedManager = DefaultToolCallingManager.builder()
.maxTotalToolCalls(budget.maxToolCalls())
.build();
ChatMemory memory = MessageWindowChatMemory.builder().build();
String conversationId = "spring-agent-" + System.nanoTime();
ChatClient client = ChatClient.create(springModel);
String turn = task;
while (!turn.isBlank()) {
var spec = client.prompt()
.system(SYSTEM)
.user(turn)
// tools(Object...) replaces toolCallbacks(), deprecated in Spring AI 2.0
.tools(callbacks.toArray())
.options(OllamaChatOptions.builder().model(context.modelName()))
.advisors(MessageChatMemoryAdvisor.builder(memory).build())
.advisors(advisorParams -> advisorParams.param(ChatMemory.CONVERSATION_ID, conversationId));
// Tool search, off unless asked for. Spring AI's name for sending tool
// definitions on demand instead of all of them up front. The catalog is still
// handed over in full; the advisor decides how much of it the model sees,
// and when.
// Exactly one tool advisor may sit in the chain, and it owns the loop, so
// whichever one this run uses is also where the ceiling has to go.
if (toolSearch) {
ToolIndex index = "lucene".equals(indexType) ? new LuceneToolIndex() : new RegexToolIndex();
spec = spec.advisors(ToolSearchToolCallingAdvisor.builder()
.toolIndex(index)
.maxResults(5)
.toolCallingManager(boundedManager)
.build());
} else {
spec = spec.advisors(ToolCallingAdvisor.builder()
.toolCallingManager(boundedManager)
.build());
}
ChatResponse response = spec.call().chatResponse();
Generation generation = response == null ? null : response.getResult();
System.out.println("answer: " + (generation == null
? "(the model returned an empty response)"
: generation.getOutput().getText()));
if (response != null) {
Usage usage = response.getMetadata().getUsage();
System.out.println("usage : " + usage.getPromptTokens() + " prompt + "
+ usage.getCompletionTokens() + " completion tokens"
+ " (provider-reported, final call)");
}
System.out.println("budget: " + budget.summary());
System.out.println("note : the advisor owns the loop, so the model-call ceiling"
+ " is enforced as the tool-call limit the manager carries;"
+ " our own counter never sees those calls, which is why it reads zero");
System.out.println();
System.out.print("reply, or blank to return to the menu: ");
turn = context.console().hasNextLine() ? context.console().nextLine().strip() : "";
System.out.println();
}
}
}
The complete SpringAgentCommand.java, collapsed because it runs to 181 lines
package com.graphqlguy.moviedb.agents.command;
import com.graphqlguy.moviedb.agents.auth.AuthSession;
import com.graphqlguy.moviedb.agents.safety.ApprovalGate;
import com.graphqlguy.moviedb.agents.safety.RunBudget;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.toolsearch.ToolSearchToolCallingAdvisor;
import org.springframework.ai.chat.client.advisor.ToolCallingAdvisor;
import org.springframework.ai.model.tool.DefaultToolCallingManager;
import org.springframework.ai.model.tool.ToolCallingManager;
import org.springframework.ai.chat.memory.ChatMemory;
import org.springframework.ai.chat.memory.MessageWindowChatMemory;
import org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor;
import org.springframework.ai.chat.metadata.Usage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.toolsearch.ToolIndex;
import org.springframework.ai.tool.toolsearch.index.lucene.LuceneToolIndex;
import org.springframework.ai.tool.toolsearch.index.regex.RegexToolIndex;
import org.springframework.ai.tool.toolsearch.index.vectorstore.VectorToolIndex;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.vectorstore.SimpleVectorStore;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.stereotype.Component;
import java.util.List;
/**
* Class 7: the same concierge with Spring AI's ChatClient owning the loop. The
* callbacks work without a bridge, because they already implement Spring AI's own
* ToolCallback interface, which is what Class 2 built them against.
*/
@Component
public class SpringAgentCommand implements AgentCommand {
protected static final String SYSTEM = """
You are a concierge for the Movie Database. Use the tools to answer
and to act; never invent ids, and never claim an action succeeded
without a tool result proving it. Ids are numeric strings like "2";
a name is never an id. Before any write, first call the read tools
to obtain every id the write needs.""";
/**
* Added to the system prompt only when tool search is on. The advisor already
* appends one sentence of its own saying that {@code toolSearchTool} exists, and a
* small model reads that sentence and still answers "the available tools do not
* include..." after a single search. These three rules make the search insistent.
*/
protected static final String SEARCH_RULES = """
Most of the catalog is hidden until you search for it. When you cannot see
a tool for what was asked, call toolSearchTool with the words of the
request. If the tools it returns fit badly, search again with different
words, one search per missing step. Report that something is impossible
only after two searches have come back with nothing usable.""";
private final ObjectProvider<ChatModel> chatModels;
private final ObjectProvider<EmbeddingModel> embeddingModels;
public SpringAgentCommand(ObjectProvider<ChatModel> chatModels,
ObjectProvider<EmbeddingModel> embeddingModels) {
this.chatModels = chatModels;
this.embeddingModels = embeddingModels;
}
@Override
public String name() {
return "spring-agent";
}
@Override
public int order() {
return 70;
}
@Override
public String summary() {
return "the ChatClient with the advisor-owned loop";
}
@Override
public String prompt() {
return "task";
}
@Override
public void run(List<String> args, CommandContext context) {
String task = String.join(" ", args);
ChatModel springModel = chatModels.getIfAvailable();
if (task.isBlank() || springModel == null) {
System.out.println(task.isBlank() ? "a task is needed"
: "No chat model configured.");
return;
}
String role = context.role();
AuthSession auth = context.login();
RunBudget budget = context.budget();
ApprovalGate gate = new ApprovalGate(context.console());
List<ToolCallback> callbacks = context.callbacks(context.tools(role), auth, gate, budget);
boolean toolSearch = context.env().getProperty("agents.tool-search", Boolean.class, false);
String indexType = context.env().getProperty("agents.tool-search-index", "regex");
System.out.println("framework : Spring AI (ChatClient, advisor-owned loop)");
System.out.println("role : " + role + " (" + callbacks.size() + " tools)");
System.out.println("tool search: " + (toolSearch
? "tool search on, " + indexType + " index, definitions sent on demand"
: "off, every definition sent on every call"));
System.out.println("task : " + task);
System.out.println();
// Memory is an advisor here, which is the framework being consistent with
// itself: the loop belongs to the advisor chain, so what the loop remembers
// belongs there too. One conversation id covers the whole exchange.
// The loop belongs to the advisor, so the model-call ceiling has to belong
// there too. A ToolCallingManager built with the run's own limits bounds how
// far the advisor will go: each further model call needs another tool call to
// justify it, so capping tool calls caps the loop.
ToolCallingManager boundedManager = DefaultToolCallingManager.builder()
.maxTotalToolCalls(budget.maxToolCalls())
.build();
ChatMemory memory = MessageWindowChatMemory.builder().build();
String conversationId = "spring-agent-" + System.nanoTime();
ChatClient client = ChatClient.create(springModel);
String turn = task;
while (!turn.isBlank()) {
var spec = client.prompt()
.system(toolSearch ? SYSTEM + SEARCH_RULES : SYSTEM)
.user(turn)
// tools(Object...) replaces toolCallbacks(), deprecated in Spring AI 2.0
.tools(callbacks.toArray())
.options(OllamaChatOptions.builder().model(context.modelName()))
.advisors(MessageChatMemoryAdvisor.builder(memory).build())
.advisors(advisorParams -> advisorParams.param(ChatMemory.CONVERSATION_ID, conversationId));
// Tool search, off unless asked for. Spring AI's name for sending tool
// definitions on demand instead of all of them up front. The catalog is still
// handed over in full; the advisor decides how much of it the model sees,
// and when.
// Exactly one tool advisor may sit in the chain, and it owns the loop, so
// whichever one this run uses is also where the ceiling has to go.
if (toolSearch) {
ToolIndex index = indexFor(indexType);
spec = spec.advisors(ToolSearchToolCallingAdvisor.builder()
.toolIndex(index)
.maxResults(5)
.toolCallingManager(boundedManager)
.build());
} else {
spec = spec.advisors(ToolCallingAdvisor.builder()
.toolCallingManager(boundedManager)
.build());
}
ChatResponse response = spec.call().chatResponse();
Generation generation = response == null ? null : response.getResult();
System.out.println("answer: " + (generation == null
? "(the model returned an empty response)"
: generation.getOutput().getText()));
if (response != null) {
Usage usage = response.getMetadata().getUsage();
System.out.println("usage : " + usage.getPromptTokens() + " prompt + "
+ usage.getCompletionTokens() + " completion tokens"
+ " (provider-reported, final call)");
}
System.out.println("budget: " + budget.summary());
System.out.println("note : the advisor owns the loop, so the model-call ceiling"
+ " is enforced as the tool-call limit the manager carries;"
+ " our own counter never sees those calls, which is why it reads zero");
System.out.println();
System.out.print("reply, or blank to return to the menu: ");
turn = context.console().hasNextLine() ? context.console().nextLine().strip() : "";
System.out.println();
}
}
/**
* The three indexes the advisor can rank with. The vector index needs an embedding
* model, so an unreachable Ollama makes it unavailable, and this reports that. A
* silent fall back to regex would leave the run measuring the wrong backend.
*/
private ToolIndex indexFor(String indexType) {
return switch (indexType) {
case "lucene" -> new LuceneToolIndex();
case "vector" -> {
EmbeddingModel embeddings = embeddingModels.getIfAvailable();
if (embeddings == null) {
throw new IllegalStateException("agents.tool-search-index is vector, "
+ "which needs an embedding model; is Ollama running with "
+ "the model named by spring.ai.ollama.embedding.model?");
}
yield new VectorToolIndex(SimpleVectorStore.builder(embeddings).build());
}
default -> new RegexToolIndex();
};
}
}
ChatClient.create(...) takes any ChatModel bean. To run the same code against a hosted provider, we change which model Boot auto-configures. The request is built step by step:
prompt()opens a fluent request spec.systemanduseradd the two messages.toolCallbackshands over the catalog.call()is a terminal that runs the whole exchange and blocks until it ends.content()reads the text of the result. Class 8 usesentity(...)on the same terminal to get a typed object.
Which Controls Survive a Framework Change
When a framework runs the loop, the safety controls of this course divide into two groups. That split is the main point of the whole track.
Controls inside the callback keep working unchanged. The login, the approval gate and the tool-call ceiling all live in GraphQlToolCallback. Every framework calls it, so they behave the same here as in Class 5 and Class 6. The approval prompt is identical whichever framework runs the loop.
Controls that lived in the loop have to move. The model-call ceiling in Class 4 was a condition in a while loop that we wrote. Here the framework runs the loop. Our counter cannot see the framework's model calls, which is why budget: reports zero of them. So we write the ceiling again, in the framework's own terms:
// built once, before the conversation starts
ToolCallingManager boundedManager = DefaultToolCallingManager.builder()
.maxTotalToolCalls(budget.maxToolCalls())
.build();
// and handed to whichever tool advisor this run uses, further down the method
spec = spec.advisors(ToolCallingAdvisor.builder()
.toolCallingManager(boundedManager)
.build());
Both lines are in the same method. Turning It On below shows them in place, because the advisor that receives the manager depends on whether tool search is on.
The loop makes another model call only after a tool call, so a limit on tool calls also limits the loop. With a ceiling of one, the advisor stops and tells the model why:
answer: Total tool call limit (1) exceeded for this turn. No further tool calls are allowed.
Class 5 does the same thing with maxToolCallingRoundTrips, and Class 6 reads the ceiling in a routing edge. So one ceiling has a different place in each of the three frameworks. The rule to remember is this: a control in the execution layer survives a change of framework, and a control in the loop has to be rewritten.
Where the Loop Went: The Advisor Chain
One part of the Spring AI 2.0 architecture matters most here, and the snippet above does not show it: ChatModel.call() does not run any tools. In 1.x, the model implementations ran the tool loop internally. Version 2.0 moved the loop out into the advisor chain. The remaining lessons plug into that chain, so we look at it in detail.
An advisor is an interceptor around one ChatClient exchange, in the same sense as a servlet filter or a Spring HandlerInterceptor. It sees the request on the way down and may modify it. It sees the response on the way back, and may act on it, or call the rest of the chain again.
Advisors run in a defined order, and Spring AI registers some of them for you. The one that matters here is ToolCallingAdvisor, which owns the loop. On each round it does four things:
- sends the conversation to the model
- finds the tool calls in the response
- runs them through your callbacks and appends the results
- repeats until the model answers with text
A chain may hold only one tool advisor. That rule guarantees that the loop has a single owner.
The hand-written loop in Class 4 is a simplified ToolCallingAdvisor. The framework's version adds ordering, streaming support and the single-owner rule. In practice the split is clear. Behaviour around every tool call in this one framework goes into an advisor. Behaviour around every tool call in any framework stays in the callback.
This architecture answers the question that the track has asked three times: where do controls owned by the loop live now? In Spring AI, they live in the advisor. The documented way to build a production approval gate is to subclass ToolCallingAdvisor and override its hooks. doBeforeCall is the natural place for a gate. The framework also enforces safety ceilings of 40 calls per tool and 150 per turn. Those ceilings arrived in 2.0.1, the version of the BOM our project pins. They are far above the budgets from Class 4, and both sides chose that gap on purpose. The framework's numbers stop a runaway loop, and your numbers describe a reasonable run.
Controls owned by the execution stay where they have been since Class 4: in the callback. That is why the transcript in this lesson shows the same approval prompt and the same budget line, without any new safety code. For the third framework in a row, the receipt reads "0 model calls" next to a real tool count. That is how a loop owned by the framework looks from the outside.
The Three Bands, One Last Time
The same three bands as Classes 5 and 6:
In the three diagrams from Classes 5, 6 and 7, the middle band stays the same. That is the portability claim in one picture.
Spring AI also puts memory and logging in advisors. MessageChatMemoryAdvisor does the job that MessageWindowChatMemory did in Class 5, and the same advisor mechanism covers memory, logging and the tool loop. One more detail is worth knowing. Custom options(...) on a prompt replace the configured defaults completely, so the model name has to be set there again. We ran into this while building these runs. The error reads model cannot be null or empty. It looks like a configuration problem, and the real cause is an options override that discarded the configured default.
The Run
If you start the application without arguments now, the menu lists all three frameworks. Each lesson of the track added its command to the same build, and your branch contains all of them:
3 agent run the agent loop: login, approval gate, and run budget
5 lc4j-agent the same task through LangChain4j's AiServices
6 graph-agent the loop as a LangGraph4j state machine, gated by an interrupt
7 spring-agent the ChatClient with the advisor-owned loop
Give the same task to agent, lc4j-agent, graph-agent and spring-agent in turn. The differences you see come from the frameworks, because everything underneath is shared:
- one generated catalog
- one
GraphQlToolCallback, with its login, gate and budget - one Movie Database service
The approval prompt looks identical in all four, and each time a different loop calls it. That shows the argument of this track in one build.
Three agent frameworks on one classpath make sense for a course. A real application picks one. They sit together here so that we can compare them.
Here is a task of the same kind, run by the fourth execution engine, with the output unedited:
framework : Spring AI (ChatClient, advisor-owned loop)
role : concierge (7 tools)
tool search: off, every definition sent on every call
task : add the movie with id 4 to my To watch this weekend watchlist, look up the list id first
APPROVAL REQUIRED: the agent wants to run a write.
tool : addWatchListItem
operation : mutation Tool_addWatchListItem($input: AddWatchListItemInput!) { ... }
arguments : {"input":{"titleId":"4","titleType":"MOVIE","watchListId":"1"}}
Execute this mutation? [y/N] approved by the human.
answer: The movie with ID 4 has been successfully added to your "To watch this
weekend" watchlist!
usage : 2,618 prompt + 802 completion tokens (provider-reported, final call)
budget: 0 model calls (ceiling 8), 2 tool calls (ceiling 6)
This time the model nested the arguments correctly. That is worth noticing, because the run in Class 4 flattened them, and the first attempt in Class 6 sent a name where an id belonged. The same model and the same schema produced three different shapes in three runs. The argument repair from Class 4 sits in the callback, and so it runs under every framework. It repairs the flattened case when it appears, and it leaves correct arguments alone. A control at the execution layer keeps working without extra attention.
The usage line is new. The ChatResponse metadata carries the provider's own token count. On providers that support it, the count also lists prompt-cache reads and writes. Class 10 turns these numbers into money.
When the Catalog Is the Problem
This run advertised a small catalog, because the allow-list from Class 2 is small. Seven operations passed curation for the concierge role, and the catalog command prices the whole catalog at 804 tokens. How much to allow is a product decision, though. A team whose agents do customer support needs more operations than one concierge screen does. Our project has a file for that case, tool-allowlist-full.txt. It holds every Movie Database operation that a support agent could reasonably need.
The wider allow-list is a second file beside the one from Class 2. We curated it by the same rules, for a team that gives agents access to the whole product:
# The same curation decision as tool-allowlist.txt, taken by a team that ships the
# whole product surface to agents instead of one narrow concierge. Every operation
# here is still reviewed by a human, and every write still passes the Class 4
# approval gate; the only thing that changed is how many operations survived
# curation. Class 7's section on tool search uses this file to show
# what a larger catalog costs and what Spring AI's tool search does about it.
#
# Point the app at it by changing agents.allow-list in application.yaml to this
# file, then rebuilding: the yaml travels inside the jar.
# Reads: the product's whole lookup surface.
movie
movies
person
search
country
tmdbSearch
tvShow
myWatchLists
publicWatchLists
watchList
# Writes a support agent may perform on a signed-in user's behalf. Each one still
# stops at the approval gate before it executes.
createReview
deleteReview
createWatchList
updateWatchList
deleteWatchList
addWatchListItem
removeWatchListItem
markWatchListItemWatched
# Account administration: never for the concierge or support roles.
deletePerson: admin
register: admin
The role suffixes work as they did in Class 2, so the concierge role still sees a subset of this file. The catalog command shows the price of the wider catalog as soon as the application uses the file.
Point the application at that file by changing one line in application.yaml:
allow-list: src/main/resources/tool-allowlist-full.txt
mvn clean spring-boot:run
Choose catalog and answer concierge. The command prices the wider catalog:
role : concierge
tools : 18
total cost : 1400 tokens to advertise this catalog (O200K_BASE)
Eighteen operations and 1,400 tokens are both manageable, and the way the cost grows matters more than the figure. Every definition goes out on every model call. So a catalog is a fixed charge, multiplied by the number of calls in a run, and the run pays it whether or not the model uses a tool. Spring's own documentation describes where that leads. A setup that collects fifty or more tools from Slack, GitHub and Jira servers often spends more than 55,000 tokens on definitions before the user asks anything. Models also choose badly among thirty tools with similar names, even when the budget has room for them.
Tool Search: Definitions On Demand
Spring AI 2.0 ships an answer to this, a second ToolCallingAdvisor. Three names are used for it, so we define them first:
- Tool search is what Spring AI calls the feature. The Claude API uses the same name for its equivalent.
- Progressive disclosure is the older interface-design idea behind it: show what is needed when it is needed. This course uses that phrase for the idea, and Spring AI's name for the feature.
agents.tool-searchis the flag inapplication.yamlthat turns Spring AI's version on.
ToolSearchToolCallingAdvisor implements tool search. The advisor indexes the tool catalog once per session. It sends the model a single built-in tool called toolSearchTool in place of the definitions. It adds the full definition of a tool only after the model has searched for that tool and found it.
It subclasses the advisor that owns the loop, so it takes over the loop. The rule from the previous section, one tool advisor per chain, means that it replaces the standard advisor. Everything below the loop stays as it was. The callbacks are the same objects, so the approval gate, the budget and the login behave exactly as before.
Turning It On
Turning it on in our project takes two dependencies, a property and a short block of wiring. The first dependency brings the advisor, the three index implementations and Apache Lucene. The second brings SimpleVectorStore. The vector index needs it, and the wiring below creates it:
<!-- Spring AI's progressive tool disclosure: the ToolSearchToolCallingAdvisor
and its three tool indexes. Brings Lucene with it. -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-starter-tool-search-advisor</artifactId>
</dependency>
<!-- SimpleVectorStore, so the tool-search benchmark can measure the vector
index beside the regex and Lucene ones. Embeddings come from Ollama. -->
<dependency>
<groupId>org.springframework.ai</groupId>
<artifactId>spring-ai-vector-store</artifactId>
</dependency>
The vector index starts an embedding client built on Netty, and Netty loads a native library when it starts. The JVM prints a restricted-method warning for that library unless native access is granted. The warning reaches the console before any logger exists, so logback-spring.xml cannot hide it. The Boot plugin can:
<configuration>
<jvmArguments>--enable-native-access=ALL-UNNAMED</jvmArguments>
</configuration>
The wiring reads two properties. Both have defaults in the configuration, and the defaults keep the runs of every earlier lesson reproducible:
# Tool search, off by default so every earlier class's run
# reproduces unchanged. Set it to true, rebuild, and the catalog goes to Spring
# AI's ToolSearchToolCallingAdvisor instead of being advertised in full.
# The index decides how the model finds a tool by name and description. Three
# values work: regex, the framework's own default, matches substrings and
# therefore reads inside names like addWatchListItem; lucene scores with BM25
# but treats each such name as a single token; vector embeds names and
# descriptions with the model named below, so it matches on meaning and needs
# Ollama to be reachable. Class 7 measures the difference.
tool-search: false
tool-search-index: regex
The vector index needs one more line, at the top level of the same file, because it embeds each tool's name and description before it can rank them:
spring.ai.ollama.base-url: http://localhost:11434
# The embedding model the vector index and the tool-search benchmark use.
# Pull it once with: ollama pull nomic-embed-text
spring.ai.ollama.embedding.model: nomic-embed-text
Spring AI 2.0 moved these keys up one level. The old key, spring.ai.ollama.embedding.options.model, still works, and it logs a deprecation warning on startup. The form above is the current one.
// Exactly one tool advisor may sit in the chain, and it owns the loop, so
// whichever one this run uses is also where the ceiling has to go.
if (toolSearch) {
spec = spec.advisors(ToolSearchToolCallingAdvisor.builder()
.toolIndex(indexFor(indexType))
.maxResults(5)
.toolCallingManager(boundedManager)
.build());
} else {
spec = spec.advisors(ToolCallingAdvisor.builder()
.toolCallingManager(boundedManager)
.build());
}
indexFor turns the property into one of the three implementations. The vector index needs an embedding model. If Ollama cannot be reached, the vector index is unavailable, so this code reports the problem when it happens:
private ToolIndex indexFor(String indexType) {
return switch (indexType) {
case "lucene" -> new LuceneToolIndex();
case "vector" -> {
EmbeddingModel embeddings = embeddingModels.getIfAvailable();
if (embeddings == null) {
throw new IllegalStateException("agents.tool-search-index is vector, "
+ "which needs an embedding model; is Ollama running with "
+ "the model named by spring.ai.ollama.embedding.model?");
}
yield new VectorToolIndex(SimpleVectorStore.builder(embeddings).build());
}
default -> new RegexToolIndex();
};
}
The default branch is deliberate. An unknown value ranks with regex, which is the framework's own default. Each of the three names selects the index it names.
What It Costs, and What It Saves
Tool search is one flag, agents.tool-search. When it is off, every tool definition in the catalog goes into every request to the model. When it is on, the advisor advertises a single search tool, and definitions arrive only when the model asks for them.
Measuring the difference takes three steps, done twice. Set both lines in application.yaml, starting with tool search off:
allow-list: src/main/resources/tool-allowlist-full.txt
tool-search: false # then true for the second run
mvn clean spring-boot:run
Choose spring-agent from the menu, type list my watchlists as the task, press Enter at the reply prompt, then type q. Now set tool-search: true, repeat the two commands, and give it the same task.
The number to read is the usage : line, which reports what the provider counted for the final model call. Here are three runs of each, on qwen3:8b through Ollama, against the 18-tool catalog:
| Run | tool search off | tool search on |
|---|---|---|
| 1 | 3,422 prompt tokens | 2,015 |
| 2 | 3,422 | 877 |
| 3 | 3,422 | 1,959 |
The off column repeats exactly, because that path does the same work on every run: the same 18 definitions go into the same request. The on column varies, because the model sometimes searches once and sometimes twice before it finds what it needs. Each search adds a round to the conversation, and the final call carries the whole conversation.
Those savings have a price, and the tool-search command reports it:
role : concierge (18 tools)
catalog : 1400 tokens if every definition is advertised up front
overhead : 293 tokens the advisor adds before it saves anything
(system-message suffix + the toolSearchTool definition)
So a single call saves roughly 1,400 to 2,500 prompt tokens, and the search costs 293. Your own numbers will differ, because they depend on the model, the task and the number of searches. Look for the pattern: a fixed cost that is smaller than the saving once the catalog is large enough.
Why the saving is in tokens. Tool definitions go into every request to the model, so a run of four model calls pays for the catalog four times. Tool search changes what is in the prompt, and the prompt is where tokens are spent.
This also explains what tool search does for MCP tools. They arrive as ordinary ToolCallback instances through an MCP tool callback provider. Fetching them with tools/list is a call from client to server, and it runs without the model. That call costs a round trip and some latency, and zero tokens. Tokens are spent when the client puts those definitions in front of the model, so the advisor saves on MCP tools exactly as it does here. It indexes both kinds of tool together, because it reads only a tool's name and description.
Which Index Ranks the Catalog
The advisor delegates ranking to a ToolIndex, and three implementations ship:
RegexToolIndexmatches substrings, and the auto-configuration uses it by default.LuceneToolIndexranks with BM25.VectorToolIndexembeds names and descriptions, so it matches on meaning. It needs aVectorStorebean.
Choosing between the first two looks like a configuration detail. Our tool-search command shows what the choice does to your catalog. It indexes the catalog the way the advisor does and prints what each backend ranks, without a model:
The complete ToolSearchCommand.java, collapsed because it runs to 101 lines
package com.graphqlguy.moviedb.agents.command;
import com.graphqlguy.moviedb.agents.toolgen.OperationTool;
import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.api.Encoding;
import com.knuddels.jtokkit.api.EncodingType;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.definition.ToolDefinition;
import org.springframework.ai.tool.method.MethodToolCallbackProvider;
import org.springframework.ai.tool.toolsearch.ToolIndex;
import org.springframework.ai.tool.toolsearch.ToolReference;
import org.springframework.ai.tool.toolsearch.ToolSearchRequest;
import org.springframework.ai.tool.toolsearch.ToolSearchTool;
import org.springframework.ai.tool.toolsearch.index.lucene.LuceneToolIndex;
import org.springframework.ai.tool.toolsearch.index.regex.RegexToolIndex;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.stereotype.Component;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
/**
* Class 7: the retrieval layer behind Spring AI's tool search, with the model taken
* out of the picture. It indexes the catalog exactly as the advisor does, then shows
* what each backend ranks for one query, and what tool search costs before it saves
* anything.
*/
@Component
public class ToolSearchCommand implements AgentCommand {
@Override
public String name() {
return "tool-search";
}
@Override
public int order() {
return 72;
}
@Override
public String summary() {
return "rank the catalog with both tool indexes, and price what tool search costs";
}
@Override
public String prompt() {
return "natural-language query, the way a model would phrase a need, e.g. add a movie to my list";
}
@Override
public void run(List<String> args, CommandContext context) throws Exception {
String query = String.join(" ", args);
if (query.isBlank()) {
System.out.println("tool-search needs a query, e.g. add a movie to my list");
return;
}
String role = context.role();
List<OperationTool> tools = context.tools(role);
// What the advisor indexes is exactly this: a tool's name and its description.
// The advisor leaves out the input schema, which is most of a definition's cost.
List<ToolReference> references = tools.stream()
.map(tool -> ToolReference.builder()
.toolName(tool.name()).summary(tool.description()).build())
.toList();
Encoding encoding = Encodings.newDefaultEncodingRegistry().getEncoding(EncodingType.O200K_BASE);
int catalogTokens = tools.stream().mapToInt(OperationTool::tokenCount).sum();
// What tool search costs before it saves anything: the advisor appends a suffix
// to the system message and advertises one tool of its own.
String suffix = new DefaultResourceLoader()
.getResource("classpath:/DEFAULT_SYSTEM_PROMPT_SUFFIX.md")
.getContentAsString(StandardCharsets.UTF_8);
ToolCallback searchTool = MethodToolCallbackProvider.builder()
.toolObjects(new ToolSearchTool(new RegexToolIndex(), 5))
.build().getToolCallbacks()[0];
ToolDefinition searchDef = searchTool.getToolDefinition();
int overhead = encoding.countTokens(suffix)
+ encoding.countTokens(searchDef.name() + " " + searchDef.description()
+ " " + searchDef.inputSchema());
System.out.println("role : " + role + " (" + tools.size() + " tools)");
System.out.println("query : " + query);
System.out.println("note : this command indexes and ranks on its own,"
+ " whatever agents.tool-search is set to");
System.out.println("catalog : " + catalogTokens
+ " tokens if every definition is advertised up front");
System.out.println("overhead : " + overhead
+ " tokens tool search would add to a run before it saves anything"
+ " (system-message suffix + the toolSearchTool definition)");
System.out.println();
Map<String, ToolIndex> indexes = new LinkedHashMap<>();
indexes.put("regex (the auto-configured default)", new RegexToolIndex());
indexes.put("lucene (BM25, StandardAnalyzer)", new LuceneToolIndex());
for (Map.Entry<String, ToolIndex> entry : indexes.entrySet()) {
ToolIndex index = entry.getValue();
index.indexTools("probe", references);
List<ToolReference> hits = index
.search(new ToolSearchRequest("probe", query, 5, null))
.toolReferences();
System.out.println(entry.getKey() + " -> "
+ (hits.isEmpty() ? "no tool matched" : hits.size() + " hits"));
for (ToolReference hit : hits) {
System.out.printf(" %-24s score %.3f%n", hit.toolName(), hit.relevanceScore());
}
System.out.println();
}
}
}
mvn clean spring-boot:run
Keep the wider allow-list configured and choose tool-search. Then ask the way a model would phrase a need, with add a movie to my list:
role : concierge (18 tools)
query : add a movie to my list
note : this command indexes and ranks on its own, whatever agents.tool-search is set to
catalog : 1400 tokens if every definition is advertised up front
overhead : 293 tokens tool search would add to a run before it saves anything
(system-message suffix + the toolSearchTool definition)
regex (the auto-configured default) -> 5 hits
addWatchListItem score 9.500
movie score 6.250
watchList score 5.917
movies score 5.179
search score 4.098
lucene (BM25, StandardAnalyzer) -> 5 hits
movie score 3.136
addWatchListItem score 2.461
removeWatchListItem score 0.981
deleteWatchList score 0.958
createWatchList score 0.936
Read the first line of each list. The request asks to add something, and only one backend puts the tool that adds things first. BM25 ranks movie above addWatchListItem. A model that receives those five tools in that order has to look past the top result to find the tool that does the job.
The cause is a tokenizer rule that we met in Class 3 for a different reason. LuceneToolIndex builds a StandardAnalyzer, which follows the Unicode word-boundary rules. So addWatchListItem is indexed as the single token addwatchlistitem, and the words add and list in the query cannot match it. The name of the movie tool, on the other hand, is exactly one of the query's words. LuceneToolIndex weights a name match two to four times higher than a description match. So movie ranks first on its name. addWatchListItem scores only through its description, which does contain Add a movie or TV show to one of your lists.
RegexToolIndex builds a case-insensitive alternation of the query's words and runs it against the raw text, so a match can fall in the middle of an identifier. addWatchListItem matches add at its start and list in its middle, which is why it ranks first there.
That result does not make regex the better ranker. It scores by counting matches and adding a small bonus for an early position, so it cannot tell that "list" is a common word and "repository" a rare one. Its substring matching also produces wrong hits. A query about pull requests matches addWatchListItem, because WatchList contains List. BM25 weights rare words the way we want, and it cannot look inside a name like addWatchListItem. Neither backend reduces words to their stem, so to both of them movies in a question and a tool named movie are different words.
The output also shows these numbers when tool search is off, and the reason matters. This command builds its own indexes and prices what the advisor would cost. It ignores agents.tool-search, and it does not run an agent. With the flag off, a run does not create a ToolSearchToolCallingAdvisor. The 293 tokens are what turning the flag on would cost.
The advice is the same for every index, and Class 3 reached it from the other direction. Search runs over the description, so write the description for the question a user will ask. The generator from Class 2 already refuses to emit a tool whose schema field lacks a description of what it does, and that rule now has a second reason. If a description only repeats the tool's name, the indexes can match only the name, and the model does not receive the definition it needed.
Watching the Advisor Work
So far we measured the separate pieces. To watch the advisor itself, we add one logging line, because the advisor works between the model and the tools, and that work does not appear on the console.
logging:
level:
root: warn
org.springframework.ai: debug
Set tool-search: true, run spring-agent, and give it list my watchlists. The console output stays the same, because the logback-spring.xml from Class 2 sends logs to a file. Read target/agents.log:
RegexToolIndex : Added 18 tools to session 'spring-agent-...'
DefaultToolCallingManager : Executing tool call: toolSearchTool
RegexToolIndex : Converted query 'manage watchlists' to ...
DefaultToolCallingManager : Executing tool call: myWatchLists
Those four lines show the whole mechanism:
- The advisor indexes all eighteen tools for this session.
- The model receives one tool in place of eighteen. It calls
toolSearchToolwith its own wording of what it needs,manage watchlists, which differs from the words the user typed. - The index answers, and only then does the model call the real tool.
The eighteen definitions stayed out of the prompt. That is why the run used 1,605 prompt tokens, against 3,422 with the feature off.
Notice the change of wording. The benchmark below cannot capture it, because the benchmark uses the same queries on every run. In a live run the model writes its own query, and it may search more than once.
Set the level back to warn afterwards, or every run leaves a busy log behind.
Measuring the Three Backends
One ranked query shows how a backend behaves. To choose a backend we need numbers over several queries, and two numbers matter: what each backend puts in the prompt, and whether it found every tool the task needs. A task like add matrix to my watchlist needs three tools:
search, to find the filmmyWatchLists, to find the listaddWatchListItem, to write the entry
An index that returns only two of them cheaply has not helped, because the agent cannot finish.
The tool-bench command reads a file of labelled queries, each line a question and the tools it needs:
# Queries for the tool-search benchmark, one per line: the question a user would
# ask, a tab, then every tool the agent needs to answer it. A task that needs
# three tools is the interesting case: an index that surfaces two of them leaves
# the agent stuck, however few tokens it saved.
#
# Every query must be answerable from the seeded catalog. An earlier version asked
# who directed Blade Runner, a film missing from every catalog here, so every backend
# scored badly because the answer was missing from the catalog.
add The Matrix to my To watch this weekend list search myWatchLists addWatchListItem
what is the plot of The Matrix search movie
which of my lists has the most items myWatchLists
what is in my To watch this weekend list myWatchLists watchList
who directed Inception search person
Every query has to be answerable from the seeded database. An earlier version of this file asked who directed Blade Runner, a film that the seed data leaves out. Every backend scored badly on it, for a reason unrelated to retrieval. We replaced it with Inception, after checking that the film is present and that the person tool reaches its director.
Point allow-list at the wider file, then run it and choose tool-bench.
The complete ToolBenchCommand.java, collapsed because it runs to 166 lines
package com.graphqlguy.moviedb.agents.command;
import com.graphqlguy.moviedb.agents.toolgen.OperationTool;
import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.api.Encoding;
import com.knuddels.jtokkit.api.EncodingType;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.definition.ToolDefinition;
import org.springframework.ai.tool.method.MethodToolCallbackProvider;
import org.springframework.ai.tool.toolsearch.ToolIndex;
import org.springframework.ai.tool.toolsearch.ToolReference;
import org.springframework.ai.tool.toolsearch.ToolSearchRequest;
import org.springframework.ai.tool.toolsearch.ToolSearchTool;
import org.springframework.ai.tool.toolsearch.index.lucene.LuceneToolIndex;
import org.springframework.ai.tool.toolsearch.index.regex.RegexToolIndex;
import org.springframework.ai.tool.toolsearch.index.vectorstore.VectorToolIndex;
import org.springframework.ai.vectorstore.SimpleVectorStore;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.stereotype.Component;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
/**
* Class 7: what tool search costs and what it misses, measured over a file of
* labelled queries. For each query the benchmark counts the tool-definition tokens
* each backend would put in the prompt, and checks whether every tool the task
* needs came back. Cheap retrieval that leaves out one required tool is not cheap:
* the agent cannot finish the task.
* <p>
* No model runs here. What varies between backends is which definitions reach the
* prompt, so that is what this counts, and counting it makes the numbers repeatable.
*/
@Component
public class ToolBenchCommand implements AgentCommand {
private static final int DEFAULT_TOP_K = 5;
private final ObjectProvider<EmbeddingModel> embeddingModels;
public ToolBenchCommand(ObjectProvider<EmbeddingModel> embeddingModels) {
this.embeddingModels = embeddingModels;
}
@Override
public String name() {
return "tool-bench";
}
@Override
public int order() {
return 73;
}
@Override
public String summary() {
return "measure what each tool-search backend costs, and what it fails to find";
}
@Override
public String prompt() {
return "definitions returned per search (blank for " + DEFAULT_TOP_K + ")";
}
@Override
public void run(List<String> args, CommandContext context) throws Exception {
int topK = args.isEmpty() || args.get(0).isBlank()
? DEFAULT_TOP_K : Integer.parseInt(args.get(0));
List<Query> queries = loadQueries();
String role = context.role();
List<OperationTool> tools = context.tools(role);
Map<String, Integer> costByTool = new LinkedHashMap<>();
List<ToolReference> references = new ArrayList<>();
for (OperationTool tool : tools) {
costByTool.put(tool.name(), tool.tokenCount());
references.add(ToolReference.builder()
.toolName(tool.name()).summary(tool.description()).build());
}
int catalogTokens = costByTool.values().stream().mapToInt(Integer::intValue).sum();
int overhead = searchOverhead();
System.out.println("catalog : " + tools.size() + " tools, " + catalogTokens
+ " tokens if every definition is advertised up front");
System.out.println("overhead : " + overhead + " tokens tool search adds to every request");
System.out.println("top-k : " + topK + " definitions returned per search");
System.out.println();
System.out.println("One search per query, without an agent or a second attempt. Each cell is the");
System.out.println("tool-definition tokens that search would put in the request, and how many");
System.out.println("of the tools the task needs came back in those " + topK + " results.");
System.out.println();
Map<String, ToolIndex> indexes = new LinkedHashMap<>();
indexes.put("regex", new RegexToolIndex());
indexes.put("lucene", new LuceneToolIndex());
EmbeddingModel embeddings = embeddingModels.getIfAvailable();
if (embeddings != null) {
indexes.put("vector", new VectorToolIndex(
SimpleVectorStore.builder(embeddings).build()));
} else {
System.out.println("(no embedding model available, so the vector index is skipped)");
}
for (Map.Entry<String, ToolIndex> entry : indexes.entrySet()) {
entry.getValue().indexTools(entry.getKey(), references);
}
System.out.printf("%-50s %8s", "query", "off");
for (String name : indexes.keySet()) {
System.out.printf(" %16s", name);
}
System.out.println();
Map<String, int[]> totals = new LinkedHashMap<>();
indexes.keySet().forEach(name -> totals.put(name, new int[] {0, 0, 0}));
for (Query query : queries) {
System.out.printf("%-50s %8d", trim(query.text(), 50), catalogTokens);
for (Map.Entry<String, ToolIndex> entry : indexes.entrySet()) {
List<ToolReference> hits = entry.getValue()
.search(new ToolSearchRequest(entry.getKey(), query.text(), topK, null))
.toolReferences();
Set<String> found = new LinkedHashSet<>();
int tokens = overhead;
for (ToolReference hit : hits) {
found.add(hit.toolName());
tokens += costByTool.getOrDefault(hit.toolName(), 0);
}
long needed = query.needs().stream().filter(found::contains).count();
int[] total = totals.get(entry.getKey());
total[0] += tokens;
total[1] += (int) needed;
total[2] += query.needs().size();
System.out.printf(" %8d %3d/%-3d", tokens, needed, query.needs().size());
}
System.out.println();
}
System.out.println();
System.out.printf("%-50s %8d", "mean tokens, and tools found of tools needed", catalogTokens);
for (Map.Entry<String, int[]> entry : totals.entrySet()) {
int[] total = entry.getValue();
System.out.printf(" %8d %3d/%-3d", total[0] / queries.size(), total[1], total[2]);
}
System.out.println();
}
/** What the advisor adds to every request before it saves anything. */
private int searchOverhead() throws Exception {
Encoding encoding = Encodings.newDefaultEncodingRegistry().getEncoding(EncodingType.O200K_BASE);
String suffix = new DefaultResourceLoader()
.getResource("classpath:/DEFAULT_SYSTEM_PROMPT_SUFFIX.md")
.getContentAsString(StandardCharsets.UTF_8);
ToolCallback searchTool = MethodToolCallbackProvider.builder()
.toolObjects(new ToolSearchTool(new RegexToolIndex(), DEFAULT_TOP_K))
.build().getToolCallbacks()[0];
ToolDefinition definition = searchTool.getToolDefinition();
return encoding.countTokens(suffix) + encoding.countTokens(
definition.name() + " " + definition.description() + " " + definition.inputSchema());
}
private List<Query> loadQueries() throws Exception {
List<Query> queries = new ArrayList<>();
for (String line : Files.readAllLines(Path.of("src/main/resources/tool-bench-queries.tsv"))) {
if (line.isBlank() || line.startsWith("#")) {
continue;
}
String[] parts = line.split("\t");
queries.add(new Query(parts[0], List.of(parts).subList(1, parts.length)));
}
return queries;
}
private static String trim(String text, int width) {
return text.length() <= width ? text : text.substring(0, width - 1) + "…";
}
private record Query(String text, List<String> needs) {
}
}
The table is easy to misread, so here is what it measures:
- One search per query, with the words above, without an agent and without a second attempt.
- Each cell is the number of tool-definition tokens that the search would put in the request. That is the 293-token overhead plus the definitions of the tools that came back. A definition is a tool's name, its description and its JSON Schema.
- The
offcolumn is all eighteen definitions, 1,400 tokens. A request carries that many when tool search is off.
Running it again gives the same result, and that is a useful property. Twelve consecutive runs produced byte-identical tables, because all three backends are deterministic:
- Regex matches text.
- Lucene scores an index that is built the same way each time.
- The embeddings behind the vector index are a deterministic computation. Asking Ollama three times for the same sentence returned the same vector, byte for byte.
So these numbers are exact, and a reader who runs the benchmark again should get the same table.
mvn clean spring-boot:run
catalog : 18 tools, 1400 tokens if every definition is advertised up front
overhead : 293 tokens tool search adds to every request
top-k : 5 definitions returned per search
query off regex lucene vector
add The Matrix to my To watch this weekend list 1400 597 2/3 955 1/3 735 2/3
what is the plot of The Matrix 1400 293 0/2 878 1/2 700 1/2
which of my lists has the most items 1400 545 1/1 885 1/1 725 0/1
what is in my To watch this weekend list 1400 533 2/2 784 1/2 520 2/2
who directed Inception 1400 293 0/2 293 0/2 658 0/2
mean tokens, and tools found of tools needed 1400 452 5/10 759 4/10 667 5/10
Every backend saves tokens, and the second number is the one to read. The five tasks need ten tools between them, and every backend missed at least half of them. Regex and the vector index found five, and Lucene found four. The three 293 cells are searches that came back empty, so the request carried only the overhead.
That is one search, in words we chose. A real agent writes its own query. The debug log above showed it rewrite list my watchlists as manage watchlists. It may also search again after a poor first result. So these numbers describe one attempt with the user's own wording. They are a useful starting point, and they leave questions open. The live matrix below puts the model back in and measures what really happens, including how often a second search happens.
A second search has a cost that this table leaves out: another model call, which sends the whole conversation again. So tool search trades fewer tokens per request against a chance of needing more requests.
How Many Definitions Come Back
We measured every number above at five definitions per search, the maxResults value that our wiring passes to the advisor. That ceiling is the most important number in the feature, because it decides how much of the catalog one search can reveal. The command takes it as an argument, so we can measure the same five queries again at any depth:
mvn clean spring-boot:run
Choose tool-bench, then type 10 at the prompt. Repeating that at a few depths shows the trade. The last column is the number of tokens that the vector index puts in the request to reach the recall next to it:
| definitions per search | regex | lucene | vector | vector's tokens |
|---|---|---|---|---|
| 3 | 3/10 | 4/10 | 4/10 | 482 |
| 5 | 5/10 | 4/10 | 5/10 | 667 |
| 8 | 5/10 | 6/10 | 7/10 | 1,050 |
| 10 | 5/10 | 6/10 | 9/10 | 1,226 |
| 12 | 5/10 | 7/10 | 9/10 | 1,340 |
The table shows three things.
Regex stops improving after five. At a ceiling of twelve it finds exactly what it found at five, because its two empty searches stay empty at any depth. A substring matcher finds a tool only when the query's words appear literally in the tool's name or description. what is the plot of The Matrix uses different words from our descriptions. A longer list helps only when the search found something.
The vector index gains the most from a longer list. It climbs from four tools at a ceiling of three to nine at a ceiling of ten. Ranking by meaning usually puts the right tool somewhere in the list, and the ceiling decides whether the returned part of the list is long enough to include it.
The cost of that recall decides the question. Finding nine of ten tools costs 1,226 tokens of definitions, against 1,400 for the entire catalog with the feature off. On this catalog, nearly complete retrieval costs almost as much as sending everything. The deciding factor is catalog size. Spring's guidance puts the threshold at roughly ten tools. Eighteen small tools are so close to that threshold that a higher ceiling soon costs as much as the whole catalog. On a catalog of two hundred tools, ten definitions are one twentieth of it, and the same recall would save a lot.
Telling the Model To Search
While measuring the runs above, we found something that looked like a mistake in our own wiring. With tool search on, the agent kept answering questions without the tool that it clearly had for them:
> which of my lists has the most items
The available tools do not include a function to compare the number of items in your watch lists.
The catalog has eighteen tools, and one of them is myWatchLists. The model had searched once and then stopped trying. The obvious explanation is that we did not tell the model that it could search. It sees a single tool called toolSearchTool, and our system prompt talks only about the Movie Database.
That explanation is wrong, and the framework's source code shows why. On every request, ToolSearchToolCallingAdvisor appends a paragraph to the system message we set:
return chatClientRequest.mutate()
.prompt(chatClientRequest.prompt()
.copy()
.augmentSystemMessage(systemMessage -> systemMessage.copy()
.mutate()
.text(systemMessage.getText() + this.systemMessageSuffix)
.build()))
.build();
The appended paragraph ships inside the jar as DEFAULT_SYSTEM_PROMPT_SUFFIX.md. It reads:
You have access to a special tool called `toolSearchTool` that allows you to
discover relevant tools on-demand. When you need to perform an action but don't
see an appropriate tool in your current context, use `toolSearchTool` to search
for it.
That paragraph and the toolSearchTool definition together make up the 293-token overhead that the benchmark reports. This tells us two things:
- The model is told how to search, in the framework's own words, so our wiring was correct.
- The advisor supplies that text itself, so the text appears exactly when the advisor does. Our own system prompt can stay the same in both configurations.
A small model may still use that instruction poorly. qwen3:8b reads the paragraph and searches once. It reads the five definitions that come back, and if they look wrong, it reports that the tool it needs does not exist. The paragraph allows the model to search, and it does not insist.
So we add our own, more insistent rules to the system prompt whenever tool search is on:
/**
* Added to the system prompt only when tool search is on. The advisor already
* appends one sentence of its own saying that {@code toolSearchTool} exists, and a
* small model reads that sentence and still answers "the available tools do not
* include..." after a single search. These three rules make the search insistent.
*/
protected static final String SEARCH_RULES = """
Most of the catalog is hidden until you search for it. When you cannot see
a tool for what was asked, call toolSearchTool with the words of the
request. If the tools it returns fit badly, search again with different
words, one search per missing step. Report that something is impossible
only after two searches have come back with nothing usable.""";
var spec = client.prompt()
.system(toolSearch ? SYSTEM + SEARCH_RULES : SYSTEM)
.user(turn)
The ternary keeps the two configurations comparable. With tool search off, the model receives all eighteen definitions and the request leaves the search tool out, so these rules would point at a tool that is missing. Spring offers a second place for the same words: ToolSearchToolCallingAdvisor.builder().systemMessageSuffix(...). That option is tidier, because the text then belongs to the advisor and appears only when the advisor does. It replaces the default paragraph above instead of adding to it. If you use it, you have to restate any of the framework's wording that you still want. We add our rules to the system prompt, and keep the framework's paragraph as it is.
The Same Tasks, With the Model Running
Every number so far came from the indexes alone, without the model. Those numbers show what a search costs. They leave the important question open: does the agent finish the job? The tool-live-bench command answers it. It runs the five tasks for real in each configuration and counts what the framework's loop does.
To count, we need a point that every round passes, and the ToolCallingManager is that point. Every round of the advisor's loop goes through it. So a manager that delegates to the real one can count each round as it passes.
/** Delegates every round to the real manager, and counts what goes past. */
private static final class Counting implements ToolCallingManager {
private final ToolCallingManager delegate;
private final Set<String> executed = new LinkedHashSet<>();
private int rounds;
private int searches;
private long tokens;
@Override
public ToolExecutionResult executeToolCalls(Prompt prompt, ChatResponse response) {
rounds++;
Usage usage = response.getMetadata().getUsage();
if (usage != null) {
tokens += usage.getPromptTokens() + usage.getCompletionTokens();
}
if (response.getResult() != null) {
response.getResult().getOutput().getToolCalls().forEach(call -> {
if (call.name().equals("toolSearchTool")) {
searches++;
} else {
executed.add(call.name());
}
});
}
return delegate.executeToolCalls(prompt, response);
}
}
It adds up the tokens of every model call in the run, including the intermediate ones, so a column's figure covers the whole task. The tool names it collects answer the completion question. A run counts as finished when it called every tool that the query file lists for that task.
The complete ToolLiveBenchCommand.java, collapsed because it runs to 235 lines
package com.graphqlguy.moviedb.agents.command;
import com.graphqlguy.moviedb.agents.auth.AuthSession;
import com.graphqlguy.moviedb.agents.safety.RunBudget;
import com.graphqlguy.moviedb.agents.toolgen.OperationTool;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.advisor.ToolCallingAdvisor;
import org.springframework.ai.chat.client.advisor.toolsearch.ToolSearchToolCallingAdvisor;
import org.springframework.ai.chat.memory.ChatMemory;
import org.springframework.ai.chat.metadata.Usage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.prompt.Prompt;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.model.tool.DefaultToolCallingManager;
import org.springframework.ai.model.tool.ToolCallingChatOptions;
import org.springframework.ai.model.tool.ToolCallingManager;
import org.springframework.ai.model.tool.ToolExecutionResult;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.ai.tool.definition.ToolDefinition;
import org.springframework.ai.tool.toolsearch.ToolIndex;
import org.springframework.ai.tool.toolsearch.index.lucene.LuceneToolIndex;
import org.springframework.ai.tool.toolsearch.index.regex.RegexToolIndex;
import org.springframework.ai.tool.toolsearch.index.vectorstore.VectorToolIndex;
import org.springframework.ai.vectorstore.SimpleVectorStore;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.stereotype.Component;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;
/**
* Class 7: the live counterpart to {@code tool-bench}. Real agent runs, one per
* query per backend, with the framework's loop instrumented so each run reports
* how many model calls it took, how many searches the model made, how many tokens
* the provider counted, and whether it ever reached every tool the task needs.
* <p>
* The instrument is a {@link ToolCallingManager} that delegates to the real one and
* counts on the way past. Every round of the framework's loop passes through it.
*/
@Component
public class ToolLiveBenchCommand implements AgentCommand {
private static final String SYSTEM = SpringAgentCommand.SYSTEM;
private static final int DEFAULT_TOP_K = 5;
private static final int MAX_GENERATED_TOKENS = 4096;
/**
* The seven columns of the matrix. The first four run the framework as it comes,
* where the only instruction to search is the one sentence the advisor appends
* for itself. The last three repeat the search backends with our own insistent
* rules added to the system prompt.
*/
private static final List<Column> COLUMNS = List.of(
new Column("off", "off", false),
new Column("regex", "regex", false),
new Column("lucene", "lucene", false),
new Column("vector", "vector", false),
new Column("regex+rules", "regex", true),
new Column("lucene+rules", "lucene", true),
new Column("vector+rules", "vector", true));
/** One column of the matrix: which index ranks the catalog, and how hard we push. */
private record Column(String label, String index, boolean insistent) {
}
private final ObjectProvider<ChatModel> chatModels;
private final ObjectProvider<EmbeddingModel> embeddingModels;
public ToolLiveBenchCommand(ObjectProvider<ChatModel> chatModels,
ObjectProvider<EmbeddingModel> embeddingModels) {
this.chatModels = chatModels;
this.embeddingModels = embeddingModels;
}
@Override
public String name() {
return "tool-live-bench";
}
@Override
public int order() {
return 74;
}
@Override
public String summary() {
return "run real agent tasks through each backend, counting rounds, searches and tokens";
}
@Override
public String prompt() {
return "runs, then definitions per search, then optional column names "
+ "(blank for 1 run, " + DEFAULT_TOP_K + " definitions, every column)";
}
@Override
public void run(List<String> args, CommandContext context) throws Exception {
int runs = args.isEmpty() || args.get(0).isBlank() ? 1 : Integer.parseInt(args.get(0));
int topK = args.size() > 1 && args.get(1).matches("\\d+")
? Integer.parseInt(args.get(1)) : DEFAULT_TOP_K;
int firstLabel = args.size() > 1 && args.get(1).matches("\\d+") ? 2 : 1;
List<String> wanted = args.size() > firstLabel ? args.subList(firstLabel, args.size()) : List.of();
List<Column> columns = wanted.isEmpty() ? COLUMNS
: COLUMNS.stream().filter(column -> wanted.contains(column.label())).toList();
if (columns.isEmpty()) {
System.out.println("No column matched " + wanted + "; the names are "
+ COLUMNS.stream().map(Column::label).toList());
return;
}
ChatModel model = chatModels.getIfAvailable();
if (model == null) {
System.out.println("No chat model configured.");
return;
}
List<Query> queries = loadQueries();
String role = context.role();
List<OperationTool> tools = context.tools(role);
AuthSession auth = context.login();
System.out.println("catalog : " + tools.size() + " tools");
System.out.println("runs : " + runs + " per query per column, "
+ (runs * queries.size() * columns.size()) + " agent runs in total");
System.out.println("model : " + context.modelName());
System.out.println("top-k : " + topK + " definitions returned per search");
System.out.println();
System.out.printf("%-14s %-40s %6s %8s %9s %10s %-34s %s%n",
"backend", "query", "rounds", "searches", "tokens", "needed", "tools actually called", "answer");
for (Column column : columns) {
String backend = column.label();
for (Query query : queries) {
for (int attempt = 1; attempt <= runs; attempt++) {
final int run = attempt;
Counting counter = new Counting(DefaultToolCallingManager.builder()
.maxTotalToolCalls(12)
.build());
RunBudget budget = new RunBudget(8, 12);
List<ToolCallback> callbacks =
context.callbacks(tools, auth, null, budget);
var spec = ChatClient.create(model).prompt()
.system(column.insistent() ? SYSTEM + SpringAgentCommand.SEARCH_RULES : SYSTEM)
.user(query.text())
.tools(callbacks.toArray())
// A ceiling on generation. Without it one degenerate reply can
// repeat itself for hours and hold the model's only slot, which
// stalls every run behind it.
.options(OllamaChatOptions.builder()
.model(context.modelName())
.numPredict(MAX_GENERATED_TOKENS))
.advisors(advisorParams -> advisorParams.param(ChatMemory.CONVERSATION_ID,
backend + "-" + query.text().hashCode() + "-" + run));
ToolIndex index = indexFor(column.index());
spec = index == null
? spec.advisors(ToolCallingAdvisor.builder()
.toolCallingManager(counter).build())
: spec.advisors(ToolSearchToolCallingAdvisor.builder()
.toolIndex(index).maxResults(topK)
.toolCallingManager(counter).build());
long tokens;
String outcome;
try {
ChatResponse response = spec.call().chatResponse();
Usage usage = response == null ? null : response.getMetadata().getUsage();
tokens = counter.tokens
+ (usage == null ? 0 : usage.getPromptTokens() + usage.getCompletionTokens());
outcome = response == null || response.getResult() == null ? "(empty response)"
: response.getResult().getOutput().getText();
} catch (Exception failure) {
tokens = counter.tokens;
outcome = "FAILED: " + failure.getClass().getSimpleName() + " " + failure.getMessage();
}
long found = query.needs().stream().filter(counter.executed::contains).count();
System.out.printf("%-14s %-40s %6d %8d %9d %6d/%-3d %-34s %s%n",
backend, trim(query.text(), 40), counter.rounds + 1,
counter.searches, tokens, found, query.needs().size(),
counter.executed.isEmpty() ? "(no tools called)"
: trim(String.join(",", counter.executed), 34),
trim(outcome.replace("\n", " "), 60));
}
}
}
}
private ToolIndex indexFor(String backend) {
return switch (backend) {
case "regex" -> new RegexToolIndex();
case "lucene" -> new LuceneToolIndex();
case "vector" -> {
EmbeddingModel embeddings = embeddingModels.getIfAvailable();
yield embeddings == null ? null
: new VectorToolIndex(SimpleVectorStore.builder(embeddings).build());
}
default -> null;
};
}
/** Delegates every round to the real manager, and counts what goes past. */
private static final class Counting implements ToolCallingManager {
private final ToolCallingManager delegate;
private final Set<String> executed = new LinkedHashSet<>();
private int rounds;
private int searches;
private long tokens;
private Counting(ToolCallingManager delegate) {
this.delegate = delegate;
}
@Override
public List<ToolDefinition> resolveToolDefinitions(ToolCallingChatOptions options) {
return delegate.resolveToolDefinitions(options);
}
@Override
public ToolExecutionResult executeToolCalls(Prompt prompt, ChatResponse response) {
rounds++;
Usage usage = response.getMetadata().getUsage();
if (usage != null) {
tokens += usage.getPromptTokens() + usage.getCompletionTokens();
}
if (response.getResult() != null) {
response.getResult().getOutput().getToolCalls().forEach(call -> {
if (call.name().equals("toolSearchTool")) {
searches++;
} else {
executed.add(call.name());
}
});
}
return delegate.executeToolCalls(prompt, response);
}
}
private List<Query> loadQueries() throws Exception {
List<Query> queries = new ArrayList<>();
for (String line : Files.readAllLines(Path.of("src/main/resources/tool-bench-queries.tsv"))) {
if (line.isBlank() || line.startsWith("#")) {
continue;
}
String[] parts = line.split("\t");
queries.add(new Query(parts[0], List.of(parts).subList(1, parts.length)));
}
return queries;
}
private static String trim(String text, int width) {
return text.length() <= width ? text : text.substring(0, width - 1) + "…";
}
private record Query(String text, List<String> needs) {
}
}
Running It, and the Protocol
mvn clean spring-boot:run
Choose tool-live-bench, then answer the prompt with 10 5: ten runs of each task, at a ceiling of five definitions per search. If you add column names after those numbers, only those columns run. We collected the results below that way, one column at a time. Between columns we restarted the Movie Database service, so every column starts from the same seeded data. That matters, because one of the tasks writes to the database. After ten runs of add The Matrix to my To watch this weekend list, the database holds ten more list entries. A column that started with them would measure a different situation from the column before it.
Seven configurations, five tasks and ten runs each make 350 agent runs. They took a little over three hours on a laptop with qwen3:8b on Ollama. The first four columns run the framework as it ships. The last three repeat the search backends, with the insistent rules from the previous section added to the system prompt.
Halfway through our first attempt at this matrix, every run stopped. Ollama serves one request at a time for a model, and a single reply had entered a repetition loop. It had generated 287,000 tokens and was still going. It held the only slot, and every later run waited behind it. That is why tool-live-bench sets numPredict(4096) on its options. Any long unattended run against a local model needs a similar limit on generated tokens.
What 350 Runs Say
| configuration | median tokens per run | median rounds | searches per run | tasks completed | tools reached |
|---|---|---|---|---|---|
| tool search off | 10,130 | 3 | 0 | 21/50 | 71/100 |
| regex | 5,592 | 3 | 1.08 | 14/50 | 46/100 |
| lucene (BM25) | 5,422 | 3 | 0.98 | 9/50 | 29/100 |
| vector | 5,364 | 3 | 1.02 | 5/50 | 24/100 |
| regex + rules | 7,738 | 4 | 1.06 | 19/50 | 61/100 |
| lucene + rules | 5,684 | 3 | 1.04 | 10/50 | 32/100 |
| vector + rules | 7,985 | 4 | 1.04 | 9/50 | 34/100 |
Tool search roughly halves what a task costs. A median run drops from 10,130 tokens to about 5,400. That saving is larger than the deterministic table suggested, and the reason is the loop: definitions go into every round, and a median task runs three of them.
It also lowers completion. With the whole catalog in front of the model, twenty-one of fifty tasks finished. The three backends finished fourteen, nine and five. The saving comes from leaving definitions out, and the agent needs some of those definitions to find its tools.
The insistent rules recover most of that, at a cost in tokens. With the rules added, regex finishes nineteen of fifty against the full catalog's twenty-one, at 7,738 median tokens against 10,130. So the useful configuration on this catalog is regex plus rules: a quarter fewer tokens for two fewer completions out of fifty.
The rules did not make the model search more. Every configuration averages about one search per run. A second search appears in seven runs of fifty at best, and the rules made it slightly rarer. The rules changed what happens after the search. Median rounds go from three to four. The agent works with the tools it found, where before it reported that the right tool was missing. So an instruction to search again made the model keep working with the results it already had.
Live ranking and offline ranking disagree. In the deterministic table, the vector index led at every ceiling above five. With the model running, regex leads and the vector index comes last. The logs suggest a reason. The model writes its own search query in the vocabulary of tool names, as when it turned list my watchlists into manage watchlists. Substring matching rewards exactly that, so an index that looked weaker offline can retrieve better in a live loop. This is one model and one catalog, so measure your own before you adopt this ordering.
What These Numbers Do Not Say
The table has three limits, and each one restricts how far its results apply.
One task fails in every configuration. who directed Inception finishes zero times in all seven columns, because qwen3:8b already knows the answer. It calls search, sees the film, answers "Christopher Nolan" from memory and skips the person tool. Our completion metric scores that run as a failure, and the user still gets a correct answer. Without that task, the totals become 21, 14, 9, 5, 19, 10 and 9 of forty, and every conclusion still holds. A benchmark task has to be one that the model cannot answer from memory.
The model is the larger limit here. Even with all eighteen definitions in front of it, the agent finished only twenty-one of fifty tasks. Retrieval has a smaller effect on top of that. On a frontier model, the same experiment would start from a much higher baseline and would probably show smaller differences between the backends.
Completion counts tool calls, and truth is a separate question. A run that calls every needed tool can still summarise the result badly. That problem is the subject of Class 8: the shape and the truth of an answer have to be engineered separately.
When to Reach for It
Spring's guidance is to turn tool search on above roughly ten tools, or when definitions exceed ten thousand tokens per request. It advises leaving it off when every session uses every tool, or when the definitions are compact. Our measurements agree, and they add three practical rules.
Turn it on to cut the token bill, and expect lower accuracy in return. Halving the cost of a task while finishing a third fewer tasks is a good trade when the catalog is large and the tasks are simple. It is a poor trade when one task needs three tools working together. Our completion numbers fell furthest on those tasks.
If you turn it on, add the search rules to the system prompt. The framework's own paragraph allows the model to search. A small model needs an insistent instruction before it works with what the search returned. On this catalog, that configuration finished nineteen of fifty tasks, against twenty-one, with a quarter fewer tokens.
Measure your own catalog before choosing an index. Ours put regex ahead live and the vector index ahead offline. That gap is a strong hint that the ordering depends on how your tool names read and how your model phrases a search. The two commands in this lesson let you measure your own catalog in an afternoon.
With eighteen small tools, the advisor's 293 tokens and its extra round trip are a large part of what it saves, so this catalog is close to break-even. The overhead is worth it when the catalog is really large. For a GraphQL agent, that usually means an allow-list that covers a whole product, or a session that also carries tools from several MCP servers.
Through all of this, the schema, the generator, the allow-list, the callbacks and every safety control stayed as they were. The decision was one setting in the configuration. That is the portability claim of the framework track, applied to a problem of scale.
Tool search is one of two answers this course gives to the problem of scale, and the two suit different teams:
- If you want a curated menu, and you want the agent to use only the operations on it, this section keeps that menu affordable as it grows.
- If the schema is too large for any curated menu, Class 3 takes the other path. It uses three meta-tools, retrieval over schema coordinates instead of over tools, and an agent that writes its own operations under validation.
In Spring AI, memory is one more advisor. The loop belongs to the advisor chain, so what the loop remembers belongs there too. MessageChatMemoryAdvisor wraps a MessageWindowChatMemory, and one conversation id covers the exchange. So the second turn of a conversation reaches the model together with the messages of the first turn.
ChatMemory memory = MessageWindowChatMemory.builder().build();
String conversationId = "spring-agent-" + System.nanoTime();
var spec = client.prompt()
.system(SYSTEM)
.user(turn)
.tools(callbacks.toArray())
.options(OllamaChatOptions.builder().model(context.modelName()))
.advisors(MessageChatMemoryAdvisor.builder(memory).build())
.advisors(advisorParams -> advisorParams.param(ChatMemory.CONVERSATION_ID, conversationId));
The three frameworks give the same behaviour through three mechanisms:
- LangChain4j carries memory in the
AiServicesbuilder. - LangGraph4j restores it from the checkpoint under a thread id.
- Spring AI adds it to the advisor chain.
The agent finishes the job in all three. They differ only in where the memory lives.
Streaming, With Tools In The Middle
The ChatClient's other terminal is stream(), and it returns the answer as a Reactor Flux you can render as it arrives:
client.prompt()
.system(SpringAgentCommand.SYSTEM)
.user(turn)
.tools(callbacks.toArray())
.options(OllamaChatOptions.builder().model(context.modelName()))
.advisors(MessageChatMemoryAdvisor.builder(memory).build())
.advisors(advisorParams -> advisorParams.param(ChatMemory.CONVERSATION_ID, conversationId))
.stream()
.content() // Flux<String>: the answer, token by token
.doOnNext(System.out::print)
.blockLast();
Our spring-stream command runs exactly this. The most interesting part of the run does not show in the streamed text. Asked which 1994 movies the catalog holds, the run made two tool calls during the stream. The advisor pauses the stream, runs the tool call and then carries on with the rest of the model's answer. The budget line at the end shows the two calls.
Familiar chat products are built on streaming with tools, and here it works without extra code. The Flux is the same reactive type as in the rest of reactive Spring. To send it to a WebFlux SSE endpoint or a WebSocket, we pass that same Flux through, without an adapter layer. As of 2.0, the stream() terminal cannot return structured entities yet. That feature works only with call(), and the next lesson works within that limit.
This course runs local models, so one platform note belongs here. Tool calling needs a recent Ollama: 0.2.8 for tool calls, and 0.4.6 for tool calls while streaming. The quality of the model still decides everything that the framework cannot control. The runs in this lesson used the same 8-billion-parameter local model as the rest of the course, and it worked. Smaller models can fail in the ways that Classes 4 and 6 documented.
Choosing Within the Track
We ran the same task through all three frameworks, and the differences fit in a table:
| Framework | What it gives you | Reach for it when |
|---|---|---|
AiServices (Class 5) | the highest abstraction: an interface and a builder | you want the framework to hide the loop |
| LangGraph4j (Class 6) | the loop made explicit: state, drawn control flow, checkpointed pauses | workflows branch, wait for humans, and must be auditable |
ChatClient (this lesson) | a managed loop with defined extension points (advisors), first-class streaming | the rest of your service already uses Spring |
The catalog, the allow-list, the login, the gate and the budgets were identical in all three. That is the evidence for the central claim of this course. The schema work keeps its value, and the framework around it is a choice you can change later. A change costs a translator and a budget moved into the new loop.
Summary
This lesson moved the concierge into Spring AI 2.0:
- The
ChatClientworks without a bridge, because the course's callbacks implement its nativeToolCallbackcontract. Registration is onetoolCallbacks(...)call. - The tool loop lives in the auto-registered
ToolCallingAdvisor, because in 2.0 the models do not run tools themselves. Controls owned by the loop belong there: approval hooks through subclassing, and the framework's own ceilings of 40 calls per tool and 150 per turn. - Controls owned by the callback kept working unchanged. A gated write passed through the same approval prompt and budget line under a fourth execution engine.
- Response metadata carries the usage reported by the provider, including cache accounting on providers that support it. Class 10 builds on it.
When a catalog grows too expensive to advertise in full, the same chain has the answer. ToolSearchToolCallingAdvisor replaces the owner of the loop and indexes the eighteen-tool catalog per session. It sends one toolSearchTool in place of 1,400 tokens of definitions. Across 350 measured agent runs, that halved the cost of a task, from 10,130 median tokens to about 5,400. Completed tasks dropped from twenty-one of fifty to between fourteen and five, depending on the index. With insistent search rules in the system prompt, regex came back to nineteen of fifty at 7,738 tokens, the best configuration for this catalog. The advisor already appends its own instruction to the system message, so the model always knows that it can search. A small model still searches once, and about one run in ten searches twice.
The choice of ToolIndex is a real decision, and the answer depends on who writes the query. Offline, with the words a user types, the vector index found nine of ten needed tools at a ceiling of ten, and regex stopped at five at any ceiling. Live, with the words the model writes, regex finished the most tasks and the vector index the fewest. Every backend searches the descriptions. LuceneToolIndex keeps addWatchListItem as a single token, so a name that a person reads as three words is one term to it.
Streaming is a Reactor Flux with tools running in the middle of the stream, and we verified it live. As of 2.0, entities work only with call(). The comparison across the track comes down to how much of the loop you want to see:
AiServiceshides it.- LangGraph4j draws it.
ChatClientmanages it and leaves defined extension points.
All three run over the same portable GraphQL tool layer.
Further Reading
- Spring AI: Chat Client API: the fluent API, advisors, and the entity and streaming terminals.
- Spring AI: Advisors: the interceptor model that owns the loop, memory, and logging; the place production gates and policies plug in.
- Spring AI: Tool Calling: the
ToolCallbackcontract this course has been building against since Class 2, and the execution limits. - Spring AI: Tool Search Tool: the
ToolSearchToolCallingAdvisor, its threeToolIndeximplementations, session scoping and the eviction strategies this lesson left at their defaults. - Smart tool selection with dynamic tool discovery (Spring, December 2025): the announcement, with the token-saving measurements across three hosted providers.
- Claude API: Tool Search Tool: the same pattern as a provider API feature, through the
defer_loadingflag; the MCP course's Class 20 covers what it means for a server you own. - Spring AI: Ollama chat: the local-model properties, including the version floors for tool calling and streaming tools.
What's Next
The track is finished and the stack is chosen. Next, we make the agent's output as typed as its tools. Class 8 covers structured output:
- GraphQL fragments become the contract for what the agent must produce.
- Java records mirror those fragments.
- Spring AI's converters and provider-native structured output make the model answer in exactly that shape.