Skip to main content

Class 8: Structured Output via Fragments

Duration: ~30 minutes | Difficulty: Advanced | Prerequisites: Class 7 completed; Ollama with a tool-calling model.

Project code

This lesson carries on from Class 7. If you followed along, keep working on the project you already have. If you skipped it, clone the agents_class_7 branch to start from the same place:

git clone --branch agents_class_7 https://github.com/graphqlguy/graphQLBeyondMCP.git

Code blocks are titled with the file they belong to, and the note in brackets says what to do with it. (new file) is a file to create. (edited, whole file) replaces a file you already have. Anything else names the one part to add or replace, as in (edited, the call method). Where a block shows part of a new file, the complete file follows in a collapsed block underneath.

So far, every answer from the agent arrives as a paragraph of English. That suits a person reading a terminal, and it is hard for calling code to use:

  • A web page that shows two recommendation cards has to pull the titles out of a sentence, and the wording can change on the next run.
  • A workflow that acts on a verdict has to search the answer for a word such as "approved".
  • A test has to make assertions about wording.

Structured output solves this. We give the model a shape, and the answer has to fill it. Here the shape is a Java record, MovieRecommendation. Spring AI derives a JSON Schema from the record, sends that schema to the model with the request, and parses the reply into the record before we see it. The calling code receives a List<MovieRecommendation>. Each entry holds an int year and a double rating, so a UI, a test or the next step in a workflow can use it without parsing English. The tools already had types. This lesson gives the answer types too, so calling code can rely on both ends of the agent.

Requiring that shape is easy, and it comes with the most important warning in this lesson. Our first run produced output with correct types, and every fact in it was invented. A schema controls the shape of an answer, and it cannot make the values true. So we handle shape and truth in separate steps, and by the end of the lesson we have a two-step pattern that delivers both.

One Shape, Written Three Times​

Structured output starts with deciding the shape. That shape then has to exist in three languages:

  • GraphQL asks the server for exactly those fields.
  • Java holds the fields once they arrive.
  • JSON Schema tells the model what it must produce.

Each language writes the shape differently, and the four fields have to match in all three. When two of them disagree, parsing fails, or a field arrives empty while everything else looks correct.

GraphQL supplies the first version, and it already has a name for a reusable shape: a fragment. A fragment is a named selection of fields. Naming one is a domain design decision, because it states what a short summary of a movie contains in this application:

fragment MovieCard on Movie { id title releaseYear rating }

A fragment belongs in an operation document, which is the file a client sends. The schema defines the types, and a fragment selects fields from them. So this definition goes in a query file, and in the SDL it would be a syntax error. For now it is a design sketch. In Page and Connection Wrappers below, it becomes part of two files we ship, the curated overrides movies.graphql and search.graphql.

The Java side mirrors it as a record, with one addition:

src/main/java/com/graphqlguy/moviedb/agents/structured/MovieRecommendation.java (new file)
package com.graphqlguy.moviedb.agents.structured;

/**
 * The Java mirror of the MovieCard fragment, plus one field the data cannot
 * supply. The first four components correspond one to one with the fragment's
 * selections, so the agent's typed output and the tool's typed result have the
 * same shape in two languages. "reason" is the model's own contribution, the
 * part that justifies involving a model at all.
 */
public record MovieRecommendation(String id, String title, int releaseYear,
                                  double rating, String reason) {
}

The first four components match the fragment's selections, field for field. The fifth, reason, is left out of the fragment on purpose, because the data cannot supply it. It is the model's own contribution, and it is the reason to involve a model at all. The record shows the design principle of this lesson: the shape of an agent's answer is the shape of your data plus the model's judgment, and we should be able to trace each part to its source.

Spring AI writes the third version for us. Its entity(...) terminal uses a BeanOutputConverter, which does three things:

  • It derives a JSON Schema (draft 2020-12, via Jackson) from the record.
  • It tells the model to answer in that schema.
  • It parses what comes back into the record.

So the record we just wrote is the only source of the JSON Schema:

src/main/java/com/graphqlguy/moviedb/agents/command/RecommendCommand.java (new file, the entity terminal)
List<MovieRecommendation> answer = client.prompt()
        .system("Format the given findings as recommendations."
                + " Use only movies present in the findings, with"
                + " their exact data; add a one-sentence reason each."
                + " If no usable movies appear in the findings, return"
                + " an empty list; an empty list is a correct answer.")
        .user("Request: " + ask + "\nFindings: " + findings)
        .options(OllamaChatOptions.builder().model(context.modelName()))
        .call()
        .entity(new ParameterizedTypeReference<List<MovieRecommendation>>() {});
// entity() is nullable: a model that answers with unparseable text yields
// null here, and the command reports that as an empty list.
answer = answer == null ? List.of() : answer;

The fragment, the record and the schema express one shape in the query language, in Java and in the model's format instructions. When the three agree, data travels from the resolver to the UI with a type at every step. Class 2's JSON Schema mapping applies unchanged on the output side: enums become closed lists, and non-null fields become required.

Two Spring AI 2.0 Switches​

Spring AI 2.0 adds two options to the same entity terminal. Each one is set at call time, as a consumer passed next to the type:

  • spec -> spec.useProviderStructuredOutput() sends the JSON Schema to the provider as an API-level field. The provider's runtime then constrains generation, so the model can only produce output that fits the schema. The format instructions are no longer in the system prompt, so every request is shorter.
  • spec -> spec.validateSchema() keeps the prompt-based approach and validates the answer when it arrives. When validation fails, Spring AI adds the specific error, such as missing required field actor, to the user prompt and sends the call again, with three attempts by default. The model reads what went wrong before it answers again, so it can correct the field that failed. Streaming stays off while this switch is active, because the advisor validates a complete response.

The second switch runs in our own process and works with any model. The first one depends on the provider. Spring AI checks whether the provider's chat options implement StructuredOutputChatOptions. When they do not, it uses the prompt-based default, so turning the switch on is safe with any provider. The documented list names the models where the provider enforces the schema:

  • OpenAI's GPT-4o and later
  • Anthropic's Claude 3.5 Sonnet and later
  • Gemini 1.5 Pro and later
  • Mistral Small and later
  • Ollama models that accept a JSON Schema

This lesson runs on Ollama, and qwen3:8b is one of the models the same page lists as unreliable. A model with a thinking mode can return its reasoning as plain text, which BeanOutputConverter reports as Unrecognized token 'The'. The docs suggest llama3.1:latest if you want to see the native path work. So the code above keeps the prompt-based default, and validateSchema() is the switch that helps on our model. Remember one more limit before you move this code to a hosted provider. OpenAI rejects a top-level array, so there List<MovieRecommendation> would have to be wrapped in a record.

Typed Output, Invented Facts​

The obvious approach is one call with both the tools and entity: the tools ground the answer, and the converter gives it a type. The project includes that version as its own command, so we can run it and read what it produces:

src/main/java/com/graphqlguy/moviedb/agents/command/RecommendOneCallCommand.java (new file, the one call)
List<MovieRecommendation> answer = ChatClient.create(model).prompt()
        .system("You research movies in the Movie Database. Use the tools to"
                + " find real catalog data answering the request; never invent"
                + " movies. Recommend movies with their exact ids, titles,"
                + " years and ratings from tool results, one sentence each.")
        .user(ask)
        .tools(callbacks.toArray())   // grounding is asked for here
        .options(OllamaChatOptions.builder().model(context.modelName()))
        .call()
        // and shape is asked for here, on the same call
        .entity(new ParameterizedTypeReference<List<MovieRecommendation>>() {});

The rest of the command is the setup that the two-step version uses as well:

  • the read-only tools
  • an anonymous session
  • a budget of eight model calls and six tool calls
  • the printing

Creating the file below adds recommend-one to the menu, because the application discovers commands as beans.

The complete RecommendOneCallCommand.java, collapsed because it runs to 81 lines
src/main/java/com/graphqlguy/moviedb/agents/command/RecommendOneCallCommand.java (new file)
package com.graphqlguy.moviedb.agents.command;

import com.graphqlguy.moviedb.agents.auth.AuthSession;
import com.graphqlguy.moviedb.agents.safety.RunBudget;
import com.graphqlguy.moviedb.agents.structured.MovieRecommendation;
import com.graphqlguy.moviedb.agents.toolgen.OperationTool;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.core.ParameterizedTypeReference;
import org.springframework.stereotype.Component;

import java.util.List;

/**
 * Class 8: the obvious composition, kept so the failure can be reproduced. Tools and
 * the entity terminal are asked for on the same call, which makes grounding and shape
 * compete for one decision. Run this next to {@code recommend} and read the tool-call
 * count in the last line: this version usually answers without calling anything.
 */
@Component
public class RecommendOneCallCommand implements AgentCommand {

    private final ObjectProvider<ChatModel> chatModels;

    public RecommendOneCallCommand(ObjectProvider<ChatModel> chatModels) {
        this.chatModels = chatModels;
    }

    @Override
    public String name() {
        return "recommend-one";
    }

    @Override
    public int order() {
        return 79;
    }

    @Override
    public String summary() {
        return "one call demanding tools and typed output at once; usually invents the facts";
    }

    @Override
    public String prompt() {
        return "request, e.g. two dramas worth my evening";
    }

    @Override
    public void run(List<String> args, CommandContext context) {
        String ask = String.join(" ", args);
        ChatModel model = chatModels.getIfAvailable();
        if (ask.isBlank() || model == null) {
            System.out.println(ask.isBlank() ? "recommend-one needs a request"
                    : "No chat model configured.");
            return;
        }
        RunBudget budget = new RunBudget(8, 6);
        List<OperationTool> reads = context.tools(null).stream()
                .filter(tool -> !tool.mutation())
                .toList();
        List<ToolCallback> callbacks =
                context.callbacks(reads, AuthSession.anonymous(), null, budget);

        System.out.println("task    : " + ask);
        System.out.println();

        // One call, both jobs. The system prompt asks for grounding, the entity
        // terminal appends the format instructions, and the model picks one.
        List<MovieRecommendation> answer = ChatClient.create(model).prompt()
                .system("You research movies in the Movie Database. Use the tools to"
                        + " find real catalog data answering the request; never invent"
                        + " movies. Recommend movies with their exact ids, titles,"
                        + " years and ratings from tool results, one sentence each.")
                .user(ask)
                .tools(callbacks.toArray())
                .options(OllamaChatOptions.builder().model(context.modelName()))
                .call()
                .entity(new ParameterizedTypeReference<List<MovieRecommendation>>() {});
        answer = answer == null ? List.of() : answer;

        for (MovieRecommendation rec : answer) {
            System.out.printf("- %s (%d, rated %.1f): %s%n",
                    rec.title(), rec.releaseYear(), rec.rating(), rec.reason());
        }
        System.out.println();
        System.out.println("type   : List<MovieRecommendation>, " + answer.size()
                + " typed entries; every field is a Java value, never text to parse");
        System.out.println("budget : " + budget.summary());
    }
}

Run This Version​

Start the Movie Database service on 8081 and Ollama on 11434, as in every lesson since Class 2. Then run this in the agents project:

mvn clean spring-boot:run

Choose recommend-one from the menu and give it the request the transcripts below use:

two great drama movies from the nineties

Read the last line of the output first, because it tells you what happened:

  • 0 tool calls is the failure this section is about. The answer came out of the model's memory, so every id, year and rating in it is invented, even though the records look correct.
  • 1 tool calls means this run consulted the catalog after all, so its ids, years and ratings come from real records.

The model decides which of the two happens, so the result changes between runs. Ten runs in a row of this request on qwen3:8b produced the failure six times and the grounded answer four times. If your first attempt is grounded, choose recommend-one again with the same request, since the menu comes back after each command. Two or three attempts are usually enough to see the failure. Each run took about ten seconds on the laptop that produced these transcripts.

Here is one run, unedited:

task    : two great drama movies from the nineties

- The Shawshank Redemption (1994, rated 8.9): iconic performances and powerful
  storytelling
- The Dark Knight (1999, rated 8.9): masterful character development and
  emotional depth

type   : List<MovieRecommendation>, 2 typed entries
budget : 0 model calls (ceiling 8), 0 tool calls (ceiling 6)

Every field has the right type and every value parses, and the facts are wrong. The catalog rates The Shawshank Redemption 9.3, so the 8.9 here is invented. The Dark Knight was released in 2008, so it is not a nineties film and does not answer the request. The budget line shows how both entries got through: zero tool calls. The format instructions and the tool definitions compete for the model's next step. For this model, producing well-formed JSON right away took priority over the instruction to look things up first. The output converter did its job correctly, and its job is the shape. A schema cannot require that the values came from a tool.

The other five failures followed the same pattern with different invented values:

  • The Lord of the Rings: The Fellowship of the Ring appeared with the year 1999. It is a 2001 fantasy film.
  • The Pursuit of Happyness appeared with the year 2006, outside the decade in the request.
  • Schindler's List appeared with the year 1994 in two separate runs. It is a 1993 film.
  • The rating of The Shawshank Redemption changed between 9.3, 9.0 and 8.9, while its title stayed the same.

The changing rating is the clearest sign of invention, because a value the model looked up would be the same number every time.

The four grounded runs all returned The Shawshank Redemption at 9.3 and Forrest Gump at 8.8, the real values from the catalog, each after exactly one tool call. A model that grounds four runs in ten is harder to work with than one that always skips the tools, because its wrong answers and its right answers look the same on screen.

This failure needs a name that your team knows: typed hallucination. It is harder to catch than the same mistake in free text. A reader treats free text as a claim to check. A typed entry has already been parsed into a record by the time anyone sees it, so every system downstream treats it as correct data.

Act, Then Format​

The repair applies separation of concerns to the conversation: each call gets one job. The diagram compares it with the failing version, and the difference is which demands reach the model on the same call:

Step one acts: the model gets the tools and writes a plain-text answer, so grounding is its only job. The findings come back with exact ids, years and ratings from the tool results. Step two formats: the findings go back to the model as text, without the tools, and the entity terminal shapes them, so the shape is its only job. Here is the command, with two calls and one job each:

src/main/java/com/graphqlguy/moviedb/agents/command/RecommendCommand.java (new file, in full)
package com.graphqlguy.moviedb.agents.command;

import com.graphqlguy.moviedb.agents.auth.AuthSession;
import com.graphqlguy.moviedb.agents.safety.RunBudget;
import com.graphqlguy.moviedb.agents.structured.MovieRecommendation;
import com.graphqlguy.moviedb.agents.toolgen.OperationTool;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.ai.tool.ToolCallback;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.core.ParameterizedTypeReference;
import org.springframework.stereotype.Component;

import java.util.List;

/**
 * Class 8: tools plus typed output, in two steps. Asking for both at once makes a
 * small model choose between grounding and shape, and it chooses badly, so this
 * command acts first and formats second.
 */
@Component
public class RecommendCommand implements AgentCommand {

    private final ObjectProvider<ChatModel> chatModels;

    public RecommendCommand(ObjectProvider<ChatModel> chatModels) {
        this.chatModels = chatModels;
    }

    @Override
    public String name() {
        return "recommend";
    }

    @Override
    public int order() {
        return 80;
    }

    @Override
    public String summary() {
        return "tools plus typed output; the answer arrives as List<MovieRecommendation>";
    }

    @Override
    public String prompt() {
        return "request, e.g. two dramas worth my evening";
    }

    @Override
    public void run(List<String> args, CommandContext context) {
        String ask = String.join(" ", args);
        ChatModel model = chatModels.getIfAvailable();
        if (ask.isBlank() || model == null) {
            System.out.println(ask.isBlank() ? "recommend needs a request"
                    : "No chat model configured.");
            return;
        }
        RunBudget budget = new RunBudget(8, 6);
        // Reads only, and anonymously: this class is about the shape of an answer,
        // so it leaves identity and the gate to the classes that teach them.
        List<OperationTool> reads = context.tools(null).stream()
                .filter(tool -> !tool.mutation())
                .toList();
        List<ToolCallback> callbacks =
                context.callbacks(reads, AuthSession.anonymous(), null, budget);
        var client = ChatClient.create(model);

        System.out.println("task    : " + ask);
        System.out.println();

        // Step one: ACT. Tools only, no output format in sight, so nothing competes
        // with grounding. The findings come back as plain text.
        String findings = client.prompt()
                .system("You research movies in the Movie Database. Use the tools"
                        + " to find real catalog data answering the request; never"
                        + " invent movies. Report the matching movies with their"
                        + " exact ids, titles, years, and ratings from tool results.")
                .user(ask)
                .tools(callbacks.toArray())
                .options(OllamaChatOptions.builder().model(context.modelName()))
                .call()
                .content();
        if (findings == null) {
            System.out.println("The model returned an empty response;"
                    + " stopping before the format step.");
            return;
        }
        System.out.println("findings (grounded, text): " + findings.replace("\n", " ").strip());
        System.out.println();

        // Step two: FORMAT. Entity only, no tools, so nothing competes with the
        // shape. The model restructures known-good facts.
        List<MovieRecommendation> answer = client.prompt()
                .system("Format the given findings as recommendations."
                        + " Use only movies present in the findings, with"
                        + " their exact data; add a one-sentence reason each."
                        + " If no usable movies appear in the findings, return"
                        + " an empty list; an empty list is a correct answer.")
                .user("Request: " + ask + "\nFindings: " + findings)
                .options(OllamaChatOptions.builder().model(context.modelName()))
                .call()
                .entity(new ParameterizedTypeReference<List<MovieRecommendation>>() {});
        // entity() is nullable: a model that answers with unparseable text yields
        // null here, and the command reports that as an empty list.
        answer = answer == null ? List.of() : answer;

        for (MovieRecommendation rec : answer) {
            System.out.printf("- %s (%d, rated %.1f): %s%n",
                    rec.title(), rec.releaseYear(), rec.rating(), rec.reason());
        }
        System.out.println();
        System.out.println("type   : List<MovieRecommendation>, " + answer.size()
                + " typed entries; every field is a Java value, never text to parse");
        System.out.println("budget : " + budget.summary());
    }
}

The last clause of the formatting prompt matters more than its length suggests. In one intermediate run on the way to the working version, the research step correctly reported that the tools had returned counts without movie data. The formatting step then invented two entries anyway, because a model asked for a list tends to fill one. When the prompt says that an empty list is correct, the model has a valid answer for empty findings. When you design output shapes for agents, always include a way to say "nothing":

  • an empty list
  • an optional field
  • a NO_ANSWER enum member

If a schema accepts only populated outputs, a model with empty findings must either fail validation or invent something, and it usually invents something.

The two-step version also costs one extra model call. That call is the price of handling truth and shape separately. It is the cheapest call in the run, because the formatting step does not call any tools and its input is a few hundred tokens of findings. Class 10 shows its exact cost.

Page and Connection Wrappers​

The intermediate failure above had a second cause, on the tool side. It also completes a point from Class 2. The movies operation returns MoviePage. The generated default selection takes scalar fields one level deep, so it selected only the page's scalar fields: totalElements, totalPages, currentPage, and five more pagination fields. The movies are under content, an object list that the conservative default leaves out. So the tool correctly answered "2 results" with only the pagination fields, and the research step reported exactly that. Page and connection wrappers always need a curated selection, because their useful data is made of objects. The curated override is the second place where the fragment helps:

src/main/resources/tool-selections/movies.graphql (new file)
# Curated override: the generated default selects only MoviePage's scalar
# fields, which are pagination metadata; the movies themselves live under
# "content", an object list the conservative default excludes. Page and
# connection wrappers always need curation, and the MovieCard fragment is
# the reusable contract for what each movie should carry.
query Tool_movies($filter: MovieFilter, $page: Int, $size: Int, $sort: MovieSort) { movies(filter: $filter, page: $page, size: $size, sort: $sort) { content { ...MovieCard } totalElements } }
fragment MovieCard on Movie { id title releaseYear rating }

The same MovieCard now defines what the tool returns and what the record holds, and that keeps the three versions in step. When the fragment changes, the tool's result changes with it, and a review of that change shows that the record needs the same edit.

We edit the search override from Class 4 to use the same fragment, so one definition serves both tools:

src/main/resources/tool-selections/search.graphql (edited, whole file)
# Curated override, refined here around a named fragment. Two things are going on.
#
# First, the union: `search` returns SearchResult = Movie | Person | TvShow, and a
# union field is invalid without a selection, so the generated document fails at
# the server with SubselectionRequired. Choosing fields across union members needs
# inline fragments and a judgement about which members matter, and a person makes
# that judgement when writing this file.
#
# Second, the shape: MovieCard is the contract for what a "movie, briefly" means in
# this application, and Class 8's Java record mirrors it field for field. The same
# shape appears in three forms: a fragment (GraphQL), a record (Java) and a JSON
# schema (the model's format instructions).
query Tool_search($query: String!) {
  search(query: $query) {
    __typename
    ... on Movie  { ...MovieCard }
    ... on TvShow { id title startYear rating }
    ... on Person { id name }
  }
}
fragment MovieCard on Movie { id title releaseYear rating }

The Movie branch now uses ...MovieCard in place of the four fields it listed before. The two other branches keep their own selections, because the shape this lesson defines is for movies. Each override file has its own copy of the fragment definition, because each file is a complete document that its tool sends on its own.

The Working Run​

Check It Works​

mvn clean spring-boot:run

Choose recommend, then answer the prompt with the same request the failing version was given:

two great drama movies from the nineties

The menu comes back when the command finishes. So we can run recommend-one and recommend one after the other on the same request, and compare the two transcripts on screen. Compare three things:

  • The budget line. 1 tool calls means the catalog was consulted, and 0 tool calls means it was not.
  • The findings line, which only the two-step version prints. It is the grounding, in plain text, before any shape is asked for.
  • The ratings. Every rating the two-step version prints also appears in the findings above it. An invented film with a plausible year is the failure this lesson is about.

recommend-one changes from run to run. This version gave the same result five times: one tool call each, and the same two films with the same two ratings. The sentences after the ratings are the model's own and change on every run. That is the reason field, which the record marks as the model's judgment.

With the override in place and the two steps separated:

task    : two great drama movies from the nineties

findings (grounded, text): Here are two highly-rated drama movies from the 1990s:
1. **The Shawshank Redemption** (1994) Rating: 9.3 ID: 1
2. **Forrest Gump** (1994) Rating: 8.8 ID: 5 ...

- The Shawshank Redemption (1994, rated 9.3): Among the highest-rated dramas
  released between 1990 and 1999.
- Forrest Gump (1994, rated 8.8): Among the highest-rated dramas released
  between 1990 and 1999.

type   : List<MovieRecommendation>, 2 typed entries
budget : 0 model calls (ceiling 8), 1 tool calls (ceiling 6)

Compare these ratings with the invented ones from the first run. 9.3 and 8.8 are the catalog's real values. They came from one tool call, appear in the findings, pass through the formatting step unchanged, and arrive as Java values. The reasons are the model's contribution, in the field that the record reserves for judgment. The two steps handle shape and truth separately, and the answer has both.

Summary​

Structured output gives the agent's answers the same kind of types its tools already had, and GraphQL fragments are a natural place to define those shapes.

A fragment names the selection, a Java record mirrors it and adds the fields for the model's judgment, and Spring AI's entity terminal sends a generated JSON Schema for it and parses the reply into the record. The same call can also use provider-native enforcement and validation retries. The main warning came from the first run. Tools and format instructions compete for the model's next step, and when the format wins, the result is typed hallucination: wrong data that passes every type check. The repair has three parts:

  • Act, then format. One call grounds the answer with tools, and a second call shapes it without them. The cost is one cheap extra call.
  • An output schema that can represent "nothing", such as an empty list, so a model with empty findings still has a valid answer.
  • Curated selections for page wrappers, written with the same fragment that the record mirrors. A page wrapper's useful data is made of objects, and the generated default correctly leaves those out.

Further Reading​

What's Next​

A typed answer arrives all at once. Class 9 covers answers that arrive in parts, with streaming and incremental delivery on the JVM:

  • what the Flux from Class 7 can carry
  • how tool calls fit into a live stream
  • where GraphQL's @defer stands on this stack today