Skip to main content

Class 9: Streaming and Incremental Delivery on the JVM

Duration: ~20 minutes | Difficulty: Advanced | Prerequisites: Class 8 completed; Ollama for the streaming recap.

Project code

This lesson carries on from Class 8. If you followed along, keep working on the project you already have. If you skipped it, clone the agents_class_8 branch to start from the same place:

git clone --branch agents_class_8 https://github.com/graphqlguy/graphQLBeyondMCP.git

Code blocks are titled with the file they belong to, and the note in brackets says what to do with it. (new file) is a file to create. (edited, whole file) replaces a file you already have. Anything else names the one part to add or replace, as in (edited, the call method). Where a block shows part of a new file, the complete file follows in a collapsed block underneath.

An agent run is slow, and most of the wait happens outside the model. A tool call goes to a GraphQL server, and the server waits on a database or a third-party API. Meanwhile the person who asked the question watches a blank screen. This lesson looks at what we can send to that person while the work goes on, and which techniques work on the JVM today.

The word streaming causes confusion, because it names three different things. A team that mixes them up can plan for a feature it does not have. The three are:

  • Model-output streaming: the answer appears word by word while the model writes it, so the user of a chat interface sees progress at once. It works today, and we already used it. The stream() terminal in Class 7 printed an answer token by token, and tools ran in the middle of it.
  • Data streaming: the server pushes events to the client as they happen. In GraphQL this is a subscription. It is mature and ready for production on the JVM, and the Movie Database service already has one.
  • Incremental delivery: the response to one query arrives in parts, with the fast fields first and the slow fields when they are ready. This is what @defer and @stream promise. It is not available yet on this stack. graphql-java supports it as an experiment, and Spring for GraphQL has yet to add it. The engine has the pieces, and the HTTP transports cannot carry them yet.

These statuses are as of August 2026. This lesson covers the three in that order.

Incremental delivery gets the longest section of this lesson, even though our application cannot use it yet. The reason is that teams plan around it. @defer has been shown at conferences for years, and some servers outside the JVM ship it, so it is easy to assume that Spring for GraphQL supports it. A design built on that assumption has to be reworked when the assumption turns out to be false. So we check what really exists, run the part that works, and finish with a technique that gives the same benefit today.

The three differ in what arrives, when it arrives and who sends it. Three timelines show the difference more clearly than three definitions:

Model-output streaming works today, subscriptions are mature, and incremental delivery is not available on this stack yet. Model-output streaming is one model answer that arrives in pieces. A subscription is many separate events that arrive over one long-lived operation. Incremental delivery is the response to one query, arriving in parts. A design that confuses any two of them expects a guarantee that the transport does not give.

Model-Output Streaming: We Already Have It​

Class 7 ended with the stream() terminal of the ChatClient, and we look at it again here. The answer arrives as a Reactor Flux<String>. When the model decides that it needs a tool, the advisor pauses the stream and runs the tool call. The call goes through the same callbacks as before, with the approval gate and the budget. The advisor then carries on with the rest of the model's answer. The user sees the answer pause and continue, which matches what the server is doing at that moment.

A Flux is the same type that the rest of reactive Spring uses. To send the stream to a browser over Server-Sent Events, a WebFlux controller returns that Flux. To send it over a WebSocket, we pass the same Flux along. The model output reaches the user without an adapter, a buffering layer or a new type.

There is one limit, which Class 7 pointed out. As of Spring AI 2.0, structured entities work only with call(). A streamed answer is therefore free text, or chunks that we join ourselves. When we want both live text and a typed result, act-then-format from Class 8 becomes act, stream, then format.

Subscriptions: The Production Push Channel​

For data that changes between requests, GraphQL has had a stable answer for years: subscriptions. A subscription is a long-lived operation that delivers events as they occur. The Movie Database service already has them. The subscriptions lesson of our Spring GraphQL tutorial built review notifications on Flux publishers. They work over WebSocket, and over without extra configuration in Spring for GraphQL.

Connecting subscriptions to an agent needs care. The obvious idea is to subscribe the agent to events, and that idea hides a mismatch. An agent conversation is request-driven: the model runs when something calls it, and its context is built for that call. A subscription is push-driven: events arrive at any moment, including when the agent is idle.

To connect push events to a request-driven agent, we have to decide three things:

  • when an event invokes the agent
  • which collected context the agent receives
  • whose budget pays for the run

These are orchestration decisions. Queues and triggers make them, and so do workflow patterns like the checkpointed graphs of Class 6. A transport feature cannot make them. A simple pattern works for many systems. Subscriptions feed the application: its UIs, caches and event logs. The application then decides when an event should wake an agent, as it would for any other worker.

@defer: Where It Actually Stands​

Now we turn to incremental delivery, which @defer and @stream promise. We check its status in three places, because a feature can exist in one of them and be missing from the next.

In the specification, incremental delivery is still a proposal. The current published GraphQL spec edition (September 2025) defines five directives: @skip, @include, @deprecated, @specifiedBy and @oneOf. @defer and @stream do not appear in it. The proposal is PR #1110 in the spec repository, at RFC stage 2. @defer comes first, and @stream will follow. The HTTP transport for it, multipart responses, is also an RFC, in the graphql-over-http repository.

In graphql-java, @defer has been an explicitly experimental feature since version 22 (April 2024), and it is still experimental in the current 26.x line. It is off by default, and a context flag turns it on for one execution. With the flag on, execution returns an IncrementalExecutionResult, and the deferred payloads arrive through a Reactive Streams Publisher. The payload format follows an older draft of the proposal, from before deduplication was added. That matters when a modern client reads it. @stream is not implemented at all: the engine's internal StreamedCall.invoke() throws an UnsupportedOperationException.

In Spring for GraphQL, incremental delivery is missing as of 2.0. The server transports and GraphQlClient both lack support for the multipart format and the incremental result type. The engine can defer, and the web layer cannot carry the result yet.

Our project turns the first half of that claim, that graphql-java can defer, into something you can run. The demo is a self-contained graphql-java schema with these parts:

  • one fast field and one deliberately slow field
  • a query that defers the slow fragment
  • the experimental flag, turned on
  • a timestamp on each payload
The complete DeferDemo.java, collapsed because it runs to 72 lines
src/main/java/com/graphqlguy/moviedb/agents/incremental/DeferDemo.java (new file)
package com.graphqlguy.moviedb.agents.incremental;

import graphql.ExecutionInput;
import graphql.ExperimentalApi;
import graphql.ExecutionResult;
import graphql.GraphQL;
import graphql.incremental.DelayedIncrementalPartialResult;
import graphql.incremental.IncrementalExecutionResult;
import graphql.schema.GraphQLSchema;
import graphql.schema.idl.RuntimeWiring;
import graphql.schema.idl.SchemaGenerator;
import graphql.schema.idl.SchemaParser;
import org.reactivestreams.Subscriber;
import org.reactivestreams.Subscription;

import java.util.concurrent.CountDownLatch;

/**
 * A minimal, self-contained demonstration that graphql-java's engine can defer,
 * independent of any web framework. One fast field, one deliberately slow field,
 * a query deferring the slow one, and timestamps proving the initial result
 * arrives immediately while the deferred part follows seconds later.
 *
 * Everything here is explicitly experimental: the capability is switched on per
 * execution through a context flag, the deferred work only starts when a
 * subscriber attaches to the publisher, and the wire format the payloads mirror
 * is an older draft of the incremental-delivery proposal. That is the state of
 * @defer on the JVM as of August 2026.
 */
public class DeferDemo {

    private static final String SDL = """
            type Query {
              "Available immediately."
              headline: String
              "Takes two seconds; the part worth deferring."
              analysis: String
            }
            """;

    public void run() throws Exception {
        long start = System.currentTimeMillis();
        RuntimeWiring wiring = RuntimeWiring.newRuntimeWiring()
                .type("Query", type -> type
                        .dataFetcher("headline", env -> "The catalog holds 12 movies.")
                        .dataFetcher("analysis", env -> {
                            Thread.sleep(2000); // a slow resolver, so its field is deferred
                            return "Dramas dominate; ratings cluster between 8.5 and 9.3.";
                        }))
                .build();
        GraphQLSchema schema = new SchemaGenerator().makeExecutableSchema(
                new SchemaParser().parse(SDL), wiring);
        GraphQL graphql = GraphQL.newGraphQL(schema).build();

        ExecutionInput input = ExecutionInput.newExecutionInput()
                .query("query { headline ... @defer { analysis } }")
                .graphQLContext(context -> context.put(
                        ExperimentalApi.ENABLE_INCREMENTAL_SUPPORT, true))
                .build();

        ExecutionResult result = graphql.execute(input);
        System.out.printf("[t=%4dms] initial payload : %s%n",
                System.currentTimeMillis() - start, result.toSpecification());

        if (result instanceof IncrementalExecutionResult incremental) {
            CountDownLatch done = new CountDownLatch(1);
            incremental.getIncrementalItemPublisher().subscribe(
                    new Subscriber<DelayedIncrementalPartialResult>() {
                        public void onSubscribe(Subscription subscription) {
                    subscription.request(Long.MAX_VALUE);
                }
                        public void onNext(DelayedIncrementalPartialResult payload) {
                            System.out.printf("[t=%4dms] deferred payload: %s%n",
                                    System.currentTimeMillis() - start, payload.toSpecification());
                        }
                        public void onError(Throwable failure) {
                    done.countDown();
                }
                        public void onComplete() { done.countDown(); }
                    });
            done.await();
        }
    }
}

Three details in this code matter:

  • The graphQLContext flag turns the feature on for one execution at a time. In practice, that is what "experimental" means here.
  • The result is an IncrementalExecutionResult only when something was deferred, which is why the code uses a pattern match.
  • The deferred work starts when a subscriber attaches to the publisher. If a client reads the initial payload and stops there, the slow resolver does not run.

A new menu command runs it:

src/main/java/com/graphqlguy/moviedb/agents/command/DeferDemoCommand.java (new file)
package com.graphqlguy.moviedb.agents.command;

import com.graphqlguy.moviedb.agents.incremental.DeferDemo;
import org.springframework.stereotype.Component;

import java.util.List;

/**
 * Class 9: graphql-java's experimental @defer, run against a self-contained schema
 * so the timestamps show the initial payload arriving before the deferred one.
 * This command runs without the Movie Database service and without a model.
 */
@Component
public class DeferDemoCommand implements AgentCommand {

    @Override
    public String name() {
        return "defer-demo";
    }

    @Override
    public int order() {
        return 90;
    }

    @Override
    public String summary() {
        return "graphql-java's experimental @defer, timestamped";
    }

    @Override
    public void run(List<String> args, CommandContext context) throws Exception {
        new DeferDemo().run();
    }
}
[t=  41ms] initial payload : {data={headline=The catalog holds 12 movies.}, hasNext=true}
[t=2050ms] deferred payload: {hasNext=false, incremental=[{path=[],
            data={analysis=Dramas dominate; ratings cluster between 8.5 and 9.3.}}]}

Check It Works​

mvn clean spring-boot:run

Choose defer-demo from the menu.

This demo runs without the service and without Ollama, because the schema and both resolvers are inside the Java class. Check the timestamps: the first line prints at once with hasNext=true, and the second line arrives about two seconds later.

The initial payload arrives after 41 milliseconds with hasNext: true, which tells the client that more is coming. The two-second field arrives when it is ready, with a path that says where it belongs. So the mechanism in the engine works on your own machine today. The comments in the demo point out one experimental detail: the deferred work starts only when a subscriber attaches to the incremental publisher. Code that treats the result as a plain value will not expect that.

Lesser-known fact

@defer has been discussed, shown in demos and shipped behind flags for a long time, in graphql-js, Apollo and graphql-java. As a result, many teams assume it is part of GraphQL. The September 2025 spec edition lists five directives, and @defer and @stream are missing from that list. How often people talk about a feature says little about how far it has moved through the specification process.

What Agents Should Do Meanwhile​

@defer would give an agent something useful: the fast half of a tool result would reach the model at once, and the slow half would follow. That shortens the wait at the tool call, which is where agent loops do their waiting. We can get the same benefit today with a design this course has used since Class 3: split the work at the tool boundary. A @defer user splits one query into a fast fragment and a deferred one. We split the tool instead, into a fast tool that the agent calls first and a slow tool that it calls when it needs the rest.

Several earlier lessons already use this shape:

  • The search-then-introspect tools of Class 3 follow it.
  • Act-then-format from Class 8 applies it to the output side.
  • The curated operations of Class 2 keep each half cheap and predictable.

When Spring for GraphQL adds incremental support, moving to it will be straightforward. The hard part is deciding how to split the work, and that will already be done.

Summary​

The word streaming covers three different promises, and they are at three different stages:

PromiseStatus on the JVMWhat carries it
Model-output streamingworks todaya Flux from the ChatClient, tools executing mid-stream, delivered over SSE or WebSocket
SubscriptionsmatureGraphQL's push channel; the application decides when to wake an agent
Incremental deliverynot here yet@defer is a stage-2 RFC, experimental in graphql-java; @stream is unimplemented

Model-output streaming cannot return entities, so we use act-then-format when we need a typed result. Subscriptions feed the application, and the application decides when an agent should wake. Incremental delivery needed the closest look, because the gap between its reputation and its readiness is wide:

  • @defer is a stage-2 spec RFC, and graphql-java supports it as an experiment with an older payload format. This lesson ran it, with the initial payload at 41ms and the deferred one at two seconds.
  • @stream is not implemented, and the engine throws an UnsupportedOperationException for it.
  • Spring for GraphQL 2.0 does not support either of them.

Until that changes, splitting the work at the tool boundary gives the same latency benefit with the tools we have. It is the pattern this course has used since Class 3.

Further Reading​

What's Next​

Streams, tools and typed answers all cost tokens. By default, the provider reports those tokens as one monthly figure, without the run, tenant or feature they belong to. Class 10 builds the accounting:

  • usage reported by the provider, turned into cost per run
  • Micrometer meters that show cost per tenant
  • a rate-based alert that catches a runaway loop in its first minute