Class 3: Schema Navigation at Scale
Duration: ~140 minutes | Difficulty: Advanced | Prerequisites: Class 2 completed; Java 21 and Maven; Ollama installed for local embedding inference. No API key is needed anywhere in this lesson.
This lesson has its own project, separate from the rest of the course: a standalone Spring Boot application we build and run. The lesson calls it the instrument, because it exists to measure. Every command reports either a cost in tokens or a score. Clone it and stay on main, which carries the project skeleton, the Movie Database schema, and a small labelled benchmark file:
git clone https://github.com/graphqlguy/graphQLSchemaNav.git
The finished instrument lives on the schemanav_class_3 branch, with the schema renderer, both retrieval backends, the benchmark harness and the agent loop in place. Build it yourself from main as the lesson goes, or check out that branch to read the finished code and run the measurements straight away:
git clone --branch schemanav_class_3 https://github.com/graphqlguy/graphQLSchemaNav.git
Every number quoted below was produced by running it, and we will reproduce each one as we go.
By the end of this lesson we will have built a search layer over a schema so large that sending it whole to a model is wasteful. Our example is GitHub's public GraphQL API, one of the largest public API schemas anyone can download. The pinned snapshot the benchmark uses is 72,484 lines of SDL, and the live schema has grown past 74,000 lines since. Two other things make it a good example. It is public, so we can check every claim here against the real thing, and many engineers have used it, so its field names mean something as we read them.
A schema this size fits inside the context window of most modern models. The difficulty is what sending it costs. Filling the window with an API description leaves less room for the conversation, and the same tokens are paid for again on every call. Most of what was sent is irrelevant to the task, and later in this lesson we measure how much it costs. This diagram shows everything we are about to build:
We will build the search layer twice, which is why the index box in the middle names two options. The first version ranks with Apache Lucene's BM25, which counts words instead of using a model. The second ranks with Qwen3-Embedding-0.6B-GraphQL, an embedding model tuned for GraphQL schemas and small enough to run on your laptop.
We will then measure both against labelled benchmark data proposed to the GraphQL Foundation's AI Working Group, so the numbers we finish with are our own instead of a vendor's. The lesson ends with a local LLM navigating the schema through the three tools in the bottom-right box, while a meter reports what every step costs in tokens. Class 2 ended by naming the search/introspect/execute pattern. This lesson builds that pattern as a working application and measures it.
That diagram already uses a dozen words that may be new. Most come from information retrieval, the branch of computer science concerned with finding relevant items in a large collection. Here is a short definition of each. Each one gets a fuller explanation later, where the lesson puts it to work.
- Schema Definition Language (SDL) is the text form of a GraphQL schema, the
type Movie { ... }syntax you already write. - Introspection is asking a GraphQL server to describe its own schema, in a query, at runtime.
- Schema coordinate is the pair of a type name and a field name written
Type.field, such asMovie.rating, pointing to exactly one field. - Corpus is the collection of documents a search looks through. Ours holds one short document per schema coordinate.
- Index is the data structure a search builds over a corpus so that it can find matches quickly.
- Apache Lucene is the open-source Java search library this lesson uses to build one of the two indexes.
- BM25 is a ranking function that scores a document against a question from word statistics alone.
- Embedding is a list of numbers produced from a piece of text by a model, arranged so that texts with similar meanings get similar numbers.
- Cosine similarity is the arithmetic that measures how close two embeddings are, scoring near 1 for texts that mean much the same and near 0 for unrelated ones.
- recall@k measures how complete a search result is. A task needs a certain set of items; recall@k is the share of that set which appears in the first k results. Finding three of four needed items scores 0.75.
- MRR (mean reciprocal rank) is a score that rewards putting the right answer near the top of the list instead of merely somewhere in it.
- Benchmark harness is the part of the instrument that runs a file of labelled questions through a search backend and reports recall, MRR and token cost.
- Token is the chunk of text a language model reads as a single piece, roughly three quarters of an English word. Models are billed per token, and every cost in this lesson is counted in them.
The instrument is a Java and Spring Boot application, and everything this lesson compares is a configuration property:
- which schema to load
- which retrieval backend ranks the results
- which of four formats renders the corpus
Each one is a flag on the command line. After the lesson, you can point the same application at your own schema at work and re-run every measurement.
The Problem: What a Whole Schema Costs to Send
Class 2 generated tool definitions from a schema by walking its introspection result. That works while the schema is small enough that the resulting catalog, or the schema itself, fits comfortably into the model's context window. Production schemas break this assumption, and the numbers below show by how much.
One Field, One Corpus Entry
First, here is exactly what is being counted, on a schema small enough to read in full. The instrument ships a trimmed Movie Database schema, a slice of this course's running example. One of its types starts like this:
"A feature film in the catalog."
type Movie {
"The film's display title."
title: String!
"Average review rating from 1 to 10, null while unreviewed."
rating: Float
}
Movie.rating is a schema coordinate: the pair of a type name and a field name, written Type.field, pointing to exactly one field in the schema. A search over the schema returns coordinates. So the instrument reads the schema once and turns every coordinate into one searchable entry. Each entry rewrites the field as a plain sentence, because questions arrive in plain words, and an entry has to share some of those words:
Movie.rating - the rating of a movie
That one sentence is a single entry. The whole set of them, one entry for every coordinate in the schema, is the corpus. Corpus is the standard information-retrieval term for the collection of documents a search looks through. Everything this lesson measures is measured against a corpus. The unusual part is what counts as a document here: one GraphQL field, instead of a web page or an article.
That entry costs 9 tokens. The trimmed Movie DB schema has 44 coordinates altogether, so its whole corpus costs about 445 tokens, which fits in any model's context window with room to spare for the conversation. On a schema that small the problem does not exist, and that is why the rest of this lesson works on a much larger one. Point the same corpus command at GitHub's public schema and it reports 6,640 coordinates and 102,203 tokens.
What a Whole Schema Costs, Three Ways
The GitHub corpus alone fills half of a 200,000-token context window, and a tenth of a million-token window, before the agent reads a single instruction. It is also the cheapest of three ways to show a schema to a model. The corpus leaves information out. It holds one flat sentence per output field, without argument lists, input types, enum values or type-level descriptions. Published SDL carries all of those, so it costs more. The introspection result for the same schema costs more again, because its JSON spells out structure that SDL leaves implicit:
Shopify's Admin API needs a million and a half tokens as introspection JSON, before any conversation is added. GitHub's figures are smaller and more typical. A model with a million-token window could hold GitHub's introspection result, and it would spend about three quarters of that window on the API description, before anyone asks the agent anything. Introspection is the most expensive of the three ways, and it is the route Class 2 used.
The Same Task Through Three Interfaces
ChilliCream measured and published the next set of figures. In an April 2026 write-up, they ran one agent task through three styles of interface and reported how many tokens the model had to read in each. The bars below follow this order:
- GraphQL with a search layer over the schema
- a plain GraphQL schema
- OpenAPI
Only the two GraphQL bars fit inside a 200,000-token window at all. The smaller of those two, GraphQL with a search layer over it, leaves almost the whole window free for the conversation the agent is there to have.
The MCP course's Class 4b covers how this problem appears through the Model Context Protocol and how Apollo MCP Server's built-in tools address it with configuration. This lesson goes underneath that configuration and builds the retrieval layer itself. Building it lets us answer three questions that the configuration hides:
- How should a schema field be rendered so that a search can find it?
- Is a plain word-counting index enough, or does an embedding model justify its extra complexity?
- How good are the results when we measure them?
What Retrieval Means Here, Precisely
Every measurement in the rest of the lesson uses four ideas, and each needs more than the one-line definition above.
The corpus is the collection of entries a search looks through. There is one corpus, and it holds one entry per schema coordinate.
Why the coordinate, rather than something larger or smaller? Every other choice in this lesson follows from that one. A coordinate is finer than a type. Suppose an agent has to answer "when did I place this order". The one field it needs is Order.createdAt. If the search returns the whole Order type, the answer is in there among eighty other fields, and the agent pays for all of them. A coordinate is also the unit an agent writes queries with. A GraphQL operation is a set of coordinates arranged into a tree, so a search that returns coordinates hands the agent the pieces it is about to assemble.
Ranking is what a search backend does: given a question, it orders every corpus entry from most to least relevant, by some definition of relevant. The two backends in this lesson differ only in that definition.
recall@k measures how complete a search result is. Every task needs a particular set of coordinates, and recall@k is the share of that set appearing in the first k results. It is a fraction instead of a yes or no, because a real task needs several coordinates at once.
MRR, or mean reciprocal rank, measures how high the right answers ranked. The score for one question is 1 divided by the rank of the first correct hit. A correct first hit scores 1, a correct second hit scores 1/2, and a correct tenth hit scores 1/10.
recall@k and MRR, Computed
Both scores are easier to see computed than defined. Finding recent open pull requests on GitHub's schema needs four coordinates at minimum: Query.repository, Repository.pullRequests, PullRequest.title and PullRequest.author. Suppose a search returns this:
| Rank | Result returned | One of the four needed? |
|---|---|---|
| 1 | Issue.title | no |
| 2 | PullRequest.title | yes |
| 3 | Repository.pullRequests | yes |
| 4 | PullRequest.bodyText | no |
| 5 | PullRequest.author | yes |
| never returned | Query.repository | yes, and it is missing |
Three of the four needed coordinates appear in the first five results, so recall@5 for this question is 0.75. The first correct hit appeared at rank 2, so this question contributes 1/2 to MRR. (The ranks above are invented to show the arithmetic. Every measured figure later in the lesson came from a real run.)
Averaged over many labelled questions, each score stops describing one question and starts describing the backend. recall@1 asks whether the first hit was right. recall@50 asks whether the right answers turned up anywhere in a fifty-deep reading. They disagree in a useful way. recall@50 credits an answer at rank 50 the same as one at rank 1, and MRR is the score that notices the difference.
Why the Task Is Hard
The task is hard because schema fields repeat names across owner types: assignee exists on issues, on pull requests, and on project items; title, name, and createdAt exist almost everywhere. A question rarely names its coordinate. It names an intent, and the hard part of an intent is working out which type it belongs to.
The published model card for the embedding model we use below records one case exactly:
| Question | Untuned base model's top hit | Tuned model's top hit |
|---|---|---|
| "When did I place this order?" | Order.completedAt | Order.createdAt |
Both candidates are on the right type, so this case is about choosing the field. Both fields are plausible, and their meanings differ in exactly the way the question cares about. General-purpose text embeddings are trained on ordinary written language, where the difference between "created" and "completed" rarely matters. Fine-tuning trains that same model further, on questions paired with the coordinates that answer them, and that is how it learns to tell two such fields apart.
The Instrument
The instrument is a command-line Spring Boot application with three commands we will use throughout the lesson, plus one we will meet at the end.
# What does this schema cost to show to a model, per corpus format?
mvn -q spring-boot:run -Dspring-boot.run.arguments=corpus
# One question against the configured backend, with token costs on every hit
mvn -q spring-boot:run -Dspring-boot.run.arguments="search movies with a high rating"
# Labelled-query benchmark: recall, MRR, tokens, and the context-budget sweep
mvn -q spring-boot:run -Dspring-boot.run.arguments=bench
Every command runs the pipeline drawn at the top of this lesson, and configuration can swap every stage of it. A single interface makes that possible. The benchmark harness, the search command and the agent tools all depend only on this one type:
public interface SearchBackend {
/** Builds (or rebuilds) the index over the given corpus. */
void index(List<CorpusEntry> corpus);
/** Returns the top-k entries for the query, best first. */
List<SearchHit> search(String query, int k);
String name();
}
The third method is the one usually left out of an interface like this. name() lets a result row label itself. That is what allows us to run the same benchmark against BM25, then against two different embedding models, and lay the three outputs side by side without tracking which was which. The code above this interface cannot tell whether ranking came from word statistics or from vectors. So switching backends is a configuration change, and the code stays the same:
mvn -q spring-boot:run \
-Dspring-boot.run.arguments="bench --schemanav.retrieval.backend=bm25"
mvn -q spring-boot:run \
-Dspring-boot.run.arguments="bench --schemanav.retrieval.backend=embedding --spring.ai.ollama.embedding.model=qwen3-graphql-embedder"
qwen3-graphql-embedder is the local Ollama name for the fine-tuned model, and the embedding section below is where we pull it and name it. Both the backend and the model are command-line overrides, so we can compare two retrieval strategies without touching the code.
Every screen the instrument prints shows one design decision: every result carries its token count. Token counts depend on the tokenizer, and OpenAI, Anthropic and Qwen models each split text differently. So the instrument uses one encoding, JTokkit's O200K_BASE, for every count, and names it wherever it prints a number. That makes each absolute number an approximation of what a particular provider would bill you. The comparisons between backends, formats and budgets stay exact, and comparisons are what this lesson is about. The goal behind every one of them is the same: the most accuracy for the fewest tokens.
Rendering the Corpus: Formatting Matters as Much as the Model
The corpus generator walks the schema with graphql-java and turns every field coordinate it finds into indexable text. It visits interface types as well as object types. On GitHub's schema, many of the most-used fields are declared on interfaces such as Node and Actor, so a generator that skipped interfaces would miss them. The generator supports four renderings. The choice of rendering matters as much as the choice of search backend, and the numbers below show it.
The Four Renderings
The four differ only in how much of the field they write down:
| Format | What goes into the corpus | The entry for Movie.rating |
|---|---|---|
RAW | the coordinate on its own | Movie.rating |
DOT_PLUS_GLOSS | the coordinate plus a gloss built from the coordinate itself, camel case split into words. It does not read the schema. | Movie.rating - the rating of a movie |
GLOSS | one plain sentence naming the owner type, the return type and the description | GraphQL field Movie.rating. Owner type: Movie. Returns: Float. Average review rating from 1 to 10, null while unreviewed. |
SDL | the field's own definition in schema syntax, description included | "Average review rating from 1 to 10, null while unreviewed." rating: Float |
Choosing between them is one switch, with a branch per format:
return switch (format) {
case RAW -> coordinate;
case DOT_PLUS_GLOSS -> coordinate + " - the " + humanize(field.getName())
+ " of a " + humanize(owner.getName());
case GLOSS -> {
StringBuilder gloss = new StringBuilder();
gloss.append("GraphQL field ").append(coordinate).append(". Owner type: ")
.append(owner.getName()).append(". Returns: ").append(returnType).append(".");
if (!description.isEmpty()) {
gloss.append(" ").append(description);
}
yield gloss.toString();
}
case SDL -> {
StringBuilder sdl = new StringBuilder();
if (!description.isEmpty()) {
sdl.append("\"\"\"").append(description).append("\"\"\"\n");
}
sdl.append(field.getName());
if (!field.getArguments().isEmpty()) {
List<String> args = field.getArguments().stream()
.map(a -> a.getName() + ": " + GraphQLTypeUtil.simplePrint(a.getType()))
.toList();
sdl.append("(").append(String.join(", ", args)).append(")");
}
sdl.append(": ").append(returnType).append(" # on type ").append(owner.getName());
yield sdl.toString();
}
};
humanize is the whole of what DOT_PLUS_GLOSS needs, and it does not look at the schema:
private String humanize(String identifier) {
return identifier
.replaceAll("([a-z0-9])([A-Z])", "$1 $2")
.replaceAll("([A-Z]+)([A-Z][a-z])", "$1 $2")
.replace('_', ' ')
.toLowerCase();
}
What Each Costs, and How Well It Searches
The two charts below show what each rendering costs and how well it searches. Both use the same row order, so read them together:
Credit for the Four Formats
Thore Koritzius published an evaluation alongside the Qwen3-Embedding-0.6B-GraphQL model card that compares twelve different renderings of the same coordinate against GitHub's schema and reports what each one scores. Two of the names here are that work's own: sdl and dot_plus_gloss. RAW is our name for what the card calls dot, and GLOSS is our name for what it calls semantic.
Four formats are enough to show the principle. The other eight are in the card's own comparison, which is worth reading in full. It also measures what happens when parts of the text are removed. With the owner type dropped and only the field name kept, MRR falls to about 0.05. That is strong evidence that the owner type carries most of the signal.
Every figure in the two charts above was produced by running the instrument against the working group's GitHub slice, so they are our numbers. They do not replace the card's numbers, because the two studies asked different questions of different data. The differences are what a second measurement adds:
- A different schema. The card benchmarks against octokit/graphql-schema, which is 64,309 lines as we fetched it. Our runs use the working group's pinned snapshot, which is 72,484 lines. They are not the same file.
- A different query set. The card uses 52 held-out queries. We use the working group's 200, scored against 1,038 plain-field targets.
- A different
SDLrenderer, under the same name. The card'ssdlwraps the field in type syntax,type PullRequest { baseRefName: String! }. Ours emits the description block and marks the owner in a trailing comment,"""Star rating from 1 to 10."""followed byrating: Int! # on type Review.
The third difference matters most when reading our chart. Our SDL places third where the card ranks its sdl first. Those are two different renderings that share a name, scored on different schemas against different questions. Neither result overturns the other. Together they show that the name of a corpus format does not tell you what text it produces. Print the text that actually reaches your index before trusting any published ranking of formats, ours included.
Reproducing the card's ranking directly would need its 52 queries, which it does not publish. If you have a labelled set for your own schema, the instrument runs the same comparison on it in one command, and that third data point would be more useful than either study here.
Where this lesson quotes the card directly, it says so.
Why a Bare Coordinate Cannot Be Found
DOT_PLUS_GLOSS reaches an MRR of 0.675 where RAW reaches 0.541. It also moves the 95th-percentile rank of the right answer from 46 down to 24, much nearer the top of the list. The model was the same in every run, so the rendering alone accounts for the gap. A bare Type.field string gives a text model very few words to match. Splitting baseRefName into "base ref name" and naming the owner type supplies the words that a natural-language question is likely to use.
The instrument shows a stronger version of this finding with one command: run bench with --schemanav.corpus.format=RAW against the lexical backend, and every metric comes back zero.
The zeros have a precise cause. Lucene's StandardTokenizer implements the Unicode word-boundary rules (UAX #29), and those rules treat a full stop between letters as part of the word, the same way they keep example.com whole. So Query.movies is indexed as the single token query.movies. No natural-language question will ever contain that token, so a lexical search over raw coordinates cannot match a single entry. The same rule is why the tantivy indexes inside Apollo MCP Server and Grafbase index rendered text instead of bare coordinates.
This applies to any retrieval system. Before switching to a better model, look at what the corpus says. Rewriting the corpus is usually the cheapest improvement. If the corpus lacks the words that the questions use, no later stage can add them back.
Backend One: BM25, Search Without a Model
The first backend uses only word statistics. It sets the baseline that an embedding model has to beat to justify its cost.
BM25 is short for Best Matching 25, the twenty-fifth in a line of ranking functions the Okapi information-retrieval project published from the 1970s onward. The arithmetic is simple enough to describe in a few lines.
BM25 scores a corpus entry against a question using three observations:
- Repetition counts, with diminishing returns. An entry containing a query word more often is more likely to be about it, and the fifth occurrence of "rating" adds less than the first.
- Rare words carry more information than common ones. Matching "pullRequests" narrows the results a lot, and matching "the" does not narrow them at all, so each word's contribution is weighted by how few entries contain it.
- Long entries get a handicap, because a long text matches everything a little.
The score adds these contributions up over the words of the question. BM25 was the state of the art for decades. When the question and the answer share their words, it still outranks systems that cost far more to run.
One of the instrument's own results shows the three observations working together and getting the answer wrong. Asked for "movies with a high rating from the nineties" on the small schema, BM25's top hit was Mutation.createMovie. Its gloss is one of the shortest in the corpus that mentions a movie at all, so the length handicap scored it higher than it deserved. The correct Query.movies, whose gloss talks about filtering by year range and minimum rating, came fourth. The arithmetic cannot tell that creating a movie is the opposite of browsing movies, because word statistics rank by the words alone. Remember this example, because the second backend exists to avoid this failure.
In the instrument the index is Apache Lucene, whose default similarity has been BM25 for years, so the integration takes a few lines:
Document doc = new Document();
doc.add(new TextField("text", entry.text(), Field.Store.YES));
doc.add(new StoredField("coordinate", entry.coordinate()));
writer.addDocument(doc);
// ... and at query time:
TopDocs top = searcher.search(parser.parse(QueryParser.escape(question)), k);
Lucene is worth learning here even if you do not deploy it, because the production indexes in this space follow its design. The tantivy library that Apollo MCP Server and Grafbase embed is an explicit reimplementation of Lucene's design in Rust, and Hot Chocolate 16's semantic introspection ranks with BM25 by default. Understanding this backend explains what those servers do behind their configuration.
How far do word statistics get on a real schema? On the GitHub benchmark below, BM25 over the DOT_PLUS_GLOSS corpus reaches recall@50 of 0.377: a third of the needed coordinates, reading fifty results deep. On the small Movie DB schema it looks better, at recall@10 of 0.60 on our smoke set, and the gap between those two figures is what this lesson is about. Lexical search gets worse as schemas grow. A larger schema has more fields with similar words, and word counting cannot tell them apart. The createMovie mistake above happened among forty-four coordinates. Among six and a half thousand, similar mistakes happen in most queries.
Backend Two: A GraphQL-Native Embedding Model
To do better than counting words, the search needs a notion of meaning, and the standard way to get one is an embedding model. An embedding model turns a piece of text into a list of numbers, which can be read as a point in a space. It is trained so that texts with similar meaning land near each other.
Meaning as Position
Picture a map with only two directions, one measuring how much a text is about time and the other how much it is about people. On that map, a question and the field that answers it land in the same corner even when they share hardly a word:
The question and the field description sit together in the top-left, because both are about time. The question about the director sits far away, because it is about a person. A real model works the same way with 1,024 directions instead of two, learned from data instead of chosen by hand. Similar meanings get nearby positions, and a search finds the nearest entries.
In practice that means embedding every corpus entry once, embedding the question, and ranking entries by how close they sit to it. Closeness is measured with cosine similarity, which compares the direction of two vectors and ignores their length. Two texts pointing the same way score near 1, and unrelated ones score near 0.
The Model We Use
Qwen3-Embedding-0.6B-GraphQL is published under Apache 2.0 and was announced on graphql.org in July 2026. It is a full fine-tune of Alibaba's Qwen3-Embedding-0.6B, a 0.6-billion-parameter embedding model. It was trained on 4,788 synthetic question-to-coordinate pairs over a 28,893-coordinate corpus. The training goal was to map natural-language questions about capabilities to Type.field coordinates, including picking the right owner type. It is one engineer's early prototype, offered as a reference point. This lesson treats it that way: we run it, report the results, and treat its numbers as provisional.
The fine-tune took about sixteen minutes on a single H100 GPU. Fine-tuning a small embedding model is among the cheapest ways to specialize one, and it usually costs far less than labelling the data it trains on. The model card's training recipe (sentence-transformers, contrastive loss, four mined hard negatives per example) can be reproduced on a rented GPU for a few dollars. If your domain has a vocabulary that general models handle poorly, one engineer can do this in an afternoon.
Running It Locally
Two commands are enough, because the model ships as GGUF builds and Ollama pulls straight from Hugging Face:
ollama pull hf.co/xthor/Qwen3-Embedding-0.6B-GraphQL:Q8_0
ollama cp hf.co/xthor/Qwen3-Embedding-0.6B-GraphQL:Q8_0 qwen3-graphql-embedder
The recommended q8_0 build is about 650 MB on disk, runs in roughly 1 to 1.5 GB of memory, and embeds a short text in tens of milliseconds on a laptop CPU. Spring AI's Ollama starter exposes it as an EmbeddingModel bean, and the backend's search is a plain loop over float arrays. With 6,640 vectors in memory, a linear scan finishes in milliseconds. A vector database would add a dependency to operate and hide the arithmetic, and at this size it would not make the search faster.
float[] queryVector = embeddingModel.embed(queryPrefix + query);
for (int i = 0; i < entries.size(); i++) {
double score = cosine(queryVector, vectors.get(i));
// collect, sort descending, take the top k
}
Notice the queryPrefix in that snippet. The model is instruction-tuned. Its own configuration embeds questions behind the prefix Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:, and embeds corpus entries as plain text. The two produce different vectors even for identical strings. Python users select it with sentence-transformers' prompt_name="query", which pulls the string out of the model's own configuration instead of hardcoding it. Ollama's /api/embed takes model, input, truncate, options, keep_alive and dimensions, and none of those fields holds an instruction. So the instrument keeps the literal prefix in a configuration property, in case you swap models, and adds it in front of each question.
Nothing applies it for you in either ecosystem. This model ships "default_prompt_name": null, so a Python caller who forgets prompt_name gets the same silent degradation a Java caller does. Skipping the prefix does not cause an error. It lowers accuracy while everything looks normal, which makes the mistake hard to notice. Qwen puts the cost at roughly 1% to 5% of retrieval performance.
A model that works this way documents how to wrap its inputs, and this one does it in one file. Its config_sentence_transformers.json declares two prompts, "query" holding the Instruct: wrapper and "document" holding the empty string, which is why questions are wrapped and corpus entries are not. The instrument's query-prefix and document-prefix properties are those two values copied across. The Qwen3-Embedding model card says the same thing in words: every query carries a one-sentence instruction, and retrieval documents do not need one. For any embedding model you use, read that file or that section of its card before you trust its scores.
Indexing 6,640 coordinates through Ollama takes a few minutes, once. The instrument batches the calls and caches the vectors on disk, one entry for each combination of model, prefix and corpus content, so every later run starts in milliseconds.
Measuring It: The Working Group's Benchmark Data
Judging retrieval quality by impression is unreliable, so the AI Working Group built labelled data to measure it against. It holds 816 natural-language questions across five production schemas: GitHub, GitLab, Linear, Shopify and Singapore's Open Data platform. Each question is labelled with the coordinates a correct answer must touch. As of August 2026, the work has reached the working group as two pull requests. The Qwen3 embedding comparison merged in June 2026 as #142, and the evaluation suite that scores it is still open as #140. The instrument's fetch script downloads the GitHub slice, 200 questions with 1,038 plain-field target coordinates, together with the pinned schema snapshot the labels refer to.
./scripts/fetch-wg-benchmark.sh
mvn -q spring-boot:run \
-Dspring-boot.run.arguments="bench --schemanav.schema.source=schemas/github-wg.graphqls --schemanav.bench.file=benchmarks/github-wg.tsv"
One Labelled Question Per Line
The benchmark file is deliberately simple, so that we can edit and extend it by hand. Each line holds one labelled question: the question, a tab, and then the coordinates a correct answer has to surface, separated by commas when a task needs several.
# One labelled query per line: natural-language question <TAB> expected Type.field
Which movies from the 1990s have a high rating? Query.movies
When did this movie come out? Movie.releaseYear
Who directed this film? Movie.directors
To point the instrument at your own schema, write a file in this format: one row per question you care about, with the coordinates you would expect an engineer to use. The loader turns each row into a record, and refuses anything malformed:
/**
* One benchmark row: a natural-language question and the schema coordinates a correct
* retrieval should surface (real tasks usually need several fields, so the ground
* truth is a list).
*/
public record LabelledQuery(String question, List<String> expectedCoordinates) {
}
What the Two Backends Score
With the benchmark rows in that shape, the measurement loop is only a few lines of logic. For each question, ask the backend for its top 100 hits, intersect those coordinates with the expected set, and average the fraction found across every question. That average is recall@k, and running it for both backends is what produces the table below (DOT_PLUS_GLOSS corpus, the same encoding throughout):
The same run as a table, with the token cost of a fifty-deep reading:
| GitHub schema, 200 questions | recall@1 | recall@5 | recall@10 | recall@20 | recall@50 | tokens@50 |
|---|---|---|---|---|---|---|
| BM25 (Lucene) | 0.098 | 0.191 | 0.245 | 0.303 | 0.377 | 707 |
| Qwen3 GraphQL fine-tune | 0.114 | 0.284 | 0.367 | 0.439 | 0.549 | 651 |
The model's announcement publishes results on the full five-schema benchmark. Field recall@50 comes out as follows:
- the fine-tuned model: 63.6%
- OpenAI's text-embedding-3-large, a hosted model with three times the vector dimensions: 59.8%
- the untuned base model: 58.5%
- text-embedding-3-small: 54.3%
The same benchmark also scores an easier question: was the right owner type found, whatever field came back? Order.createdAt and Order.completedAt both count as finding Order. On that score the large hosted model leads slightly, 87.8% against 85.6%. So the two models rank differently depending on the question. Before choosing one, decide whether your retrieval needs the exact field or only the right type.
Our 0.549 and the published 63.6% are in the same range, and they come from deliberately different experiments:
- We score only plain
Type.fieldtargets. The benchmark also labels argument coordinates such asQuery.repository(name:), which our index does not hold, so 591 of the raw targets are excluded. - Our corpus rendering is
DOT_PLUS_GLOSS, and the published run uses the suite's owncoord-return-desctemplate. - We run the GitHub slice alone, which is the largest and plausibly the hardest of the five schemas.
The instrument's numbers are consistent with each other, which is what comparing backends, formats and budgets requires. Reproducing the suite's exact pipeline is a fine exercise once its pull requests merge.
Reading the Numbers
The recall table supports two readings. The encouraging one is that a 0.6-billion-parameter model beats word statistics by twenty recall points at every depth, and that on the published benchmark it beats a large hosted embedding model at the task it was tuned for. Domain fine-tuning works, and it works at a scale one person can afford.
The second reading limits the first. Even the better model finds only about half to two thirds of the needed coordinates in a fifty-deep reading. So retrieval alone is not enough to navigate a schema. A design that trusts a single search to return everything an agent needs will fail often. Search has to sit inside a loop that can look again, expand what it found, and verify. That is where this lesson goes next, after one more measurement.
Two later numbers are worth reading alongside the retrieval figure, because they measure different things and are easy to confuse with it. They come from elsewhere in the same suite: PR #140 defines five benchmark families, and the two below are its strategies and agent families, not the retrieval one this lesson reproduces. Retrieval quality, the recall figure above, asks whether a single search surfaces the coordinates a task needs.
A schema-slicing strategy, meaning retrieval plus expansion around what it found, is scored differently. The best one fits the complete required field set into 12.4k tokens on average, with 92.6 percent perfect recall. End to end, an agent using search and then execute answered 85.0 percent of the benchmark's questions. One search finds half to two thirds of what a task needs, and the assembled slice is right nine times in ten. The loop closes that gap with:
- a second search with better terms
- an introspection call that shows a neighbouring field
- a validation error that sends the agent back for another look
The Budget Dial: Big Schema, Small Brain
Reporting "tokens@50" quietly assumes an agent reads fifty results. Real agents work within a stricter limit: a token budget, set by context-window size, latency and cost. So the benchmark also answers a more precise question: given B tokens of search results, how much recall can each backend deliver? It walks the ranked list, takes hits until the next one would exceed the budget, and scores what fit.
| Budget (tokens) | BM25 recall | Embedding recall |
|---|---|---|
| 125 | 0.239 | 0.365 |
| 250 | 0.290 | 0.437 |
| 500 | 0.351 | 0.525 |
| 1,000 | 0.405 | 0.591 |
| 2,000 | 0.435 | 0.622 |
Reading down a column shows what more context adds. For the embedding backend, going from 125 to 2,000 tokens raises recall from 0.365 to 0.622, with clearly diminishing returns per token. Reading across a row shows what a better model adds at the same cost. One comparison matters more than the rest. The embedding backend on a 500-token budget reaches recall 0.525, where BM25 reading 707 tokens' worth of results reaches 0.377. Quality and cost are linked. A ranking that puts right answers higher lets the agent stop reading sooner, and every result it skips leaves room in the context.
The budget dial also helps with teaching, because it reproduces the effect of a big schema on a small one. Set schemanav.retrieval.context-budget-tokens low enough on any schema, and you get what a huge schema does to a normal context window. Relevant coordinates fall below the cut, and the quality of the ranking decides everything. The same dial also works in the other direction, on small schemas. Below a certain size, navigation costs more than it saves, and the Movie DB numbers later in this lesson put a figure on where that line falls.
From Hits to Queries: Semantic Introspection
A ranked list of coordinates is not yet a query. The agent still has to learn each type's exact shape, the arguments, and the path from the query root to the field it wants. The working group's proposal for this gap is the semantic introspection RFC. Its design is worth reading closely, because it standardizes the two-step loop that this lesson builds by hand.
The Two Proposed Meta-Fields
The RFC proposes two extensions to introspection:
__search(query: String!, first: Int! = 10, after: String, minScore: Float): [__SearchResult!]!
__definitions(coordinates: [String!]!): [__SchemaDefinition!]!
Both sit on Query, next to the __schema and __type fields that introspection already provides.
__search is the discovery half. Its query argument takes a natural-language description of the capability the agent wants, such as Find a user by their email address. The other three arguments shape the result list. first caps how many results come back, at ten by default. after pages forward through a long list. minScore sets a floor, so weak matches drop out.
What a Search Result Carries
Every hit comes back as a __SearchResult, and most of the design is in that type:
type __SearchResult {
coordinate: String!
definition: __SchemaDefinition!
pathsToRoot: [[String!]!]!
score: Float
cursor: String!
}
Those five fields answer three questions at once: what did we match, what does it look like, and how would we reach it?
The first two are the match itself. coordinate is the schema coordinate that matched, the Type.field pair from the start of this lesson: Query.user, User.email. definition is the full introspection record for that coordinate, so a single call gives the agent the type, its arguments and its description.
The next two are bookkeeping. score is the relevance the ranker assigned; it is nullable, and the RFC only asks that implementations keep it between 0.0 and 1.0 where they do return one. cursor is the opaque pagination token that pairs with the after argument above, so passing the last result's cursor back as after fetches the next page.
pathsToRoot is the one worth studying, because it carries what a ranked list otherwise leaves out. A coordinate on its own tells the agent that User.email exists. It does not tell the agent how to reach User.email from the query root, and without that the agent cannot write an operation. Each path is a list of coordinates running from a root field down to the match. Given this schema:
type Query {
userByEmail(email: String!): User
}
type User {
email: String!
}
a hit on User.email comes back as [["Query.userByEmail", "User.email"]]. The outer list holds one entry per route, because a definition reachable through two different root fields gets two paths. When the match is itself a root field, that path holds a single element: [["Query.userByEmail"]]. Two caveats come from the RFC itself. Implementations may return a partial list of routes, so an agent cannot treat the paths as exhaustive. And the editors have flagged the placement of pathsToRoot on __SearchResult as unsettled, on the grounds that it arguably belongs on the definition types.
That leaves definition. Its type is a union, which changes how a caller reads it:
union __SchemaDefinition =
| __Type
| __Field
| __InputValue
| __EnumValue
| __Directive
Those five members are exactly the things a schema coordinate can name, and each one is an introspection type that already exists today. The union is why a caller reads a result through inline fragments: ... on __Field { name args { name } } when the hit is a field, ... on __Type { fields { name } } when the hit is a whole type.
__definitions, the Direct Lookup
__definitions is the second meta-field, and it is the plainer of the two. Hand it a list of coordinates and it returns their definitions in the same order. Resolving Query.userByEmail to its definition today means walking __schema and __type down to the field; __definitions returns it in one call. That makes it the natural follow-up to a search, though the RFC points out that any tool that already knows the coordinate it wants can use it on its own.
Implementations must maintain an index of the schema, and the RFC leaves the ranking mechanism open. Lexical, semantic or anything else satisfies the contract.
How Settled This Is
It is easy to read an RFC as more settled than it is, so here is its exact status. The RFC is a working document by Pascal Senn and Michael Staib of ChilliCream, and its stage and champion fields are empty as of August 2026. Hot Chocolate 16, released May 2026, is its origin implementation, enabled with a single option and on by default only in development mode.
Our own measurements support one design decision in that implementation. Hot Chocolate 16 ranks with BM25 by default, which keeps operations simple. An embeddings-backed option is announced, and as of August 2026 unconfirmed as shipped. The RFC uses "semantic" for the interface, meaning a search by meaning from the caller's side, and leaves the ranking mechanism to the implementation. That is what lets a server start with word statistics and add embeddings later, with the contract unchanged.
The numbers in this lesson support that choice. On the working group's GitHub questions, BM25 reached recall@50 of 0.377 where the tuned embedding model reached 0.549. Both searched the same DOT_PLUS_GLOSS corpus, the rendering that turns each coordinate into a short sentence. So a lexical index, without a model to deploy, reaches roughly two thirds of the tuned model's recall. That recall depends on the corpus rendering: over raw coordinates, the same index scores zero.
Their deep-dive post is the fullest published explanation of the design, and the token measurements quoted at the top of this lesson come from it.
For this course, the RFC matters as a direction, so it does not matter that Hot Chocolate runs on .NET. The part to remember is the shape of the interface: a search that returns coordinates, definitions and root paths. The instrument's agent tools implement that same shape in Java.
The Agent Loop, Watched Closely
Everything so far measured retrieval on its own, without a model. The final section hands it to an actual model and follows what happens step by step. The instrument drives the agent loop by hand so that we can watch each step. A framework would run the tools out of sight.
The Loop, and Why Every Turn Resends Everything
An agent is a conversation in a loop. The model receives the conversation so far, and either answers or asks for a tool call. The loop executes the tool, appends the result to the conversation, and sends the whole thing again. That is the most important detail in this section. The entire conversation is resent on every call, so every tool result is paid for again on every later turn. A 4,000-token introspection result costs 4,000 tokens multiplied by every turn that follows it. That multiplication is why the size of tool results decides most of an agent's cost, and Class 10 prices it.
The Three Tools and the Context Receipt
The three tools are search, introspect and execute, built on this lesson's own layers:
searchSchemaruns the configured backend inside the configured token budget.introspectTypeprints one type's full definition.executeGraphqlvalidates the operation against the schema with graphql-java, and posts it to a live endpoint when one is configured. GitHub's real endpoint requires authentication, so for GitHub's schema the run stops at validation.
The chat model is whatever Ollama serves locally. The runs below used an 8-billion-parameter model, so everything from retrieval to reasoning ran on one laptop. We have not measured any of this against a paid hosted model, which is why nothing in the lesson asks for an API key. We would like to add that comparison later.
Each run ends with a context receipt, which separates two kinds of token count that are easy to confuse. Prompt and completion tokens are the provider-reported bill: what the model processed and produced, resending included. Tool payload tokens are what the three tools injected into the conversation, measured with the instrument's own token counter. That second figure is the part schema navigation controls. Here is the receipt from a real run against GitHub's schema, with the embedding backend and a 500-token search budget. The task was "get the titles and author logins of the most recent open pull requests in a repository":
answer: query { repository(owner: "...", name: "...") {
pullRequests(first: 5, states: OPEN) {
edges { node { title author { login } } } } } }
context receipt (what this run cost):
model calls : 6
prompt tokens : 33,607 (provider-reported; the conversation, including every tool result, is resent on each call)
completion tokens : 2,726 (provider-reported model output)
tool payload tokens : 7,620 (counted here, with one encoding throughout; what the three tools injected)
for contrast : handing the model the whole schema instead would put 102,203 tokens into EVERY call
Compare the last two lines, because together they turn the argument of this lesson into a measurement. The agent spent 7,620 tokens on schema knowledge and 33,607 prompt tokens in total across all six calls, and produced a valid operation against a schema it was seeing for the first time. The alternative was 102,203 tokens of schema in every call. That is half of a 200,000-token window, used before the conversation starts and before anyone asks the agent anything. Every number in that comparison comes from the instrument's own output, and the next section shows how much a rerun of the same command varies.
The Same Command, Five Times
The receipt above is one run, and running the same task again does not reproduce it exactly. Five consecutive runs of the identical command, against the same schema with the same model and the same 500-token budget, came out like this:
The order the model asked for tools in, run by run:
| run | model calls | what the model asked for, in order |
|---|---|---|
| 1 | 8 | search, introspect, introspect, execute, execute, introspect, execute |
| 2 | 7 | search, introspect, introspect, introspect, execute, execute |
| 3 | 6 | search, introspect, introspect, execute, execute |
| 4 | 6 | search, introspect, introspect, execute, execute |
| 5 | 7 | search, introspect, introspect, execute, execute, execute |
All five reached a valid operation, and the receipt above is one of the cheaper ones. Remember three things from that table before you try to reproduce any number in this lesson.
The shape is stable and the count is not. Every run searched once, introspected two or three times, then attempted execution two or three times, because a rejected operation sends the model back for another look. That is the loop working as designed.
The prompt-token spread is wider than the tool-payload spread, and the reason is the resend. One extra attempt adds the whole conversation again, and the conversation is longest near the end of a run. So an extra round trip near the end costs the most.
Runs 3 and 4 spent identical prompt and payload tokens, and produced different amounts of output. Compare runs by the figures that the instrument counts itself. Figures that the provider reports change from run to run, because the model's output changes.
When Inlining Wins: The Small Schema
Running the same task on the trimmed Movie DB schema shows the opposite result, and the instrument measures both sides. Setting schemanav.agent.inline=true prints the whole schema into the system message and takes the search tools away, leaving executeGraphql as the only tool. Everything else stays the same, so the only difference between the two receipts is where the schema knowledge came from.
Three runs each, same task, same model, same 500-token budget:
| Movie DB, 44 coordinates | model calls | prompt tokens | completion tokens |
|---|---|---|---|
| navigating: search, introspect, execute | 5 to 6 | 6,366 to 8,796 | 2,857 to 3,013 |
| inlined: the whole 928-token schema, execute only | 1 to 2 | 1,209 to 2,524 | 831 to 1,507 |
Inlining wins clearly on this schema:
- It used a third to a fifth of the prompt tokens.
- It finished in one or two model calls, against five or six.
- Its operation passed validation on the first attempt in every run.
One run answered without calling executeGraphql at all. That is a warning: with the whole schema in its prompt, the model sometimes skips checking its work.
The first row of the table shows why. Navigation has a fixed overhead: a search call and one or two introspection calls before it can write anything. On a 928-token schema, that overhead costs more than the schema itself.
When Inlining Fails Silently: The Large Schema
Running the same inlined command against GitHub's schema fails without reporting any error:
schema in prompt : 260,153 tokens (what the instrument put in the system message)
model call 1 : 20,482 prompt tokens (what the provider reported evaluating)
We gave the model a quarter of a million tokens of schema, and it read about eight per cent of them. The run did not report an error, a warning or a truncation notice. It continued as though the whole schema had arrived.
We measured the cause. num_ctx is Ollama's option for the size of the context window, given in tokens. Sending the same oversized prompt at three different settings of it gives an exact relationship on this stack, Ollama 0.32.15 with qwen3:8b:
num_ctx (context window, in tokens) | prompt tokens actually evaluated |
|---|---|
| 4,096 | 2,050 |
| 8,192 | 4,098 |
| 40,960 (this model's full context) | 20,482 |
Every row lands on the same ratio. The model evaluated half of num_ctx, give or take two tokens, and held the other half back for generation. Applied to the GitHub run, that is where the missing ninety-two per cent of the schema went:
A second test shows which half of the prompt survives. Put a distinct marker word at the very start of a long prompt and another at the very end: the model reports seeing only the one at the end.
The prompt is capped at half the context window, and the half that survives is the tail. An oversized prompt therefore loses the system message and the first fields of the schema, and the run carries on without raising an error.
An inlined large schema does not stop the run with an error. The agent works from the tail of a document and does not learn what it lost. So every later failure looks like a mistake by the model.
Navigation has the opposite property here, and it is the reason the pattern exists. It puts at most the search budget into the conversation. The agent works from a small slice that it definitely received. If the slice missed something, that shows up as a validation error the loop can act on.
Dynamic navigation is a tool for large schemas. Below some schema size, it is better to skip it. Knowing where that line falls for your own schema is the reason to run the instrument against it.
A Failed Run, and What Fixed It
One failed run is preserved in the project's git history because it teaches more than the successes. In the first Movie DB attempt, the model skipped introspecting the MovieFilter input type. It guessed the field names three ways, trying yearRange, then year, then variations, and spent the entire step budget on validation errors. Here is one of those errors as the service returned it:
{
"errors": [
{
"message": "Validation error (WrongType@[movies]) : argument 'filter' with value 'ObjectValue{objectFields=[ObjectField{name='yearRange', value=ObjectValue{objectFields=[ObjectField{name='from', value=IntValue{value=1990}}, ObjectField{name='to', value=IntValue{value=1999}}]}}]}' contains a field not in 'MovieFilter': 'yearRange'",
"locations": [{"line": 1, "column": 16}],
"extensions": {"classification": "ValidationError"}
}
]
}
The only useful phrase is the last one, contains a field not in 'MovieFilter': 'yearRange'. It comes after a full dump of the argument's syntax tree. The model pays for the whole message on this call and on every later call, and the useful part is the easiest to miss.
Two small changes fixed it. The first compacts graphql-java's verbose validation errors down to their message and appends a hint: "introspect the types named in the errors before retrying". The second adds one hard rule to the system prompt: every name the model writes must have appeared in a tool result first. The rerun solved the task in six calls, and comparing the two transcripts shows why the introspect step exists. Agents do what specifications and error messages make easy. So an error message that agents read is part of your API's design.
The Standards Picture
This area changes quickly. The three paragraphs below say which parts of this lesson are settled and which are still early, as of August 2026.
Settled. The search/introspect/execute pattern itself ships in production tools: Apollo MCP Server, Grafbase and Hot Chocolate 16 all implement it. The corpus-formatting effect is robust and cheap for anyone to reproduce. Local embedding inference is mature and reliable.
In motion. The semantic introspection RFC still lacks both a stage and a champion. The working group's evaluation suite is an open pull request, though the model comparisons alongside it merged in June 2026. And the embedding model is a single engineer's prototype, published with a documented regression. Its rate of picking the wrong field on the right owner type rose from 6.3% to 10.3% against the base model, and the author asks readers to treat it as a baseline.
Worth watching.
- Progressive subschema disclosure through a
@require_graphql_subschemadirective, in working group issue #54. Meta engineers opened it, and describe an internal implementation with the GraphQL server and the agent code co-located. - Alternative SDL renderings written for models to read, in issue #32.
- The code-execution pattern that this course's closing lesson covers. It skips runtime navigation by handing the agent a generated, typed SDK. Anthropic reports a reduction from 150,000 tokens to 2,000 on one workload.
If that last pattern is widely adopted, today's meta-tools may turn out to be a temporary design. The schema descriptions and the measurements from this lesson stay useful in both cases.
Summary
Production schemas are expensive to send whole. GitHub's costs 102,203 tokens rendered as a searchable corpus, half of a 200,000-token window. In ChilliCream's measurement, the same task measured 665,564 tokens as OpenAPI, 133,441 as a raw GraphQL schema and 59,067 with a search layer over it. Agents therefore need retrieval over the schema instead of the schema itself. The unit of retrieval is the field coordinate, and the hard case is telling apart two fields that differ only in their owner type.
How the corpus is rendered is the cheapest way to improve quality. Raw coordinates are close to unsearchable, and a lexical index cannot match them at all, because Lucene's tokenizer keeps Type.field as a single word. Spelling the coordinate out raises embedding MRR from 0.541 to 0.675, over the same corpus with the same model. It also moves the 95th-percentile rank of the right answer from 46 to 24.
Lexical BM25 is the baseline that is free to run, and the production servers named in this lesson use it today. The GraphQL-tuned 0.6B embedding model beats it at every depth on the working group's GitHub data, 0.549 against 0.377 at recall@50. A better retriever also saves tokens: given a 500-token budget it outperforms BM25 given three times that.
Even the better retrieval finds only half to two thirds of what tasks need, which is why search belongs inside an agent loop with introspection and validation. On the large schema the context receipt shows 7,620 tokens of schema knowledge replacing 102,203 per call. On a 44-coordinate schema it shows the reverse: navigating spent 6,366 to 8,796 prompt tokens where inlining the whole 928-token schema spent 1,209 to 2,524.
Inlining a schema that is too large fails silently. If you remember one finding from this lesson, remember this one. On Ollama the prompt is capped at half the context window, and truncation drops the front, so the system message and the first fields of the schema go first. No error is raised, and the agent works from the tail of a document, without being told what it lost.
So the right choice depends on the size of your schema:
| If your schema is | Then |
|---|---|
| small enough to inline, as Movie DB is at 44 coordinates and 928 tokens of SDL | inline it. Navigating spent three to five times the prompt tokens to reach the same answer, and an inlined schema cannot return the wrong slice. |
| large, as GitHub's is at 6,640 coordinates and 102,203 tokens | search, introspect and execute inside a loop, and spend the effort on how the corpus is rendered. |
| somewhere between the two | point the instrument at it and read the receipt. That measurement is the answer, and it takes one command. |
Further Reading
- A GraphQL-Native Embedding Model for AI Agents (graphql.org, July 2026). The model announcement, with the benchmark construction, the formatting experiments, and the training recipe this lesson quotes.
- Semantic Introspection RFC (GraphQL AI Working Group). The
__searchand__definitionsproposal; short, readable, and the clearest statement of the two-step loop as a standard. - Semantic Introspection deep-dive (ChilliCream, April 2026). The origin implementation's design reasoning, including the token measurements across OpenAPI, full-schema, and search-layer interfaces.
- Solving context explosion in GraphQL MCP servers (Grafbase, 2025). The post that first described search/introspect/execute, the pattern this whole lesson builds on.
- graphql-embedding-model and the model card. The training pipeline, evaluation suites, and the worked examples of owner-type disambiguation.
- AI Working Group evaluation suite, PR #140 and model comparisons, PR #142. Where the 816-query benchmark lives: #142 merged in June 2026, #140 is still open and carries the five benchmark families this lesson quotes from. The instrument's fetch script downloads its GitHub slice.
- Code execution with MCP (Anthropic, November 2025). The strongest published alternative to runtime navigation, and the source of the 150,000-to-2,000-token figure.
What's Next
The agent can now find its way around a schema of any size. That also means it can find things we would rather it did not use. Class 4 takes up safety. It covers:
- complexity caps, the first defense against a query that asks for everything
- persisted-query allow-lists scoped to the agent role
- keeping mutations behind human confirmation
It builds on two earlier pieces. The allow-list work from Class 2 bounds what the agent may call, and this lesson's navigation layer is how the agent found it.