Programmatic Tool Calling: The Architecture Replacing JSON Function Calls

0:00 Every AI agent you have ever built calls its tools the same way. The model emits a structured JSON object. Flood that agent's context with 128 tool schemas and its accuracy gets measurably worse,

0:12 two and a third points worse on average, measured across 14 different production models. Now change exactly one thing. Let those same models write a short Python script instead of emitting JSON.

0:22 Under the identical flood with identical tools, they do not get worse. They get five and a half points better. One interface degrades under load, the other improves under load. Nothing about the

0:33 task itself changed. That is context rot, measured on the precise thing your agent framework is doing all day, every day. And it comes from a paper submitted 11 days ago, which did not come out of a

0:44 frontier lab at all. It is from four researchers working inside PricewaterhouseCoopers, the big four accounting firm. 14 models, 309 benchmark tasks, eight categories, one

0:56 question underneath all of it. So should your agent still be emitting JSON or should it simply be writing the code for itself? The answer depends on how many tools your agent has, how deep its call

1:06 chains run, and uncomfortably which model you are paying for. The paper is the bitter lesson of tool calling. Eshan Patel, Sahil Sen, Elias Lumer, and Vamsi Kumar Sabbella from the commercial

1:18 technology and innovation office at PricewaterhouseCoopers. That byline is worth sitting with. This is not a frontier lab publishing a flattering result about its own models.

1:28 It is a consultancy that has to make agents work for clients on whatever model the client already bought. So here's what the frameworks do today. You hand the model a pile of JSON tool

1:37 schemas. It emits one structured call. Your code runs it, feeds the result back into the conversation, and the model goes round again for the next one. A turn-by-turn

1:47 negotiation. Every link in a chain costs a full inference turn, and every raw tool result lands back in the context window whether the model needed it or not. That was the right design when

1:57 models could not be trusted to write code. Programmatic tool calling changes the shape of it. The same tools get compiled into typed Python function stubs, one per schema,

2:07 carrying the schema's own argument types. The system prompt hands the model that stub module as source code. Then the model writes one short script that imports the stubs and calls them. The

2:17 agent loop runs it in a shell sub process, captures what it printed, and stops before a second inference ever happens. Here is the paper's own worked example. Get the circumference with

2:27 radius 7 and the area of a square with side 5. Under JSON, that is two tool call objects across two model turns with a round trip in between. Under code, it is

2:37 one call containing four lines of Python. Import both stubs, call both, print both results. One turn, one sub process, both answers. The rest of the paper measures whether that survives

2:48 contact with reality. They measured it on the Berkeley function calling leaderboard, version 4. 309 entries across eight categories on 14 models spanning 20 months from November 2024

3:00 through July 2026. Every run at temperature zero and scored deterministically. The headline result, programmatic tool calling matches or beats JSON in 11 of the 14. And the two

3:12 largest gains land in the same family, GPT 5.6 Sol and GPT 5.6 Terra at 10.6 points each. Sol climbs from 72.2% to 82.8. Terra from 72.5 to 84.1. All five Anthropic models clear baseline on this

3:29 main benchmark, too. From dead level on Opus 4.8 to six and a half points on Sonnet 4.6. So, the split is not Anthropic against OpenAI. It is newer models against older ones.

3:41 Which sounds like a clean story right up until three of the 14 did not merely fail to improve. Did they slip a little? No, they fell off a cliff. GPT-4o, the model from 2024 drops 26.9 points.

3:54 GPT4.1 from 2025 drops 19.8. And GPT5.4 Mini, released only in March of this year, drops 24.3. On the chaining tasks, it is starker. GPT4.1 scores 98.1% under Jason, the

4:11 highest number anywhere in that table, and collapses to 40.4 under code. 57 points gone on identical tasks. This is the most instructive result in the paper because the cause is not what you would

4:22 guess. It is not an orchestration failure. Those models are not confused about which tools to call or in what order or with what arguments. They write Python containing two literal

4:32 characters, a backslash and the letter N, where a real newline belongs. The subprocess reads that as a syntax error and the script dies before a single tool gets called. Every entry needing more

4:43 than one line scores zero, which is why chaining is their worst case. Every intermediate step needs a multiline script. This is a serialization defect, a text encoding

4:52 mistake in models that otherwise write Python perfectly well. And the control sits one row above them in the same table. GPT5 Nano, smaller and earlier and cheaper, does not do it. Same system

5:04 prompt, same instruction to use real new lines. A smaller, older model gets it right while a bigger, later one does not. The paper calls that a capability gap and says the cause is outside its

5:15 scope, which is the right amount of humility. Read it as a buyer, though, and it says something sharper. Three shipped, paid production models will break if you move them to code mode over

5:25 an escaping bug. So, set those three aside and look at where this earns its money because what matters is not the average case, but the shape of the task. Start with chaining. 52 entries, chain

5:36 lengths from two calls up to 20, weighted toward the long end. Claude Sonnet 5 goes from 80.8% to 96.2. Opus 4.8 from 80.8 to 94.2. Though on that same table, Haiku 4.5 goes the

5:50 other way from 88.5 down to 73.1. And critically, the advantage scales with depth. At chain lengths of 12 or more, the paper measures an 18.8 point gap, and at short chains that gap does

6:04 not exist at all. JSON pays one inference turn per link. The script pays one turn total, which shows up on the clock as well as the scoreboard. Code mode finishes chaining tasks in roughly

6:15 half the wall time for 13 of 14 models at ratios from 0.32 to 0.96 of baseline. GPT-5 is the exception, and a useful one. It runs at 2.8 times the baseline

6:27 latency. It's extended reasoning output is long enough that writing the script costs more time than the saved turns give back. Fewer turns does not automatically mean faster. Then the

6:37 calls go wide instead of deep. Parallel fan out. 32 entries from seven simultaneous calls up to 48, and 13 of the 14 match or beat baseline. GPT-5 jumps from 71.9%

6:50 to 96.9. Because in code, fan out is a loop or a parallel gather block with no ceiling on how many calls fit, since it is only Python. In JSON, the model must emit

7:01 every parallel tool call object inside one response, and past a certain width, it starts dropping them. So, the paper goes hunting for the wall, probing Claude Sonnet 5 under plain JSON

7:12 at 60, 70, 72, 75, and 100 parallel calls. At 70 calls, perfect enumeration. At 72, it drops to 75%. At 100 calls, it scores zero. Not degraded. Zero. The model stops issuing calls altogether,

7:29 while programmatic tool calling holds at 100% on the same entries. But before anyone turns that into a law about JSON, GPT-5.6 Soul holds 100% under plain JSON all the way out to 100 calls.

7:42 So, the cliff is not a property of the format. It is how one vendor's models serialize a parallel tool call block. So, what does switching actually cost you? There is a clean number for it, and

7:52 it is the single most useful thing in the paper. Token cost crosses over at about 26 tools. Below 26, code mode costs more because the system prompt carries the whole stub module as prose,

8:04 while JSON passes its schemas through the API's tools parameter. Above 26, JSON costs more because the response must enumerate every tool call object. At 30 tools, it is 3,559

8:16 tokens against 3,380. At 48, 5,097 against 3,535. Code mode barely moves. JSON grows linearly. Though on deep chains, it reverses where code mode uses about half

8:30 again the input tokens, and output tokens do not differ between the two at all. So, here is the question worth stopping on. How many tools does your agent have registered right now? Do you

8:40 know the number? Because that number is the entire decision. Under about 26 tools with shallow chains, JSON is cheaper and simpler and better supported. So, leave it alone. Over that

8:51 line, or with chains more than a handful deep, you are paying a tax on a format you never chose. Now, the part most coverage will not reach, and the part that decides whether

9:01 you should believe the rest. Four limitations go in writing. You already heard one of them, the token overhead, and a fifth sits in an appendix. First, the benchmark uses echo return stubs.

9:12 Every function returns its own arguments back verbatim instead of calling a real API. So, what is measured is argument serialization accuracy, not end-to-end tool use. If your tool's return value

9:24 feeds the next call, this study does not cover you. Second, the ablation samples are small, 31 to 52 entries per condition. The paper states plainly that individual model results carry wide

9:35 confidence intervals and should be read as directional, and that only the aggregate patterns are reliable. So, the 18.8-point chaining gap, the GPT-5 fan-out jump, the Sonnet-5 cliff,

9:47 treat those as arrows, not measurements. What survives is the aggregate. 11 of 14, 13 of 14, and the direction of travel under a flood. Third, and here they are a little

9:57 generous to themselves. They cite an outside audit that found 20% evaluator human mis-alignment in this benchmark, and say their deterministic score side-steps that path because the

10:07 mis-alignment sat in the language model judge mode. So, I went and read that audit. It is by Vishvesh Bhatt and colleagues, published at the end of June. Its own table lists

10:17 this benchmark's evaluator as an abstract syntax tree plus simulator state, which is deterministic, carrying a 20% error rate across 200 audited tasks.

10:27 The language model judge benchmark in that audit is a different one at 30.5%. So, the 20% is not in the path they side-stepped. It is in the path they used. To their credit, their next

10:38 sentence concedes the labels may carry noise. And fifth, the one in the appendix. Average the per category scores across all 14 models, and the overall comes out at 78.6 for JSON

10:50 against 77.0 for code. On that macro average, code mode is behind. On the parallel categories, 14 points behind. The paper explains it. That gap is largely the three broken models whose

11:02 new line bug hurts most on multi-call scripts, which is true. But, notice which number went in the abstract and which went in appendix A. 11 of 14 leads. 77 does not. And one more,

11:14 because I ran the numbers myself. That headline plus five and a half under context flood, take out the three models with the new line bug, which scored so badly in the filtered condition they had

11:24 nowhere to go but up, and the remaining 11 average plus 0.6, not negative. Just nothing like five and a half. The claim that survives is the narrower one. Under a flood of 128 schemas, code mode holds

11:37 stable while JSON degrades. Stability is the finding. The size of the gain is mostly three broken models recovering. So, a hedge paper, real caveats, a small subset. Why care at all? Because of what

11:50 is already shipped while the measuring was not happening. The idea itself is not new. Code Act from Xingyao Wang and colleagues at ICML in 2024 showed Code Act beating JSON on multi-tool tasks.

12:02 Up to 20% higher success with up to 30% fewer actions, concentrated in exactly the parallel and compositional cases this paper isolates. Then, in September 2025, Cloudflare shipped Code Mode, and

12:15 Kenton Varda's framing of why is still the best line in the debate. "Models have seen an enormous amount of code," he wrote, "and almost no tool calls." Getting a model to use tool calling, he

12:24 said, "is like putting Shakespeare through a month of Mandarin and then asking him to write a play in it." Six weeks later, Anthropic published its own version of the argument with a number

12:33 attached. A workflow that burned roughly 150,000 tokens passing tool definitions and intermediate results through the model got rebuilt around code execution and came back at about 2,000. 98.7%

12:46 less. Then, in February 2026, Cloudflare put a number on the schema problem that makes this paper's 128 schemas look quaint. Their API has over 2 and 1/2 thousand endpoints, and exposing every

12:59 one as a tool would cost more than 2 million tokens, more context than any frontier model has. Through Code Mode, that same API fits in roughly 1,000 tokens, a 99.9% cut. That is not a

13:12 benchmark result, but a production platform saying the schema dumping approach does not fit inside the machine. And then, the 9th of July, 2026, OpenAI ships GPT-5.6,

13:23 the Luna, Terra, and Sol family with programmatic tool calling built into the responses API as a native feature. Model written JavaScript in an isolated runtime orchestrating the tool calls

13:35 itself. Now, hold that against the paper's table. >> [snorts] >> The three models with the largest gains in the study are the three models OpenAI

13:42 shipped that feature on. The paper does not draw that line. I am drawing it, and its harness is its own Python setup rather than OpenAI's feature. Anthropic ships the same capability under the very

13:53 same name, programmatic tool calling, where your tools appear to Claude's code as async Python functions it can fire off in parallel with a gather block, which is almost line for line the

14:04 construct in this paper's own parallelism example. And today, as this goes out, Deep Seek released its agent harness in developer preview under an MIT license. Four runtime modes, and one

14:15 of them is code mode, tools exposed through an SDK, so the model composes multi-step operations inside a single TypeScript program. So, the timeline reads like this.

14:24 A 2024 paper suggests it. One infrastructure provider and three frontier labs ship it across 2025 and 2026. And only now, 11 days ago, does anybody run a controlled comparison. The

14:37 industry shipped first and measured second. So, here is where I land. Programmatic tool calling wins, and it wins for most people building agents right now. Anyone on a current

14:47 generation model with more than about 26 tools registered or chains deeper than a handful of steps. Three receipts earn that. 11 of 14 models match or beat their own baseline on the main

14:58 benchmark. 13 of 14 under parallel fanout. And roughly half the wall clock time on chaining. Those are the aggregate patterns the paper itself calls reliable. Who it is not for. If

15:08 your agent has eight tools and calls them one at a time, JSON is cheaper and simpler and better supported. And the paper's own crossover maths says so. And here is the concession I'm not going to

15:19 bury. On the macro average across all 14 models, code mode is behind. 77 against 78.6. I would still switch because that deficit is three models with a text

15:30 escaping bug, but you should know the number exists because the abstract will not tell you. JSON tool calling was designed for models that could not be trusted to

15:39 write working code. That was a reasonable decision at the time. The constraint expired, the format did not, and the thing that made agents reliable is now the thing capping them.

15:49 Which leaves the question this whole result raises, and I do not think it has an obvious answer. If the model composes tools better than your framework orchestrates them, what exactly is the

15:58 framework still for?