NVIDIA NOOA vs LangChain: Half the Tokens, Higher Score

0:00 This is a Python class. It is also in full an AI agent, one that Nvidia Research put on the internet in July. Look at the second method. Its whole body is three dots. That is not a

0:10 placeholder somebody forgot to fill in. Three dots is the instruction. It means a language model writes and runs that method at runtime every time you call it. Now look at the method above it.

0:20 Ordinary body, ordinary Python. That one just runs the same way it ran yesterday with no model involved at all. Same class, same file. Two completely different execution models sitting four

0:32 lines apart. And the only thing deciding which is which is whether you bothered to write an implementation. No prompt template, no tool schema, no callback to register, no graph to draw. The dock

0:43 string is the prompt. The fields are the state. The type annotations are the contract. That is the entire interface. Nvidia calls it Noah object-oriented agents. The code went public on the 20th

0:55 of July and the paper followed it 2 days later, four weeks old. 1,700 stars, 240 forks, Apache 2.0 pushed again this morning. It got posted to Hacker News three separate times. The first

1:08 submission got four points, the second got three, the third got two points, and one comment, nine points total for a state-of-the-art agent harness out of Nvidia Research. So, one of two things

1:19 is true here. Either the people who build agents for a living read this and shrugged or almost none of them have read it at all because the claim in that class is not a small one. It says most

1:30 of what an agent framework sells you is machinery you already own and forgot you owned. Here is the paper's own description of the problem. Traditional agent development it says is split

1:40 across prompt templates, tool schemas, callback code, and workflow graphs. Four separate places to edit. Four separate things to keep in sync. None of which your type checker or your test suite

1:51 understands. Noah deletes all four. The prompt template goes first. Your doc string is already a natural language description of what a method does. So the dock string becomes the prompt and

2:02 there is nowhere else to look. The tool schema goes next. You already annotated your arguments and your return type and the annotation already says what shape a valid answer takes. So the annotation

2:13 becomes the contract and a bad answer fails as a type error rather than as a bad vibe. Then object state. Every framework has invented somewhere to stash what the agent knows between

2:23 turns. A memory store, a scratch pad, a growing transcript. But a Python object has had fields since 1991. So the fields are the state and the loop. Orchestration in Noah is ordinary Python

2:35 you can read, step through, and edit, which means the model can edit it too. Because to the model, it is just more code. That last part is the one that changes how it feels to use. The model

2:45 does not get a flattened text dump of your data. It gets the live object by reference with a bounded preview and it acts by writing Python against it control flow and all. If it needs to

2:56 check 20 orders, it writes a loop instead of you making 20 round trips. And there is a sixth idea which is the one most frameworks get backwards. The context the agent is carrying and the

3:07 history of what it has done are an interface the model itself can call and manage, not a dashboard you watch from outside. The paper is blunt about where it draws that line. If only the

3:17 developer can see it, it did not happen. Which raises the obvious objection. This is a nice story for the developer, but the thing on the other end is a language model. Can it actually drive an

3:26 interface like that? So they measured it. 88 tests across 36 families, typed calls, stateful mutation, error recovery, batching through generated loops, task decomposition. Every test

3:39 five times against 10 different models from small local ones up to the frontier, 4,400 runs in total. 97.9% of them passed. The small cheap models passed 96%. The Frontier models passed

3:52 99. And the worst model in the whole set still cleared 91. Whatever else is hard about building agents, reading a Python object is not the hard part. There is one number in that table worth sitting

4:03 on. With reasoning turned off, Nvidia's own smallest model, 30 billion parameters, pass just over half the tests. Turn reasoning on, same model, same tests, 85%. The interface did not

4:15 change. The thinking budget did. Reasoning is doing the work a bigger model would have done. And none of this is tied to Nvidia's models, which I half expected it to be. It routes through

4:25 light LLM. So the same class runs against Anthropic, against OpenAI, or against a model on your own machine through Alama or VLLM with no API key at all. The one hard requirement is a

4:38 recent Python 3.12 or newer. So the models can use it. The harder question is whether using it wins anything and that is where the paper stops being a design document and starts being a

4:48 scoreboard on S.E. bench verified real issues real repositories the benchmark everybody in coding agents is judged on Noah reaches 82.2% at the highest reasoning setting for scale the

5:02 framework that invented this write code as your action idea scores 68.4 on the same benchmark switch both to a different model and run them head-to-head Noah lands at 79.8 11.4

5:15 points clear of the thing it builds on terminal bench 2.0 O, which is multi-step work in a command line. It goes from 46% with reasoning off to 73% with reasoning on. And with reasoning

5:27 off, the setting where the harness has to supply the discipline instead of the model, the gaps are widest. 8 and six points ahead on SWE, 11 and N points ahead here. Turn reasoning up and they

5:38 shrink. At the very top setting on this benchmark, one rival passes it. The paper prints that, too. But the number that actually matters is not any of those. It is this one. Noah reaches

5:49 82.2% using about 28 model calls and 1.1 million tokens per task. A rival harness spends 66 calls and 2.2 million tokens, twice the spend, and scores four points

6:02 lower. Part of that is a rule about quitting. A different rival stops the moment the model replies without calling a tool. And on Terminal Bench, 77% of its failed runs gave up inside 10 steps.

6:14 In Noah, the model cannot just announce it is finished. It has to hand back evidence and a command you can run to check. They also pointed it at security work on Cyber Gym, which asks an agent

6:25 to find and repair real vulnerabilities with the network cut off. It solves 86.8%. Two closed specialized systems beat it. It is the best open- source score on

6:34 that board and it beats the closed system from the lab that makes the model it was running. Then there is the one I keep coming back to on ARC AGI 3 grid games where the agent gets no rules and

6:45 has to work them out by playing the same model scores 13.3% alone and 85 inside this harness 6.4 times from the rapper. The paper says the evaluation budgets differed. So read it as indicative. The

6:59 reason is worth knowing because it is the object model doing the work again. The agent plays, works out the rules, then writes executable code that predicts what the game will do next, and

7:09 keeps it. In 22 of 25 games, it held on to a working model of the game it was playing. Strip that memory out, give it a plain markdown file to scribble in instead. And on the earlier fleet, with

7:21 the earlier model, the score falls by nearly 12 points. Same model, same games, same skill. The only thing that changed is whether what it learned survived the night. Which brings us to

7:31 the part that gets left out of the write-ups. Every number I have just given you was produced by Nvidia running Nvidia's harness published in Nvidia's paper. Would you move a production

7:41 system onto a benchmark the vendor ran and no outside team has ever reproduced? Here is what makes this one unusual. Nvidia got there first in writing before any reviewer did. The repository is

7:52 named with a labs prefix. The package is on version 0.0.8. 8. This is a research team publishing an experiment and Nvidia's own blog calls it an open experimental surface, not a replacement

8:05 for the harness you are using. And then there is the safety note which I would like to just read to you because vendors do not write paragraphs like this. It says the framework checks generated code

8:14 and blocks dangerous imports. And then it says these are defense in-depth guardrails, not a containment boundary. The containment boundary is OS level isolation. Always run this inside a

8:25 container or a virtual machine. Do not rely on the validators alone. And it explains why, which is the part I would put on a poster. A checker that reads Python cannot make that promise because

8:36 the language will not let it. Open a file and you have arbitrary file access. Import by path and you have loaded anything on disk and reflection reaches whatever is left. The same restraint

8:46 runs through the comparison. The paper puts Noah beside 14 other agent frameworks. Langraphph and Langchain, Pyantic AI, OpenAI's agents SDK, Microsoft's agent framework, Google's

8:59 ADK, and nine more, and scores all 15 against six capabilities. And the bar it sets is specific. It only counts if the model can see it. A tracing dashboard does not count. Automatic compaction

9:11 does not count. A hidden callback does not count. Most systems it finds have a version of each idea and expose it to the developer instead of the model, but it will not claim to have invented any

9:22 of them. The paper's own words. The community is already converging on several of these ideas, often as experimental or partial features, and the comparison is published to encourage

9:32 further adoption. That is a research team saying its main contribution is putting six existing good ideas on one surface. It is a much smaller claim than the video title makes, and it is the

9:42 reason the claim holds up. So, we are back at nine points on hacker news and now the question is sharper. Something can be well-built, well-measured and honest about its own limits and still

9:52 land in complete silence. Look at who made it. 15 names on the paper. The first listed author and the last listed author also wrote Nvidia's blog post about it 7 days after the code went up

10:03 207 commits in 4 weeks. About 60 stars a day from a standing start on the strength of a repo link and one blog post. Three people did most of it. Ricardo Syl Vera Cabrell, Severign

10:14 Clinger, and Allesio Davoto. All three on the paper account for roughly 85% of every commit in the repository. And here's a detail I did not expect. One commit in four carries a co-author

10:26 trailer naming a claude model, including the first one, 300 files, 280,000 lines. The initial public release of Nvidia's agent framework co-authored by an agent. Now, the scale, 1,700 stars against

10:39 Langchain's 144,000. On the Python package index, four orders of magnitude fewer downloads a month than Langraph. Being early and being ignored look identical from the outside, right up

10:51 until they don't. So, here's where I land. The bet is right, and the software is not ready. And those are two different sentences that people keep collapsing into one. The bet that the

11:00 abstraction was already in the language I think is correct and the receipts are the token numbers not the accuracy ones. Twice the spend for a worse score is what the extra machinery costs you and

11:10 that is measurable in a way that taste is not. The software though is 0.x research code whose API can move between releases whose scores have never been reproduced by anyone outside Nvidia and

11:22 whose safety model ends at a paragraph telling you to supply your own container. A type check tells you a value has the right shape. It does not tell you the refund was authorized, but

11:31 the programming model is free to steal. You do not need to install anything to stop hand assembling prompt templates around objects you already defined. And if you are a Python team whose agent

11:41 touches real typed objects, that idea is worth more than the packages, which leaves the question the whole paper is really asking underneath the tables. If the framework does dissolve into the

11:52 language, no templates, no schemas, no graphs, just a class, then what is actually left to compete on? The harness around the model or the model itself.