The End of LLMs? (NCM)

0:00 I went through the Sudoku table in a new Stanford preprint expecting a close race. A small model writing one word at a time, the way your chatbot writes, solved about 1% of the hard boards. One

0:11 the same size that drafts the whole grid first solved 86%. Nearly every chatbot you use writes the same way, just at a vastly bigger size, one token at a time, a token being a word or a piece of one.

0:24 It's why long answers trickle onto your screen. This month, two papers went after that habit from opposite ends. Next concept modeling drafts the whole answer roughly, then sharpens it. Hybrid

0:35 agents lets a model think without words and write only to use a tool or answer. End of LLMs? No. What made me care was a line in OpenAI's reasoning docs. The reasoning tokens you can't see are still

0:47 billed as output. So, I wanted to know if moving past word-by-word changes that bill or just the vocabulary. That popular picture is fair. Three Blue One Brown's explainer calls a language model

0:58 a function that predicts the next word, and nearly all are trained that way. But, in that Sudoku table, the word-by-word model got 128 passes. The drafter stopped at 12. One question I'll

1:09 carry to the end. If a model thinks without words, can anyone read what it thought? By the end, you'll know how both work, what they won by the authors' numbers, and why your laptop isn't

1:20 faster yet. Karpathy's line, written in early 2025, was that most LLMs you've been seeing are clones as far as the core modeling approach goes. He put a little squiggle

1:30 in front of clones. The one that means roughly it's a very diplomatic squiggle. He was writing about Mercury, which he called the first large diffusion-based LLM, and that's where this trail starts.

1:41 Writing in order has a built-in cost. Inception, the company behind Mercury, spelled it out on its blog. A token can't be generated until all the text before it has been generated. So, a long

1:52 answer is a long chain of steps, and every step waits for the one before it, however fast each step runs. Reasoning models make that chain longer. Before they answer, they produce

2:02 reasoning tokens. OpenAI's docs say those tokens take up room in the context window and get billed as output, even though the API doesn't show them. You wait for words you won't see, and you

2:13 pay for them, too. One person who's pushed against that chain for years is Stefano Ermon, a Stanford professor working on diffusion for text since 2019.

2:23 The pitch for diffusion is that the network can change several words at once in parallel, instead of only adding one at the end. With two former PhD students, Aditya Grover and Volodymyr

2:33 Kuleshov, he founded Inception Labs. That idea shipped. Mercury reports about 1,000 tokens a second on Nvidia's data center chips, and that's the company's own figure. So, parallel writing works

2:45 outside a lab. But, diffusion models still make many passes over the whole text, and that's the crowd NCM ends up racing. A second team went after a different

2:54 waste. In late 2024, Shib Ao and colleagues at Meta's research lab, FAIR, and UC San Diego published Coconut. Right before a model picks a word, it holds a long list of numbers called the

3:06 hidden state. Coconut skips the word and feeds that hidden state straight back in as the next input. That had an odd side effect. One of those continuous thoughts could encode several possible next steps

3:17 at once, so the model searched breadth-first, keeping options open rather than betting on a single path. Hold on to that hidden state. Hybrid agents, the second new paper, builds on

3:28 the same move and uses Coconut as a baseline. The third lead is about seeing the thinking at all. In September 2024, Mozilla's Marco Figueroa said that probing the reasoning

3:38 of OpenAI's O1 model put him on OpenAI's ban list. Other users reported warning emails for asking about the model's reasoning. Those accounts come from news coverage at the time. OpenAI's stated

3:50 reasons for hiding the raw chain of thought were internal monitoring, user experience, and competitive advantage. One objection in that coverage was that the warnings hinder red teaming safety

3:59 research. Either way, you are paying for a part of the answer you weren't supposed to ask about. So, we've got two old problems. Writing in order is slow, and thinking in words is billed by the

4:09 token. NCM goes after the first, and it borrows its plan from image generation. Some image models predict a whole picture at a coarse scale first, then each sharper scale in turn. And NCM's

4:21 say they follow that next scale approach. Picture a sketch that sharpens. The first pass sketches the entire answer roughly, and every later pass adds

4:29 detail everywhere at once. Part one is a tokenizer, which turns text into codes. NCM encodes a passage into a handful of levels, and every code is picked from a fixed list called a codebook. The top

4:41 level holds the fewest codes, a coarse summary of the whole passage. The bottom level holds one code for every token. Each level stores only what the levels above it missed. The first level makes

4:52 its rough guess. The next records the difference between that guess and the encoded passage, which is called the residual. The level after that records whatever still off.

5:01 Add every level together, and a decoder can rebuild the text. There's a training trick that makes the sketch worth having. A technique called nested dropout trains the coarse levels to be

5:11 useful on their own, so the top of the stack carries meaning without leaning on the detail underneath. Part two is the generator, a transformer of about 100 million parameters for most tasks. It

5:21 predicts every code in a level at the same time in a single forward pass, then moves down to the next level. Coarse to fine, a pass per level, and no position waits for the one before it.

5:33 Part three is the decoder. It takes the levels, sums together, and maps them back to a token at every position at once. That's the whole loop. A few passes, each touching the entire answer,

5:43 instead of a pass for every word. That pass count is where the speed comes from. On XSum, a new summarization test, NCM used eight passes. The strongest diffusion model, Aurora LM, used 128.

5:57 NCM also scored a little higher on the summary metric. The authors timed it at 31 times faster. That was measured on a single Nvidia H100. On the other two test sets, the gap was smaller, but

6:09 still big. Back to that Sudoku table. Every cell in a Sudoku depends on the others, so I expected drafting the whole grid to help. I didn't expect it to help this much. Difficulty was set by how

6:20 many clues each board started with. So, the hard boards gave the least to go on. Both models were tiny, about 25 million parameters each. Divide one pass budget by the other, and the word-by-word model

6:31 had more than 10 times as many passes. It still failed almost every hard board. The fairer rival is diffusion. Its strongest entry there, the plus version of FMLM, solved 81% of the hard boards.

6:43 So, against another parallel method, the drafting model wins by a few points. Against word-by-word, the margin is enormous. Before that 31 times lands on a slide, check who it beat. Every NCM

6:55 speedup is measured against diffusion language models of similar size. Not against the normal chatbot, and not against the modern word-by-word LLM. The largest generator they tested has

7:06 725 million parameters, research scale, nothing like a chatbot. On translation, NCM scored lower than a diffusion baseline called E2D2, while using far fewer passes.

7:18 It's an odd thing to be glad about, but a loss printed in the authors' own table makes the wins next to it easier to believe. The paper's limit section is also where my sketch picture breaks. In

7:29 an image, one wrong detail in a fine layer barely shows. In text, the authors write, "A single token error can change the meaning, stop code from running, or invalidate a Sudoku solution." They add

7:40 that the coarse-to-fine order is fixed, and quality is capped by how well the tokenizer rebuilds text. So, NCM's result is narrower than its headline and still worth your attention. Writing a

7:51 level at a time, every position in parallel, beat diffusion on summarization in far fewer passes, and edged it on Sudoku, all on small models. The code and models are public on

8:01 GitHub, so you can download it today as research code. The second paper starts from the thinking. Its lead author, Jai Russo, opened the announcement on X with this: "Your LLM

8:12 agent has to think out loud, token by token, before it can search, run code, or act in the world. We humans don't." An agent here is a model that switches between thinking and using tools, like

8:24 running Python or searching the web. Hybrid agents keeps the tool calls and the final answer in text, and moves the thinking in between out of words. Its GitHub page puts it as: "Think in latent

8:34 space, act through text." Latent here just means in those hidden numbers, not in words. Step one, the model writes a special token that means start thinking in latent mode. From then on, it stops

8:46 choosing words. Its last layer hidden state, that same list of numbers from the coconut story, passes through a small adapter and comes straight back in as its next input. That

8:56 adapter is called Hybrid Link. It keeps the original state and adds a small learned correction on top. On the smallest model they used, it's about 10 million parameters, a tiny sliver of the

9:06 whole. Step two is the exit, and it's the neatest part. After every silent step, the model checks which entry in its vocabulary the adapted state sits closest to. When the closest one is the

9:17 closing latent token, it leaves latent mode. It stops thinking when its own thought most resembles the word for stop thinking, a mood I recognize. Step three, it writes ordinary text, a Python

9:28 call, a search query, or the final answer. The tool's result comes back in. Everything stays in the KV cache, the stored record of what the model has already processed. So, text, silent

9:40 steps, and tool results all remain working memory for the next round. The team trained this on top of Qwen 3, an open model family at several sizes, and released the larger checkpoints.

9:51 Per problem, the agent took roughly 170 to 180 steps in latent mode, thinking that never turned into a single written word. Across 11 benchmarks, from competition math and medical exams to

10:03 web research, the authors report about a third to a half fewer tokens overall. Fewer tokens usually means less waiting and a smaller compute bill. They also report inference about twice as fast.

10:14 And cost came out around a third of what the text agent spent, the biggest of their headline gains. Those numbers come with catches. The comparison is against text-based agents

10:23 the authors trained with a reinforcement learning method called GRPO, not against frontier APIs. And the main text doesn't spell out how cost was calculated, so that ratio can't be split

10:34 into GPU time versus price per token. Now, the question from the start. If a model thinks without words, can anyone read what it thought? Hybrid agents says yes, by design. Each silent step gets

10:46 matched to its nearest word in the vocabulary. And strung together, those make pseudo text, a rough readout a person can inspect. Read that claim slowly. The paper shows the readout

10:56 reads sensibly. It doesn't show the readout matches what the model computed. Those are different promises. A sentence that makes sense is a hint about the math behind it, and no more than a hint.

11:06 So, back to that receipt. Nothing on OpenAI's bill changes today. In the authors' tests, against agents they trained themselves, silent thinking cut the tokens an agent writes. And what you

11:17 can read shrinks to a nearest word readout, a clue to where the thinking went, with no proof of how it got there. Which brings us to your laptop. Hybrid agents' own limits say current serving

11:28 engines don't support switching between text and latent thinking across turns, and testing covers only Quen 3 with Python and search tools. There's no setting for this in Ollama, just a paper

11:39 and a GitHub page. NCM sits in the same spot. Small research models, public code, and nothing that makes your local chatbot faster today. Both papers are preprints submitted days apart this

11:50 October and every speed, cost, and accuracy figure from the two papers is the author's own. My call. Yes, writing word by word is the bottleneck labs are now attacking from both directions at

12:01 once. One paper paralyzes the writing and the other takes the thinking out of writing. The payoff is speed and cost on research size models. If you build agents and pay per token,

12:12 I'd watch hybrid agents most closely since cost is where it claims the biggest gain. If you run models locally, change nothing yet. If you work on text generation, NCM Sudoku result is the one

12:23 I'd want to see reproduced at a larger size. That Sudoku table is the image I'd leave you with. Same size, far more passes, and the model writing one word at a time solved about one hard board in

12:34 100. So, where does word by word waiting cost you most right now? The slow typing or a reasoning bill you can't read?