End of Tokens? What Meta's New Byte Model Actually Proved

0:00 Three numbers stuck with me. 496, 675, and 15,717. That's strawberry. The way GPT4's tokenizer hands it to a model, and it's the usual explanation for why chatbots

0:13 miscount the Rs. Those numbers stand for stir, A, O, and berry. Not one chunk is a single letter. So counting the Rs means proofreading a word the model was handed as three ID numbers. Now a dev

0:26 post from October 9th says tokens are a crutch and bite models win the long game. Behind it is a real paper breaking the token ceiling by Khalani Marata and colleagues at the University of

0:37 Washington and Metaphair posted September 11th. I wanted to know whether this is the end of tokens or a smaller result in a bigger headline. So I went through the paper its worked example and

0:47 its scaling tables and the older bite papers that came before it. Bite models read text one bite at a time. In this paper, small models that learn from a bigger teacher and read tokens win early

0:58 and then stall. The bite reading ones start worse and they're predicted to finish higher on less data. That part of the headline holds, but the model teaching those bite students is the 8

1:08 billion llama 3 and llama reads tokens. By the end, you'll be able to explain how a token model teaches a bite model and what the predicted win is worth. And hold one question for later. Does any of

1:20 this fix strawberry? Start with why tokens exist because they're a good idea. A tokenizer chops text into common chunks and gives each chunk an ID number. In this paper, the token student

1:31 works with a vocabulary of 128,256 chunks. Chunks buy you short sequences. In this setup, a llama 3 token averages about 4 and a half bytes of text. Do the division. And on average, a paragraph

1:45 written as bytes needs about four and a half times as many positions as it does as tokens. Fewer positions means fewer steps for the model to grind through. You pay for that with the strawberry

1:55 problem. Once stir, o, o, and berry become three IDs, the letters inside them aren't part of what the model receives. Whatever it knows about spelling, it had to pick up some other

2:05 way. A tech briefly report from December 2025 caught chat GPT on GPT 5.2 still answering two Rs. That report shows an open AI tokenizer, likely a newer one, splitting the word a little differently.

2:19 A bite model skips the chopping. There are only 256 possible bytes and each plain English letter is one of them. So, strawberry arrives as 10 separate symbols with all three Rs sitting right

2:30 there. Each side gives something up. Tokens give you a huge vocabulary and short sequences. Bites give you a tiny vocabulary and long ones. What's still open is which trade wins once you train

2:41 for long enough and what that costs. Ifittes are that obvious, you'd expect someone had tried them years ago. someone had Google's BT5 paper from May 2021 trained bite level models and found

2:54 them significantly more robust to noisy text and better on tasks that hinge on spelling and pronunciation. Then in December 2024, Meta published the bite latent transformer or BLT. BLT groups

3:07 bytes into patches cut wherever the next bite gets hard to predict, so easy stretches cost fewer steps. Meta scaled it up to 8 billion parameters and 4 trillion training bytes. According to

3:17 that dev post, a September thread on X presented BLT as the brand new breakthrough, which makes the brand new breakthrough about 21 months old by my count. Maret's paper is a separate study

3:29 about distillation. I'd assume the teaching part at least was new. Bo and colleagues paper from February 2026 says otherwise. They distilled Llama, Quinn, and O Elmo models into bite models that

3:42 kept most of their teachers performance using about 125 billion bytes. Reading that left me with the real question. What's left for this paper to add? Two things. An exact way to turn a token

3:53 teacher's guesses into bite guesses. And a long scaling study, a sweep of models around a billion parameters trained on up to about a trillion bytes to see where each kind tops out. First, the

4:05 teaching. In distillation, a big teacher model reads the training text and at every position gives a guess, a probability for everything that could come next. A small student trains to

4:15 match that whole guess rather than only the one answer that really came next. Here, the teacher is the 8 billion llama 3. All its guesses get written to disk once, a step the paper calls logit

4:26 dumping, and the students train from that saved record. Logits are just the raw scores behind those probabilities. saved in full, that record gets heavy fast. A token teacher's full guess at

4:37 one position has a score for every entry in a vocabulary of over a 100,000. So, the usual move is top K truncation. Keep the few highest guesses and throw the rest away. Bite guesses don't need that.

4:49 With 256 possible bytes plus a few markers, the whole guess fits per the paper. No truncation is needed and stored teacher outputs come out at roughly 1/5 the size. But that skips the

5:01 hard part. A token teacher never guesses bytes, only whole chunks. So, how do you turn a teacher that's thinking eso into a guess about the single next letter? Figure two in the paper answers with a

5:11 worked example, the word tiramisu. For tiramisu, the teacher's tokens are a start marker, then t irram, eso, and an end marker. A paper about the future of language models, and its central example

5:24 is dessert. Stand at the spot right after. Here, the bite student has to predict one bite. Meanwhile, the teacher's guess covers the whole next chunk spread across candidates like ESO,

5:34 ISK, and the short chunk is marginalize it. The older conversion starts out easy. For the first bite, it adds up the probability of every candidate that starts with the same letter as ESO.

5:46 Their total is the chance that letter comes next. Picture the teacher's guess as one full jug and each possible next bite as a cup. Every token starts with some bite. So on that first step, each

5:57 token pours into its first letters cup. ESO, ISK, and ES all land in the same cup. Nothing spills. Two byes in, it gets harder. Marginalize it now keeps only the tokens that match the bites so

6:09 far. With two letters already matched after, only ESO and ISK are still in the running. But the short chunk ES match two, and it ends right there. Anything after it would need a fresh guess from

6:20 the teacher, and this conversion never asks. So its share 0.125 gets dropped. Call it 1 and 8 poured on the table. Then it tops up what's left. Isu and ISK get rescaled to 0.8 and 0.2. So the cups

6:35 add back up to one. Everything looks tidy again. They're just no longer exactly what the teacher said. Marginalize it then only approximates. Whenever a candidate token ends before

6:45 the bite being predicted, a slice of the teacher's guess gets thrown out. What's left gets stretched to cover the gap. Over a long training run, the student learned from a bent copy of its teacher.

6:55 End of token fixes this with one new symbol, a marker inserted after every teacher token in the bite stream. Adding it takes a student's vocabulary from 260 symbols, the bytes plus a few special

7:07 markers to 261. Written out, tiramisu becomes t, then a stop marker, then the four bytes of irram, stop. The three bytes of ISO, stop. From here on, the student predicts those markers right

7:21 alongside the letters. Back at the spot, two bytes into ESO. Now, the student has three choices instead of two. U, K, or end of token. Whatever belonged to the short chunk I, S, the part that spilled

7:33 before goes to that marker. After the stop, the next bites come from the teacher's guess at the next position, which by my reading is already sitting in the saved record. In the cup picture,

7:44 that's one extra cup labeled stop sitting exactly where the spill used to land. Nothing hits the table, so nothing needs stretching. Marati's team claims the teacher's distribution comes through

7:54 exactly where the cups mislead you is time. A real conversion doesn't pour once. Within a chunk, each new bite goes back to the same teacher guess and filters it again by what's been read so

8:05 far. Those cups only show one bite of that. No extra teacher time either. End of token needs no additional calls to the teacher, so it fits the save it once lowjet dump from earlier. Students pay

8:16 for it though. A llama 3 token averages about 4 1/2 bytes. So a marker turns up about every 4 and 1/2 bytes. The paper prices those markers at roughly 31% more compute for the same amount of training

8:28 data. So the whole trick is three moves. A token teacher guesses chunks. Each chunk gets filtered bite by bite. One extra stop symbol catches what used to fall through. If you train small models

8:39 yourself, that 31% is the number to circle. If you only call models through an API, skip to the results. They'll tell you whether any of this reaches you. Do exact guesses pay off? A fair

8:51 test needs a fair fight. So, the paper builds students with the same body. 25 layers, each numbers wide. One reads tokens. One reads plain bytes, 260 symbols. One reads bytes plus the end of

9:04 token marker, 261. Same body, different weight. The token student has 1.81 billion parameters. The bite students 1.28. Most of that half billion difference sits in the token models

9:17 vocabulary. It keeps a row of 248 numbers for every one of its 128,000 tokens. Byte students keep 261 rows, which is shorter than plenty of lock file diffs you've scrolled past. Each

9:29 student trains on up to about a trillion bytes from the Llama 2 pre-training mix. Then it scored on reasoning and questionans answering benchmarks, a Python coding test, and translation

9:39 between English and German both ways. If tokens were simply the worst idea, you'd expect the bite students to lead from the start. They don't. At small training budgets, the token student is ahead,

9:50 which fits the short sequences from earlier. Then it flattens. Keep adding compute and the token curve levels off while the bite students keep improving and pass it. Where they cross is the

10:00 headline. Where do the curves end up in the papers table five? At one learning rate setting, the distilled token students ceiling is 48.4% average accuracy. Distilled end of token lands

10:12 at 52.4. So the abstracts up to 4% means four points. 48 to 52, not 4% of 48. And the abstracts word for when this happens is asymtoically, which is a lot of weight

10:25 to hang on one adverb. Efficiency is the part I'd remember. End of token matches the distilled token student with about one6th of the training data and its stored teacher outputs are about a fifth

10:36 the size. Those curves also predict wins over models you could deploy up to 6 and 1/2 points over the 1 billion llama 3.2 up to 8.1 over the 1 billion Gemma 3 and up to 2.1 over Gemma 2B. My read is

10:51 those are points too the same as the four point ceiling gap even though the abstract says percent. One small mess here on benchmarks. The abstract says eight, but figure 1's caption says those

11:02 averages cover six. That leaves it a little fuzzy which tasks sit behind those comparisons. Now trace the headline numbers back. Both ceilings and the wins over outside models are

11:12 predictions from curves. No train model scored them. Marati's team fits two curves. First comes bits per bite, how surprised a model is by each bite of held out text tracked as compute grows.

11:24 Lower is better. At the end of that formula sits a constant, a floor the curve bends toward and never crosses. A second curve turns bits per bite into benchmark scores. Chain the two. Push

11:34 compute out toward infinity and that floor becomes the ceiling in the headline. Every trained model here sits around a billion parameters. 52.4 sits at the far end of a fitted curve.

11:45 There's no checkpoint for it you could download. And why get the lower floor, the curve doesn't explain. All it does is fit. Change what you fit to and the curve moves. According to Alpha Archive

11:56 summary, an alternative validation set shrinks the predicted gap from four points to 1.8, 50.5 against 48.7. Alpha Archives figure hasn't been matched to the paper's own table yet. Then there's

12:09 running it. End of token models, the authors write, remain significantly expensive at inference time, and they don't claim a speed win. In a paper, significantly is about as close as the

12:19 authors get to raising their voice. Their limitation section adds that the study covers dense transformers at roughly one fixed size. So it says nothing yet about the much bigger models

12:29 you chat with. And the teacher, the model every bite student learns from is the 8 billion llama 3, a token model. So end of tokens keeps its question mark, at least for this paper. Every bite

12:41 student here learned from a model that reads tokens, which brings back the question from the start. Does this fix strawberry? I went down the paper's benchmark list expecting at least one

12:50 spelling task, reasoning, question answering, a coding test, translation. Not one of them tests spelling. For spelling evidence, you need the older work by T5 found bite models better on

13:02 spelling sensitive tasks back in 2021. And BLT's paper reports 99.9% on its spelling tasks against 1.1 for Llama 3. So look good for Strawberry on evidence from 2021 and 2024. What this paper adds

13:17 is a cheaper way to train small bite models. Copy a token teacher's guesses without dropping any and you match the token student on about a sixth of the data. Whether it's students spell better

13:27 is a fair guess. Nothing in this paper measures it. My call. If you train small models for a long time, bite students are the bet to watch. They're predicted to top out four points higher or under

13:38 two on another validation set by alpha archives reading and to match the token student on about a sixth of the data. Doing it exactly costs roughly 31% more training compute for those stop markers

13:50 and the model costs more to run. If you serve large models, ship on device or care about speed per request, I wouldn't plan around this yet. In this paper, the biggest model is the teacher and the

14:01 teacher reads tokens. Next time a post says bytes beat tokens, I'd check three things. Whether the win was measured or predicted at what model size and whether a token model did the teaching. for this

14:13 paper that's predicted about a billion and yes back to those three numbers 496 675 15,717 gpt4's tokenizer sees strawberry that way and in this paper a model that reads

14:27 in chunks like that is still doing the teaching so for what you build is a small model that sees every letter worth longer training and a costlier model to