Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Meta and UW's paper shows that byte-level student models, trained via knowledge distillation from token-level teacher models, have training curves that eventually surpass the teacher's, reaching higher predicted accuracy at scale. The alignment problem between token teacher and byte student is solved cleanly by inserting an end-of-token marker. The practical takeaway is that tokens aren't obsolete yet since byte models remain expensive, but the curves suggest bytes could win for small models given enough data.
Key points
Byte student models' training curves predict higher final accuracy than token student models.
An end-of-token marker enables exact alignment between token teacher probabilities and byte student predictions.
Only models around 1 billion parameters were tested; byte models remain computationally expensive.
The paper did not test spelling improvements; earlier BLT work provides that evidence.
Authors recommend keeping tokenizers in production today but watching small models with long training.
Tools mentioned
Techniques
- knowledge distillation
- byte-level modeling
- tokenization
- end-of-token marker
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Strawberry is three tokens to GPT-4, and that's why a chatbot can miscount its R's. The pieces are s t r a w and berry. None of them is a single letter, so the model counts letters it never sees. A
September paper from the University of Washington and Meta AI builds the opposite. Small models that read raw bytes, basically single letters, taught by a token model. Meta's
8 billion Llama 3. Can letter readers beat chunk readers? The paper's answer, byte students start behind and curves fitted to their training runs say they finish ahead. You'd expect the teaching
to fall apart. The teacher guesses chunks, the student guesses letters, and they don't line up. With one extra symbol, they line up exactly. The paper's example word is tiramisu,
and the teacher's tokenizer serves it in uneven layers. A lone t, then i r a m, then i s u. The student reads it byte by byte with an end of token marker where each chunk ends. Picture the teacher's
guesses as water in pipes branching at every letter. After t and i r a m, the student's odds for each next letter are the water flowing down that letter's branch. In the paper's figure, the
branch for i s holds three quarters of it. Some of that water belongs to a chunk that stops after the s. The simpler method, grouping by shared letters, lets it leak. End of token
drains it into the marker instead, so the student gets the teacher's exact odds with no extra teacher runs. The picture breaks here. Pipes carry a steady flow, but the teacher pours a
fresh set of odds at every chunk. Maritai and colleagues measured the early part. Extended out, those curves capped the token student at about 48% average accuracy. The byte student
levels off higher, near 52. Nobody's trained one that far yet. Only students around 1 billion parameters were tested, and the authors say byte models are still expensive to run. This paper
didn't test spelling. Evidence that bytes spell better comes from BLT, Meta's earlier byte model. Tokens aren't over. Ship a model today and I'd keep the tokenizer since bytes start behind
and cost more to run. Watch small models given lots of training. That's where the curve say bytes could win first. The byte student that might win still learns from a chunk reader.