The Local Coding Breakthrough We Waited For? (Bonsai 2)

0:00 a 27 billion parameter model squeezed into 5.9 GB, small enough to sit on the laptop you're watching this on. And the claim is that it keeps 98% of its benchmark score. That went up yesterday

0:12 from a lab called Prism ML open weights, free to download the same hour they announced it. What they compressed is the new Quinn 3.8 at 27 billion. The compressed build is called Bonsai 2.

0:24 Here's the pitch a lot of you will hear. Stop paying for a coding assistant every month. And run this one on the machine you already own. Two things stand between you and that. 98% is an average

0:35 and 5.9 GB is a file, not the memory it runs in. Start with the model underneath at full precision. Quen 3.8 is about 54 GB. You need a data center card just to hold it. That base model is Alibaba's

0:49 out in August and it's dense, which means every one of those 27 billion parameters fires on every token. That's exactly what makes it heavy to run and exactly what makes it worth shrinking.

1:00 We covered the first bonsai on this channel back in July. It fit a phone and it kept most of its brain, at least on paper. Then somebody put that first bonsai through a real agent coding test,

1:10 the kind where the model has to run its own commands. It scored 7.9%. The benchmark average looked great. The agent fell over. So this video asks one thing in three parts. how the squeeze

1:23 works, where it actually costs you, and whether it really runs in 6 GB once you're doing real work. Take the squeeze first. Quantization just means every weight in the model, and there are

1:33 billions of them, gets stored in fewer bits than usual. Normally, that's 16 bits each. Bonsai stores most of them in under two. It forces every weight to one of three values, minus1, 0, or + one.

1:46 That's the whole idea, and it's called turnery. Think of it like sorting every number into three bins. below zero, zero, or above. You throw away the exact value and keep only which way it leaned.

1:57 Three values is about one and a half bits of information. Group the weights in blocks of 128. Keep one full precision scale for each block, and the average lands at 1.76 bits a weight. The

2:09 part that keeps this from falling apart is a rotation done before the rounding. It spreads the handful of extreme weights across the group. So, no single rounding error gets to do much damage.

2:18 Do the arithmetic and 16 bits down to 1.76 is about 9 times smaller. 54 GB becomes 5.9. There's a bonus. When every weight is minus 1, 0, or + one, the math stops being multiplication. It's add,

2:33 subtract, or skip. That's cheap, and it's a big part of why a model this size can run on a plane machine at all. Now, why three values and not two? The first Bonsai shipped an even smaller build

2:44 that used just minus one and plus one. One bit a weight under 4 GB. That one had a problem. Hand it a task and it would ramble past 14,000 tokens and never actually finish. Adding back the

2:55 zero, that one extra state is most of the difference between a model that stops and a model that doesn't. Turnary is the compromise that ships. One more catch before we move on. that 5.9 is the

3:07 ideal packing. The file you actually download in the format that runs today is 8.6 GB. Part of that is the runtime, which can't store 1.76 bits exactly, so it rounds up to a bit more. And part of

3:20 it is the eyes. Of the 27 billion parameters, about half a billion make up the vision tower, the piece that reads images, and that piece stays at full precision. So, it doesn't get the

3:30 discount the rest of the model does. Right? The file is small and it's fast. Now, the question that actually decides whether you can use it. What did 98% cost you? That 98 is an average across

3:42 20 different tests. Pull the average apart and it stops being one friendly number. Coding barely moves. The original score is 82. The compressed one 81 12. Writing a function, fixing a

3:54 failing test, a quick refactor, the day-to-day stuff, it handles. This compression is almost free for that. And that's the good news up front. Math is basically untouched and instruction

4:05 following even ticks up a hair. So far, the squeeze looks kind, then it bites. The two things an agent leans on hardest, calling tools and running multi-step tasks, take the biggest drop

4:15 of anything it does with text. There's a clean reason. A single code completion is a short task done in one shot. An agent that has to chain 10 tool calls without losing the thread is a long one.

4:26 And low bit rounding is a small error that compounds at every step down a long chain. In plain terms, that's the gap between a model that suggests a fix in the chat and one you trust to open the

4:37 files, make the edits, and run the test on its own. The first is help, the second is an agent, and the second is where the points went. Reasoning slips almost three points, too. Envision drops

4:48 the most of all, three full points, because the part that reads images, remember, got squeezed the hardest of anything. So, if the plan was to feed it screenshots and have it read your

4:57 screen, that's the one job where this compression hurts you most. So, the read on quality is two sentences. As a coding assistant that answers one question at a time, 98% is basically real. As an

5:09 agent, you point at a task and walk away from, that average is hiding the exact skill you'd be relying on. And we've seen this movie that 7.9% from the first bonsai was the same shape of claim, a

5:21 strong average, then a collapse on a real agent harness below a 9 billion model, a quarter of its size. The new one does score higher on the tool calling test, about 75 out of 100, up

5:32 from the last generation. But here's what's still missing, and it's the number that matters. An independent agent harness run on Bonsai 2. The paper score went up. The real world score, the

5:42 one that caught the first Bonsai out, doesn't exist yet. The first hands-on test did land today. Bjan Bowen loaded it on a single graphics card and put it through a live run, which is the first

5:53 real look most people will get. The release thread filled up fast, too. Passed 1,000 votes in a few hours. The local model crowd pulling it apart in real time, and the early picture lines

6:03 up with the benchmark shape, reliable for focused coding help. Still unproven the moment you hand it a long multi-step job and stop watching. So that's the quality question mostly answered. Strong

6:14 assistant unproven agent. Now the other half of the title 6 GB. Six is the weights at rest sitting on your disc. The trouble starts the moment you load them to run. In the format that runs

6:25 today, the model sitting in memory is 8.6 GB. You're already past six before you've typed a single word. Then it grows as you work. Every token you feed it and every token it writes back gets

6:37 kept in what's called the context cache. That's the model's short-term memory for the conversation so far, and it stacks up fast. Prism ML measured this themselves. Hold a 100,000 tokens of

6:47 context, which is roughly a medium code file plus a real back and forth, and the whole thing needs 14.7 GB. So, let's line those numbers up. The figure on the box is six. Loaded, it's 8.6. Give it

7:00 real work, and it's closer to 15. Here's the part that's clever, though. That same 100,000 tokens on a normal 4-bit version of this model would cost 25 GB. The turnary build nearly halves it. So

7:12 the compression doesn't just shrink the file, it buys back the context room, and context is the thing you run out of first, which sets up its own irony. On paper, this model can hold 260,000

7:23 tokens of context. Your card cannot. You'll run out of memory long before you run out of context window. So on the 8 GB card a lot of people actually own the weights fit and a short chat fits. Open

7:35 a big file in a long session and it spills over into system memory and slows right down. The machine that runs this comfortably at real context is a 16 GB one which is still a laptop, not a data

7:47 center, just not the 6 GB the headline sold. One practical wrinkle worth flagging. This doesn't drop straight into the Alama you might already have. It needs the lab's own runtime on

7:57 Apple's framework or a patched build of the usual engine. A year from now, that's probably a non-issue. Today, it's an extra step, and running it isn't exotic. You pull the 8.6 GB download,

8:09 point the runtime at the folder, and it's answering in a few seconds. No cluster, no queue, no monthly bill. When it does fit, it's quick. On a top-end gaming card, it generates around 130

8:21 tokens a second. on an M5 Max laptop about 47 which is faster than you can read and it sips power doing it. On one older card it's roughly a thousandth of a watt hour per token. A small data

8:33 center card pulling just 72 watts still runs it at 30 tokens a second. This is a model you could leave on all day. You don't even have to install anything to see it work. It runs in a browser tab on

8:44 your graphics card through the web at around 55 tokens a second on that same laptop. It isn't text only either. It reads images a 27 billion multimodel model answering you inside a browser out

8:56 of under 9 GB of memory. Two years ago, that sentence was a joke. And it's Apache licensed, so you can actually build on it at work, ship it in a product, no permission and no fee. For

9:08 something this capable and this small, that's the part that should make a team pay attention. So, here's where I land. Bonsai 2 is a real local coding assistant and the best one anywhere near

9:18 this small. For a solo developer on a 16 GB machine, it can take over the routine coding help you're renting by the month. The concession is just as real. The 98% is an average that hides a weaker agent.

9:30 The memory to run it at real context is closer to 15 GB than six. And no independent test has proven the long hands-off agent work yet. One more warning buried in that table. compressed

9:41 the model and one of the 20 scores actually went up. That doesn't mean squeezing a model improves it. It means that particular test stopped telling the two apart, which is a reason to distrust

9:52 the average, not to trust the model. So, the thing to be annoyed at isn't the lab. It's the habit of reading one average and stopping there. The average said 98. The coding row said the

10:02 assistant is fine. The agent row said, "Don't trust it alone yet." Same table, three different answers, and only one of them fit on the announcement. Against the first bonsai, this is still a real

10:13 jump. That one was July, and its whole story was size. The first 27B that fit a phone. This one is September, and its whole story is quality. Better base model, and this time, the coding

10:24 survived the squeeze. The part that didn't survive is the part every small model still struggles with. So, it's a fair place to be losing. So, can you use the computer you already own for coding

10:35 help? Yes, today for free for handing an agent the keys. Not yet. And the number to watch was never the file size printed on the box. It's how much of your own context you can afford to keep in

10:47 memory. If a clear look at one model a week is useful to you, subscribe and I'll keep pulling the averages apart.