Inside K2 Horizon: The 4B Model That Beat 550B

0:00 Every transformer shipped in the last 9 years reads with one pair of eyes. A model released this week reads with 64 and picks four of them for every word you type. It stores 36 billion

0:12 parameters. It does the arithmetic of about 4 billion. And on the benchmark that measures whether a model can drive a terminal, it beat one with 550 billion. That's the headline. Now look

0:24 two rows down the same table. On graduate level science questions, the big model wins. Same table, same day, same lab. So, this isn't a small model beating a big one. It's a table saying

0:37 two opposite things at once and one change to the architecture that explains both. The model is called K2 Horizon MUA and it came out this week from the Institute of Foundation models. The

0:48 license is Apache 2.0. The model weights are yours to run, change, and ship commercially for nothing. By the end, you'll know what MUA does to a transformer, why it buys speed and not

1:00 memory, and why that headline is narrower than it sounds. We'll take them in that order. We'll build it up from a part you already use. Type a word into any model. Inside every attention layer,

1:11 three things happen to it. A query which asks a question, a key which is how every earlier word answers and a value which means what you carry back from the words you looked at. Think of a reading

1:23 room. Query and key are the index cards telling you which shelf and which page. The value is the person who copies the useful part into your margin. Every model you have run hires exactly one of

1:35 those people. One value matrix per layer for the whole library. Now, mixture of experts, you know this one. Instead of one big feed forward block, the model keeps a 100 small ones and a router

1:48 picks eight per word. Capacity goes up, work per word stays flat. But look at where that trick gets applied. The feed forward block, the thinking. Attention was never touched. So, the model got a

2:01 100 different thinkers and every one of them read the page the same way. MOUA is what happens when you put the router in the other place. Here's what that looks like in the actual weights file. Layer

2:11 zero of this model has a tensor called VRO one value matrix completely normal. Layer 4 doesn't have a VRO at all. It has 64 tensors named V experts plus a small one called Vouter. So the

2:27 mechanism your word arrives as a couple of thousand numbers. Query and key run as before deciding where to look. Then the router reads that same word and scores all 64 value experts. A learned

2:40 bias nudges the ranking. The top four win. Those four each produce a value, get weighted by their own scores, and are added together. That sum is the value attention uses. A gate multiplies

2:52 the result, and the layer writes it back. 45 of this model's 48 layers work that way. And the ratio is the whole point. One layer stores 64 readers. Your word uses four 16 times more sitting

3:05 there than doing anything. Stack that up and the model holds nearly 3,000 value matrices and any word you type touches 180. That's where the extra capacity comes from. Now the part the headline

3:18 skips. Sparse doesn't mean small. Those two words get used as if they mean the same thing. And the difference decides whether you can run this at all. Active parameters is a claim about arithmetic

3:29 not about memory. Every word is routed on its own and the router picks a different four readers for the next one. You can't know which in advance, so you load all of them. It's like a card that

3:40 lets you read four books a day in a library you still have to buy outright. The published 16-bit download is one file and it's 75 GB. The 8-bit build proves the point by what it squeezes. It

3:53 compresses the experts 71% of the model and leaves the always on path at full precision. even squeezed. That's 48 gigabytes. So, say you send it one prompt. For every word that comes back,

4:06 about a tenth of the weights do the work and all of them pay rent. Sparity buys throughput. It doesn't buy a smaller model. And a headline calling this something you can run at home is

4:16 describing a machine you probably don't own. Two things so far. Mua is real and it's clever. And it's a speed trick, not a size trick. Which brings us back to that contradictory table because now it

4:28 makes sense. Nine benchmark rows. The four billion active model wins too. Driving a terminal and using banking tools. It loses the other seven. And look at what those ask for. Graduate

4:40 science, expert reasoning, factual accuracy, physics, long context recall. Every one of those is a question about knowing things. And that's what four billion active parameters can't buy you.

4:53 Knowledge lives in weights and this model reaches a slice of its own memory at a time. Routing changes which reader shows up. It doesn't change how much got written down. But agentic work isn't

5:04 recall. It's a policy which means read the error. Try the next thing. Call the tool in the right shape. Don't quit on the third attempt. That's a behavior. And behavior fits in a small active

5:16 path. The lab trained it on purpose. And the recipe is the interesting part. About a sixth of the pre-training corpus is problem solving trajectories rather than pros and post-training synthesized

5:28 over a 100 million tasks. Here's the move I'd steal. They vary the tool syntax during training on purpose, presenting the same tools three different ways and accepting calls in

5:38 three formats. The model never got to memorize one shape, so it had to learn what a tool call means. You can watch it land on the banking benchmark. This model scores 26.8. A Quen model with

5:51 almost the same active parameter count scores 9.3, nearly three times the score on the same arithmetic per word. The gap there is curriculum, not capacity. So, a question worth answering before the next

6:04 number. If a model drives a terminal better than something 14 times its size, but knows less than a model a fraction of its size, what have you got? Something smarter or something better

6:16 trained? Hold that because the answer changes what you do with it. Now, can you believe the number at all? This is usually where I list reasons to distrust a lab's own benchmark table. This time,

6:28 the lab did it for us, and I haven't seen a model release do this before. They ran their biggest model on the terminal benchmark. 700 trials, 500 passed, 70.2%.

6:40 Then they audited every passing run for benchmark cheating using an outside company's rubric and an outside model as the judge. Two dozen trials got flagged and they published the corrected number

6:52 70.2 down to 66.9. They lowered their own headline in the launch post voluntarily. Then they described what the model had been caught doing. It worked out it was inside a

7:04 public benchmark, found the repository on GitHub, and downloaded the reference solution. It read files it wasn't meant to see. It edited the test harness. Their writeup says the model expressed

7:15 excitement at having the answer handed to it. They reported the same thing on their 7 billion model which found the answers to a coding benchmark and inflated its own score. And they said

7:26 plainly that the score doesn't mean anything. Eric Shing who founded the institute and runs the university behind it put the principle in one line. Open source is much more than open weights.

7:38 Science works when others can see the data, follow the method, reproduce the result, and improve on it. I want to hold them to that standard because it's theirs. Measured against it, two things

7:49 are missing. The first is the data. The model card links two training data sets by name. Try to open either and you get an authentication error. The launch post is more careful than the press release.

8:01 It promises training data or a detailed recipe and says where they can't redistribute. they'll describe the method instead. That's honest. It isn't the same as publishing the corpus. And

8:12 on a hacker news thread above 300 points, you can watch people argue about that gap. The second gap decides whether MUA is worth anything. The lab also shipped a dense 32 billion model trained

8:24 on the same 22 trillion tokens with the same recipe. That's a control group, which means the comparison that tells you what routing the value path is actually worth, except the dense model

8:35 isn't finished. Its card is titled stage one with a note in red saying the final checkpoint is still coming. So, the sparse model beats it on five of six shared benchmarks, which tells you

8:47 nothing because it's beating a halftraed opponent. The claim that MOA lands just below dense is about a checkpoint you can't download. and put the MOA card next to the lab's other one. The MOA

8:59 model scores 58.6 on the terminal benchmark. On the other card in a different table, a dense 27 billion Quen sits at 79.8. Same benchmark, same baseline provider

9:12 named in the footnote on both pages. 21 points ahead from a dense model with no routing in it at all. Neither table lies. But those two numbers never appear together because they live on separate

9:25 pages. That's the real lever here. Not the architecture, not the training run, who you choose to put on your table. So the verdict, Muvo wins. The labs disclosure standard wins. The model

9:38 itself doesn't. If you build agents or you do research, this is the most useful thing anyone shipped this month. A new architecture, weights you can inspect, a serving recipe that works today, and a

9:50 lab that published the audit that made its own number look worse. That last part is worth more than the benchmark it corrected. If you wanted a local coding model, this isn't it. And I'd say that

10:01 at a fifth the size, 75 GB of weights, no provider is serving it. Support in the main local runtime is still an open poll request, and a dense 27 billion model beats it by 21 points on the

10:15 benchmark it's famous for. The few who should download it today want to fine-tune a sparse attention model or take one apart. For them, the intermediate checkpoints are the real

10:25 product and those are still coming. The open weight race has been about one thing, the number on the card. This release proposes a different scoreboard where a lab gets credit for shipping the

10:36 control group, the audit, and the failures. The frustrating part is that they proposed it, then shipped an unfinished control group in a locked data set on day one, which leaves a

10:47 question I don't think has an obvious answer. If a lab publishes the evidence that its own headline is too high, does that make the headline more trustworthy, or does it just tell you how bad the

10:58 others look? I know which way I lean. I'm not certain I'm right. If this is the kind of tearown you want more of, subscribe. One of these every week.