They Removed Quarter of an AI’s Brain. Here’s What Happened......

0:00 I opened Mercedes' research log with the question you'd want answered before cutting layers out of your own model. Take six out of the 8 billion Qwen 3, retrain nothing, and what does it

0:10 forget? Their table gave me the answer. The reasoning tests keep up to 94% of their scores, while the facts test falls most of the way to guessing. The write-up names what happens next in one

0:20 short line. It is a cliff. Take 10 layers out, and on facts it's guessing outright. That split is what hooked me. A model that still handles a common sense question, but misses facts it used

0:31 to get right. So, I went through the log, the Lacko paper, the code's readme, and the other layer cutting studies. Mercedes' headline leads with how much reasoning survives, and the Lacko paper

0:41 behind it kept most of its task scores with no training at all. Read that way, big models carry spare layers you can delete nearly for free, and it reads like something new. Part of that holds.

0:52 Other researchers removed up to half a model's layers and kept most of their question answering scores after a small repair. But, that same table shows the facts going. The method dates from

1:02 February 2024, and the code doesn't merge the way Mercedes read the paper. So, I'll carry one question to the end. Why would a cut spare the reasoning and take the facts? By the end, you'll know

1:13 how the merge works, what this model forgot, and whether a cut like this belongs near a model you care about. Start with why the headline felt believable. Lacko is a paper by Yifei

1:23 Yang, Zhaoying Zou, and Hai Zhao, first posted in February 2024. Its idea is to collapse several later layers into an earlier one with no training afterwards, and keep most of what the model could

1:35 do. Their results backed it up. Take about a quarter of a Llama 2 model's layers away, and it kept over 80% of its average task performance. The paper was accepted to Findings of EMNLP later that

1:47 year, so this is peer-reviewed work. When Juhi, writing Mercedes' log this March, applied it to the 8 billion Qwen 3, I expected the same shape. A gentle slide across every test, a little lost

1:59 everywhere. The first thing that didn't fit came near the end of that log in Mercedes' own conclusion. They call LeGO a step before fine-tuning, not a compression technique on its own. That's

2:09 an odd way to describe a free shrink. To see why they'd say that, you need the two kinds of test. The reasoning ones are short multiple-choice common-sense questions.

2:18 MMLU is the one Mercedes treats as the knowledge test. It's multiple-choice, too, four options per question. So, a model that knows nothing still scores about a quarter. Multiple-choice hands

2:29 out a quarter of the marks for free. Now, the cut itself. Models like this are stacks of layers. Your text goes in at the bottom. Each layer reworks it and hands it up, and

2:39 the top predicts the next word. Each layer is a big block of weights, the numbers it learned in training, and Mercedes counts 36 layers in the 8-billion parameter. LeGO's first step

2:49 decides what merging even means. Pick a base layer. For each layer above it that you want to fold in, subtract the base's weights from that layer's position by position, and add the difference onto

2:59 the base. The paper calls the rule reserving differences while seeking common. Second, it works from the top of the stack downward, folding in a few layers

3:08 at a time. It only works inside a band of layers you choose, so you can fence off the ones you don't want touched. It also leaves a minimum gap between one merge and the next. Third, every merge

3:19 gets checked. A few calibration sentences run through the old model and the merged one. LeGO compares what comes out the top using cosine similarity, a score for how closely two lists of

3:29 numbers point the same way. Below the threshold you set, the merge is thrown out. That check is the part I'd watch. With no retraining, the only judge of a merge is whether a set of sample

3:40 sentences still comes out looking alike. Mercedes fenced off the bottom and the top of the stack. Their check ran on 10 sentences, where the paper used about 500. That's a lighter review than most

3:51 pull requests get. Their repositories readme list different settings, so I'm going by the blogs, which match their results. A few merges, working down from high in the stack, squeezed a block of

4:01 neighboring layers into one. Six layers gone, 30 left, and nothing retrained. That works out to about a sixth fewer layers. The file shrinks by less because the parts that turn words into numbers

4:13 and back weren't touched, and the log doesn't give the new model size or speed. Up to here, a sensible rule and a sensible check. The part of Mercedes log that surprised me most is about the

4:24 code. They describe the paper's formula as averaging the layers together. The official code they ran doesn't average anything. When it finishes, the merged layer holds a copy of the last layer it

4:34 folded in. Think of the merge as directions. The formula, as the paper prints it, measures every difference from the original base and adds them all at once. The code measures each

4:44 difference from wherever the base has got to so far and updates the base after every step. Follow that through. From the base, add the gap to the next layer, and you're standing on the next layer.

4:55 From there, add the gap to the one after, and you're standing on that one. Keep going, and you finish on the last layer with every earlier step written over. Done in one go, the paper's sum is

5:06 neither an average nor a copy. Done step by step, it lands on the final layer. Mercedi read the paper as averaging, and when they tried averaging, the model wrote gibberish, sometimes mixing

5:17 languages at random. The directions picture breaks in one place. On a map, any spot between two pins is somewhere you can stand. A layer is a huge grid of learned numbers, and a blend of two

5:28 working layers isn't guaranteed to be a layer that works. That's Mercedi's guess at why averaging failed, and they don't test it further. That's Mercedi's own account of their code, and a second

5:38 builder backs it up. An uploader on Hugging Face called AlifFunk made smaller Qwen 3 models the same way and describes the merged layer as the last layers weights with the base layers

5:48 small scaling settings. It's norms kept. In practice, the merge keeps the top layer of the block and lets the rest go, which is a very polite way to delete them. A block of layers became one

6:00 copied layer approved on a handful of sentences with no repair. What did the model keep? On the short common sense tests, most of it. The best, Winona Grande, kept about 94% of its original

6:12 score. The weakest, Hella Swag, still kept close to 4/5. If the paper's overall scores told the whole story, you'd expect the knowledge test to land in that same range, maybe a little

6:22 lower. It didn't. MMLU fell from about 77% to 31. That's barely clear of guessing. Of everything it scored above guessing, call it an eighth survived. Then they cut further. Eight layers out,

6:35 guessing. 10 out, still guessing. The reasoning tests, meanwhile, held on to most of their scores. That's the cliff the log describes. By the first cut, most of what the model knew above

6:46 guessing was already gone, and by the next cut, it hit the floor and stayed there. Why it collapses that fast instead of fading, the log doesn't say. Six layers cut, reasoning mostly fine,

6:57 facts mostly gone. Say you put that model behind a help chat. On short common sense questions, it would mostly hold up. On anything that needs a remembered fact, its knowledge quiz says

7:07 to expect barely better than a guess. On short tests, reasoning survives, and that's the claim to check next. In October 2025, KU Wang, Shiwei Liu, Jonas Geiping and colleagues looked at long

7:20 reasoning, where a model writes out its working before it answers. Pruning even one or two layers could make that collapse, while short tasks looked stable the whole time. Supervised

7:30 fine-tuning, the usual follow-up training, didn't bring it back, and every reasoning test in Mercedes' results is a short multiple-choice question. So, the headline needs one

7:39 word added. Short reasoning survives. On that evidence, the short benchmarks can stay green while long reasoning breaks underneath. Anyone who's shipped software knows how that release goes.

7:50 That leaves what fixing the damage costs, and the research gives very different answers. In March 2024, Andrey Gromov, Daniel Roberts and colleagues chose which block

8:00 of layers to remove by how similar the layers were and took out up to half with little loss on question answering tests. They got there with a small healing fine-tune afterwards using QLoRA, a

8:11 cheap way to fine-tune a squeezed copy of the model on a single GPU. NVIDIA went bigger. In August 2024, its team cut half the layers out of the 8 billion Llama 3.1 to make a model called the 4

8:23 billion Llama 3.1 Minutron. They ranked layers by removing each on its own and measuring how much worse the model got at predicting text. The first and last layers mattered most. They didn't ship

8:35 that raw. They retrained it with distillation, the big model teaching the small one on 1.4 trillion tokens. Afterwards, it scored close on MMLU to a version they'd shrunk by trimming width

8:46 instead and ran at about 2.7 times the original's throughput. The repair then runs from a single graphics card to a full-scale training run. In both stories, the cutting was the cheap part.

8:58 Mercer is aiming at the cheap end. They expect LoRA, the same kind of light fine-tune, to bring MMLU back to somewhere between 45 and 55. That's their projection, not a result they've

9:09 reported, and even if it lands, it's well short of where the model started. Back to the question from the start, why a cut would spare the reasoning and take the facts? Mercer's answer is that the

9:19 middle layers hold much of the model's factual knowledge, so cutting them cuts it. It fits their tables. It's their reading of the scores, though, not a measurement inside the model.

9:29 The other papers don't agree on where knowledge sits. Gromov's group left it open. Either deeper layers are underused by today's training or the shallow layers do the storing. Song and

9:39 colleagues in a 2025 paper put knowledge and retrieval in the shallow parts, with deeper layers mattering once the model has to write text. Song's group adds a warning that lands right on this story.

9:50 Tests that only score how likely each answer is make most layers look removable. Where you think the facts live depends partly on how you measured. My read,

10:00 none of these sources pins a fact to a layer in the 8 billion coin 3. Mersity found a sharp result, and the is still open. That's why this result is worth more as a research question than as a

10:10 shortcut. It points at a few problems new work can go after. First, where facts are stored. A model that fails its knowledge test, but keeps its common sense ones is a

10:20 clean test case for finding which layers do the remembering, measured directly instead of inferred from scores. Next, the cliff and the gibberish. Neither the log nor the papers explain

10:30 why the facts fall off so suddenly, or why a blend of working layers stops working. Both questions are still open. Last, the cheapest repair. Somewhere between a single GPU healing run and

10:41 over a trillion tokens of distillation may sit the smallest fix that brings the facts back, if one does. Testing it needs long reasoning and real text generation that places Wong's and Song's

10:52 groups found the damage. Then there's the idea that this is new. Laco is from February 2024. Short GPT from Bai Chuan and others came out the next month, showing that many layers are

11:04 highly similar, and ranking which to remove with a score it calls block influence. Put together, the method has been around since early 2024, and Mersity's run skips the repair step that

11:15 the strongest results here relied on. Its downside sits in the very table it came from. To Mersity's credit, the bad news is right there in their own write-up, which is more than a lot of

11:25 launch posts manage. My call, it's a cheap, useful first cut from 2024 with its facts problem in plain view. On its own, it's a starting point and a couple of years old.

11:36 Where that leads you depends on the job. Say you're shrinking a model for short tasks with fixed answers like those common sense quizzes that don't lean on remembered facts. If you can fine-tune

11:46 afterwards, a Llama style cut is a cheap place to start. Mercedes 30-layer model is on Hugging Face, so you can try it on your own tasks first. If your model has to know things or reason at length,

11:57 don't ship a cut model raw. Budget for the repair and test the long version of your task, not just a quiz. The rule I'd keep, measure the exact skill you need at the length you need it before and

12:09 after the cut. A model can pass the averages and still forget what it knew. That line in the log, it is a cliff, turns out to describe the whole trade. On one side, a model that still handles

12:19 short questions. On the other, one that can't recall most of what it knew. If you've cut layers out of a model and healed it, I'd like to hear how much training it took to bring the facts back

12:29 and whether they all came back.