This 27B Model Improved Itself. Zero Human Answers?

0:00 A model on Hugging Face says it improved itself with zero human answers. Its score on a hard science test went up 5.24 points. On that test, 5.24 points is about 10 questions out of 198.

0:14 10 questions could mean the model taught itself real science, a lab grading its own homework, or plain luck. You'd want to know which before you download it or start worrying about where this is

0:24 heading. Start with luck. Is plus 5.24 a real gain or just noise? The model is Darwin 27B RSI from Final Bench and Vidraft, first posted around the 27th of September. RSI stands for

0:39 recursive self-improvement. The card's whole account of how that worked fits in one sentence. The model works on problems, judges its own work, learns from what it produced, and repeats. The

0:50 results table on GPQA Diamond, that hard science test, a single sample goes from 72.85 to 78.09. That's the plus 5.24. There's also a

1:01 majority of 16 score, where you ask 16 times and take the most common answer. That one goes from 79.80 to 83.59, a gain of 3.79.

1:13 Below that, Super GPQA, a bigger sibling test, gets a plus 4.03 with no starting score shown. That's a bit like being told about your raise, but not your salary. Sampling settings are listed.

1:26 Temperature 0.6, which means the model's answers vary from run to run. Hold on to that. GPQA Diamond is 198 multiple-choice questions, four options each. Domain experts got them right.

1:39 Most non-experts got them wrong. Pure guessing lands at 25%. One question out of 198 is worth about half a point, 0.505 to be exact. Divide 5.24 by that and you get about 10.4. So, the headline

1:54 gain is roughly 10 more questions answered correctly. One detail, 72.85% of 198 is 144.2 questions. You can't get 2/10 of a question right. Those scores look like

2:07 averages over several samples, which fits the temperature setting. That's my reading of the numbers, not something the card states. Noise then. 198 questions is a small sample. If the

2:18 model's real skill is about 75%, a different set of 198 equally hard questions could land a few points either way. That's the standard error. Take 0.75 * 0.25 / 198 and take the square

2:32 root, about three points. Temperature adds its own wobble on top because borderline answers flip between runs. Compare two scores measured separately and a five-point gap wouldn't mean much

2:43 by itself. That's the version of this story that ends in it's just noise, but the card says every gain is statistically significant in paired tests and pairing changes the math. Both

2:53 models answer the same 198 questions. A question both get right or both get wrong cancels out. Only the flips count. If most of them go the new model's way, that can be a real signal even though

3:05 each score on its own carries about three points of uncertainty. I wouldn't call plus 5.24 noise because a paired test can catch a gain that size. I'd call it unverifiable from what's

3:15 published. Before leaning on that significant sentence, I'd want the P values and the per question outputs sitting right next to it. Every number here is the lab's

3:24 own, not independently checked. Darwin 27B RSI starts from Darwin 27B Opus and that model has its own card from April. It reports 86.9% on GPQA Diamond. The new card starts the same pairing at

3:39 72.85, 14 points apart on the same 198 questions. The Opus card explains its number. Plain greedy decoding, where the model always takes its single most

3:50 likely answer, got 148 of 198 right. That's 74.7%. Then it retried only the 50 it missed with eight votes each and a verification step and landed at 86.9.

4:03 Neither card has to be wrong. They're measuring different things. One answer per question versus a second pass with retries and checking. The takeaway for you is simple. Don't compare benchmark

4:14 scores across two cards unless the method lines up even when it's the same lab and the same model. And the loop itself, the card says it plainly, the training procedure itself is not

4:24 released. You can't check the number of rounds, how much data went in, or the training method. You can't check how the model judges its own work, either. And you can't reproduce it. That also means

4:34 zero human answers can't be checked. We don't know where the practice problems came from. The card does disclose a small overlap with MMLU, a widely used test set. 31 problems, about 0.22%,

4:47 and it says no answer labels were used. Credit for flagging it. The name gets slippery here. The same lab's bigger sibling, Darwin 180B RSI, describes its RSI loop differently. Its answers are

4:59 checked against verifiable references, like answer keys and executable checks. Answer keys were written by someone at some point. A model judging its own work is a model marking its own paper. Same

5:10 three letters, different teacher. For the 27B, the card says it judged itself, and the judge is exactly the part we can't see. My verdict on Darwin, roughly 10 more questions, possibly real under a

5:22 paired test, produced by a loop you can't inspect. Interesting model, not yet a breakthrough you can check. The headlines say recursive self-improvement is already happening at the big labs.

5:33 What those labs claim is bigger than Darwin, and when you read the actual wording, it's more careful than the headlines. OpenAI first. In October 2025, Sam Altman set two targets, an

5:44 automated AI research intern by September 2026 and an automated AI researcher by March 2028. On the 6th of September, OpenAI said it had hit the first one. Read the

5:56 definition though. The intern carries out well-defined research tasks under human direction, including work that would take a skilled researcher several days. OpenAI reports 3.1 agent work days

6:06 for every human work day. That's a busy intern, but it still has a supervisor, which is a long way from OpenAI automating research. Anthropic's version is about code. Its

6:17 write-up says that as of May 2026, Claude authored more than 80% of the code Anthropic merges. Humans still pick the problems and judge the results. 80% of merged code and 80% of research are

6:29 different claims. Code is the part that's easiest to count. Anthropic's forecast is that tasks taking a person weeks could come into range in 2027. Google DeepMind has the smallest example

6:40 and one of the cleanest. In May 2025, Alpha Evolve used Gemini to speed up a matrix multiplication kernel inside Gemini by 23%. That cut Gemini's training time by 1%. 1%. The loop is

6:53 real. It's closed and it's about as cinematic as a change log entry. As for a big DeepMind plan, what's on the record is a Reuters report from August that Sergey Brin is steering

7:03 resources toward recursive self-improvement. Alibaba's Qwen team went furthest on automation, at least in its own telling. At Apsara on the 22nd of September, it said Qwen 3.8 Max ran a

7:15 month of fully automated runs and completed 33 iterative cycles. Its score on artificial analysis, a third-party index that tracks model benchmarks, went from 40 to 45. Alibaba also says Qwen 4

7:27 is in training and Qwen 4.5 and 5 are planned at 5 to 10 trillion parameters. For scale, Darwin is 27 billion. Line them up and there's a pattern. OpenAI and Anthropic keep humans

7:40 choosing the work and judging it. DeepMind's clearest closed loop is one kernel. Alibaba's automated cycles come with a score Alibaba reported. AI is doing more of the building. The deciding

7:50 is still where the evidence gets thin. The strongest counterweight came in August. MIT Technology Review covered a Princeton-led test. Claude Opus 4.8 agents got 6 days and $3,000 to do

8:02 research and write papers. Human reviewers rejected them. The agents could do all the engineering, but they were unambiguously bad at carrying out the research itself.

8:11 The code compiled. The papers didn't. Jack Clark called it a bearish signal for short timelines. That's the same gap as the Darwin card at a bigger scale. Doing the work is getting cheap. Knowing

8:22 whether the work is any good is the hard part, and it's the part of self-improvement loop leans on. What about the future? There's one measured trend in all of this. METR's time

8:32 horizon 1.1 from January tracks how long a task AI agents can finish. For models from 2024 on, that length doubles every 89 days. Over 2019 to 2025, it was about every 7 months. 89 days is roughly four

8:48 doublings a year if it holds. METR also says its measurements above 16 hours aren't reliable with the current task suite. The ruler runs out right where the interesting part starts. For the

8:59 next year or two, the only anchors are the labs' own dates. Anthropic, weeks-long tasks possibly in range in 2027. OpenAI, an automated AI researcher by March 2028. Those are targets. Watch

9:13 whether they're met and by whose measurement. Five years has passed every dated target here. So, the METR trend is the only thing you could stretch that far with a ruler that stops at 16 hours.

9:24 10 years out, the evidence doesn't support a forecast, and I'd rather leave that open than make one up. My call. If you run open models, treat Darwin 27B RSI as a 27B with a lab-reported gain of

9:37 about 10 questions and an unpublished recipe. Test it on your own tasks before swapping anything. If you're tracking recursive self-improvement, skip the adjectives

9:46 and follow the dates, 2027 and March 2028, plus any check done outside the lab, which leaves a question under all of it. When a model judges its own work and a lab reports its own score, who

9:58 checks the grader?