Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Anthropic's Opus 5.5 rewrote HAProxy in Rust, passing nearly all of the original's regression tests in 9.5 hours at 51% lower cost than Fable 5.1, but the key question is which tests failed and whether the rewrite is production-ready. The most practical takeaway is a Hacker News user's harness that tested models on 12 patches with 14 known bugs: Opus 5.5 found 8 for $15.40, while Fable 5.1 found 7 for $66.34. The real value is in building your own answer key of already-fixed bugs to measure what a model actually finds in your codebase.
Key points
Opus 5.5 rewrote HAProxy to Rust in 9.5 hours, passing nearly all original regression tests.
Opus 5.5 cost 51% less than Fable 5.1 on the HAProxy rewrite task.
A Hacker News user tested models on 12 patches with 14 known bugs.
Opus 5.5 found 8 of 14 bugs for $15.40; Fable 5.1 found 7 for $66.34.
CodeRabbit evaluated Opus 5.5 as a code reviewer, catching a race condition in cal.com.
Tools mentioned
Techniques
- code rewrite with regression test validation
- stacked pull request conflict resolution with AI coordination
- formal verification via Lean modeling
- bug reproduction by injecting NaN values
- headless browser testing for game validation
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
A low balancer written in C gets rewritten in Rust by an AI, and the new version passes nearly all of the original's own regression tests. That's Anthropic's reported result for Opus 5.5
on HAProxy. It sounds finished. So, what's hiding in nearly? Now, scroll your feed and launch week, clip after clip of results like that. Most of them don't show you test coverage, sample
size, or how much the creator's setup did the heavy lifting. So, here are nine things people built or fixed with Claude. Each with its own version of nearly. Eight are credited to Opus 5.5,
and one's a Claude code project where the model isn't named. Every number's credited to whoever reported it, with visual builds mixed in so there's something to look at between race
conditions. Engineering goes first. Number one is that rewrite, and the appeal's obvious. Modernize old software without breaking it. In an internal test, Anthropic had Opus 5.5 and Fable
5.1 each translate HAProxy to Rust, with HAProxy's own regression tests as the judge. Both passed nearly all of them. Opus finished in 9 and 1/2 hours, Fable in 12, and Opus cost 51% less. That's 2
and 1/2 hours faster at roughly half the price on real infrastructure. The catch is that one word. Anthropic doesn't say which test failed, and in a low balancer, the failing test is the story.
Because that's the behavior you'd be shipping broken. Nearly all tests passed is also how a lot of incident reports begin. There's also no word on whether either rewrite is production ready. That
caveat's mine. A green suite only tells you the rewrite matches what the test check. Part of why this could work is that HAProxy came with its own regression suite for the model to aim
at. If your old service doesn't have one, writing it comes before any rewrite. So, before you try this, which tests failed and what behavior do they cover? Number two is smaller and
stranger. Spearliss takes a photo and redraws it as one single continuous line, then films that line being drawn, like a pen that never lifts off the page. The part worth copying is where it
runs. Everything happens in your browser, your photos never uploaded, and after the first visit, it works offline. So, as far as I can tell, once it's built, there's no AI service in the
loop, which makes it a rare AI project that keeps working after you cancel the AI subscription. Claude Code wrote the tool once, and the
tool doesn't need it anymore. Two catches. Line art mode first downloads about 54 MB of models, which then run on your own device. That's why it can go offline,
but that first visit isn't free on a phone plan, and the readme credits Claude Code without naming a model, so we can't say this one was Opus 5.5. The question for your own tools, which parts
really need a hosted AI service while they run, and which parts could a model just write for you once? Number three, one change breaks the next 39. Picture 40 stacked pull requests, each built on
top of the one before. Resolve a conflict near the bottom, and it ripples up through every PR above it. Christian Rivera, a staff software engineer quoted on Anthropic's launch page, describes a
multi-day rebase of exactly that stack. One Opus 5.5 session directed a dozen more sessions and laid out every conflict plainly. That's one coordinator with 12 workers under it, all on one
stack. The part I'd copy is what it didn't decide. The calls it couldn't make, it held for him, and it framed them so clearly that
after hours away, he could answer them in minutes. All 40 pass CI the next afternoon. Two things to weigh. It's a customer quote on the vendor's own launch page, so read it as a testimonial
rather than a benchmark. And pass CI is only as strong as your CI. Green test over code they don't cover tell you nothing about that code. So, ask this, of all the calls it made without asking,
how many would you have made differently? The quote doesn't give that number, and it's the one that decides whether you'd hand it your stack. Number four is the first visual build, and it's
the best case here for turning an explanation into something people can play with. Ryan Sale had Opus 5.5 build an interactive camera lens lab. Turn the
focus ring and the glass elements inside the lens shift. As they move, the sharp plane slides through the scene with them. There's an aperture control running from F2 to F16 and a label for
the sharp zone. That sharp zone is depth of field, one of those ideas that takes a paragraph to explain and about a second to understand once you're dragging the ring yourself,
Sale's own figures. One run, 1 hour 26 minutes, $25.66 at API rates. For a working optics demo, that's a tempting price tag. Those figures describe his run, and they
aren't a price you can reproduce. A report on the build says his prompt told the model to scan files from his earlier, sometimes unpublished projects, and he ran it at maximum effort. So, the
model started with his old work to read. If you tried your own version, what would the model need to read first? And how much of Sale's result came from those old files rather than the prompt?
Number five, both jobs succeeded and your database is still wrong. Coda Rabbit evaluated Opus 5.5 as a code reviewer, and it caught a race condition in cal.com that Coda Rabbit's production
setup missed. Picture a retry counter sitting at zero. Two background jobs read it at the same moment. Both see zero, both add one, and both write back one. The counter says one. One retry
just went uncounted. Every step in that story reports success, which is the most annoying way possible for data to go missing. The fix is to stop the jobs from doing the arithmetic. Each job
tells the database to add one to whatever's there, so two jobs make two. Now, the full report, which is mixed. On 80 known bug patterns, Opus scored 51 to the baseline's 49. On 13 hard cases, 10
to five, but it missed nine bugs the baseline caught and used roughly 40 to 60% more tokens. So, Code Rabbit reports the cal, calm catch, and it also reports the nine misses. Would you run it
instead of your current reviewer or next to it? With those misses on the table, I'd run it next to it. Number six is a developer's GitHub repo. Three browser games, each one a single file, each
built from a single sentence prompt. The games are the bait, but the testing's the part to study. Every game ran in a headless browser, which means a real browser engine with
no window, driven by a script. One got a soak test of 320 seconds, a bit over 5 minutes, across a full day-night cycle. That's long enough to see the whole cycle play out instead of judging the
first flattering frame. Then, after launch, the pelican game started going black, occasionally, which is the worst kind of bug because you can't fix what you can't make happen.
So, the model made it happen. It injected NaN values, the not-a-number result you get when floating-point math goes wrong, and watched. The bloom effect, the glow that spreads bright
pixels into their neighbors, spread those invalid pixels into black blocks. The fix came in two layers. A cleanup pass scrubs bad pixels before bloom runs, then it went upstream and fixed
two places that produced the bad values, including the Fresnel term on the ocean. That's the map that decides how much the water reflects, depending on the angle you look at it. Guarding the symptom and
fixing the cause is the move you'd want from a human, too. One caveat, this whole investigation is the repo's own account, written in its readme in Chinese. The loop's still worth
stealing. Build, test, reproduce, fix. And before your agent touches a fix, ask it, "Can you make the bug happen on demand?" Number seven is the one I'd copy first. Would AI catch issues you
already know are there? A Hacker News user, GWD, built a harness to review patches posted to a mailing list, then tested models on it. He took 12 patches containing 14 issues he already knew
about and counted what each model found. That's the whole trick. It's a test where you already hold the answer key. Opus 5.5 found eight of the 14 for $15.40. Fable 5.1 found seven for
$66.34. So, the run that cost about a quarter as much also found one more issue, which is not how you get what you pay for is supposed to go. He calls it a relatively
small sample himself. The costs are the equivalent API figures his CLI reported while he was on a subscription, and he's already had to correct one number in his table for the older Opus 5, and eight of
14 still means six missed. Which leaves you one job, build your answer key. A handful of bugs you've already fixed with their original patches is enough to start. Number eight is a company demo,
so it's labeled as one. Higgs Field showed Opus 5.5 running on Higgs Field's own platform, turning one photo of a house and its floor plans into a full 3D model in Blender. Then it built an
offline viewer in the browser. You can step through each construction stage, x-ray the walls, and walk through the furnished interior. A static spec became something you can walk around in,
and that idea travels well beyond houses. What it leaves out. The published clip is 30 seconds long, and it doesn't show how much manual work or how many attempts went into it. Every
demo leaves the same thing out, how many tries and how many human fixes. Number nine closes the loop, AI checking the tools used to build AI. Boris Cherny says a couple of short prompts had Opus
5.5 build a Lean model of the Claude agent SDK. Lean is a language for writing math that a computer checks. A Lean model of some code is a precise description of how
that code supposed to behave. Where a property you'd expect doesn't hold in the model, you've got a place to look in the real code. So, his result was 16 pull requests fixing bugs and race
conditions. His post also says Claude is excellent at Lean. That's a glowing review of Claude's Lean from someone who, by his own account, doesn't know Lean well. Now, the word he used. His
post says "formally verify". What he describes reads more like bug finding, and one commentator says exactly that. Any conclusion holds only as far as the model's assumptions match the real SDK.
Which leaves one thing to settle before you try it. Who checks that the Lean model describes the SDK you actually ship? So, the decision. If you maintain a real code base and want one thing to
try this week, copy GWD setup or Code Rabbit's bigger version of it, collect bugs you've already fixed, feed the model the original patches, and count what it finds. It's a boring setup, and
it's the only one here that measures your code. The Rust rewrite, the 40 PRs, and the Lean model are worth watching, but each one leans on something you'd need to trust first. A test suite, a CI
setup, or a formal model. The Lens Lab, Spiralist, and the games are where I'd steal ideas. Especially the games repo's habit of reproducing a bug before fixing it. The house stays a demo until someone
shows their attempts. Mhm, mhm. And the question I'd most like answered is the one we started with. When Anthropic says nearly all of HAProxy's tests passed, which ones didn't?