Pi Agent vs OpenCode: Same Qwen 3.8 Model, Completely Different Results

0:00 20 numbered balls bouncing inside a spinning heptagon. Two screenshots of the same animation side by side. The left one was made by a coding agent called Pi. The right one by a different

0:10 agent called Open Code. Underneath both of them, the same model is doing the work. QN 3.8, 27 billion parameters, same weights, same graphics card. One RTX 3090 in a stranger's house serving

0:24 both agents. He says the left one came out much better. Within a day, 152 upvotes, 103 comments. The argument underneath them is one question. Does a picture of bouncing balls tell you

0:37 anything about the programs that drew it? Because if the model underneath is identical, then whatever changed has to be living in the wrapper around it. And this one is settleable. Both wrappers

0:46 are open source. Both are TypeScript. Both are MIT licensed. So this video reads the source of both harnesses line by line. Instead of arguing about heptagons, what turned up was a two-line

0:58 rule inside one of them. Rewriting your sampler settings for 381 days. And a stranger ending than the thread reached. The man with the screenshots was probably right for a reason he never

1:08 named and could not have measured with the test he ran. A model on its own cannot open a file, run a command, or notice that a test failed. It emits tokens. The harness is the program that

1:19 turns those tokens into work. Which means the harness owns a list, the system prompt, the tool definitions, the loop that runs a tool and feeds the result back. What stays in the context

1:29 window and what gets thrown away. And the sampling parameters on every request. Every item on that list is a decision somebody made for you in a file you have probably never opened. Open

1:39 Code is the bigger of the two projects. Just under 200,000 stars. MIT licensed TypeScript and 5,200 open issues. Pi is the smaller one. 94,000 stars. Mostly written by Mario Zechner with Flask's

1:54 creator Armin Ronacher second. 137 open issues. 38 times fewer at half the stars. Not a scoreboard, a measure of how much each program does on your behalf. So, take the claim seriously and

2:06 check it. He says open code compresses the conversation far earlier than Pi does. On a 100,000 token context with a 32,000 output limit, he watched compression start at 67,000. Pi held out

2:19 until around 90. That first number falls straight out of one file. Open code decides when to compact in a source file called overflow.ts. And the rule there is one subtraction.

2:29 Your usable context, it says, is your context window minus the maximum output tokens. Once the running total reaches that, everything older gets summarized away. And the maximum output tokens is a

2:41 constant sitting on line 18 of another file. Output token max equals 32,000. The function beneath it takes the smaller of that and whatever your model advertises, which makes it a ceiling,

2:52 not a default. Declare 128,000 output tokens in your own config and you still get 32. So, do the subtraction on his machine. 100,000 minus 32,000 is 68,000. He

3:05 watched it start at 67. The code reproduces his observation to within a thousand tokens. Pi's rule has the same job and a different shape. Compact when the context exceeds the window minus a

3:16 reserve, and that reserve is a flat 16,384. On the same window, the cut lands at 83,600. 15,600 more tokens of conversation

3:27 before anything is thrown away. He said 90,000, the code says 83,600. His direction is right, the figure is his own. One more branch matters. Declare an input limit in your config

3:39 and the reserve is capped at 20,000 instead. Which is why a workaround posted to that same subreddit works. Fill in all three limits and the early compaction stops. So, the mechanism is

3:49 real, it is in the source, and it explains his number. But look again at what he was testing, 20 balls in a spinning heptagon, a physics animation in one HTML file, one prompt in, one

4:01 answer out, no tool loop, no long session, no accumulated history. A task like that does not come near 67,000 tokens. The compaction rule he identified cannot fire on the thing he

4:13 showed us. So if the mechanism he named could not have run, what is left to explain the difference on his screen? Commenter said it in a sentence, "I don't see how

4:23 one-shotting some HTML is a comparison between the two harnesses. Run it through some multi-step exercises with plenty of tool uses." Go back to that list. Most of it needs a long session

4:33 before it changes anything. Two items reach the model on the very first request, the system prompt and the sampler settings. Open code keeps its system prompt as handwritten text files,

4:43 one per model family, Anthropic, GPT, Code X, Gemini, Kimmy, Meta, Trinity, and one called Beast. Eight families with a prompt of their own and Quinn is not one of them. The function that picks

4:55 is 22 lines long and a local Quinn falls out the bottom of it into a file called default.txt. 8,528 characters, around 1,400 words, before

5:06 your tools or your project files are added to it. It opens with you are open code, an interactive CLI tool, and it tells the model to fetch web pages from the open code documentation site. A

5:17 prompt written for a hosted assistant handed to a model running on somebody's desktop. Pi has no prompt file at all. It assembles one in code from a template 1,352

5:28 characters long, more than six times smaller. Inside it, one sentence saying you are a coding assistant, the tools you were given, and where the docs live. Skills work the same way, a name and one

5:40 line of description with the instructions loading only when the model chooses to read the file. A commenter on the heptagon thread described that design better than the marketing does.

5:49 Quote, "Pi is better because it actually does not do anything extra. No built-in guidance for agent about how to code." He wrote, "No clever truncation or injection. Maybe the gunk embedded into

6:00 open code were useful when models were not as well trained in coding. Now they just get in the way." That is one man's opinion and you have the file sizes to weigh it with. The second thing reaching

6:10 the model on turn one is not an opinion. Five days before those screenshots went up, a user filed a bug against open code and the title is most of the story. Incorrect sampling parameters are

6:21 hardcoded based on model name. He had pointed open code at a local llama.cpp server running Qwen, set his sampling on the server, and found the harness overriding it on the way past.

6:32 The rule was two lines in a file called transform.ts. If the model ID contains a string Qwen, temperature is 0.55. If it contains Qwen, top P is 1.0. A

6:44 substring match on a name, any model with Qwen in its identifier from any provider, local or hosted. Qwen's own model card asks for temperature 1.0 and top P 0.95 in thinking mode. Top P at

6:58 one means no probability filtering whatsoever. At 0.95, it trims the unlikely tail off every token choice. Those two settings give you a different model to talk to. Both lines went in on

7:10 the 2nd of August, 2025, in a single commit whose message reads, "Ignore fix dev branch." The rest of that commit is quote marks and line wrapping. A change to how every Qwen request gets sampled

7:22 wrote in on a formatting pass and it stayed there for 381 days. Then, 3 days after that bug report, somebody posted the same finding to the local model subreddit. 59 upvotes and a link to the

7:34 commit with a note that it carries no explanation whatsoever. 10 hours later, a pull request opened. Title, "Remove Qwen sampling defaults." author, a GitHub account called open code agent

7:46 with the word bot in square brackets after it. Its summary reads, stop forcing temperature 0.55 and top P1 for every Qwen model. At the bottom it says requested by Aiden

7:57 Kline via Slack, the same maintainer whose name sits on the 2025 commit that put the lines in. It merged 9 minutes and 1 second after it opened. 13 lines added, two removed, zero comments on the

8:09 thread. The lines that went in include a test asserting that for Qwen 3.8 the temperature, the top P and the top K all come back undefined. Nothing sent unless you asked. The fix reached users a day

8:21 and 7 hours later and the heptagon post went up 18 hours after that release. So we cannot tell which build he was running. Updated and his comparison is clean. Not updated and one agent was

8:33 talking to Qwen at temperature 0.55 while the other was not. He does not say. It is not in the config he published and it is not in the screenshots. If you want to know how

8:43 badly a single run can mislead, there is a published one that shows it. An engineer put Qwen 3.8 through Pi on a benchmark called Slop Code Bench and published all of it. Versions, reasoning

8:54 effort, random seed, container image per checkpoint records. The result is sobering. Five strict checkpoints out of 39. Zero of eight problems finished correct, but look at the comparison

9:05 table underneath because two of its rows are the same system. DeepSeek V4 Flash on Pi run A and DeepSeek V4 Flash on Pi run B. Run B took five checkpoints out of 17, top of the table, above every

9:20 other system on it. Run A took one, bottom of the table, same model, same harness, same benchmark. Five times as many solved on one attempt as on the other.

9:30 That is the ceiling on every one-shot comparison, including the one that opened this video. The people who built the benchmarks work this out first. Terminal Bench, the public leaderboard

9:40 for agents in a shell, does not rank models. Read the columns. It ranks agent and model pairs, and it prints the reasoning effort beside each one, which lets you match on effort and read the

9:50 wrapper straight off the board. Gemini 3 Pro under an agent called Terminus 2 scores 73.9%. The same model at the same effort under Google's own Gemini CLI scores 65.8.

10:03 Eight points from the wrapper alone, and the wrapper that loses is the one built by the company that made the model. It repeats down the table. GPT 5.5 is 83.1 under Code X and 78 flat under Terminus

10:16 2. Matched on effort, the harness is worth between 2/10 of a point and 8.1 points on that board. Smaller than the figures people throw around. Larger than most model upgrades. And then the

10:27 receipt that made me stop reading and start writing in the footnotes under Qwen's own benchmark table, four rows below the scores the headlines quote, "For SWE Bench Pro, for NL2 Repo, for

10:38 Deep SWE, for their own in-house benchmark, evaluated with the Claude code harness, Alibaba ships its own coding agent, and when it needed those numbers to mean

10:48 something, it ran its own model inside a rival's harness. So, here is the call, and it is narrower than the threads. If you run an open weight model on hardware you own, and you set your own

10:58 sampling and your own context, install Pi. The reason is not code quality. The one published run has Qwen and Pi at five strict checkpoints out of 39, and a user in that same thread reports edits

11:11 failing more often in Pi than in Open Code on identical settings. The reason is what each program does when you say nothing. Pi's test suite asserts that if neither

11:20 you nor your config sets a sampling parameter, the request goes out without one. Open Code's default was to read your model's name and guess. That assertion now exists on both sides. Pi

11:30 wrote its version on the 3rd of August. Open code wrote its version on the 18th. 15 days apart, and one of them was written after somebody complained. Now, the case against my own verdict, because

11:41 it is real, Open code wins the interface. Three separate people in the thread that praises Pi complain about Pi's interface, and one of them says that is the sole reason they have not

11:51 moved off Open code. It also carries twice the stars. And Pi's own read me says it ships no permission system at all. It runs with whatever access you gave the terminal. If you want a sandbox

12:02 by default, Open code is the safer program to hand a shell to. And that argument comes from Pi's documentation, rather than from its rivals. I would still take Pi at twice the price. A

12:12 harness whose defaults I can read in one afternoon is worth more to me than one that is pleasant to look at. And I bet with a date on it, before the 31st of December, at least one more major coding

12:24 harness strips name matching heuristics out of its request path in a commit that says so. The fixed cost two lines. The bug cost a benchmark result. What is worth carrying out of this is that the

12:34 villain here has a name, and the name is the invisible default. Both of these programs make choices for you that you did not agree to and cannot see from outside, and so does every agent on that

12:44 leaderboard. When your model feels dumber this week than it did last week, that is usually why. So, when you sit down to benchmark a model tonight, ask yourself what you are

12:53 actually measuring. Is it the weights, or is it somebody's opinion about how to talk to them?