Run Ornith 1.5 35B Locally: 155 Tokens/Sec on Your GPU?

0:00 One forum title, four numbers, a 35 billion parameter coding model running on a consumer graphics card, 155 tokens a second on an RTX 4090. That is the first number and it is the loudest. Then

0:13 a second machine entirely, an RTX 3060, 12 GB of video memory, 16 of system RAM, 170,000 tokens of context, 53 tokens a second coming out, 900 going in. That 3060 launched in February of 2021 at

0:29 $329. So, the claim on the table is that a 5-year-old budget card runs a frontier-shaped coding agent at speed. Every one of those four numbers can be

0:38 checked against a file, not argued about on a forum, checked with arithmetic. One of them holds up almost exactly. One is off by a whole card class. One has no post behind it that a search can find.

0:50 And the fourth, the 170,000 is true and misleading at the same time, which is the worst of the four because the context window a machine can hold and the context window a model can actually

1:00 use are two different numbers. The screenshots going around this week measure the first one. So, let us do the arithmetic properly. Or 1.5 went up on hugging face on the 18th of August and

1:11 the team announced it the next day. three models, a 9B dense, a 35B mixture of experts, and a 397B. All MIT licensed, so the weights are yours, and there is no bill per token.

1:24 The training pitch is the unusual part. According to the team's own writeup, the model was not fed a fixed pile of human written tasks. It proposes progressively harder tasks itself, generates a

1:34 scaffold for each one, instructions, tools, decomposition, and then produces a solution roll out. reward from that roll out flows back through all three stages. Better policy, harder tasks,

1:46 better scaffolds, better training signal, round and round. It is also the kind of paragraph that makes a forum immediately suspicious. One commenter said, "The writeup deserved a Nobel

1:56 Prize in vague but wildly grandiose promises, which is unkind and not entirely wrong. Underneath all of it, this is a fine tune." The config file names the architecture as QN 3.5 mixture

2:07 of experts and the community read is that 1.5 is post-trained on QN 3.6. Ornith puts that base model in its own comparison table which answers the usual complaint about finetuners hiding what

2:19 they started from. The 35B is where this story lives. 36 billion parameters in total. 256 experts. Eight of them fire on any given token which works out to roughly 3 billion parameters doing

2:32 actual work. That ratio is the entire trick. You pay for 36 billion in storage and you pay for 3 billion in speed. Hold on to that because in about 4 minutes it becomes the whole answer. The scores the

2:44 vendor publishes are strong. 67.8 on terminal bench 2.1 under one harness 68.5 under another. 79 on SBW bench verified 59.6 on S. We bench pro against the base model it started from. Those

2:59 are wide margins, not rounding. And credit where it is due because this is rarer than it should be. The footnotes under that table name, the harness for each row, the temperature, the context

3:10 window, and the fact that every result is the mean of five runs. The SWE bench rows even state that git history was stripped and network access disabled, so the model could not look up its own

3:20 answers. It is still the vendor's own measurement. Terminal bench keeps a public leaderboard of agent and model pairs. And as of the 20th of August, ornith does not appear on it. Not 1.5,

3:31 not 1.0, not one row. Which brings us to the part you can settle for yourself. File sizes. The official GGUF repository list the Q4KM build of the 35B at 21.71 GB. Full precision is 71. And for a

3:47 sense of what you have escaped, the 397B at the same quantization is 240 GB. An R TX49090 has 24 GB of video memory. So the weights land with about 2.3 to spare, which is comfortable until you

4:03 remember you have not loaded any context yet. The 3060 has 12. The weights do not fit. Roughly 10 GB of this model has to live in ordinary system memory and get read back across the bus while it works.

4:15 Now put 170,000 tokens of context on top of that and the whole arrangement should collapse. It does not. And the reason is sitting in the config file in plain text, 40 layers. And the layer list says

4:27 only every fourth one is full attention. 10 layers keep a cache that grows as you talk. The other 30 are linear attention and their state does not grow with the conversation at all. Two key value heads

4:38 head dimension 256 keys and values two bytes each at half precision. Multiplied out across the 10 layers that care and one token of context costs about 20 kilob. So 170,000 tokens is about 3 1/2

4:52 GB. Quantize that cache to 8 bits, which most local setups do, and it is under two. Had all 40 layers kept a cache, the same window would have cost close to 14 GB, more than a 3060 has in total, four

5:05 times the memory for the same conversation. That one architectural choice is the difference between a video and a joke. So the memory claim survives. 21.7 of weights plus under two

5:16 of cash is 23 and a half and a 4090 holds 24. It fits barely and only if you quantize the cache. Leave it at half precision and you are over the card by more than a gigabyte. Speed is harder

5:29 because speed needs somebody who owns the card and is willing to publish what they ran. On the 20th of August, somebody did a full run of Ornith 1.5 on an RTX 4070 Ti. 12 GB of video memory.

5:43 the same class as the card in our title. 32 GB of system RAM Q4KM llama.cp build number and flags all in the post. Prompt processing 650 to 700 tokens a

5:57 second. Generation sustained 50 to 56 in an isolated bench with no context to carry 65 50 to 56. The title said 53. On a different card, in a different house, on a different day, the headline decode

6:12 number lands inside a measured range. That is the one claim in this title that survives contact with somebody else's hardware. Prefill is where it comes apart. 900 tokens a second on a 3060

6:24 against 650 measured on a 4070 Ti, a card with 40% more memory bandwidth. The slower card is supposed to be faster by a third, and there is no configuration in the post that explains it. and the

6:36 155 on a 4090. A 4090 moves a thousand gigabytes a second and 3 billion active parameters at this quantization is under 2 GB of reading per token. Divide one by the other and the physical ceiling is

6:49 somewhere near 550. So 155 is entirely plausible. It simply has no post behind it. Plausible and sourced are not the same word and a title is not a citation which raises the question the whole

7:02 genre keeps skipping. If the number is right, what exactly did it measure? Go back to that 4070Ti run for one detail. The context is set to 32,000 tokens, not 170,000, 32. And the author says so

7:17 himself in the post. The context is admittedly tight. Same model, same quantization, same 12 GB as the card in our title, and he allocated less than a fifth of the window the title quotes. He

7:28 is not being modest. he has used it for why open the model's own community tab. On the 29th of June, a user opened a thread on the 1.0 repository with a title that is the entire problem in one

7:41 line. Reasoning degradation and tool calling failures at 70 to 80,000 context tokens. His method is the part worth stealing. 20 files, 200,000 tokens in total, loaded one at a time. analyze,

7:55 fix, rewrite, move to the next. Gradually filling the window instead of declaring it in a launch flag and calling that a benchmark up to 65,000 tokens fully functional. Everything

8:06 works. 70 to 80,000 noticeable degradation. 90 and above completely broken, looping, re-checking the same file it had already finished, describing tool calls instead of making them. A

8:19 second user running an entirely different job on entirely different work hit the same threshold independently though he added that it does eventually reason its way back out which the first

8:29 thread had not seen. Two strangers, one number, no coordination. That is ornith 1.0 tested in June and pinning it on 1.5 without a rerun would be unfair. What is fair is the shape it exposes because

8:43 that shape is not about ornith at all. A context window is a memory allocation. The number in your launch flag says how much cash you are willing to buy. It says precisely nothing about whether the

8:54 model can still think at the far end of it. The throughput post measure the allocation. That is benchmark worship wearing a hobbyist clothes. And it is why the same readers are disappointed by

9:04 the same model month after month. None of which makes this a bad model. It makes the numbers around it soft. And those are different problems with different fixes. So here is the ledger

9:14 both directions. A commenter lined Ornne's 35B up against CUNE 3.8 27B on every benchmark where both had published figures. Terminal bench 73 against 68.5 Q deep 42 against 22 Quinn badly.

9:32 Humanity's last exam without tools 30.8 against 25.6 Quinn again. GPQA diamond ties. Ornith takes NL2 repo. Three losses, one tie, one win. And the man who compiled it flagged that several

9:47 Quinn cells have no official figure at all. And deep sui is the row to squint at. Ornith 1.0 scored a flat zero. 1.5 scores 22. One commenter's red was blunt. That jump is the public deep SOE

10:01 data set arriving in the training mix. Not new debugging ability. That is an accusation, not a finding, but a jump from exactly zero is the right place to make it. Then there is the setting the

10:13 announcement thread skip. An independent tester ran 1.0 against 1.5 on his own task battery. 12 tests, five runs each. At stock settings, the new model refused eight of the 12. Its package default

10:26 temperature is not8. The model card itself recommends not 6. Retuned to the older model's sampling parameters. The same weights went 12 for 12 and faster with less memory overhead. The upgrade

10:38 is real. It is just not a drop in and the launch day screenshots left that out. One last thing for anyone about to download this. A team that Quantis' models for a living measured each build

10:49 against full precision. The stock Q4KM holds 91% top one agreement. Their alternative is a gigabyte smaller and holds 92.7. Of the six builds they measured, the file most people will grab

11:02 is the worst quality per bite on the shelf. So where does that leave the card on your desk? Ornith 1.535B is the model to put on a 12 to 24 GB card this week. And I would take it over

11:15 waiting for something tidier to arrive. Three receipts, 21.7 GB of weights, plus under two of cash at 170,000 tokens. A memory shape that would have cost four times as much on a conventional layout.

11:28 50 to 56 tokens a second measured by a stranger on 12 GB. Config published an M it so it cannot be taken back. metered or read on the way past. The concession is real and it is not small. At stock

11:42 settings, it refuses. QN beats it on three of the five benchmarks where both show up. Its own predecessor came apart between 70 and 90,000 tokens of filled context and no rerun on 1.5 has been

11:54 published. If your work really lives at 100,000 tokens, this is not your model yet. Take the 9b at 5.6 GB. But the villain in this story is not ornith. It is the screenshot. A tokens per second

12:06 figure printed beside a context size measures how much memory somebody was willing to buy and the local scene reads it as a measure of what a model can do. We built a culture of receipts and then

12:17 agreed not to read them. Which leaves one question worth sitting with. If a 5-year-old $300 card can hold something this capable, hardware stopped being the interesting constraint a while ago. So,

12:28 who publishes the first local benchmark that reports its number as the context fills and gets worse on