I REALLY Want Open Models to Win, But They Just Don't Cut It (Yet)

0:00 Every single time a new open model comes out like a new Kimmy GLM or DeepSeek model, I always get excited because they show these really promising benchmarks and they say that they match or approach

0:09 the strongest proprietary models for AI coding. And I think to myself just for that day, oh man, we finally have gotten to the point where openweight models are as good as Frontier models for AI

0:20 coding. So we don't have to rely on the big companies anymore like Anthropic and OpenAI for everything. But then I actually test the model and I'm disappointed. every single time. But

0:31 exactly how these open models disappoint me actually has a lot of nuance to it because genuinely they are impressive and they do match the performance of the best Frontier models on a lot of

0:41 different things for a fraction of the cost. Like looking at Terminal Bench 4.0 here comparing Kimk 3.1 Pro Max to GPT6 Astra and Claude Opus 5.5, it's even a bit better in this benchmark. And so the

0:53 question I had to ask myself is what gives? How come Kimmy, Quen, GLM, all these companies can release these open models with benchmarks saying that they're better than the best frontiers,

1:02 but then when I actually try to use it to build real things, it just falls apart. And so that's what I want to cover with you in this video. Why I can't use open models for all my AI

1:11 coding, the testing that I did so I can really prove it to you, and then also the solution because I really am rooting for open models. I do think this is the future and so I want to incorporate them

1:22 as much as I can and there are a lot of ways to do so even if we don't hand over all of our workflows. Now don't get me wrong, I love using my Codeex and Cloud Code subscriptions. I'm using them every

1:32 day. But those subscriptions are not guaranteed to be around forever and we all know the rate limits are getting worse and so I don't want to be dependent on that. And if it ever got to

1:40 the point where I had to pay the API pricing for Opus 5.5 or GPT6 Astra, it would not be realistic for me financially to be using it for most things just considering the amount of

1:50 tokens I'm using every day. Now, there are a lot of other reasons to root for open models and invest into using them, but we don't have to get into that right now. I want to focus on the performance

1:59 here. This is the big dichotomy that I've seen. Frontier models and open models are equivalent in performance when you give them shorter tasks even if they are more complicated and require

2:10 some deeper reasoning. And so there's a lot of different tasks that I've given both Frontier and Open models and I've had a separate agent review them and then I've also reviewed the outputs

2:20 myself and it is basically equivalent. And of course using the open models here it's a lot cheaper. And so for these kinds of tasks in fact for most business automations and use cases for agents

2:32 open models win. I've even seen a lot of studies and professional opinions circulating recently that say things like openweight models will run 90% of enterprise agentic workflow steps and

2:42 they usually cite within around 2 to 5 years cuz we're not at that level of mass adoption yet but we're getting there very quickly and I'm even seeing this myself as I work with companies and

2:51 I see how they're very cautious for token spend especially within the last half year and for a lot of these open weight models they want to run it locally because they're dealing with

2:59 sensitive data and the thing is nine and 10 agent workflow steps are routine and openweight models handle them at par for a quarter of the price of even the cheapest closed model something like

3:10 Haiku 5.5 and so the point is where we really need to have agents built into larger business automations it's just a step in the workflow where we don't have to have GPT6 Astra or Opus 5.5 most

3:22 things you can have a weaker model that gets the job done and is a lot cheaper and that's where open weight models win every time and I say weaker but really I should correct myself because like I was

3:32 saying for the shortrunning tasks. The open models are just as good as the frontier models where they really fall apart. And this is what matters a lot for agentic coding is for longer running

3:42 tasks. If we're using these open models to drive a larger harness like my AI software factory for example, that's where I see them disappoint time and time again. And I'll show you exactly

3:52 how. So what I have here is a large simplification of my AI software factory. This is generally what my AI coding workflow looks like because I don't just open up Claude Code, Kimmy

4:03 Code, whatever, and send off a request. I have skills and rules that guide myself and my agent through the process of planning together, implementing based on that plan, and then having myself and

4:12 another agent review and play test to make sure we're confident for the merge. I do this for every bug fix and every new feature that I'm implementing. The problem here is there is a lot of

4:22 opportunity for drift that comes naturally with any complex code base and larger process you're going to be working with. And this is where open models fall apart. Even Kimmy K 3.1 Pro,

4:32 which I keep referencing because it is really the latest and greatest open model. What it comes down to, I've kind of thought of a word for this. It's all about the attention. It seems like open

4:41 models aren't able to pay attention to all of the rules and context it needs to be following when we have longer running processes. Like when I'm working with Opus 5.5, I have all my rules and a

4:53 complex code base and it has to understand where to integrate things and what to review. It always just seems to get me and get the code base even when I have it running for a long time. But

5:03 that's not the case with Kimmy. It does good for individual edits if I keep it very very granular. But if I really try to expand the scope, it just starts to forget rules that I have for it in the

5:13 codebase. Like even the most basic things in my global rules like don't touch the PRD, it's read only. I've had to try to do that. It makes that attempt quite a few times as I've done testing

5:22 over the past couple of weeks and it just constantly gets stuck because I have strict guidelines for what is allowed to merge in my codebase. I have a pretty high bar that Kimmy seems to

5:32 never really be able to reach without iterating over and over and over again. And that actually makes it so that Kimmy is even more expensive than Claude just because of how many tokens I have to

5:41 spend fixing all the little problems that come up. And we can even see this play out when we look at the larger list of benchmarks, not just the ones that Kimmy wants you to see like Terminal

5:51 Bench 4.0. So yeah, definitely for some Kimik 3.1 and a lot of the other open models are better or on par than the Frontier models. So there's something to it. But if we look at the other

6:03 benchmarks that are focused more on long horizon tasks like Terminal Bench Science, Frontier Code 2.0, there is a very very big difference. This is more in line with what I've seen with Kim K

6:14 3.1 Pro and the other open models seemingly falling flat on their face compared to the Frontier models like Gemini 4 Carbon and Claude Opus 5.5. And so for how I actually want to

6:25 incorporate models in my Aenta coding workflows, this is more my experience, not this. As much as I wish Terminal Bench 4.0 was accurate for my work. Now, I also don't want you to be under the

6:37 impression that Frontier models have totally solved longrunning tasks. There's still a struggle there. I've seen this in a bunch of research, including my own, that Frontier models

6:46 do still struggle on long horizon tasks. It's just they struggle a lot less than open models, like GPT 5.6 Soul being at a 41.6% in this study and then Kimmy K3 only being at a 22.1%.

7:00 That is a big difference. So, we still need harness engineering as a whole. We need the guidance and the process for even the best models to get any kind of accuracy with these larger AI coding

7:11 workflows. But even if we have a lot of that guidance and process set up, it still doesn't seem to help these open models nearly as much. The sponsor of today's video is Triple 10, and they

7:22 built programs for senior engineers who want to get really good at building AI systems. AI has actually been the top reason companies gave for job cuts in the US this year. But the need for

7:33 engineering hasn't just disappeared, it's shifted. And most of the growth in job postings has gone to senior roles. You can pick up an AI tool in a weekend, but getting real experience managing AI

7:44 systems in production is really hard to do on your own. And that is what triple 10 is for. There are two programs. AI system engineering and AI machine learning are made for engineers with a

7:55 couple of years of production experience. So this is not a beginner program. You build five projects that come together into a real system like a retrieval pipeline with eval and a

8:04 support agent hooked into Jira with MCP and you defend your projects in front of a real engineer which I really like. Triple 10's expert projects are public on GitHub too, so you can see the kinds

8:15 of systems that you build before signing up. And the instructors have public profiles you can look up yourself like Muhammad Sheriff who's worked at AMD, Microsoft, and Amazon. If you have a few

8:25 years of engineering under your belt and you want to go deeper on AI systems, I'll have a link to both programs in the description. So, everything I've covered with you here is based on all my

8:35 research and my experience doing a lot of comparing closed and open models over the course of this year, but also I want to make things interesting for you and really quantify what I'm seeing. And so

8:46 I put together a very controlled test using my AI software factory to build the exact same application with Claude GPT6 and Kimmy K 3.1 Pro. And so every single model here got its own AI

8:59 software factory to build the exact same video game. I chose a video game because they're fun for me, nice to visualize for you, and I wanted something reasonably complex, not just a simple

9:08 web app, so I can really push the limit of these models. So the game that I built is called Northstar. I have my real factory building the full version of this right now, but this is the

9:17 version that I built with Claude using Opus 5.5 for the planning and reviewing and sonnet 5.5 for the builder, the kind of workhorse. And so this is a very very simplified version of the game because I

9:30 just wanted to run about 12 laps in total of the factory. So there's 12 tickets that I built for this game. And it's pretty good overall. Like this is a really solid with the physics. I mean

9:40 the combat is kind of silly with the animations. Like my guy looks kind of bad here, but the full game loop is working here. This was built by Claude within my software factory. So, here's a

9:50 version of the game built by GPT with six Astra for the planning and reviewing and then the workhorse with 6.1 soul. And so, the HUD looks pretty ugly here, but if we actually go into a raid, it

10:00 looks good. So, of course, there's going to be little issues to fix with all of these proof of concepts because I didn't spend a ton of time on it, but this one had 11 tickets in total to get to where

10:09 we're at now. So, there's some hit boxes left over. looks like some debugging things. So, doesn't look as good as Claude. It also decided to do a much more zoomed out view. So, each one of my

10:18 factories got the exact same starting point for the artwork, which is why it looks so similar. But again, like Claude, it did a pretty good job overall. My goal here was absolutely not

10:27 to make a polished version of the game, but to run a bunch of tickets on a more complex codebase just to get a feel for how well it was able to build out the features and test its work. And then

10:37 finally, this is the game built by Kimmy, which has a much more zoomed out view like the one built by Claude. And then if I start a raid here, the UI actually looks kind of clean compared to

10:46 Claude and GBT. I like it. But the problem is, and we'll get into this in a little bit, it didn't build nearly as much as far as the features in the game because it spent so much time fixing all

10:56 these little things and iterating on each of the individual tickets. And so, for example, I don't really have time to show this here, but between raids, there's a shop where you can upgrade

11:05 your character with the claw in OpenAI versions, but Kimmy never got around to building that. So, I know this is kind of a silly example, and it is just one application, but I wanted to build

11:15 something concrete to give you some tangible numbers here for what I've been seeing building dozens of applications with Frontier and Open models. And so there's not a lot of data points as far

11:24 as the sheer number of tickets, but I did spend tens of millions of tokens with all the testing I did here, really letting it iterate and struggle through things so I could get the real data. And

11:34 this is in line with everything I've seen throughout this year. So Claude did 12 out of 12 tickets for the game in about 14 hours. And so each time it went through the full loop of the factory, it

11:44 took about an hour, which is pretty good considering how big each ticket was. and it went through the full process of planning, building, reviewing, play testing, and merging. And then going to

11:53 OpenAI, it took twice the length and didn't even quite finish. It did 11 out of 12 tickets. And so, even between Frontier models, there's a pretty big difference here. There's a reason why

12:03 I'm using Cloud Opus 5.5 to drive a lot of my Aenta coding workflows right now. Pretty much all of them. But overall, the curve looks pretty solid between the two, right? Like it wasn't super stuck

12:14 on anything, and the end result was pretty good. But then when we get to Kimmy, look at this. There's a couple of points where it stalled for hours and hours and only got through five out of

12:24 12 tickets in 44 hours, which is why it didn't have things like the shop at the end of the raid. It just got so stuck iterating and all these little things that came up. It was so frustrating

12:34 because it would constantly try to do things that the rules told it not to. And so that's why the reviews would catch it and then it would have to iterate. It was just constantly not

12:42 obeying everything that the factory had for it. Now, I got to say, maybe I could change the factory to be a bit more lenient, let some of these minor issues in so that at least Kimmy would look

12:52 closer to Claude and GPT, but I want to keep my entire review process. I've spent so much time refining this in my AI software factory over time that I don't want to have to conform to weaker

13:03 models just so that more things go through. When I switch between Frontier models, they all seem to just get my factory, get my codebase, and my rules. But open models, it really does seem

13:13 like a true flaw where for these longer running tasks, they just aren't able to juggle and pay attention to everything they need to. I also have some more interesting numbers to show you. At

13:22 least I find this fascinating. So the total cost of building everything here with all 12 tickets with Claude was $255. Now obviously I use my subscription for

13:32 everything here. This is just what it would have been with API costs, which is really the only thing that I can truly quantify. And so that's pretty good overall. With GPT, it was $635.

13:44 It didn't even fully finish. It ran for twice the length, but it was actually more than twice as expensive as well. Pretty interesting. And then with Kimmy, it was $180, which really isn't cheaper

13:55 than Claude because it didn't actually finish everything. And so that's the thing, just because a model is cheaper doesn't mean that it's actually cheaper per task because it has to iterate more.

14:05 If it's failing more often, it can even become more expensive. That's what I see with these openweight models a lot of times in my coding workflows. And we can see this with the very next row. Claude

14:15 was $21 per ticket, GBT63, and Kimmy almost $36 for every single ticket. And also, I had to intervene a lot less when I had Claude running the factory. So I also have a part of the system here

14:27 where I basically have a manager over the entire workflow that alerts me when there are uncertainties that need my decision or at least the manager would decide for me. And so with that Claude

14:37 most of the time is able to go through things very smoothly. Open AAI generally about once per ticket it would bring one thing to my attention but Kimmy needed help for a couple of different instances

14:47 every single ticket it was creating and that also was pretty frustrating for me. Claude is just able to be a lot more autonomous overall. it has a lot more confidence in its implementations which

14:57 seems warranted as well, right? Like we don't want unwarranted confidence where it's just making a ton of mistakes, but Claude is solid and it knows it is. And then the last interesting thing here is

15:06 I had a separate agent do a blind code review of the whole code base. Claude got an 8 out of 10, GPT a 7 out of 10, and Kimmy a 6 out of 10. And a 6 out of 10 is okay. I mean, Kimmy got to

15:18 something decent eventually. It was just pretty painful to get there. Okay, so at this point we have covered the problem in a lot of detail. We know why we can't just hand our longer running agent

15:28 coding workflows over to a cheaper open model. Even if we let it iterate a lot, it ends up getting even more expensive than just having Claude handle our software factory, our coding workflow,

15:38 whatever. And it's going to be a lot more reliable. So with that, let's talk about the solution, or at least what I'm doing right now. The eventual goal, at least for me, is to get to the point

15:46 where I can use open models for everything. And I firmly believe that in the future we're going to get there. We're not going to be locked into using Frontier models for anything if we don't

15:56 want. It's just like we can see we're not there yet. But that also doesn't mean that I have to ditch the idea of using open models altogether in my agent coding workflows. Something I've been

16:07 experimenting with a lot this year is mixing models and providers. Using the right model for the right task depending on how much it costs and how good it is. And so, for example, using Opus for the

16:20 planning and reviewing in my larger agent workflow and then using Kimmy as the workhorse. And these kinds of combinations have worked really well for me. Now, I'll have to put out more

16:30 content where I really show you how to set up this harness. And there is a video I'll link to right here on my channel where I get into these numbers in a lot more detail. But at a high

16:37 level right now, I just want to show you what it looks like to mix models and how we really don't have to sacrifice the reliability of the code to get a lot of cost and time savings. And so what I did

16:48 with this test is I had a control group where I had a lot of agent coding workflows that I ran through Opus for everything. It did all of the planning, implementing, and reviewing. And then

16:58 for my other test, I had Opus do the reviewing and the planning, but Kimmy was the workhorse that did the implementation, which is where a lot of my tokens go. And what I found is the

17:08 results were pretty equivalent between these two groups, which is incredible. So, we look at the cost here. For simple tasks, it was really cheap between the two. But for more complicated tasks, it

17:18 was more than twice the cost using Opus for literally everything, even though the results were pretty equivalent both for simple and complex tickets that I had it implement. This is crazy. It all

17:28 comes down to the fact that if we use the more powerful model for the planning, that's where we need to load in all the context and follow all the rules. So, if we scope that down into a

17:37 plan document, our weaker but still pretty capable model is able to execute on that very well. This is the solution right now. We can still use openweight models, just not for everything in our

17:48 larger workflows. So, that's my main recommendation right now. If you want to save on tokens, make your workflows faster, experiment with open models, use them for the workhorse that always

17:57 writes the code, but then you just sandwich that with the planning and reviewing that uses a frontier model and exactly how you set that up. There's going to be a lot more content I put out

18:06 soon where I cover that. So, also let me know in the comments what part of this you're interested in. Like, do you want me to show more data? Do you want me to show you how to really set up these

18:14 systems? I'd be curious what you're looking for. And so, I hope that you appreciate this. Overall, the problem with open models, getting into the solution, a lot more I want to do on

18:22 this soon. If you appreciated this and you're looking forward to more videos on agentic engineering, openweight models, I would really appreciate a like and a subscribe. And with that, I will see you

18:31 in the next video.