Is Kimi K3 Really That Good?! (Don't Just Believe The Hype)

KimikoK3 was released just last week, and it is the most powerful open weight model to ever be released. And looking at the benchmarks here, you would think it's one of the most powerful LLMs period. It's better than GPT 5.5 and Opus 4.8, and it's almost as good as Fable 5 and GPT 5.6 Soul, at least for agentic coding tasks. But I'm here to tell you, I'm not buying this, and you shouldn't either. Now, don't get me wrong, KimikoK3 is a genuinely impressive model. You can do very very long agentic tasks with it. But KimikoK3 and really a lot of these open weight models like GLM and MiniMax, they have these certain failure modes, I'll call it, that we don't have to deal with with other models like GPT and Opus. That's what I want to cover in this video with you. I want to show you how KimikoK3 is very powerful. In fact, sometimes the raw output quality is better than Opus 4.8, but there are these failure modes we have to deal with that make it so that KimikoK3 isn't actually as reliable. Things that you need to be aware of and even engineer for when you're using it for your agentic coding workflow. So, we'll get into all of that here. And going along the same lines here, these reliability issues that we have with open weight models, they never come through in the benchmarks. And I'm sure you come to the same conclusion where the benchmarks are not really an accurate representation of how well the model's going to work for you when you're using it in the real world for your agentic coding workflows. We need to design our own tests to really push the limits of these models. I'm never going to be using KimikoK3 as my daily driver over Opus 4.8, even if they're the same speed and price. And so, I want tests to really reflect what I'm going to be using. And so, that is exactly what I built here and what I'm sharing with you, a comprehensive benchmarking solution that I've created and I've used it to compare Opus 4.8, KimikoK3, and KimikoK 2.7, the predecessor. And I brought it through a bunch of real engineering tasks on one of my repositories, as well as what I call trap tasks that I've engineered to shed light on the failure modes with these models like Kimik3. The kinds of things that you will see come up in your AI coding workflows that you need to engineer for. And I've spent literally millions of tokens doing the benchmarking here. So a ton of takeaways to share. I have some pretty strong opinions of Kimik3. As much as I love the model because it's open weight and it's cheaper than Opus, it's definitely not as good. And so I'll show you the results of the real engineering tasks that I did first. So I was walking it through AI coding workflows of planning, implementing, and validating dozens of times for each of the models here. And sneak peek, you can see that Kimik3 is almost as good as Opus, which is pretty awesome to see, but definitely not better like the benchmarks said. So we'll cover that, and then we'll get into some of the trap tasks that I have. So I spent a lot of time engineering these to show the reliability issues of models like Kimik3. And so I'll walk through all the different tasks, the different prompts that I have, the issues that this is showing for these models. And we can see the results here. So yet another sneak peek. The failure rate across all the testing I did here for Opus is 8%. And then for Kimik3, it jumps all the way up to 36%. That is not a good number. Now, I have engineered things to specifically target the weaknesses of Kimik3. And so it's definitely an exaggerated difference of reliability. But I did this very intentionally because I want to have a conversation with you about these issues and how we can actually address them when we're using the model for real. Because these are the types of problems that can creep into your agenda coding workflows. You got to be aware of this if you want to use open weight models. And the reason you want to use these open weight models in the first place is because they're generally a lot cheaper than the frontier model lab equivalents. And so for example, Kimik3, just looking at OpenRouter here, it's $3 for every 1 million input tokens and $15 for every 1 million output tokens. And then, looking at Opus for an example, it's almost two times as expensive. And yes, we have these reliability issues that we'll get into, but a lot of times these models are going to be very head-to-head. So, you can have almost half the price of a model, and you can also use it through a subscription just like Opus. Like right here, I have my Kimi code subscription for using Kimi K3. And I did hit my weekly usage limit, unfortunately, because I have been doing so much benchmarking, like I'm showing you in this video. Uh but most of the time I don't hit my limits. Like you can go so far with Kimi compared to using Fable or Opus. And Kimi K3 is also the most expensive OpenWave model right now. There are so many other really powerful models, like GLM 5.2, MiniMax M3, that can get the job done for a lot of tasks. And so, what I'm experimenting with, what a lot of people are right now, is mixing models for a larger workflow, like using Fable or Opus or GPT 5.6 Soul for planning, and then for the workhorse, doing a lot of the implementation and validation, using a model like Kimi K3 or GLM 5.2. So, it's worth testing out these models, getting an understanding of how they work and their weaknesses, so that you can build these more efficient workflows, so you're not always hitting your rate limits in tools like Claude code. All right, so with that, let's get right into the benchmarks. Now, full disclosure, I built all these benchmarks myself to explore for my sake, and hopefully yours as well, to figure out how good is Kimi K3 really. And I want to create this so I can use it to test other models in the future as well. And so, I've really tried to target the weaknesses of OpenWave models that I've seen when I've used them out in the wild. And so, these benchmarks get a little specific, especially in the last part of what I'll show you. So, maybe you do things differently. Maybe you think there are some flaws in my benchmarks. Just let me know. I'm not trying to say it's perfect here, but I've definitely gotten some really good takeaways that I'm excited to share with you here. All right, so let's get into the first benchmarks here. This is the real engineering tasks that I walked Kimmy K3, Opus 4.8, and Kimmy K2.7 through. And I did dozens of workflow executions to get to the averages that we're seeing right here. We'll talk about the numbers in a little bit, but let me show you what I'm really testing here. I have a bunch of GitHub issues that I have created in a real project, not just a demo that I made for the benchmarking here, but something that I've been working on a lot and even sharing on my channel. So, these are real engineering tasks that I had each of the models go through. So, they range in complexity, really simple things all the way to more complicated features and bugs that have to be fixed. And I always have people tell me like when they want to benchmark models, they wish they had the time to walk the different models through the exact same task instead of just doing different tasks and trying to kind of vibe and figure out which one is better. And so, I took the time to do this here. You're welcome. I took so many tokens, but I did run exactly the same GitHub issue through each of the three models with the workflows that guide everything. And so, to make all this happen, I'm using my open-source harness builder Arkon. I'll link to a video right here where I cover it. I don't really need to get into the weeds of it right now. It's not super important that you understand it, but essentially Arkon allows me to build these larger workflows that string together multiple coding agent sessions because I want to do a full process of planning, implementing, and validating, and you can't shove all of that into a single coding agent session or you're just going to get terrible results no matter the model you're using. Right? So, I want to right here like for the planning step, I'm using Kimmy coding Kimmy K3. So, this is my Arkon workflow for the Kimmy K3 testing. So, I plan with Kimmy, I output a document here that's going to be read by the implementation step again using Kimmy. And then for the rest of it, it doesn't really matter the model. I'm basically just comparing when I'm using a certain model for planning and implementation, right? So, Kimmy K3 right here. And then if I look at the other workflow, this is Kimmy K2.7 both for the planning and implementation. And then the same thing with Opus. And so, it's the exact same workflow. The only thing that's different is my configuration for the large language model that I am using. Everything I'm showing you here, I'll have in the GitHub repository that I'll link to in the description. So, that has all the workflows. It has the prompts that I'll show you for the other benchmarking that I did. And so, I'm documenting this all in public for you if you want to run these benchmarks exactly as I did. The sponsor of today's video is QA Tech and they're solving one of the biggest problems with AI coding right now. The reality is writing the code is no longer the bottleneck. Trusting it is. Because your agent is able to ship three or more features in the time that it used to take to write one. And so, it's not realistic for us to click through everything to confirm our flows are working as we're doing our manual and regression testing. And data backs this up. Most developers don't trust AI generated code fully, but only about half are actually verifying everything before they ship. Now, that's exactly the gap that QA Tech fills. It's an AI testing tool with autonomous QA agents that test your application as a user would. So, you have no selectors or brittle scripts. So, I have all my test cases here. And then to add a new one, for example, I simply have to describe in plain English what I want to test and the agent is going to figure out the steps and also adapt as my site changes so the tests don't go red. And so, I've been creating a lot of different tests for my main Dynamis landing page as you can see right here. And then I can even have a conversation with the agent directly in their dashboard to create multiple tests at the same time. They also ship an MCP server so I can create and edit my tests to directly with my coding agent like Claude Code so I never have to leave my editor. And you can also tunnel to your local dev server so you can verify changes before you commit. And on top of that, they also have a GitHub integration through GitHub actions. And so, my QA Tech agents can review my pull requests right as I open them. So, the way that QA Tech has put this is that scripts verify clicks, but agents verify intent. And that is powerful. Their customers are already seeing real results. Like UpLead Sales has replaced 320 hours of manual testing every single month. The whole idea here is you have your coding agent write the code, QA Tech is giving you the agent to verify the code. And they also have this guided proof of concept, helping you set up all of your core flows. I'll have a link to this in the description. So, anyway, for each of these Archon workflows that the models were walked through, the GitHub issue is the input, and then there's a GitHub pull request as the output. And then that pull request goes into a separate Archon workflow for evaluation that judges against a seven-dimension rubric. And so, it's one through 10 for each of the seven things. That's why it's a max of 70 for the score per workflow execution. So, it's judging things like, you know, how good is the implementation overall? Could there be less over-engineering? Is there good testing and documentation to go with the PR? That kind of thing. And it also tracks the cost. So, we have a lot of data to go off of here. And so, first of all, we have two categories because I took the GitHub issues and I split them into the simple ones and the more complicated ones. Because for the simpler GitHub issues, you'll see that we really can get away with using the cheaper models, so we can be more efficient in our development overall. And so, with Opus 4.8, it got a score on average of 64.3 out of 70 handling these GitHub issues. Pretty good. Now, it's far from perfect, but I made the evaluator pretty picky. And so, this is a pretty solid score. It's like, you know, when you have a movie that's rated 8.5 on IMDb, that's actually one of the best movies of all time, right? It's not It's not a 10 out of 10, but it's still very good. And so, it also took $1.60 on average per task here. And then for Kimi K3, it's very very close. We have a difference of .2 on average, which is just a rounding error. It basically did as good as Opus, and it was less expensive per task. Now, it's actually closer in price than you would think, which tells us that Opus was able to solve the problem with less tokens overall, cuz it's almost double the price in general, but not double the price for this task. And so, I think like Opus still handled things better, but the raw quality of the pull requests in the end is pretty much equal. And then Kimi K2.7 is still pretty good overall, and it is significantly cheaper, but yeah, you definitely want to jump to Kimi K3, and you want to be at that level at least. So, I wouldn't recommend in general using Kimi K2.7 to drive an entire workflow, even for simpler things. And now, this is where things really get interesting. Going to the complex builds here, we have a real discrepancy that is forming. So, the score is of course dropping across the board. Opus 4.8 is now a 62.2 out of 70. So, still pretty good score overall, but also it's a lot more expensive. Take a look at this. The price difference is insane. I had to spend a lot more tokens to reach the pull request for each workflow. Told you, I spent so many tokens on this benchmarking here. And then with Kimi K3, it also is still pretty solid. At least it's above a 60 out of 70, but now there is a real discrepancy here. It's not just a rounding error anymore. Now, at this point it is significantly cheaper than Opus. So, that's actually interesting. I don't really totally know why the benchmarking produced these results, but it also shows you here that when you really want to be efficient, you probably should use the cheaper models for at least some part of your workflow, right? Like the optimal setup is usually something like the more powerful model for planning, and then the workhorse is going to be something like K3 and that really shows here in the benchmarking. And then of course Kimi K 2.7 continues to fall even more and yeah, for more complex building you definitely don't want to use a smaller model like this and yeah, it's cool it is even cheaper but yeah, you're going to have to spend more tokens correcting things in the pull request anyway. And so we really do see Opus start to shine here for more complex builds. Definitely not what the benchmarks were telling us. And this discrepancy here which I mean the issues I had you can get so much more complicated than even that. Like for real real real work Kimi K 3 is definitely not going to feel as good for you as Opus 4.8. And the reason for that and and the reason that we have this discrepancy here is those reliability issues. That's what I want to get into with you now. We'll go over to the other set of benchmarks. All right, so now going over to our second set of benchmarks. This is a lot more similar to what you see in the benchmarks online. But like we already established public benchmarks can't be trusted. That's why I want to build my own set and really identify and shed light on some of the issues that we have with models like Kimi K 3. And yeah, these public benchmarks a lot of interesting tidbits here for why we shouldn't trust them. The most interesting one though is that large language models are trained on a lot of the answers for the questions that we have in these benchmarks. So they're really over tuned over trained on these benchmark type questions and tasks. And then also I don't always really agree with how the benchmarks are judging things in the first place. Like you know, the human picking the one of two generated apps when really that has nothing to do with the code quality. Things like that I don't really agree with and so that's why I wanted to build my own set here. And I have three suites of tests. I have the easy control and all the models passed all of the tests here. This is just establishing the baseline to make sure that my benchmark setup is working. Then I have the harder tasks, the traps that I've designed for open weight models and then some more advanced ones as well. Really just to see if I can push the limit of these models. So, for our easy control, this is just simple coding and debugging tasks and reasoning that all the models knocked out of the park. So, that's good, but let's get to the hard traps now. I spent a lot of time designing these to figure out the weaknesses of open weight models like Kimmy K3. Now, it takes a while to explain all of them. So, I'm going to focus more on the ones that really made Kimmy K3 fail. But, there are quite a few interesting ones here. The kinds of things that you will legitimately have come up as you're using them for agentic coding. And so, for example, by design symptom. This is where you tell the coding agent you have some problem in your code base, but it's actually you not understanding that it's working as intended. And so, sometimes coding agents will try to fix something even though they should realize that it's more your misunderstanding. Right? Because sometimes you're just not going to fully understand a problem. We don't want to rely on our understanding to always be perfect to avoid the coding agent making drastic mistakes. Another one that Kimmy K3 really didn't do that well on, all open weight models tend to fail on, is the false premise. And so, that is where we ask it to fix a problem that doesn't even exist in the code base. So, it's kind of similar to by design, but it's not like it's something that's working as intended. It's more the problem just doesn't exist at all. And so, coding agents will invent a solution to something and treat it like the problem's really there, when really you'd hope the large language model would be intelligent enough to discover as it's going through the code base that, oh, the user is wrong. This isn't actually an issue. Maybe there's this other problem that they're seeing or this other symptom. Okay, so with that, let's get into the actual results here. This is the big takeaway. A lot of interesting things to cover. And so, for each one of the tasks, I had the model go through it five times. A lot of tokens. Opus only failed twice out of the dozens of tasks that we walked it through here. Kimmy K3 and specially 2.7 it didn't do that hot because we really are identifying the problems here with actually just large language models in general. It's kind of interesting because the only two that Opus failed on are the false premise tests. Where again, we tell the agent there's some problem that doesn't actually exist. So, the LLM spins its wheels trying to identify where the problem is, and oftentimes it'll hallucinate a location when really the issue just doesn't exist at all. And so, it's interesting that the same one that Opus failed on is also the one where Kimmy did the worst on. So, it really it's like all the large language models suffer from the same issues. It's just more exaggerated in models like Kimmy K3, at least for some things. But then, there's definitely some things where Kimmy K3 is kind of disappointing where Opus totally knocked it out of the park. Like hidden invariant, for example. This is a test where we have it edit one file, but there are rules identified in some other context for the agent that tell it how to work with that file. And sometimes coding agents will ignore those things, even though they're core files like our mission.md or the rules.md, and it should be able to identify like Opus every single time is like, "Okay, before I edit this file, let me get a lay of the land in the code base and understand other contexts that might affect how I edit this file." Kimmy did not always do that. And then, it's a similar kind of thing with the berry constraint here, where we have a rule that we need the coding agent to follow, but we have that a lot earlier in the conversation. And so, we see this with models a lot with context rot, where they start to ignore instructions we have previously in the conversation. And open-weight models like Kimmy K3, that just seems to affect them a lot more. And then, one more here. They're more prone to sycophancy. So, making a mistake because they're just trying to appease you, which is also kind of similar to the idea of a false premise. And so, we definitely see a pattern here of when these models don't perform as well as the larger ones like Opus or GPT. It seems like Opus 4.8 is able to just think for itself more and be more exploratory. I think that's the way that I'd really put it here, where it's able to tell you when it thinks that you're wrong, like identifying a false premise. It's able to explore more and understand like, okay, here's the context that I need before I just go and edit this file. So, it's able to understand hidden invariants where Kimi K3 is just going to dive right in, right? Like it seems like models like Kimi K3 are more to the punch, do exactly what you tell them, and not think for itself or explore anything more that might aid in the current task. And so, when you have these really well-scoped tasks, Kimi K3 is going to knock it out just as much as Opus 4.8, even better. That's a lot of what we see in those public benchmarks. But for real work, where we might not be fully confident, we might not have the right guidance, or there's a lot more context the agent needs to explore than you'd have in these benchmarks, that is where Opus 4.8 and other models like GPT 5.6 Soul are just straight-up better. And so, I don't even really have to go through the advanced stuff here. It's just the same story. The problem here is that, yes, K3 is incredibly impressive, but it's just an issue of reliability. So, it's not necessarily the capability, like the raw output, especially for really bounded task, is sometimes even better than Opus. But it's just we need a model that can really think for itself and understand all the context it needs to load in without us having to handhold it through every step of the workflow. That is where Opus shines. And so, this continues to reinforce the idea that I'm keep trying to present here, where you want to use the more powerful model for planning. Because when you're planning, that's when you're going to identify any kind of false premise, any other invariants that you need to identify, and then you send the implementation off to the workhorse once you have all those things identified and corrected. Because that's when you're going to rip through a very solid implementation. So, I know it took a bit to get to the conclusion here, but I hope that you found that helpful as well as just seeing the full process and all the benchmarking that I'm doing cuz I'm going to keep doing this for new models as they come out to just keep helping you identify what are the models that you should care about, where do they fit in your AI coding workflow. And so if you appreciated this and you're looking forward to more of this benchmarking and AI coding workflows in general, I would really appreciate a like and a subscribe. And with that, I will see you in the next video.