Plan with Claude Opus, Build with Kimi K2.6? LIVE Mixed-Provider Benchmark

We are live. Time to do some more experimentation with our dark factory. So the last live stream I was doing some testing mixing providers. So using Archon as our harness to build workflows that combine Opus and Kimmy K 2.6 to handle something end to end but not relying on you know the big model for the entire thing. being able to use Kimmy for a lot of our research and implementation validating and then opus for the planning. Right? So it's like that one step that has the most reasoning required. That's where we lean on the bigger model. But in order to be very token efficient for the rest of the workflow, we use Kimmy. And so let me actually pull this up here. But I have the the Kimmy for code or the Kimmy coding subscription that I'm using. So I have my enthropic subscription when I'm using Opus. And then I have my uh Kimmy code subscription for every other part of the workflow besides the planning itself. Um so this is sorry this is the wrong page here. Uh Kimmy code console. I need my Kimmy code console because I want to have this up for the stream as well. Um is this it sometimes? Okay. For some reason it's hard to find the right place in the Kimmy uh UI. But anyway, we'll get back to that. So, what I'm going to be doing today is I'm going to be doing some live benchmarking. And so, let me pull up the right details here. Okay. Yep, there we go. So, what I'm going to be doing today is I'm going to be testing different combinations of Opus and Kimmy for these Archon workflows. And so as preparation for our stream today, I created a bunch of Archon workflows already. U let me bring that up here. So I have that right here. So this is my repository for the dark factory. This is the codebase that builds itself autonomously. No human allowed for writing or reviewing code. And so I'm using archon as the orchestrator, right? So like these workflows are what guide the entire evolution of the codebase. So triaging issues and doing all my testing, handling each one of the issues, merging things to production. I have agents handling every single part of it with Archon workflows. And so I've built four workflows here. So KK, KO, OK, and OO. Let me explain this really quick. So I'm going to open up my benchmark playbook here. So for each one of these workflows, I'm using a different combination of Kimmy and Opus. So for benchmark OO essentially I'm using opus for both planning and implementation and then for okay it's using opus for planning and then Kimmy for the implementation really the rest of the workflow and then I flip it right so like just Kimmy for planning and then opus for everything else and so really what I want to test here is what is the optimal way to mix providers and the reason this is so incredibly important right now is because the rate limits are getting really bad for all the frontier models. Our subscriptions are getting more and more rate limited. It's just becoming more expensive to run the more powerful models, but we don't want to just ditch them entirely. Instead, what we can do is lean on the cheaper models when possible, only using the more expensive ones when we really need it. And Archon is perfect for this because it allows us to build these workflows that mix the providers very seamlessly. So for example, this is the okay workflow. So we're using opus for planning, Kimmy for implementation. And so the way that this workflow works is it takes a GitHub issue as context like here is the task that we want to work on, the feature we want to implement or the bug we want to fix. So it fetches the context of the issue and then it feeds that into our first model here which is going to do the exploration. So we're just going to use sonnet for exploration because really like the benchmarking here is like for the planning and implementation specifically, right? So like here's our prompt. You're exploring how we're going to handle this issue and then we pass it to planning. So based on the issue context and the exploration of the codebase that we did, now we're going to create a planning artifact. So a markdown document that outlines the entire scope of work for this issue fix. And so we're using Opus for the planning and then we pass that plan to our implementation. And so for this workflow, this version, we're using Kimmy K 2.6 for our implementation using PI as our coding agent. So whenever we go to Enthropic, we're using Cloud Code. Whenever we're using Kimmy, we are using Pi. And I'm just going to be using my Kimmy 4 code subscription. So I already have my credentials set up for both on my machine. So for our live stream today, we get to jump right into it. we get to do some benchmarking. I'm going to actually execute all of these workflows on a few different issues that I have in the dark factory repository. So, real work and I'm going to benchmark it across many different or a couple different issues. So, we're going to actually have uh 12 workflows running at the exact same time. I hope my subscription rate limits don't get hit here. Uh cuz we're going to we have four different workflows and I want to run each of them on three issues. So we have that matrix there of 12 archon workflows that we're going to have running at the exact same time. And we have like a little bit of self-review as well. And then we create the pull request. Every single one of the four workflows that I have here follows the exact same setup. It's just using different models at different steps. So like this one uses Kimmy for planning and implementation. This one is opus for everything. And then this one uh sorry I know the KO is one I didn't show already. This one uses Kimmy for planning and Opus for implementation. And the really cool part about Archon workflows is mixing providers, switching between providers and individual models is so easy because we can do it at an individual node level. Like I can just specify for the planning step in this workflow. I'm using PI with Kimmy and then implementation using Claude with Opus. And so really the only difference between these workflows is just swapping a couple of those parameters. I just wanted to have them all as separate workflows just so that we have an artifact for each one of the tests that we are running here. And then uh the other thing that I have here is I have a couple of issues that are already selected. So just so that we didn't have to like go through that process together, there are three issues that I have identified in the codebase right now. And each one of them well one of them is like pretty easy. So, I want to have kind of like a control that's like super easy. But then a couple of the other issues that I've identified, they require a good amount of reasoning. Now, there they don't require that much as far as like actual lines of code changed cuz I don't want these workflows to take too long for the live stream, but it does require like a a pretty deep understanding of the codebase, like having a good plan and knowing how to validate things after the implementation. So I I spent a good amount of time making sure that we got some good use cases, some good issues for our testing today. And so now really all I have to do is run these 12 workflows. Like we're going to get into the meat of the live stream actually very fast here, which I'm excited for. Um so all I'm going to do is within my second brain conversation here, I'm going to ask it. All right. So, do you have all of the workflows ready to execute on the three issues? And do you have full context of the issues here with the cache race, port mismatch, and RRF determinism? Okay, so I'm just going to like set the stage with my second brain a little bit before I use it to invoke the archon workflows. I just want to make sure that it has the context it needs to invoke them, right? Because like these workflows live in the dark experiment repository. So, I just need to make sure it understands that. And so yeah, I've got the the markdown for my issue candidates as well. So I I think it'll read this if it hasn't already. Um yeah. Okay. So it it does need to do a little bit of exploration first. So okay, it found the repo. Let me investigate. Let me inventory the benchmark workflows. Read the issue candidates. Okay, good. So it's it's exploring this right now. And uh then I guess the other thing I haven't talked about just as we're waiting for it to load context is we have to talk about the scoreboard. So how are we going to evaluate the effectiveness of the implementation for the issue? And really my plan for that is just to use my second brain. So my second brain when it evaluates the output of each one of these workflow runs, it's going to look at the pull request and it's going to uh evaluate a couple of different benchmarks. So we have the the evaluator YAML here. So use this when scoring a benchmark pull request against seven dimensions. And so the seven dimensions, uh let's see if I can find where that is. Here we go. So, we're going to be judging each one of the pull requests um on a scale of, you know, 1 to 10. Did it actually address the issue we asked for is and then for the root cause, is it structurally sound fix or is it a band-aid that papers over the symptom? Uh and then scope discipline because this is something that I've noticed a lot of models not do the best with like Kimmy and Miniax is sometimes they'll go out of scope and they'll touch more than they actually need. So, you know, 10 a score of 10 is like it's very surgical, right? like it only touched the things it really needs to to handle the issue. And then one is it's sprawling with a ton of unrelated changes. And then um we're we're also rating like subtle correctness like how it handles edge cases and async operations like making sure there's no foot guns. Uh code quality overall. I mean, there's definitely some overlap here, but I'm I'm just trying to make it so that the evaluation is really really nuanced because otherwise it's if I'm just leaving it up to the coding agent to describe very generally how good the implementation is, I don't really have confidence in that. So, even though there's a little bit of overlap here, I'm allowing that because I'm trying to just be like extremely specific in my instructions for how we're evaluating this. And then you know like how good are we at adding tests and then the plan implementation fidelity. So like this is where we have the um the evaluator here look at the plan that was created either by Opus or Kimmy and then compare it to what the diff actually delivered right cuz we're looking at the pull request the final diffs in GitHub and then comparing it to what we have in the plan cuz this will show us if like Opus did a good job planning but Kimmy just messed it up or vice versa, right? like we want to make sure that we're correctly evaluating the plan as it relates to the implementation. And so then we just have structured output in this archon workflow here that gives us the JSON for each of the scores. And so we'll have a final score adding everything together. And so the highest that a single workflow run can score is a 70 out of 70. And uh from some of my like initial smoke testing that I did, a lot of these workflow runs were getting like you know 60 out of 70 something around that. Uh but I haven't like done the entire test with all 12 workflows. So I'm going to be exploring this in real time with you guys here. So let me go back to my second brain and see where we are at. Um okay. Only open issue on the repo is 2020. Yeah. So we have to create the issue still. So that's fine. I'm not sure why it's still loading context here, but I'll let it keep loading, getting things ready, and I'll head on over to the chat while I wait for that. So, it's a good good time to pause anyway. All right, so good to have all you guys here. Hey. Hey. All right. Uh, let's see. Hey, Cole. Uh, what do we use to replace the Archon document server we used to have as an MCP? I got a lot of API documentation. So, um, honestly, these days coding agents can just search the web really, really well to pull any kind of documentation. If it's something that's like for your own APIs, then I would just have it as like markdown, like extract the documentation as markdown that you just put in the AI layer for your coding agent. Um, yeah, like the reason that I ditched all like the rag stuff from the past version of Archon is it's just coding agents don't need that anymore is really what it comes down to, right? like that. That's why the new version of Archon as the harness builder, the power in that is it's the layer above the coding agent instead of the old version of Archon just being tooling for the agent. And so, so tooling gets replaced, but the orchestration layer doesn't really. Um, okay. So, what is it saying here? Workflows are ready, but the issues are not. Okay. Um, three issues aren't filed. Overnight prerun never happened. Well, I don't need that. I know I don't have full context of two of the three. Okay. So, let's let's go ahead and tell it to read this here. Uh so, I'll copy the path, paste it in. Uh this gives you all the context you need for the issues that we need to file. So, I don't want you to do it yet, but let me know when you are ready to file the three issues and then run all 12 workflows to uh handle these issues with the different workflows that we have. All right. Cool. Out here doing God's work, Cole. I appreciate it. Yeah, I'm excited for this. We're we're we're going to solve the Okay, here here's the main question that we're solving right now that came up in my last live stream. If you want to mix providers for the sake of token efficiency, is it better to use the more powerful model during implementation or planning? So, in my last live stream, I decided I'm going to use Opus for planning. The idea being if the structured plan is really thorough enough and outlines like all the parts of the codebase that you have to touch and what the validation strategy is going to look like and everything, then you don't need as powerful of a model for the actual implementation cuz it's following a really speced out blueprint already that has all the validation and like here's the task list and here's the success criteria, right? That's one way to think about it. But then you could also argue that it it's still better to have the more powerful model writing the actual code changes because you don't want any kinds of hallucinations not following a plan to a te and even if the plan isn't as good because you use a cheaper model well at least the more powerful model is going to catch those things during the you know self-re and implementation and things like that. And so that's what I'm trying to figure out here really. And um yeah, like this live stream is not just for show. Like I I actually really want to figure this out because I'm starting to use Archon for a lot more of these mix provider workflows. And even if you aren't mixing providers, like you still just want to use anthropic for everything or codeex for everything, this idea still applies to different models, right? Like when do we want to use Sonnet versus Opus or GBT 5.4 Nano versus GPT some, you know, codeex model. That's the kind of thing that I'm I'm looking for. Um, okay. I'm going to just copy this information. So, sorry. I I realized that it was fixing a different issue. So, I'm going to copy this. Uh, here are the issues we are going to address. So, I did a little bit of prep off camera that I think made this a little bit stale, this issue candidates markdown. That's why I'm just sending it in some context from that conversation here. All right. I've been thinking about this type of approach for a while, but haven't tried it yet. Well, I hope that this uh live stream can inspire you to try it because uh it's very easy to set this up in Archon. All right. Uh is this functionality multi-providers available in Archon version.32? Uh yes, it is. Yep. So, we've had support for multi-provider for quite a while now. So all you have to do as long as you have the authentication set up in the machine where you're running the archon workflows all you have to do is uh in each individual workflow like well sorry in the workflow in each individual node you just have to specify the provider and model uh so like pi with Kimmy for coding and then um like right here using uh claude with opus And uh obviously the the big caveat here is if you are going between providers, you can't continue the same conversation, which a lot of times to keep your agents focused, you don't want to anyway. But like you'll see here that when I do my planning uh with Claude or with Opus here, I am uh setting context fresh. So it's a brand new coding agent session. And then going into implementation, same thing. Now, they're still communicating with each other because the planning step creates the plan markdown and then I instruct the implement step to read it. So, it's not like we're totally losing context, but it's not that same context session that we're continuing. There's no way to do that when you're mixing providers. But, yeah, like I said, most archon workflows are built around that idea anyway. All right. Uh problem is that cloud performance is degrading notably since the 1 million context window and with uh 4.7 it continues to degrade. At this point I'm using Opus 4.7 as code writing machine at best. Yeah, that's the other thing is along with the rate limits getting harsher. It's just like the literally the model output quality is worse over time because our subscriptions are subsidized. It is pretty unfortunate. So I still consider Opus to be the best at coding in general even with some of the degradation that like I I agree is there to an extent. It still is the best but uh it definitely feels like the writing's on the wall that it's going to get worse and worse to the point where we just can't rely on it. Like that's why we have to explore other models like Kimmy and just like all these other open source ones that are being released. All right, cool. So let's see where we're at. Uh, here's the readiness report. I'm not go yet because issue two has a real problem that needs your call. Okay. Well, we'll we'll try to figure that out. Uh, verification results. File ready. File ready does not exist in the port mismatch does not exist in the code. Okay. Uh, all right. Well, okay. If issue number two doesn't work, let's just find a different issue. So, I need you to dig deep and find another issue. Let me know when you're ready to address that so we can kick off all 12 workflows. All right. Uh, so I mean I got to be honest, I didn't spend too much time figuring out the right issues to address because I thought Claude would just nail that. But I guess I guess uh the initial session I had that planned this hallucinated and this isn't a real problem. Um, yeah, like I' I'd spend a lot of time like looking at different issues, but I didn't like evaluate it because I didn't want to like run it end to end. Anyway, we'll we'll just find another issue here. Okay. Best control candidate is likely a genuine real mechanical small diff defect. And the most trustworthy source is the team documented issues. Okay. I don't actually know what it means by team documented because there's no team running the dark factory. But anyway, we'll see. We'll see what it comes up with here. And then in the meantime, I want to try to get my uh Kimmy for code console up. Uh yeah. Okay. This is This is what I was looking for. Console. Okay. Kimmy.com/code/conole. Okay. I'll remember that. So, so far I used So, in the last live stream I used 5% of my weekly limit with Kimmy, which I used like millions of tokens in the last live stream. So, that's really not bad at all. Like, you get pretty high usage with the Kimmy Code subscription. So, I'm on the $40 a month subscription. Um, lot cheaper than Claude Code. gives you a lot of tokens. And then for my enthropic subscription, let's see where I'm at with usage there. I'll actually pop this up quick. Uh so claw.ai. Let me pull up my usage. I'll show both as we run the workflows here. And I just want to monitor that to make sure we're not screwing ourselves over. Okay, let's see. This is good. Okay, so this is my usage right now. So, I've used 11% within my 5 hour window and then uh 60% for my weekly limit. That's resetting tomorrow, which is actually not bad. Last week, my rate limit was hit way faster. And I've done about an equivalent amount of work this week and I'm only at 60%. And I hit 100% like Tuesday last week. So, they've actually let up on the rate limits this week for some reason. It was like worse last week. I don't know if you guys have experienced that as well, but it's not as bad as it was last week. So, I'm not complaining. Um, cuz last week I did actually have to buy I had to buy a second Anthropic subscription, which was really unfortunate. All right. See where we're at now. Um, okay. Still trying to find Yeah, this this is kind of annoying. It has to find an issue, but we'll let it we'll let it find that. All right, going to take a sip break. [clears throat] Let's go back. I'll just kind of wait for it to finish here. All right, Jeff said, uh, I think they did something last Thursday. I noticed my weekly limit reset on Thursday. Really? Mine did not reset on Thursday. But I guess it's kind of different for everyone, though. I don't know. But uh I mean, good for you if it reset early. That's nice. All right. Come on. Find an issue for me. I just need something here. Let's go back over to the the workflows here because I I kind of want to just like show how this works more exactly while we wait. So each one of these workflows that I've built, they're a more simplified version of the fix GitHub issue workflow. So this is the core workflow for the dark factory that handles any issue that we give. And so the way the dark factory works is the input for the factory to run is always a GitHub issue. If there's a bug I want to fix or a new feature I want to build, I always create an issue. And then it also creates issues by itself when I have it do regression testing once a week. So once a week I have it run a super comprehensive workflow to test all user journeys and edge cases and everything and then create any GitHub issues for anything that it encounters during its um its regression testing. And so when an issue is created, there's a orchestrator that I have running on a VPS in the cloud and it triggers every half hour to look at these issues and it'll label them. So basically doing the triage like what's the priority and what's the complexity and then it'll handle them one at a time. So it'll run the archon workflow to go from uh issue all the way to pull request. You can see that I got like a couple of test things here just as getting ready for the live stream today. And so for the sake of simplicity because I don't I'm going to run 12 workflows here. I want them to be fast. I want them to be token efficient. I've made a very much a simplified version of this workflow for each of the four that we have here. And so we get the context of the issue and then we explore it. Right? So the way that we communicate between nodes in archon is we can access the output field. So we take the ID of the archon workflow node right here and then we do output. That's going to give us the last bit of output from that node, which in this case is going to be the the issue context here, right? GitHub issue view, issue number. We're getting the number, title, body, label, author, and so that is sent in as context. And so now we're going to explore the repository to look for files and modules involved in the issue, existing patterns and conventions to follow, just curating all that initial context that's then going to be sent into our planning step. So explore and then turn that into a structured plan. So in archon the way that the nodes communicate to each other is either just you know straight up looking at the output of the past node um or looking in the artifact directory. So every single work tree that's spun up for an archon workflow always has a dedicated space for its working space. And so this is where we can output any markdown documents that we want future nodes to read. So right here we're just giving a summary of our exploration. You know like here's what the issue asks. Here are the relevant files, relevant patterns, risks and edge cases. Putting all that in a markdown document so the next node can read that. So it fetches the issue context again just the output of that first node reading the exploration markdown and then based on that it's going to write out a plan with the approach files to change specific changes right so we're getting a lot more detailed here like initial exploration goes into like okay now let's get a concrete plan and honestly like this prompt could be a lot bigger because there might be more of an approach that I want it to follow more specifically for how I want it to create this plan but again I just want to be really concise here because if the workflow prompts aren't optimized, that's okay because this the point of our stream here is to benchmark different models, not to make an ideal workflow. Like definitely my usual planning prompt is a lot longer than this. But I hope you get the idea here where it's more about like let's have something simple and easy and fast to test that's good enough where it's still like a pretty legitimate workflow, but it's more about the different models we use for the same prompts. Uh, so let's let's go back and see where we're at now. Okay, so all three now verified. Uh, so we're good to go. Orphan video row. Okay, so we have like a super easy fix again and then the two same like higher depth fixes. Okay, so uh once I say go, it'll file all three issues with the label so the factory knows that we need to work on it right away. Then we're going to dispatch waves of four. So wave one is going to be all four cells against issue number one. Then wave two on issue number two. Wave three on issue number three blocking between waves. After all 12 pull requests land, we're going to run the benchmark evaluator workflow over each. This is what evaluates against the seven criteria that I showed earlier. I showed that in the prompt. And then uh who dispatches the playbook says run the waveloop from a plain terminal not inside cloud code. Uh well they actually don't care about that. Um, clean slate. Let's see. So, yeah, you are going to dispatch these workflows here. So, I'm not really sure why it says run in a plain terminal. I want you to go through all of the waves. I want you to run the evaluator after all the pull requests land. And I don't care about the clean slate. So, I'm not worried about that. So, yeah, I want you to go ahead and kick off our testing here. All right. Cool. So, we are going to have my second brain manage this entire process. And like I said, because the workflows are simpler, hopefully it doesn't take too long here, but it'll be cool to uh watch them go cuz as the workflows are running, we can see the issues that get created. We'll see the pull requests get created eventually as well. And then I also have the archon web UI up so that we can uh you know track the status of the running workflows and see the ones that are running as they get kicked off here. So, it has to, of course, spend its time thinking before we can then get into the workflow runs. Anthropic had a big compute deal with SpaceX, so they changed the limits. Yeah, that's probably part of it. The thing is though, they announced the SpaceX partnership the same week the rate limits were awful. So, I don't really know why. I guess maybe there's a bit of a delay in getting some more hardware set up or whatever, but yeah, that probably is part of why it's better now. All right. Uh, so yeah, let me go back to my second brain. See, okay, we're still thinking. Honestly, there's going to be quite a bit of downtime just as we're waiting for the workflows to kick off here. So, one thing I was interested in doing potentially is trying out the new anti-gravity. I think that would be neat to do just as something that while we wait unless you guys have more like questions or stuff for the chat. But uh I have not tried the new anti-gravity with the new Gemini 3.5 Flash and I'd be very interested in doing so. Uh I actually have not used anti-gravity in quite a while to be honest. So I don't even know how to check for an update. Oh, maybe I did already update. Am I updated? How do I check editor settings? Can I could I just ask it? What version of anti-gravity am I using? Oh, no. It's downloading an update now. Okay. Yeah. So, I think I have to update this still. [laughter] All right. We'll get this updated. We'll we'll test this out in the same time. At the same time here. Okay. Installing update. Cool. News came out today that they are approaching they are approaching um Microsoft for compute too. Oh, that's cool. Okay, I didn't actually know that. Here, hold on. Let me take a look. Um Anthropic and Microsoft partnership. Let's see. Um well, they had their partnership at the end of last year. for more compute. Let's see. Uh I'm not seeing anything specifically. Oh, here. This might be it. Anthropic is in talks of using Microsoft AI chips two hours ago. Oh, but I have to subscribe to read it. Okay. Well, there's something here, but I can't read it right now because I'm not subscribed to the information.com. Uh, not really interested in paying for that. I guess that's like all there really is right now, though. Interesting. Cool. All right. Uh, where are we at with anti-gravity download? Okay, cool. Because I I've uh heard that the new anti-gravity is insane. Like, so incredibly fast. All right, so let me sign in my Google account here. Yes. Let's try it out. Open anti-gravity. Dope. All right. And then, well, actually, I should check on my second brain, too, and see where we're at. Uh, three issues are filed. Um, let me check all three and fix. Okay. So, I think it's going to start by invoking issue number 229. Okay. So, let's look refresh. Yep. Okay. So we have the issues here. Hybrid retrieval is non-deterministic. We got the easy bug and then we have the video cache problem. And so let's just make sure that we actually invoke an archon workflow. Uh so I think it's starting with 229. All right. Well, I'll let it keep going here. So let's jump back over to anti-gravity. Um I'll just uncheck that. Don't I don't know. Uh, dark theme plugins are packaged collections of skills in MCP to help the agent in anti-gravity. Um, yes. Yes. Yes. Sure. Let's just do those. Okay, cool. Oh, wow. This looks a lot different now. They is there. There's not even like a classic IDE anymore. It's totally just uh you manage your projects and conversation. That's pretty cool. I mean, I I appreciate that because I know the IDE needs to change. Um yeah, like if I click on conversation. Okay. Create an HTML. Um well, here I'll say research archon.diy diy and github github.com/colium0000 and then build a single HTML file that gives a comprehensive overview of archon. Let's try this. I I want to just do like a really quick test here. Apparently it's like insanely fast. We'll see. All right. So, while it does that, let's go back over. Okay. So, wave 1 is dispatched and running. Here's where we are. So, we have the files or the issues created. Wave 1 is running. So, we have four workflows running against issue number 227. So now, if I go up to our UI here, there we go. Sure enough, we've got KK, KO, OK, and OO all running. So, different combinations of Opus, and Kimmy handling the exact same issue. And so once these are created, then it'll go on to wave two handling that second issue. And then it'll handle the third one. And then we'll have 12 pull requests in the end to evaluate and see uh what actually scored the best. And if I wanted to be extremely comprehensive, I would probably have more than just three issues. But I think that's already going to kill my rate limits here. So I'm I'm going to be happy with this to start. And then maybe I'll keep doing these these benchmarkings if it uh feels useful. So yeah, this is pretty cool. So we're off to the races here. So now we can just kind of keep poking around with other things like anti-gravity just while we wait for these workflows to finish. Hopefully it doesn't take too long overall. It says that it'll only take 3 to 5 minutes for the wave to finish based on the smoke testing that I did yesterday to prep for the live stream. I don't really trust that though. I think it might be longer, but we'll see. All right. Okay. So it gave us an implement. Oh, this is cool. So, I have research archon and unfortunately I can't seem to zoom in. Control+ isn't working. Oh, wow. Okay, so I can zoom out, but I can't zoom in. So, I have to just click this manually. Plus isn't working for me. So, I just want to zoom in for you guys. There you go. So, I've researched Archon and designed a comprehensive single file HTML page overview. I've created the implementation plan and please review it for approval. Okay. Um, wow, this looks really neat. Okay. So, modern premium styling, deep slate to indigo dark mode color scheme with electric purple and cyan accents. Uh, let's do purple. Let's do pink and green colors. Like the archon logo. I want to change that. I think that'd be that'd look a lot better. Uh, the harness builder for AI agents. That also I don't really like that. Okay. Well, it's not that impressive honestly in its reasoning capability here. uh cuz it's the it it is the uh opensource harness builder, not the harness builder for AI agents. So, I think it kind of got the headline wrong. Okay. Pending message. So, it's working on this. Okay. Yeah. I mean, the user experience here is pretty good. Okay. Yeah, it's fast. Wow. Nice. Um anti-gravity is so bad and not finished. Gemini 3.5 flash just for for the price is really bad, but Composer 2.5 is really good. Okay, interesting. Good to know. Well, yeah, I'll see my take on it here as well. Um, I've not tried Composer 2.5, though. Um, did anti-gravity fix the Windows sandbox issue? I believe so, but I'm not like 100% on that. Uh, the news was posted all over X an hour ago. Just search Microsoft Enthropic. Oh, okay. Yeah, let let me let me do that then. Um x.com. See, I think I'm signed into my X on this. Yeah. Okay. Uh Microsoft Anthropic. Uh yeah. Okay. This Yeah. Talking to rent Microsoft AI server chips. Startup seeks to boost community power and meet demands for its services. Okay, so official Bloomberg. That's cool. All right. Accept. Oh man. Well, this is a mess. Okay, never mind. I guess I can't read that. [laughter] All right, but anyway. Yeah, this cool. It's official. It's official. All right. What do I think about Andre joining Anthropic? I think it's cool. I'm actually kind of surprised cuz he was working on Eureka Labs and he was so um gung-ho for like education and in his tweet he's just like we'll come back to that where uh where did he say that um Andre car I think well okay let me just pull this up again yeah see personal update I joined anthropic I remain deeply passionate about education plan to resume my work on it in time it's so vague it's kind of funny like I understanding because Anthropic is such an impressive company right now. As much as it's, you know, kind of frustrating in some ways the the direction they're going with the subscription, like overall, they're still an impressive company. So, it's cool to see him join because I know he's going to help them a lot. Um, but yeah, it's kind of funny that he's just like, "All right, that's Eureka Labs. Like, we'll come back to that later." I'm sure he took it out of his bio already. Uh, yeah, he took it out of his bio already. [laughter] [gasps] Um, I wonder if he still talks about it on his website here. If I search for Eureka, I don't even remember how to how he spelled it. Yeah, it's not there anymore. It's a bummer. It's just gone. Goodbye. I thought that was going to because he basically he was building like a aentic um like an agentic first education platform that basically be like you it's like Corsera. Okay, I don't know for sure this is what it was, but I think it was like Corsera, but every student has their own personal tutor that walks them through the courses and helps them absorb all that information and go through exercises and things like that, which was going to be really cool. Um Oh, I guess. Okay, I see Eureka Labs. So, he has it here. It's just an image. Um, but he didn't like actually build up the platform, right? Like he just like was doing a little bit of like education on YouTube to get started and then now he's scrapping that for now. All right. So, yeah, let's go back and see where we are at. Okay. So, wave one is still running. In fact, let me see if I view the logs here. I'm curious where we're at right now. Um, oh wow, something is really off with the logs here. That's weird. Okay, Kimmy might have botched something for this. Interesting. Okay, we'll see where we're at with that. Is that the same for all of them here? If I open up Okay. Yeah. So, okay. Okay, I think something went wrong with tool calling with Kimmy cuz they had a failure there. We'll see. But that might that might botch the benchmarks. Um, where are we at with KO? Yeah, it did the same thing. Okay, something's weird with Kimmy here. Or maybe it's like a bug with how Kimmy interacts with Archon. I'm not really sure. Not a problem for this one, though. Using Kimmy for implementing. Interesting. Okay, well, we'll let it finish here. Let me go back to anti-gravity. Okay. Um, all right. I've completed the research and built the single page HTML overview for Archon matching your brand guidelines. Project files been created in a proposed subdirectory. Here it is. So, can I actually open this? Because I thought there was a way to preview stuff in anti-gravity so I could view the the website live. Is that not a thing? It gave us a whole walkthrough and everything. That's cool. Uh yeah, it looks like I can't open it directly here. But anyway, I can copy the path and then I'll just go to a new terminal here. Oh, hold on. All right, let's open this up. Open with Google Chrome. >> [laughter] >> That. Wow. This looks really cool. Holy cow. For no guidance at all except like two tiny little things. This is This is good. This looks good. Look at that. We got the installation commands. That looks cool. Enforcing structural determinism. Raw AI assistants are infinitely capable but highly variable. Archon solves this variance by acting as a rigid portable harness around cognitive models. I this isn't really how I would personally explain it. So maybe it's not the wording that Gemini 3.5 Flash did the best with, but as far as actually making a site that looks good. This looks good. Archon is the Docker for AI. It's the CI/CD for LLMs. Again, maybe not the best explanation, but oh wow, like [laughter] that's pretty good. plan, implement, execute. Nice. Authoring custom workflows. Yeah, I mean it it's it's pretty insane how much context it pulled to be able to create these examples. Just blitzing through that in just minutes. Architected for speed and trust. Get or work tree isolation, composable node mix, self-healing loops, multi-channel UI, TypeScript and button core, workflow marketplace that we were working on in the last live streams on my channel. Very cool. Interactive integration. This is nice. Is archon. Okay. Frequently asked questions. It's kind of cool that it added this. Is archon an alternative to claw or cursor? No. Archon sits a layer above the coding assistance just like I said earlier. Uh, think of tools like cloud code or cursor as the execution engine or cognitive worker of your pipeline. Archon acts as the product manager or orchestrator holding these models to structured boundaries. Yeah. What models are supported out of the box? Connects natively to any model provider supported. Okay, that well it totally elucinated that. That's actually terrible because we don't we don't have anything to do with the AI SDK from Versel in Archon. Uh that's that's funny. So wow it seems like okay and this is the consensus that I've heard in general from other people is Gemini 3.5 flash is really good at um building frontends like design is good. Like this doesn't feel like an LLM generated website. It well I mean some things like like this specific component here kind of does. I feel like every lovable and bolt website ever has this like rounded box little thing at the top of of the hero. Uh but for the most part like it doesn't feel super LLM generated like you would usually have with claude or or GPT but uh at the same time Gemini 3.5 Flash is not as smart as something like Opus or GPT codecs. Like it it it straight up hallucinated things like this that is actually pretty unacceptable. Like that's ridiculous. Um but yeah, like that that's why here's another good example. Actually, this could be another really cool live stream or maybe something I'll do now if I have time is uh building an Archon workflow that uses Gemini 3.5 Flash to design the front end and then Opus or something like Kimmy to uh actually create the content, right? So it doesn't hallucinate things like that. That could be pretty cool. Um how to get workree isolation works in practice. When you run archon run, the engine runs get workree ad pointing to a unique branch. All subsequent bash scripts, llinter rules, cognitive code rewrites occur inside the completely isolated directory. Once the workflow is complete, Archon pushes the branch to your remote, opens a pull request, and deletes the work tree structure cleanly, which is true, but it depends on the workflow. Like not every archon workflow does code changes and ends in a pull request. But like generally, yes, that's how you use archon. Uh, but yeah, definitely like not the most accurate information here. This is pretty good. link to the documentation that works. GitHub repository workflow hub. Um, well, it didn't quite get that right. It's workflow. So, another hallucination. So, yeah, it's like design's good, information's bad. Badish. [snorts] That's pretty cool, though. Um, I'm trying to think like if I wanted to I wonder can I use Pi with Gemini 3.5 Flash? That would be the question. I have to ask. Okay, because unfortunately we can't use the Gemini CLI with Archon because they don't have an SDK. And [snorts] Anti-gravity has an SDK, but it's Python only. So, we also can't use that with Archon. So, if I want to use Gemini 3.5 Flash in an Archon workflow, it' have to be through Pi, I believe. Or maybe I could overwrite Cloud Code to point to it. I'm not totally sure. But I also don't want to interrupt that when um I have the other our other benchmark going here. [snorts] Okay. So, give a status update of wave one. I guess we're going to have to wait for these waves to finish here. Maybe I will try to build that workflow. That could be cool. All right, let's see. So, I'm just going to ask it to do an update. I'll check the UI at the same time. Uh okay. So it looks like two of them are done. So if we look at benchmark OO, we can see that it finished. [snorts] Okay. Create PR standard error failed to create a pull request. Oh no, it did create a pull request. Okay. I think I just had to iterate once or something. But we have the pull request created. Bench O implement address issue number 231. Okay. Okay. We're getting there. We're getting there. So, it's working. It's working. Let's see what uh my second brain says. Wave one finished. Let me read the completion summary and verify all four PRs. Okay. Did it actually finish? Because it says these two are still running, which is kind of weird. Um, okay. It's creating the pull request, but yeah, it actually did make the pull request. We have K O K. Okay, there's one we're missing. We're missing um Okay, we're missing Okay, we're missing Opus for planning and Kimmy for implementation, but we are almost there. I'm not sure why it said I might just need to close this one. I'm going to close this issue here. You can see that we have the scoring on this as well, but I'm going to Yeah, I'm going to close this pull request. That's the one for yesterday when I was just testing things. So, we we only have we actually only have three right now. So, I'm not sure why my second brain said. Okay. Okay. Yeah. So, the retry is now running solo. Uh why did we have to retry? I'm curious if there was like an intermittent failure or something. Oh, okay. It's Opus subprocess crashed. Archon retry then hit the guard. No PR. Uh, this looks like a transient crash. Okay. This happens unfortunately with the clawed agent SDK. This happens once in a while where it just crashes. So, I have like retry mechanisms built into Archon like you guys see right here. But, it is unfortunate. This happens with um honestly everything except codecs like Kimmy Kimmy does the same thing where it'll like hang sometimes and so yeah the agent SDK will just crash and you have to retry a node or a workflow but we're getting through it. Um you know what I actually might do here open sub process. Okay. Uh let's actually kick off wave two and three in parallel. I just want to get through everything here. So I want you to invoke all the workflows in parallel. So, we should have nine running at the same time here. And then I want you to run the benchmarking after they're all complete. So, I'm going to speed things up a little bit. The only reason I had it split into different waves here is just for so I can track things closer and for the sake of uh not hitting like rate limits just in case. But I think I'll actually be okay. Um cuz yeah, like these are pretty small PRs overall. So you can see I with the archon workflow even though I'm going through all the planning implementing validating like I didn't actually use that much of my limit yet. And then if we refresh the console for Kimmy I have used yeah only 3% of my 5hour window. So we're we're going to be good to just rip through everything here. So let's just do that. All right. Cool. Um, anti-gravity settings model show model usage. Okay, let me let me try that. Anti-gravity file. Wait, where? Oh, settings here. Um, show model usage. Wait, really? I've already used 20% of my Gemini 3.5 Flash. That's kind of ridiculous. I just created a single page and it use it used 20%. Wow, that's unfortunate. That is ridiculous. I created a single file and it used 20%. Man. All right. Um, added a sequential think MCP server and mempalist anti-gravity. That's cool. Nice. All right. Uh, wow. I can't believe that used 20%. [laughter] Okay, I'm going to I'm going to try I'm going to allow it to iterate here and see if we can make it better. So, uh, okay. The the design of the website looks fantastic, but the information is hallucinated in a lot of different places, like using the Verscell AI SDK, assuming every Archon workflow ends in a pull request. I need you to do extremely extensive research of the Archon codebase to make sure you're validating every single one of the claims and uh then update the the page accordingly. So I I probably could be more specific, but I just want to see what happens when I let it run through something bigger. I want to see how fast it is. Um okay, allow running a get clone. No, I have archon cloned locally already. So just find that. I don't need it to clone it again, but I can understand why I'd want to. Okay. Yes, I'm going to allow always. Okay. I don't really know how to do yolo mode or whatever. So, I guess I'll have to keep approving things here. But, okay. It found it found the the code base, so we're good. So, I guess the first time it only just explored five. Yeah. So it it stayed to the web and now now it's actually looking in the codebase. So that that this will be better. So maybe maybe Gemini 3.5 flash isn't that bad. I just it wasn't good or specific enough in my request. We'll see. Um wait till I use a cloud model. Oh boy. [laughter] Oh boy. All right. Okay. One honest flag then I proceed. Nineway parallelism is more subprocess contention than the four-way. So, expect a higher chance that one or more cells crash. That's fine. I don't really care. Uh, I'll get a notification as each job finishes. So, nine workflows are running concurrently. Okay. So, we got we got everything that that we can running right now. Nice. Okay, cool. So, we're handling issue 229 and 228. And then we have the one retrying for 227. Okay, very good. Good. Let's see how our rate limits progress here. So, where are we at with Kimmy right now? Uh 5%. Yeah, we'll we're definitely going to be good. We'll rip through all of this. Cool. So, let me go back to anti-gravity here. All right. This is fun. Just trying out anti-gravity kind of at the same time here. uh conducted a thorough multi- package audit of the local repo. I bet it divides several crucial areas where the documentation was hallucinated like the versel AI SDK reference. Yes. Okay. So, um AI providers and SDKs, workflow modalities, engine node type, blah blah blah. Looks good. I'm probably just going to approve this and see what it builds. Um yeah. Okay. This is Yeah, it's looking a lot better. Okay. Got the primitives for the Archon workflows as well. Uh, production gradient switcher showcase. Cool. Relational database. Okay. Yeah, this looks good. This looks good now. All right. Let's go ahead. Let's see how fast it makes the index.html here. Wow. Look at how many tokens it's producing here. Like that is actually very impressive. But after it generates this, we'll look at the rate limits again because it's probably going to be like I have, you know, 30% of my usage left for the entire week. Uh, I don't have the $100 a month plan that Google just released. So, I definitely I could up my rate limits a lot with that. I think I'm just paying $20 a month for anti-gravity right now or whatever like the initial subscription was that they released when they when anti-gravity version one came out. All right. Uh, yes. And always allow select string. Sure. Oh, I shouldn't have to keep approving things. I don't really know how to change that. I assume it's something in my settings that I can tweak for permissions. Um, let's see. Yeah, I could probably tweak that. Anyway, I could go look at my rate limit now, but I want to wait until it's done done and then see what what we have. Okay. Uh, I think it's already done. Okay. Yep, it is. So, let's uh copy this path. I think it's the same. Was this updated? Yeah. Okay, this was updated just a minute ago. Okay, here's what we got now. The deterministic open source harness builder for AI. Oh, I guess that part didn't change. Um, what did change? Trying to see. Well, for sure we have that Versel AI SDK hallucination at the bottom. Um, let's see. Yeah, this is a lot better now. [laughter] What providers and models are supported? Archon integrates directly with native vendor SDKs for maximum speed of control. Out of the box core providers are Claude, Codeex, and Pi. Expose a unified harness for 20 LMS via simple provider model ref. Awesome. Do workflows have Well, okay, this is kind of weird that it decided to add this. So it seems like taken my feedback and kind of overfitted for the site. No, Archon workflows are completely generic DAGs that I also don't want to just like call them DAGs because people don't even really know what that is, right? Direct asyclic graph. People don't know that. They don't care about that. They just want archon to work. Uh while they can run isolated get work trees and submit pull requests with scripts, they can also run directly in an active branch. That's also not the way I'd frame it. So, okay. Still overall not super impressed with the reasoning power of Gemini 3.5 Flash, but the design is pretty incredible. And okay, let's take a look at the rate limit now. So, models, [laughter] it used another 20% just to edit that single file. Holy cow. Wow. Wow. Okay. Well, let's try Opus now because Yeah, I'm I'm about to be even more disappointed, I'm sure. Um, okay. What could I do now? Okay, I'm now do another deep dive into the codebase just to make sure that you're covering all the different ways that we can use Archon because the one thing that the website doesn't really talk about right now is how we can really use Archon to turn any AI coding process into a more deterministic and reliable workflow. And we can even use Archon for more than just AI coding. Like we can really use it to create workflows for any agentic process that we have for any business use case, AI coding workflow, anything at all. All right. So, now with Opus, we'll see how much we rip through the this limit right here. [laughter] Not excited to see this, but it'll be funny. All right. And then while we wait for that, we can also, excuse me, let's go back to my second brain and see where we're at with the workflow runs. So, give me a status update for both waves. And then hopefully that uh wave 1 workflow that had to be retrieded is done as well. Let's uh take a look. So, let's refresh rate limits and see where we're at. Um, apparently we haven't used more opus, which is kind of weird. We've used more Kimmy, though. Okay, that's kind of strange. Wait, why is that? That's weird. The workflow is finished and my rate limit didn't go up for Claude. That is strange. I hope it's not using I hope it's not using my API credits. I might need to check that quick. Oh boy. Uh that would suck. Claude console. I got to check my usage really fast. I have to do that um off camera here. Let's see. No. No. I'm good. I I actually have no idea why my rate limit didn't like I've only Okay, so it went up 1% now. So, I'm using something. That's weird. I don't know. Turn off extra usage to be sure. I mean, I I could turn off extra usage, but you can see here that I'm not actually using it. I have my spend limit and I haven't used a single percent of it this week or this month. I mean, I think it's the workflows are are working. We Let's see if I refresh. I should have pull requests. Yeah, I mean I got all the PRs. This is working. Okay. Yeah, it looks good. I don't know. That's weird. I It might It might be because I oversimplified things. Like honestly, I probably should have done more complex issues. It didn't use a lot of my my limit because it is such a simple test cuz right like I don't want to have like a 30 minute run for every single one of these workflows. It's going to use way too much. It's going to be too slow for the stream. So if I want to do like a kind of second round of benchmarking, I'll do the same process but just more elaborate workflows and more complicated issues is really what it comes down to. So this is just kind of me testing my initial benchmarking harness with you guys. And hopefully we'll still get, you know, like some kind of answer at the end for what combination is best in the end. So we have uh one more to be done. All 12 will be in and I'll immediately kick off the evaluator across every PR seven dimension opus scoring then aggregate the scoreboard. Waiting on that final completion notification. How close are we to that workflow being done? It has been a while because it's still stuck on that very first workflow that had it had to retry which I don't like because where are we at right now? If I view the logs, we're currently implementing. We're we're still 10 minutes in the implementation. Yeah, the tool edit failed. So, this this happens a lot with Kimmy unfortunately where the uh tool edit will fail. Like the API will just hang. I think that's actually what's happening with this workflow here is the workflow is just hanging because the Kimmy API had a fluke. It that happens like one in every four workflows or maybe like one in every eight workflows that I run with Kimmy. It's kind of become a problem like sort of disappointing. Um then I there's a lot of other open source models I'm interested in trying with this harness as well. Quen 3.6 would be another really good one. Um, honestly, I could just go over to Open Router and look at the more popular models. Um, so yeah, I mean, Gemini 3.5 Flash, of course, that would be cool to try as well. Um, it's not super cheap. Let's see what else is there. What? I don't even know what Recraft is. Oh, image gen. Yeah. Yeah. If I wanted Oh, yeah. I need Sorry, I need to go most popular. Okay, let's see. Yeah, so Deep Seek would be another one that I'd maybe want to test with this harness. Um, Owl Alpha. Is this new? Oh, that's not new. I haven't heard of this. High performance foundational model for agentic workloads. This could be a cool one to try as well. Um, what else do we have? Step 3.5 flash. Maybe this is what I'm using right now. I've tried this with the dark factory as well and it's okay. Miniax is more reliable than the Kimmy API. Uh like if you go directly to the provider APIs, but it's not quite as powerful as in terms of like the raw output as Kimmy K 2.6. Uh this model not super interested in trying. GLM 5.1 I'd also want to try. Um I'm surprised that Yeah. So Quinn 3.6 plus yet another one I'd want to try. I'm not really interested in trying the super small ones like Gemma for example. I don't think that's going to perform well at all. Uh yeah, I think that pretty much is the list of what I'm interested in trying. So we got like four or five other models that would be cool to try with this benchmarking just trying with Archon workflows in general. So I can set up Pi to go directly to the providers like Kimmy when possible or if I need to just like go right through open router like to use Gemini 3.5 flash I I could do that as well. Okay. So yeah we are we are ticking up slowly in our rate limits here. All right. Um where are we? Implement finish now in self-review. Okay. So it is almost done. Implement node logged. Six tool edits failed. retry warnings more than the usual single transient, but it did complete with a produced diff. So Kimmy just fought the edits a bit um consistent with why this cell was slow, right? So yeah, Kimmy is just kind of not the most reliable. All right. All right. Uh does Kimmy have a hook failed call to reset when it fails like Claude? I believe so. But the other thing is a lot of times these failures can come with the Kimmy API just hanging. So it doesn't actually return back to the harness. It just sits there and the harness waits for the Kimmy API to return the tokens. Um, so I think that might be partially why it's slow as well. I don't Yeah, Kimmy is really weird here because you can see like tooledit failed, tool edit fail, tool edit failed, and then like all of a sudden it outputs this really weird text with a ton of new lines. There might be a bug in Archon as well, but I've never seen this with any other model before. Like, if we look at any other logs for anything with Opus, like you you you don't have that anywhere. So, yeah, I don't I don't know what's up with that. All right, but anyway, we are at the self-review step. So, we're almost there. I might actually just ask it to push forward. Uh what if we just end with a pull request and kill the workflow? Like, I don't want to go through all the self-review and wait for that right now. So, I want you to just like go in there, create the poll request, assume the review is good, and then end the workflow, and then go on to the benchmarking, the reviewing. All right? Because just in case there's more failures from Kimmy, I I don't want to mess with that right now. I just want to get to the the results here. All right. Uh, howdy colon chat. Hey, Leon. Good to see you here, man. All right. Uh, had to switch from Kimmy to GLM. Uh, MIMO 2.5 Pro might be good. I have I've not considered that one. Let me go back to open router here. Mimo 2.5. Mimo. This one right here. It's a flagship model. Um, pretty cheap. It's about as cheap as uh Kim K 2.6, right? Let me look at the pricing here because I'm I think I have my pricing right, but there's so many models I have in my head now. Uh, yep. It's about it's about as it's a little bit more expensive for input tokens and a little bit cheaper for output tokens. I guess that's cool. Um, yeah. See, looks decent on the benchmarks. Well, kind of. This is pretty low. Okay. Yeah, I'd be down to try this one as well. It's the cool thing about Open Router is I can just point Pi to Open Router and try any models that I want. Probably won't do that today though, I think, especially because we're getting close to the point where we're going to get our review in. We're getting there. But anyway, let's go back to anti-gravity while we wait for this and we'll see. Okay, so it claud launched to research sub aents to research the workflow engine documentation and examples. Oh, and and then it errored. [laughter] We got an error. Um, our servers are experiencing high traffic right now. Please try again in a minute. Oh boy. Uh, the sub agent hit capacity limits. Okay, so well that that doesn't bode well for anti-gravity. Um, and then if we look at our usage, oh my goodness, [laughter and gasps] what what we have 20% remaining. And and it also even though I switched the model to opus, it didn't use my opus rate limit. Like what is happening? That's wild. I think that must be a bug in anti-gravity. Or maybe like it doesn't actually allow me to There [laughter] I hit my limit. There we go. That's the anti-gravity experience right now. Wow. May maybe anti-gravity doesn't let you switch the model mid-con conversation, but it should definitely tell me if that's the case because I switched to Opus and it still was using my Gemini uh rate limit. And then also, this tells me I'm at 20% still, but then the conversation itself says my quota is reached. Uh that is pretty frustrating. Wow, what a bad experience. Okay, but anyway, I'm still impressed with the UI that Gemini was able to create. That sucks that I hit the limit already, especially considering like, okay, we hit our limit extremely quickly, but at the same time, this is not actually that expensive of a model. Like, we looked on Open Router. It's way cheaper than Sonnet and Opus. It's like haiku pricing, but the rate limit is still awful in anti-gravity. Like, that that doesn't feel right. Anyway, um, which compatibility are am I running Kimmy with, OpenAI or Anthropic? So, I'm using Kimmy through PI. I believe they use the anthropic compatibility. And when I initially was using Kimmy K2.6 with my dark factory, I was actually using it through cloud code. So, I was using the anthropic compatible endpoint that uh the Kimmy API gives us. um that gravity works via anti-gravity. That That's right, Jeffrey. Yep. Um you popped up in my feed, wanted to see what you're up to. You're the goat, Cole. I appreciate Leon. You are too. Thanks to your or thanks to your invaluable contribution to the AI dev community. It is uh my pleasure, my friend. Keep it up, man, with your channel, too. All right. Anti-gravity does that a lot. Yeah. I mean, a lot of what I'm showing you guys here is the general experience that I've heard from others as well. Like, okay, so like quick recap. Anti-gravity, the rate limits suck. Gemini 3.5 Flash is really good for UI and design, but bad for everything else, at least for like the limits that you get. That's that's kind of what it comes down to. And then, for some reason, I wasn't able to actually use my opus or my my clawed part of the rate limit. Oh, no. I guess I did. It went down, but then for some reason, even though I was using that, it's still like, okay, now now all you guys saw that flash before your eyes like it I I have not been prompting it for the last 10 minutes and then like all of a sudden it just boom, it's gone. It's like, wow, that is a that is a face palm right there. So, okay, apparently it did actually start using my Opus rate limit, but there's just this massive delay in this tab right here, which is pretty frustrating because I've never had that with any other provider. Like if I go to claude here, I can refresh and it's, you know, up to date every 10 seconds if I wanted to. Same thing with the Kimmy console. So yeah, not the best experience overall. All right. Well, anyway, let's go back and see where we are at right now. Uh, it's already running. How do you want to handle it? Uh, kill it now. Handmake the PR. Yeah. Yeah. I just want to Well, it actually might have already finished now. Now that I was wait Oh, it did already finish. Okay, hold on. Let me interrupt. Um, okay. Never mind. And by the time I answered your question, the workflow actually finished. So now we have all the pull requests ready for you to benchmark and review. And we can validate that in the UI as well. Let's go back to pull request for the dark factory. Do a refresh. And yep, sure enough, we got 12. There we go. So we have uh three okays, we have uh three KKs, we have three OO's, and then three KO's. [laughter] Looking good. All right. So, we got okays and knockouts and then we and we got some ooze and some and some KKs. All right. So, let's uh let's see what happens with the reviews now. I don't actually know how long the review is going to take. So, maybe Well, I guess I was going to I was about to say we'll keep messing with anti-gravity while we wait, but we can't because I already hit my freaking rate limits. So, that's the exploration. It's kind of funny. This actually I'm really glad I did this because I was thinking potentially for my Saturday live stream uh because I'm I'm live streaming right now every Monday, Thursday, and Saturday at 9:00 a.m. Central time. So, for the live stream this weekend, I was thinking about doing some anti-gravity exploration. But even if I got the $100 a month plan with anti-gravity, I don't think the rate limits would have lasted through a normal live stream, even if I did a shorter like one or one and a half hour live stream. So, I'm glad that I explored this now so that I know that I can't do that for an entire live stream. The limits don't even last enough, which is pretty disappointing. Um, I guess I don't know exactly the rate limits between the plans. So, maybe I should look that up. U Google anti-gravity plans cuz if it is like, you know, 10x the usage or something, maybe. Um, okay. To be honest, I might actually be on the the IND the free plan right now. Maybe. So, I don't I don't really know. I don't even know how to check. Maybe. Let me let me actually try to check that right now. How do I do that? Account. Um, manage your plan. Okay, it says I can manage my plan here, but there's nowhere to actually manage my plan. That is kind of frustrating. Um, how do I tell? All right, I'm going to I'm going to sign in on my browser here and see. Okay, look look at this, guys. What is this? It says my account isn't eligible for Google anti-gravity benefits. What? What? I What? Okay, I'm I'm done. Uh, goodbye, anti-gravity. I I think I'm done with you. That is ridiculous. All right, close. Close. [laughter] [sighs and gasps] All right, that's enough of that. So, all 12 confirmed, one pull request per cell per issue. Kicking off the evaluator across all 12 now. Seven dimension opus scoring per PR. Running them in batches of four specifically to avoid the opus subprocess contention. Actually, just go ahead and run them all right now. They're fast enough where we should be good because I just I just want to see the final scores to get that uh that final takeaway for us for the live stream here. So, if I see Yeah. So, I have four evaluators running right now. Um Okay. Okay. And then after you kick them all off, I want you to give me a status update of the first four that you've ran. And then uh based on the smoke testing that I did yesterday, how long can I expect for each one of these to take? All right. Okay. Um working with Quinn 3.6 27B dense model and archon works pretty well for implementation tasks. That's cool. So, planning and reviews for Claude and Codeex implementation with a smaller Quen model. That's neat. That's very cool. Um, I only use anti-gravity to use my bonus Claude. I use Gemini through the CLI. Yeah, that's that's smart. Yep. U, it will switch to the Claude model eventually. Yeah, it did. It did. Uh, you have to use your core Google account instead of an alias email account. But I am that is my core Gmail account. I don't know. I'm sure it's there could be some user error, but still a very like the using anti-gravity just within the last hour was a frustrating experience even besides the rate limits. All right. Uh Christian said, "Just because a good cup of coffee always helps." Well, thank you very much, Christian, for the support. That means a lot. I'll uh I'll go take my wife out for coffee again. Thank you. All right. Um, yeah. And then Claude is still thinking here. It it started to do that recently. For better or for worse, before we have our first token generated, it always does a lot of thinking or at least like quite a few times in a conversation. It'll just like sit there for a couple minutes thinking. And now, okay, now finally, there we go. So, switching to all at once. First day. Ah, great. I didn't want it to stop. Oh, goodness gracious. Okay. Okay. No, I didn't want you to stop the batched job or Oh, wait. Oh, wait, wait. I understand. I understand. No. Okay. So, I want you to keep the other four running. Just fire the next eight and then then give me the status. Okay. So, it might be too late. It might have actually Oh my goodness. Uh status there. It might have already stopped it. We'll see. Um shoot. Yeah, I think it did. Unless the the evaluator ran end to end. Oh, no, no, no. It actually did already run. Okay. Yeah. So, there we go. So, read, write, structured output. It gave a score and then posted to the PR. Okay. Yeah. Yeah. So, we have let's see um actually the first four already finished. So, you just need to run the last eight. Okay, there we go. So we can actually take a look at the results here. I'm not sure which poll requests exactly number 240. Okay. So let's look at at number 240. So to number 240 is opus for planning and Kimmy for implementation. All right. And so the benchmark okay m maybe maybe my issues were a little too simple here. But also I think this was the simplest issue out of all of them. But it gave it a 67 out of 70. So it solves the issue very directly. Root cause good scope is good. It's a very very um minimal change like right like there's no scope creep. Um subtle correctness tiebreak uses ascending text PK which is consistent across SQL and Python paths. Correctly inverts the score blah blah blah. minor caveat relies on ID being non-null and unique, which it is as a primary key. Um, okay. Well, honestly, this is stupid feedback, so it probably should just be a 10 out of 10 cuz why the hell would an ID uh ever be null or not unique when that's enforced at the schema level, but whatever. Um, okay. So, code quality, uh, new lambda slightly less tur uh, but standard pattern for compound keys. Okay, whatever. Okay, so overall the feedback is like kind of like whatever. I guess I may probably want to like iterate more on how it really judges things and and I should just have run this with more complex issues. But anyway, we'll see once we get the final results of everything. So, I'll have my second brain here. Um, oh, it says everything's already scored. Wait, wait. So you're telling me that every every workflow has already been run? So we we might actually be good. For the record, all 12 are scored. Want me to build a full where? Yeah. Okay. Okay. We might actually be done. I didn't realize that uh it ran already. So this initial um wave like that it ran here. I guess that did actually get through everything. I didn't realize. I didn't know the workflow is that fast. Okay. There were two phases, both fully done. Uh, okay. Good. Yeah. So, I want you to build the scoreboard and I want you to tell me where the Opus premium pays off versus where Kimmy matches it and the cost per quality read. So, give me that uh in summary here. All right. So, we have the we have the final result. We're about to get the final result here, which is good timing for our live stream. Um, I just would say that like if I were to do this again, I would definitely pick more complex issues. So, I'll probably do that maybe for another live stream later. Could be something for this Saturday. We'll see. Or I'll just have to run it off uh off stream and then share later what the results are. I I wanted to be more safe than sorry and pick simpler issues just so that we aren't sitting here for a three and a half hour live stream as things are running through a really complex implementation and evaluation. But, uh yeah, I think this will still be hopefully valuable to see. All right. First, I need Okay, so we'll let it kind of do this and uh while it does that, let's uh go back to the chat here. By the way, Cole, maybe you can build a hot swap API plan switcher to change the subscription when the quotas hit. That's a good idea. There might be something out there already. I'd be surprised if there isn't, but if there's not, like that would be something cool to build. Um, Cole, do you have do you have to have a Google work account to get that gravity deal? Maybe it might be it, John. I'm not really sure. I'll have to look into that. Uh, why not customize your Claude status line? You could have a ton like context usage limits. Claude could pull all the stat into the status line for you. I've experimented with them before. Uh, honestly, I don't really have a reason to not have it right now. I probably should because yeah, I agree. They're useful definitely. Um, nine router. Uh, so from what Chris said, nine router, GitHub, unlimited free AI coding, connect cloud code, codec, cursor, etc. to free via 40 plus providers. Oh, that's interesting. This So maybe not quite I don't know this is quite what Chris is going for, but this looks really cool actually. Huh. That is pretty neat. I'll have to check this out. Well, cool. All right. Well, anyway, let's uh go to the moment of truth here. Um, all right. So, I guess it hasn't produced it yet. All right. So, we're almost there. We're almost there to the point where it's going to tell us like here's where we should use Opus. Here's where we should use Kimmy. And if there's anything that's not perfect here, I'll just have to rerun the benchmark. But I can do the exact same process just with more complicated issues. So, let's see. I guess we got a bit of a long drum roll here as Claude is thinking. Come on, Claude. Come on, [snorts] man. Hurry up. [laughter] Of course, when you want it to be fast, it's super slow. Oh, it's it's unraveling. Stop unraveling. It's almost done thinking. This is crazy. This is like a simple request. I'm surprised it's taking a while. [laughter] Okay, there we go. All right. All right. Here we go. Here's the full read. It's thinking. Well, it's producing tokens. It's going. It's going. Come on. Almost there. Wow. It's a slow day for tokens, I guess. Okay. [laughter] All right. Oh, this is this looks good. This looks good. So the best score it says is u okay this actually looks great. So overall, Opus did the best when it was only Opus, but using Opus for planning and Kimmy for implementation was almost as good. And then it was uh somehow the worst when it was Kimmy for planning and then Opus for implementation in that's actually really interesting. Like maybe the the using the same model, even if it's weaker, is better across the board um compared to using the weaker one for planning. Like I I think what this tells us is it's just it's really really important to have the more powerful model for planning. The clean the clean way to read this is marginal. What does upgrading each node actually buy? So if we um put the planning to opus then we get plus 14 points right. So upgrading the planner to opus is plus 9 on average. Upgrading the impleered opus apparently is minus one. So again, there might it might not be the perfect benchmarking when I should have a more flesh shell process and complicated issues, but I still think that this this is interesting because Kimmy matches opus for implementation as long as we're using opus for the planning. Um, so basically the bottom line is reasoning matters at the plan step, not the build step. Spend Opus on planning and let Kimmy implement. So there you go, ladies and gentlemen. That is the the final say here. And I'll probably run the same exact thing with something more complex just to really be sure. But this this definitely matches my intuition and what other people have experienced as well. And like yeah, that's what I already have for the dark factory. And then yeah, the caveat is uh solve issue is saturated at 10 out of 10. So like yeah, they're maybe a little bit too simple like we like I I mean I keep repeating myself there. You guys know what I'm talking about. Okay, so there we go. that that's the final result for a benchmarking here. It is ideal to have the most powerful model for planning and that allows you to then use something cheaper and faster for really the rest of your workflow if you want. Like the workflow that I'm using for my dark factory right now is is uh Kimmy for explore, opus for planning and then Kimmy again for the implementation and the validation and all the selfcorrection that we wanted to do at the end of the workflow before the pull request. So, pretty cool. Uh, the results make sense. Good planning results in good outcome. The old saying, garbage in, garbage out. That is right. Yeah. Yeah. Cool. Well, then another cool thing I'd want to try is using like an even weaker model for implementation, right? Like how fast and cheap can we make it, assuming we have a really well-crafted spec with Opus. So, could be another thing to try in another live stream. But, yeah, I'm going to go ahead and uh call the live stream here because we got the benchmarking done. This was really fun to build this and, you know, try out all of the different Archon workflows that I built for the sake of the stream here. Uh, pretty neat. So, yeah, hope to see you guys in the next live stream again this Saturday at 9:00 a.m. Central time. If you're in the Dynamus community, I hope to see you guys in the Dynamus community. And, uh, yeah, go go go try anti-gravity for the two seconds that you get to because it is kind of cool to see the front ends that it can build. And, of course, try Archon. Just go to to archon.diy DIY and uh we got the GitHub repo here with instructions to install it super easily. You get to package up any AI coding process, any agentic process at all as an Archon workflow makes it super easy to run them in parallel and make them more reliable. Um so yeah, with that, hope you guys have a fantastic rest of your day and I will catch you all in the Dynamis community or the next live stream. Take it easy everyone.