Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Opus 5.5's effort levels produce dramatically different results for the same complex task, with Extra delivering the best balance of quality, cost, and runtime. Max was the most expensive and slowest but not the best, while Ultra Code underperformed due to a possible bug. The sweet spot for demanding agentic work is Extra, not the highest setting.
Key points
Opus 5.5 was tested on six effort levels with the same prompt to build a 3D conference world.
Extra effort produced the best output, balancing quality, cost, and runtime.
Max effort cost $50.38 and ran 2h28m, but was not the best result.
Ultra Code cost $18.69 and ran 1h35m, but felt buggy and lacked dynamic workflows.
None of the runs used subagents, and only High asked a single question.
Tools mentioned
Techniques
- Effort level tuning
- Slash goal prompting
- 3D world generation from video assets
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Opus 5.5, Opus 5.5, Opus 5.5, Opus 5.5, Opus 5.5. This model is literally everywhere and for very good reason. It's smart. It's cheap. It has amazing taste. It is an incredible AI model. But
with every AI model, you have a choice of effort. Whether that be low, medium, high, extra, max, or ultra code. So in this video, I gave Opus 5.5 the same exact prompt, and I ran it on every
single effort level. And we're going to be comparing the results. We will look at the quality of all of the different actual outputs, but then we're also going to look at how long each one of
these ran, how much it cost us if it was API billing, the total tokens, how many checks they ran, and how many questions they actually asked me throughout the process. And the results that we got are
not what I expected at all. So, I can't wait to share this with you guys. Let's not waste any time and just get straight into this one. Okay, so let's jump right into this one. I want to start off just
by showing you guys the actual prompt that we used that we gave to every single one of these different agents. I'm going to go into the files over here and we're going to open up this prompt
markdown file and I'll show you what we got. So, here is the / goal that I fed in. I said, "You need to create me a 3D world that is a realistic tech conference that I can walk around in
thirdp person view. You will look at this folder, which is my event recording assets from AIS Live." In this folder is a frame IO folder of 105 GB of video recordings. This was a completely
virtual event. Everything was recorded and all of the recordings are right here. I said, "Your goal is to take this event and turn it into a 3D explorable world that makes me feel like I actually
went to a real in-person conference with different rooms, different tracks, different stages, blah blah blah. Feel free to use key.ai if you need to generate any images or videos. And you
can also use anything else inside of my Herk 2 project, which is like my AI operating system." I said, "You'll be judged on creativity, design, physics, and the overall feel as I explore the 3D
world that you built." And that was basically the end of the instructions. So, as you can see on this lefth hand side, I ran this through all of the different effort levels. Let's start off
with low and work our way up to ultra code. All right, so here we have the result from low. Let's open this up and take a look. So we have AIS Live AI services summit in person finally and we
are able to click around. First of all, this doesn't feel very branded. Like this isn't the AIS Live logo. It's not even our colors. So I don't love that. But let's get in here. Okay, this is way
too bright. Um, we have a map in the top right. We have a city back here. I can't tell what city this is. Okay, this is Chicago, which is kind of cool because, you know, I live in Chicago. But
anyways, on the top right we can see a map. We have a lobby. We have an expo hall. We have VIP lounge. Main stage. Also, the map shows where every other person is. And that's syncing live. So,
we can see registration. We can see day one. Hyper agent keynote live debrief. Cool. So, it actually does know the agenda. And then there's day two. So, it found that. That's good. We have these
little balls in here that I can hopefully kick around. Okay. The fix. Oh, look at this. If I go over here, all the people just disappear. Very bad. Very bad. Okay. So, let's see. Can I
sprint? Okay, I can move a little faster. I'll first go over here. There's some swag. Um, AI+ certified glido. Okay, so it's got the actual booths that we had in the virtual event. We had
booths. So, that's pretty cool. Little place to take pictures. Room C. Right now, we have Tangy Frederick doing a workshop. Okay, but this isn't a video. As you can see, it's just an image.
She's not moving. So, that's just an image. These people are disappearing. They must be ghosts. Let's go over here to room A. We have Liberty White. Okay, very cool. Your first 30 days in
automation. Once again, it's just a still image and the people are glitchy, so not very good here. I'm going to go over to the main stage and see what we've got going. Okay, cool. So, we have
a main looking stage. The people are glitching really bad. That's not very good at all. Our video is actually moving. Like, I saw my face here and I saw Devon's, but now they're gone. So,
I'm not sure what happened. Okay, it's it looks like it's more of a slideshow. Nothing is actually being played still. Anyways, let's come in here. We have more booths. We have Hyper Agent. We
have Claude Code. We have more swag. Room B's Dave Evelar. I'm assuming it's the same exact thing. We've got coffee. And then I'm assuming VIP lounge, VIP access only. That's pretty cool. But
there's really nothing going on right here. That screen's way too bright. Okay, so I think you guys understand the vibe that we're getting here from Opus 5.5 on low effort. And here's where
things get interesting. How much do you guys think that this ran for? How long? This one ran for 16 minutes and 43 seconds. How much do you think this cost? $3.91. If this was API billing,
I'm obviously using my subscription here, but we're just going to calculate this in API billing. Total tokens was 191,000. It did 22 checks. So the verification 22 times it opened up the
browser and ran different sorts of checks. So 22 categories of checks. And how many questions did it ask me? It asked me a total of zero questions throughout this /go prompt. Okay. So
let's open up medium effort and see what we've got. Okay, here we go. Medium effort. We've got Nate Herk. We've got my badge. This is AI's live branded. So this is already looking a little bit
better. This feels like our color schemes. It used our brand guidelines. Day one build. Day two earn. VIP. Cool. Okay. I'm going to enter the venue. Okay. Wow. So similar sort of vibe. a
city in the background. This doesn't look like Chicago, though, does it? No, this looks like a Honestly, this looks like a madeup city. Either way, it's funny that they decided to do that.
Let's see if I can move a little faster. Oh, wow. The people are interacting with me. Watch. If I go up to this guy, he just like raised his arm. Well, now he doesn't want anything to do with me at
all. But all of these little robots here must be making decisions. I'm not sure if they're using Jev. They're definitely not. I didn't tell it to. Actually, my Jev key is in the back. I don't know.
Maybe it did use it. Anyways, we can see here we have workshop room C, agents lab. Nice. So, this one is actually being run. You can see that is a real video being played by Tangy. Everyone
here is actually like working on a laptop. They're not glitching. This is pretty cool. Also, my badge is on my chest, which is pretty cool. I can come this way. We have a map up in the top
right, as you guys can see. But I can come this way. We have an expo hall. This is where we have the Glido booth. And this is actually playing Yeah, it's playing the video of us talking about
Glido. It's playing the video of Ed and I talking about our certification program. We've got the AIS Plus logo back here, which is kind of in a weird spot. This is a speaker slides and
takeaways. So, wow. This is all the resources that we gave out after the event. They're all sitting right there as well. We can see we've got a community spotlight. So, this is Aiden
talking about his deal that he landed and it's playing live. These people are watching. They're pretty engaged. We've got hyper agent. That was that's what I meant if you guys saw those people like
raising their arms saying hi. That that was kind of funny. Look, there he goes again. Anyways, okay. Where am I now? Now I'm in the main lobby. We've got a coffee bar. We've got a big logo, which
is the real logo. It's too bright, but we have the logo. We can see if we can come in here to the foundation track. We've got Sabrina Romanov and Liberty White. So, different trainings right
there. We can come into this room. This is the advanced track. So, what's going on here? We've got Dave Eblar and Salmon talking about some different stuff in there. And now, let's go take a look at
the main stage. Oh, wait. There's a video of me up there. Is that like a VIP section? Yeah, we'll go check that out in a sec. But anyways, this is the main stage. This looks very, very nice. We've
got a big stage. We've got like four people sitting up here. We've got the the three screens of Alex up there with Hyper Agent. Am I allowed to get on stage? Oh, it lets me up on stage. Okay,
that's pretty nice. All right, guys. Let's take a selfie. Let me get everyone in the background. Get in here. Anyways, that is pretty pretty cool. Not every seat is filled, though, so we need to
work on that. But anyways, I'm going to run back over and see what that VIP section was. Okay, VIP lounge. I feel like this is like a we're at the airport or something. Okay, cool. So, now we've
got the VIP sessions up here. VIP Q&A with Nate. Live video playback right here. Very, very cool. And we've got like a bar or something. Awesome. I would say that this is a pretty good
output. Now, as far as the stats here, this one took 1 hour and 13 minutes to run. It would have cost us $1244. It used 419,000 tokens and it did 23 checks and it asked us a grand total of
zero questions once again. All right, let's move on to high. This was already a pretty decent output and Enthropic themselves in their how to prompt Opus 5.5 video or sorry, not video article,
they said to just start on medium and scale it up or down if you need. So, that was a medium output. Let's move over to high and see what we got. Real quick, guys, I need to take a second to
tell you about the sponsor of today's video, Hostinger. So, both of these models just built me a working version of the same thing. And now I'm right where I always end up with something
finished sitting on my laptop and no quick way to get it online. And that's the gap that Hostinger's connector closes. It's a free extension for your editor that pulls your Hostinger account
into whatever you're already coding in. So, VS Code, Cursor, Cloud Code, Codeex, you name it. You sign in once with a single click, and from there, your agent can deploy the site, point a domain at
it, set the DNS records, and check on your VPS without you ever having to leave the editor. So, whichever one of these you end up liking better, the thing it built is a couple minutes away
from a real URL on managed hosting. Connector is free on every hosting or plan. So, if you still need the hosting underneath it, grab the unlimited plan with the link in the description and use
code Nate Herk for 10% off. That also comes with a free domain and business email for the year. and it's still the cheapest way that I found to get something you built onto a real URL. So,
let's get back to the video. Okay, once again, very, very branded. This is even better of a loading screen than the previous one. We've got this nice little effect in the background. We've got the
logo. We'll enter the venue. Okay, here we are. This looks pretty nice. We start outside and you can see that we have these flags for all the speakers. Wyatt, Casper, Alex, Ed, Aiden, Sabrina,
Liberty. This is pretty cool. We got live blocks here. It took this picture from me. Your host, Nate Herk. John Dave. Nate Herk. There we go. Okay, the doors. Awesome. They're automatic
sliding glass doors. I love that. We can see VIP check-in. We can see GA. We can come over here and we can check out the expo with different booths, community spotlight. You can also see that in the
top left I have a passport. So, it's like it will be showing how many of the places I've visited. All of these are real playback. We've got a resource wall with all of the different speakers.
They've also got a networking session over here. So, I'm going to come over real quick and see what that's all about. So, we've got the AIS cold brew bar. We've got different community
members that were spotlighted or highlighted. We've got the VIP wing. Wait, what? Pick up a wristband. Oh, I have to actually go get the wristband. Okay, let me check in real quick.
Wristband's already on. Wait, what? Okay. Oh, okay. Now the doors have opened for me. Cool. I can come into here. Oh, that just goes to the main stage. VIP lounge. This is a Q&A going
on, it looks like. Very cool. I mean, I'm very impressed by how it's able to do this. Wow. Okay, so this is really good. What we did is we had VIP breakout rooms with different people. You can see
that there's different rooms, different members of the AIS team going into stuff. This is really cool. This is very cool. That's a much better VIP experience than what was shown in the
first piece. Okay, VIP afterparty. Look at this. We've got a dance floor. We've got all these elements in here. We have the actual VIP afterparty playback right here. And there's a DJ booth. That is so
funny. There's a bit of a bug right here, a glitch right there. But this is awesome. Oh, cool. So, when I'm in here in the main stage, we get closed captions. You can see right here in the
bottom of my screen, we're getting these closed captions of Wyatt actually talking up here. We've got lights. We've got the panel. Very cool. Nice main stage. I'm going to go over here. We can
go to the foundation, advanced, and the enterprise tracks over here. So, let's see. We have anatomy of the three real deals. We've got hyper agent. We've got the evals with Nate and Ed in here.
We've got Dave going on in the advanced stuff. This is really nice. I mean, obviously each each of these outputs so far, low was okay, medium was better, high has been even better. Let's see if
that trend continues. And let's go ahead and see what this cost us. So, high ran for 1 hour and 7 minutes, so a little bit quicker than medium. It would have cost us $16.31.
It used half a million tokens, 59,000. It did 22 checks. And it also asked us, well, actually, no, I was wrong. This one asked me one question. And spoiler, it was the only one that asked us a
question throughout all of this. So, let's see. We've got three left. Extra, max, and ultra code. Let me pull open extra and we'll see what we got. Okay, so this one looks pretty good. I would
honestly say so far the loading screen high was the best, the one that we just saw. But anyways, let's enter AIS Live. Whoa. Okay, so we have like little sound bites. I can chat with people. The tool
war panel settled a few debates for me. Nice. Good perspective there. We're outside again. We have these different banners, although they're all the same. they don't say like different people's
names. So, big AIS live logo. Workshop wing is over here. And let's go through the sliding glass doors and see what we've got. So, we've got the AIS cafe. The map is in the bottom right. And it's
not too descriptive. I like how the other maps have told us what like where things were. But this one looks very professional. We can see here's the main stage. Let's hop in here real quick.
They've all got these balls flying around, which I think is pretty funny. The AIS beach balls. We have me up there speaking. I think I was introing one of the days. Let's keep on moving here to
the workshop room on this left side. Okay, so here we've got the Hyper Agent Theater. We've got this sponsored session in here by Hyper Agent, but it's showing us what's coming up in here as
well. It's really funny that we can chat to people. Salmon built a voice sales rep live. The price at right room was packed. Did you grab the VIP companion guide? That's so funny. We have the
advanced track in here once again. We've got live playback. Is that live playback? Oh, okay. It started once I got in, but I can take a seat. Oh my. I can watch this. I can stand up. I want
to sit in the front. That's pretty cool. That's very nice. I like that. And you know what I noticed so far? The actual character that I'm playing kind of looks like me. I think it modeled it off of my
thumbnail pictures or something. Anyways, we have Sabrina in here. Room host here. Grab any open seat. Okay, cool. And I really like the sitting functionality. It's kind of funny. Like
we could actually attend this workshop and participate. Anyways, it's showing us the speakers. It's showing us the agenda. There's a little red carpet here to take some pictures. We can strike a
pose. Oh, wow. That's pretty cool. Resource library. Get AIS Plus certified. Glido, Hyper Agent, AIS Plus. Three real deals. Awesome. I mean, I would definitely say so far each one is
getting better. And we didn't even check out the VIP section yet. The VIP lounge. Let's go up here real quick. Hopefully, I can get in. Nice. We've got tooling reset. These are the different rooms we
could go in. So, once again, I could grab the worksheet and I could try to understand how to price my stuff. This is so cool. This is definitely better than the previous one where we kind of
just like looked at stuff. Awesome. I can get behind the bar. I can come in here. This is very nice. Okay, so as far as statistics go, this one ran for an hour and a half. It cost 25.92 bucks. I
don't know why I say point. $25.92. It was 7323,000 tokens and 34 checks. So, it had the most checks so far by far. And it asked us zero questions. I can't wait to see what we got here from
Max and Ultra Code. Okay, here is Max. Loading screen is boring, but it's on brand and it has our logo. So, that was good. I like that. We will go ahead and enter AIS Live. Oh, nice little
animation here that brings us in. Once again, the character looks like me. They've all looked like me. I mean, sort of. We have city in the background. This looks like Chicago. Like I mentioned
earlier, a lot of these are playing sounds and I'm not including that because it would be very distracting for you guys to try to listen to what's going on as well as me speaking. So,
there is like some slight music in all these. I I hate how this is walking. This walking is really really bad. I mean, the walking Yeah, I don't like this at all. So, that's not great. But
besides that, let's go in and explore. Notice these shadows when I walk in. They like really switch. I'm not sure why that is. But anyways, we can chat with people in here, too. Hyper agent
booth is right where you walk into the expo. All good. Okay, cool. I can keep clicking E to make them change what they say. We've got the speakers right here. That looks pretty good. Although, we
definitely had everyone's profile picture, so I'm not sure why that's not included. There we see people taking pictures right here. I love that. And it saves the little picture. Okay, the map
also isn't super like doesn't give me a great explanation of what's going on, but I like these booths. These are cool. I think these booths are the best that I've seen so far. Like, they just look
nice. They've got reps. There's nice like slides behind them. Yeah, these booths are cool. Okay, we have a little featured theater going on over here. It's Casper. Although, why is this not
playing? I feel like this should be playing, right? Like in the other ones, they were always playing. We can talk to some other people over here. Coffeey's free. Blah blah blah. Amy Simpson, Matt
Wolf. Nice. Okay, this is just the networking area that we're in right now, but we can see in the top right, we can also see what's live on the main stage. Right now, it is a tool war panel. So,
let's head in here. We've got Devon, Cole, Dave, and Russ chatting in here. We've got sort of AV, some light stuff going on back here. Switch the main stage to what actually matters right
now. So, I can switch the topic. Cool. So, I just switched to me and Matt. We can go to anatomy of three real deals. That's pretty cool. The stage looks good. We've got a nice little panel up
here. Can I get on the stage? Nice. Nice. Well, I can't go too far, actually. All right, everyone. Let me take the selfie. Everyone get in this. I can also take a seat in this audience
over here and just enjoy the session. Very cool. Very cool. Okay, let's go over here. I see an upstairs section. So, it's funny how they all choose to put the VIP section upstairs. I mean, I
don't hate it. Oh my gosh, they have an escalator. No way. I'm going to chat with this guy on the escalator. Glenn has 15 years of agency experience. His land and expand stuff was gold. Great
work, Glenn. Cool. So, I'm going to I can't even get past this guy, though. Oh, I had to jump over him. Okay. VIP level badge required. Oh my gosh. You have to Are you kidding me? I have to go
get my badge. Okay, cool. Now it shows that I'm an actual VIP and I can go up here to the VIP section. We have nice little working sessions back there which we can hop in. I wonder if it'll let me
like sit down in here. I can just chat. Can I participate? It's not letting me sit and participate. That's okay. We've got the pricing war room. Oh, this might be the afterparty. Let's go see what's
going on back here. Or maybe I just have to enter from over here. Okay, that's weird. I just had to enter from over here. This afterparty is not as cool as the other one. But anyways, let's go see
what's going on back here in the workshops. Okay, that was not good. Look at this. You can see everything and I just glitched and now boom. So that's not good. I would say overall I mean you
guys are getting the vibe of how this works. But I would say that the one before which was I believe high. I liked that one better. I can't sit in these chairs either. Yeah. So I don't like the
walking in this one and I don't like the vibe as much and there's some bugs. So so far if we want to go to our list, I like extra. Extra was the one that I liked the most so far. But anyways, this
one was max. This one was max right here. So, let's see how long this ran for. 2 hours and 28 minutes. So, it ran for a long time. $50.38. 1.18 million tokens. So, it actually hit
a compaction and had to autoco compact. And then it did 51 checks. Did it really though? Cuz there was a lot of bugs in there. And anyways, this one asked us zero question. So, so far each time
pretty much has gotten more expensive and took longer besides right here. But these basically took like very similar amount of time. But every time it's used more tokens because they've been
thinking more and then you know those tokens are going to cost more. But anyways, let's move on to the final one which is Ultra Code. So we would really hope that this one is the best one. So
let's hop into this local host and see what we've got. Okay, cool. Look at this badge. It's a nice badge. Host all access. We've got a nice little visual right here. We'll go ahead and enter AIS
Live. Cool. Okay, welcome in, Nate. I like the walking. It feels realistic. I like the logo, although it's missing the little red dot that makes it look like live. The map in the top right is
labeled a little bit better so I can see what's going on. I'm going to come in here and collect my VIP wristband real quick. Okay, nice. It's also telling me what to do. So, in the top left, it says
scan in at the VIP gate at the lobby east wall. So, I believe east would be this way, right? Never eat soggy waffles. Yeah, VIP wings. Scan wristband. Okay, cool. Now, I'm in the
VIP section. I can see these different rooms. The tooling reset, live video is being played. I'm able to see the closed captions right there of what's being talked about. It's also playing the
sounds, but I'm just not playing the audio for you guys because I don't want to overwhelm. Once again, this one's working with Cody and Mustafa in there. That's awesome. Live video. The video
doesn't play until you walk in, though. So, honestly, I think that's a good call. As soon as I walk in, though, the video starts. Nice. Nice touch. All these rooms. Awesome. Yeah. I mean, this
feels very premium. Here's a pricing war room. Let's hop in here. Me and John in there. And then we have nice the afterparty. This afterparty is not as busy yet. And we have more beach balls
for some reason, but this afterparty is cool. I mean, it gives us the good vibe. And there's the playback right here of our afterparty Q&A. This is all live as well. Sweet. Okay, let's head into the
main stage. It's also prompting me to grab an aisle seat at the main stage, which is straight through the expo. So, actually, let's go through the expo first.
What are you building? There's a lot of people talking about different things over here. Wow. There's also like a little a basketball thing. Can I throw it? I can. Do I have to look up to throw
it up? Okay. Well, not not great. But anyways, we have an AIS Plus booth. We have the Glido booth. Is that playing live? Yeah, that definitely is playing live. Nice. We have the Hyper Agent
booth. We have other stuff over here. Okay, cool. I'm going to go into the main stage and see if we can grab an aisle seat. As soon as we walk in, everything starts playing. We've got a
very nice stage vibe. How do I grab an Alisa seat, though? There we go. I had to find the right one. Grab the aisle seat. There's no one on the stage, which is weird. I liked when they had people
on the stage in the past versions. Let's take a quick selfie. Anyways, we got me and Pat up there. Pat is all dressed up like a construction worker. As you can see, we were doing a little mock
discovery call in that example. I'm going to come back through the expo and we're going to go out here to the workshop wing and just check out if these rooms basically are the exact same
as they should. Now, I can't actually chat with people. I used to be able to in the previous versions chat with people, which I thought was a really nice touch. And we have workshop a
foundation track. Can I sit down? I can't sit down. I I don't know. We do have Liberty actually talking right now and she is speaking and we can hear it. So, that's nice. But it's not letting me
sit down. And look at this. I'm getting pretty glitchy right here. That That was glitching the way I was walking. It like wasn't letting me walk. That's not good. Same thing. We got this advanced track
in there. Awesome. So, overall, they have a very similar vibe. I I will say I'm impressed by the way they were able to tell a story out of what we were doing. Speaker takeaways library. Okay,
this is cool. I don't think we saw this from different places, but these are like the resources and showing some cool stuff. Oh, wow. I can actually open all of these things. And we can take
pictures right here as well. Nice. Take a photo. I can save this as well. Like I could actually download this. And now we have this photo that we just took at this AIS live event. All right. Well, I
think it is time for me to make some conclusions. But first, let us see what this run cost us. This took 1 hour and 35 minutes. So, it was much faster than Max. It only took $18.69.
Wow. So, it was a little bit more expensive than high, cheaper than extra, and much cheaper than max. It also ran 66,000 tokens and 42 checks with zero questions. Now, one other thing that's
interesting to note is that all of these runs, none of them used a sub agent. I looked through and I made sure none of them used sub aents. They didn't want to delegate any work around, which was
interesting. So, these tokens are what was reflected inside of that session. Obviously, like I said, this one went over, you know, 950K, so or whatever the compaction window is. I never usually
let it get that high, but because this was a slash goal and I wasn't involved, this one had to compact, but the rest of them just ran in this one session. And these are the overall stats. And also,
real quick on the Ultra Code stuff, guys, I don't know if you've been noticing this, but when I've been running Ultra Code lately, it's just felt weird. It's felt a little buggy. I
a couple times I've ran it and been like, is it even running Ultra Code? It did quite a few more checks than these other ones, but for some reason it just didn't feel right because essentially
what Ultra Code is is it's extra effort and then it's just like using more dynamic workflows in order to do things. And so through all my digging on the session logs and even when I was
watching this thing build in Ultra Code, it wasn't spinning up any of these dynamic workflows. And I tried this a few times. So I don't know if that's a bug right now in the cloud code harness
or if it's just with Opus 5.5, it's a little bit worse with Ultra Code right now or something. But either way, these are the actual bulk effort levels. And this all seems to make a lot of sense
when you kind of look through how they progress. So, let's take a look at this. Max cost versus low. We had 12.9x on the cheapest run compared to the most expensive run, which I believe was $3.98
to $50.38. So, that was low and max. As far as max checks versus low, we had a 2.3x multiple. The total across all six was 127 bucks and Ultra Code was $18.69. Let's look at speed versus cost here.
So, let me zoom out a little bit so we can see this whole thing. So, on the X-axis we have runtime. In the Y-axis we have cost. So, I feel like you know what the best would be is in the bottom left,
but not really. So, anyways, you can see that low was cheap and fast, max was slow and expensive. But this sort of chart typically makes sense. As you increase in effort, it's going to cost
more and it's going to run a little bit longer. That makes sense. Now, let's see growth relative to low. So we have runtime in blue, API cost in orange, tokens in green, and checks in yellowish
gold mustard. And by the way, the reason why Ultra Code shows up like this is because it actually uses extra effort level. It's just incentivized and it uses more like dynamic workflows and
things like that, which is why this, you know, this makes sense because it basically was using extra under the hood. That's also why Claude tagged it here in orange. Anyways, if we continue
back down here, this typically makes sense, right? As the effort level increases, once again, these metrics are going to increase. runtime, API cost, tokens, and checks. Same thing here with
runtime. It's just kind of giving us more individual line charts now for each of these different metrics like API cost, checks, total tokens, cost per check, and all the numbers in one spot.
So, pretty cool data. I will say nothing here is too shocking. What was more shocking to me was those results. My top two contenders were high, which is this one, and extra, which is this one. So, I
need to go back in here and just remember what I thought about them. I really liked this feel. This one also just feels the smoothest. The physics were nice. The sliding glass door was
nice. I didn't really notice many bugs in this one, which is what I really liked. I can't remember if this one was one where Oh, I couldn't talk to people though. I could just walk right through
them. I couldn't sit in this one either. Here is another little visual thing where I basically just walk right through this wall. So, don't love that. But I think was this the one where I
could sit in these sessions? No. Okay. So, I don't think this was my winner then. This is extra high. I think this is the winner. Yeah, I think this is the one that I liked the most. I loved this
whole vibe. I love that I could chat to people. This was definitely the one where we could come in here and we could sit wherever we wanted. Take a seat, stand up. I could read these three deals
and I could chat with them. I also realized that there was little sections to do mock discovery calls in here, too. We have some swag and totes, which is real physics. I like that. This was the
one where we could sit down everywhere. Yeah, I really, really liked this one. Although I think the one downside about this one was that it didn't have like a VIP afterparty because I think this was
the lounge and I think this was the only piece of the VIP section which was these being the different rooms that you could come in and sit in. But besides that, it didn't have a great VIP experience
compared to some of the other ones that we saw. So my winner here is definitely going to be Extra. Extra did a phenomenal job. It was about half the runtime and half the cost of Max. So
Max, I think was just way too much for not enough good. I think that highs was decent. It could have with maybe one or two more prompts gotten to where I really liked it. But for a slash goal,
extra delivered an amazing result here. I didn't love medium and for a lot of my knowledge work and stuff I'm doing, medium works just fine. But for this task specifically, I needed a lot of
reasoning. It had to go through tons of stuff. It had to go through tons of videos. It had to find a lot of things inside of my projects. It had to create an experience and tell a story out of
everything. I think Extra did a phenomenal job. In general though, I liked a lot of these outputs, but Extra is the one that I'd want to start from right now. If I wanted to really make
that like a super super polished and cool app and world, I would start with Extra's output and probably keep iterating with Extra. So, anyways guys, that was the experiment. I hope that you
found that insightful. I hope that you learned something new and if you did, please give it a like. It helps me out a ton. And as always, I appreciate you guys making it to the end of the video
and I'll see you on the next one. Thanks everyone.