Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
In a 12-use-case head-to-head, Opus 5.5 beat GPT-6 Astra 8-4, but Astra was 45% faster and 38% cheaper overall. The real difference is style: Opus delivers more creative, tasteful outputs on vague goals, while Astra is a more efficient, obedient worker that wants a specific spec. Both are strong on routine tasks, so the choice comes down to how you prompt them.
Key points
Opus 5.5 won 8 of the 12 use cases against GPT-6 Astra based on subjective quality.
Astra was 45% faster and 38% cheaper overall despite being 2.5x more expensive per API token.
Both models scored near-perfect on a coding challenge, but Astra finished in a fifth of the time at half the cost.
The presenter found Claude/Opus models more creative and tasteful, while GPT models are more pragmatic and require specific prompts.
The presenter noticed Astra's Canva drawing output degraded compared to earlier versions.
Tools mentioned
Techniques
- slash goal prompting
- agent loop execution
- browser use automation
- isolating agents in separate work trees
- iterative feedback prompting
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
So, I spent all day playing around with Opus 5.5 and GBD6 Astra, and I just put them head-to-head on 12 real use cases. I'm talking things like building websites or editing videos and Instagram
reels, planning trip itineraries, creating real professional branded deliverables like slide decks and spreadsheets and analytics and P&Ls, also creating real worlds like 3D
playable games, or 3D experiences to help you actually learn new concepts. We'll also look into browser use and things like creating carousels. And on all 12 of these experiments, I will show
you how long it took each model to run and how much it cost each time. So, I don't want to waste any time. Let's just get straight into this one. Okay, so last night I made this type of
comparison video between Opus 5.5 and GBD6 Soul. Today, we're doing it with GBD6 Astra. In this video, I'm using both of these models, Opus and Astra on high effort. And I'm using these inside
of my subscriptions, but I'm going to tell you how much this would have cost as if it was API billing, just so we can get a better I don't know, we can contextualize it a little bit better
because I don't know if you're on a 20 bucks a month plan or 200. So this is the API billing for Opus 5.5 and GBD6 Astra. What's really interesting is that Astra is 2.5 times more expensive than
Opus 5.5. So is it going to perform 2.5 times better than Opus 5.5? That's what we're going to see. So in all these experiments, I'm basically shooting off the exact same prompts to both models in
their respective harnesses and then we're going to judge the results. Now, as far as my video yesterday, I did make a mistake and I had two of those experiments. The agents kind of like
worked together, so I made sure that didn't happen this time. What I obviously should have been doing there is what I normally do when I run these test videos is I put them in different
work trees so they literally can't touch. So, sorry about that guys. I apologize. I hope the video was still insightful. Yesterday, I had a million things going on and I was honestly
really tired at the point of filming, but I wanted to get it out as soon as possible. So, sorry about that. But, let's get into today's comparison. So, if you watched yesterday's video, then
you're going to recognize some of these experiments with Opus 5.5, but I decided to also run those exact same ones with Astra because I thought they were really good use cases and things that I would
actually do with these models. So this one is Opus 5.5 with perk form. Same prompt, same brand guidelines as you can see. I like how it put the keys and the can on different layers than the
background. As we scroll, we get some text coming in. We come down here to an AI generated image that it made. Coffee in one hand, protein in the other. We can see that we get the scroll effect of
the text right here. It's very simple. I don't like how this ore is so tiny. That definitely could have been bigger. I really like this effect where you're pulling the tab and you're getting this
basically animated. I think it created a video from this, but it's really trying to piece this all together and tell a story here. I like the images here. I like the drop shadow. I like how the
text is coming in as we scroll. I really like this animation where these are sort of like rotating into, you know, the front view. And then that's basically FAQ last animation right here. And then
that's the end of this page. And here is Astra's attempt at this same site. You can see similar style, similar feel, but obviously a little bit different. I like the big text in the back that says one.
I think that was a nice touch. We have some layering going on here as well. We'll continue to scroll down. We have text sort of coming in, but not the exact same way that it happens in the
Opus version. I like how these two things sort of merge into one. I think this could be cleaned up. It's not super smooth. I like the vision, but it could have definitely been done better. Now, I
really like this view. We're getting these different flavors here and we can switch between this. This feels very I don't know. This feels premium. It feels very clean. It doesn't feel AI generated
here. And I like how the colors changing in the background over here. I think this is really nice. We also have a large ice cube and some coffee coming in right here. FAQ. And then we come down
to basically the end of the site. Now, to be honest, I think overall I enjoyed the experience more on the Opus site. It just felt a little bit more premium to me. I also just kind of prefer the dark
mode typically, and this was kind of like it felt more like this one consistent bright throughout. I mean, that's obviously not true, but I liked the overall vibe of this one better,
especially the hero section. The hero section here is like the first impression, and this looks a little bit cheaper. So, as far as um quality, I'm going to give this one to Opus. But if
we take a look at the actual pricing, here's what we got. So, Opus was 40 minutes, whereas Astra was 32 minutes, and Opus was $18 bucks.32 and Astra was $11.33.
So, it's very interesting to me, and I want to see if this is consistent throughout. Astra is a 2.5 times more expensive model, but in this case, it was faster and cheaper. Ultimately, I'm
still going to give this round to Opus 5.5. I think when it comes to creativity and design and some of this like visual judgment, I think that Opus has Astrobe in most cases, but we will see if that
holds true throughout the rest of the experiment. So, so far, Opus is up one to zero. And real quick, guys, both of the sites that you just saw, as well as a ton of other sites that I actually
have live, used this completely free skill that you guys can go grab right now. The link for this will be down in the description, and it has all of these skills and prompts and ideas about
building good websites inside of it. So, go grab that for free. Let's get back to the video. So, moving on here to number two. You guys know this is a use case that I love. I think it's a really good
benchmark. It shows what the models are really capable of. I gave them both this folder that has 105 gigabytes of recordings from our last event, AIS Live. And I basically said, "Hey, I want
you to go through this, create me a 30 second sizzle reel that we can use to market our next event in October, October 17th and 18th. And I wanted to edit all this with hyperframes." So,
let's go ahead and give both of these videos a watch. We'll do Opus 5.5 first and then Astra. Hello. Hello, AIS live. My >> Oh my god, I'm so excited for this.
>> Let's Let's get some energy going in here. >> It's been a great two days. I think this is so so good. You guys saw this exact same version yesterday if you
watched my comparison yesterday. But I think it's incredible. I mean, I think one thing that we don't even call out enough is how good it was at sinking to the beats of the music. This scene, for
example, it was going bump and everything was coming in with the music. It was super energetic. There was a lot of layering and depth. Like it wasn't just, hey, here's a recording and here's
some text. It was like, "Hey, let's put a recording in the background and then overlay another recording on top with the drop shadow and we have like an overlay and we have the opacity turned
down." Like, I think that this was a really, really well done video. And here we have Ash's version. Let's go ahead and give it a watch. Let's get you guys paid. Right
now, we're up to $32,000 with them. one workspace, my entire agency, a small folder on my desktop. Okay, so not nearly as good. The beginning felt awkward. It was very
quiet. The music was very like slow and it also didn't feel like it came in until maybe 5 seconds in. I think Asha does a really good job with vision and screenshotting things and putting in
B-roll. That makes sense, but the energy was not there. The creativity was not there at all. Like I said, in general, it feels to me like Opus and Claude models are just way more creative and
have better, I don't know, taste in a lot of ways. And yeah, let's take a look at the pricing of this one. Opus ran for 31 minutes and cost 10 bucks, whereas Astra ran for 39 minutes and $21, almost
22. So I think across the board, Opus wins here. It was faster, cheaper, and the output was better. So, we've got Opus 5.5 coming in so far at up 2 and 0 or 2 to zero against Astra. Let's move
on to number three. So, for number three, I gave them this recording of myself and I told them to make an Instagram reel for me, which is using hyperframes. And there's also a skill in
my student kit that basically helps them um create these Instagram reels. So, they're given pretty much the same stuff, but the outputs are very different. So, let's take a look at GBD6
Astra. Stop prompting Claude. Andre Karpathy thinks there's a much better way to work with AI. And his method has three layers. Layer one is the spec. Instead of giving Claude a task and
hoping it understands you, work with it to create a detailed spec first. Have Claude interview you about what you're actually trying to achieve, then break the work into smaller checkpoints before
it starts. Layer two is the verifier. Before Claude does anything, define exactly what a good result looks like. Then give it ways to actually check its own work, whether that's another AI
model or real data or tests that it can run on its own. And layer three is the environment. Build a workspace where Claude already has your instructions, your knowledge, your skills, and your
rules every single time you use it. So instead of becoming better at writing prompts, you're building an entire system around Claude that makes it better every time you use it. All right,
so that was Ashstro. Let's take a look at Opus' output. Stop prompting Claude. Andre Karpathy thinks there's a much better way to work with AI. And his method has three layers. Layer one is
the spec. Instead of giving Claude a task and hoping it understands you, work with it to create a detailed spec first. Have Claude interview you about what you're actually trying to achieve. Then
break the work into smaller checkpoints before it starts. Layer two is the verifier. Before Cloud does anything, define exactly what a good result looks like. Then give it ways to actually
check its own work, whether that's another AI model or real data or tests that it can run on its own. And layer three is the environment. Build a workspace where Claude already has your
instructions, your knowledge, your skills, and your rules every single time you use it. So instead of becoming better at writing prompts, you're building an entire system around Claude
that makes it better every time you use it. So just ask yourself, which one of those was easier to watch? Which one felt quicker? I think Opus's felt way better. It was more engaging. I think
some of the sound effects were a little bit too much, but it was way more engaging. There was more going on. The B-roll was more interesting. The animations were more interesting. I just
think that this one was a much better output. GPT Astros just felt bland, you know? It just felt very vanilla, very mundane. I don't know. It it it wasn't very
creative. And what I think is really weird about this is as I've been using Astra more since Opus dropped and since Soul dropped, Astra feels like it went down. It feels like it downgraded.
You'll see later I have some other experiments where I was like Astra usually does this better, but it feels like Astra got worse because I actually made a video of me editing with Astra
and it was really really good. And since then the Astra editing outputs have like definitely degraded. So I'm not sure what's going on there. But as far as this specific experiment, Astra ran for
about half the time that Opus ran and cost $2 less. So about N bucks compared to 11 bucks. And overall, the output here is going to go to Opus 5.5. I mean, Opus 5.5 is a very, very impressive
model to me so far, especially when it comes to creative taste and judgment and things like that, vision. I know these first three experiments were definitely more on this design side, but we're
going to get into some other experiments. So, let's move on to number four. Okay, so in this one I gave Astra a ton of data, ton of mock data, and I told it to build me an investor deck, a
financial report, an analytics dashboard, and a client landing page. So pretty big request. And one of the things you'll notice here is that Astra asked me a question in the middle of
this goal because I did slash goal prompts for all these. Now, Codeex in general, I think it's a harness thing, is way better than Cloud Code at asking you questions, and it will keep the goal
going in case you don't answer, but it asks me way more questions, which I really like because it helps me make sure that the outputs I'm getting are more tailored towards me. So, anyways,
let's real quick go through Astra's deliverables to me. The first one we have is an investor deck with 17 slides. We can see that it generated some Bright Path branding up here. It generated this
little image. We have some big hero stats right here. The slides are very simple. There's not too much going on. Um, I think that they're these could look a little more professional if there
were like lines and sort of some boundaries between, but overall, they're not too wordy. They're pretty clean. They're also just very simple. It definitely used a Google sheet on the
back to generate these visualizations, but there's nothing too impressive about this. I mean, I think it did a good job going through the corpus of information I gave it on the mock company data. But
besides that, it probably could have leveraged, you know, more images a little bit better or probably could have had a logo to make this feel more branded. Nothing too crazy, but it
definitely did a decent job. We'll open up the Google Sheet financial report. We have some topline metrics over here as well as um some charts to go with it, some visualizations of the data. It's a
big summary. We've got some assumptions here. Um we have a cash plan, we have P&L, we've got revenue. I mean, it's formatted nicely. It's branded. And I'm not sure if this stuff has formulas in
here that would change. It looks like some of these have some very basic formulas going on with like some sums and things of that nature, but not all the sheets do or not all the cells.
Sorry. These ones are just very basic sums. Um, marketing people and checks. Okay. So, nothing too crazy like I think that this style of output could have been generated with, you know, GBD516
soul or even Luna. That's a very simple output. Anyways, let's take a look at the analytics dashboard. We'll just look at this right here in the Codeex desktop app. We have um this is pretty much
exactly what I expected Astra to design. It always kind of looks like this. It's not terrible, but it just looks very very simple. We have recurring revenue here. Can we switch between these? No.
We can switch between the dates. Q1, Q2. We can see other stats down here. What are these on the side? We have sales pipeline. Okay, this has some filtering for us as well. We've got the customer
health. We've got the What is this? acquisition efficiency. We have financial performance. And then we also have um execution tracker and some other data notes down here. Okay, nothing too
crazy. And then finally, we'll open up the client landing page. Actually, I'll just open this up in the actual browser here. So, make your next move a clear one. Bright path. We have the nice
little logo behind. I like how this is on an angle. Not too bad. Let's keep on scrolling here. Number comes up. Actually, that's interesting. Only the 412 was an animated number. If you watch
this, I don't know why I didn't do that with these numbers as well. I will deduct one point for that. This number came in nice. We have this little dashboard. I do like this element. I
think this feels pretty professional. And as we keep scrolling down, we have some pricing. We have FAQ. Okay, so that's basically it. Nothing too crazy at all. I don't know. I feel like it
could have maybe leveraged using some image generation a little bit more and done a little bit more, but it's not too too bad. Let's head over to Claude and see what we got from Opus 5.5. Okay,
same exact thing over here. Here we're getting a little bit of a sneak peek to the landing page, which looks like it could be cooler. It's kind of weird because of the desktop app view, but
anyways, let's start with the investor deck. I'm going to just keep this in here. Um, see what is happening. Know what to do next. So, I like how there's an image in the back. I don't know. It
just feels pretty cool. Growing service businesses still run on spreadsheets. We have an image right there. One view of the See, this is what I meant. Like, Opus is putting at least some lines and
it's just giving it a little more depth than what Astra did earlier. Astra also chose to connect right to Google Sheets and just build it in there. whereas Opus did this in a just a PowerPoint, but
we've got some images here to go along with these stats. I think that this slide looks pretty good. We've got a table there. We've got some other stats, more tables, more charts here, different
colors. It's kind of emphasizing these three. Got a little bit of a churn versus new visibility to recommendations. 68 people. Okay, this would be filled in with actual numbers
for what we're trying to see. This is where Astra asked me what numbers we wanted to put here. Oh, I can actually edit move this in here. I didn't even know that. But Astra asked me what I
wanted to put for these these metrics. Whereas Opus didn't even say, "Hey, do you want me to make these up or do you have these metrics?" Astra stopped and asked me that. But anyways, as far as
the design of this one, I like the design here a little bit better. Let's see what else we've got. We will go to the 12 tab financial report on analytics. This one's definitely more
verbose. You can tell right away this one feels more like an analysis compared to just having a spreadsheet. But as we go through unit economics, cash and runway forecast, they feel very similar.
Like there's nothing too crazy except for this one. This is something I do want to call out. This sheet has way more references, more references, more formulas. And I do think that's a huge
deal. Like having a sheet by be dynamic where I can change the numbers and see what actually changes is definitely a an important factor, right? like you don't want to just have a static sheet that
you would have to have one of your agents regenerate every time you wanted to change some numbers. So I think the fact that they do look sort of similar. I mean Opus is using more like colors
and stuff in here but Opus has more formulas and I think that that is a big deal when you're creating spreadsheets. The analytics dashboard as far as a design doesn't look anything too much
better than what we saw from uh Codex, what we saw from Astra. They're very very similar. We've got revenue, we've got customers, we can go on light mode, dark mode. We can do different quarters,
and we can see some other sort of analysis here. I would say that nothing about either of the dashboards was too like stand out compared to the other. And then we'll go ahead and look at this
landing page that I built. Okay, so you can see that this is a clawed artifact. See what is happening, know what to do next. We have a little bit of a dynamic image in the background. Okay, so we
start to scroll in. We can click into these things or no, no. Okay, these are just showing us what's going on. Six locations. Your numbers live where the work happens, wherever the work happens.
We have some different pictures here. We're definitely getting more of a story being told here compared to I don't like how this is right aligned. That seems a little bit odd. Don't love that. We have
this dashboard where as we scroll, we're seeing numbers move along. So, I really like that element. We have more text, more numbers coming in down here. Got a quote, an image, different pricing. We
can see how this would scale up. Okay, that's not too bad. Overall, Opus is definitely doing a better job with the design because it's not just the design. It's also it just feels more like it's
telling a story. Now, of course, some of the quality judging I'm doing here is very subjective. Like, some of you guys might have way preferred Astra's website. Some of you guys probably
prefer this one. So, it's hard to say like subjectively, but when we look at the actual cost, you can't argue with the numbers. Opus came in here about 7 minutes, 6 minutes faster. 7 minutes
faster, as well as only about a dollar more expensive. So, I would pay the extra dollar here to get the outputs that we got here from Opus. And this is just crazy. I was not expecting this to
be, you know, so heavily in Opus' favor, but not only from the design. There were way more formulas inside of the sheet with Opus. So, anyways, let us move on to the next one. Okay, so this one is a
game. I had them generate this game for me where we're sort of like trapped in a museum. I'll actually show you. I said to build a polished browser game where the player escapes a miniature museum
after closing. include three connected rooms, an inventory, five link puzzles, contextual hints, sound controls, a complete ending. I'll be judging you on creativity, realism, lighting, design,
the entire experience around the game itself. So, let's hop back into this tab. This is what we got from Opus 5.5. I'm just going to say begin again. And what I like about this is, let me unmute
this tab. You can see that it basically like runs us through a story. So, there's music playing and it's sort of setting the scene. You were only meant to sit down for a minute
and now it looks like you got stuck inside the museum. I don't know if like I wasn't reading that. So, I don't know if like we fell asleep and then they closed and now we have to escape or
something, but the lights are off. The whole place is dark except for that moonlight. Okay. So, I can walk around like this. I can control my mouse.
As far as like the the graphics of this are not too bad, especially when you consider how long it it ran for. You can see like there's glare. I'm not necessarily going to like play this full
game and figure out how to escape, but it's not too bad. I mean, it's obviously a decent experience. The one thing I'd say is my mouse is doing that thing again where it's not like locked in.
Meaning, if I move my mouse out of frame then, oh, I can still control it. So, I'm I have to hold to look around though. So, it's a little bit weird, but I'm still my mouse isn't in frame at all
and it's working. Anyways, this is not too bad of an output at I like the flashlight as well. I can examine things like stopped at 10:47 the minute I got locked in. The pendulum hangs dead
still. So yeah, it's kind of like an escape room. You just have to find find clues, try to figure out how you can find things to help you get out of this room.
I can't see anything at all here. I can read this little piece. We can see this locked at 10:47. So I'm not doing a very good job at actually like figuring out what to do to escape. But so far, I
mean, the the graphics and the realism, not too bad at all. I like it. Okay, so here is Ash's version. It's called The Last Curator. The doors close at 9. You stay a little too long. Let's step
inside. So, this is not as immersive right away. This is more like a It feels more like a puzzle that I'm looking at rather than I'm inside. But we can see we can click into different things. We
can try to access a code to open up the cabinet. We can click into the clock. Something is missing. The mechanism needs the winding key, the brass nighting gale. I sing when the sea, the
sky, and the hours agree. Okay, a bit of a riddle there for me to figure out. We have, "If you were still here at 9, don't force the door. Restore the closing ritual. Begin with the
collection cabinet. Our oldest treasures always come first. Count their gold marks." We have eight dots, three dots, five dots. Let's see if I can open the cabinet with that. What did I say? 853.
Open the cabinet. Not quite. The mechanism settles back into silence. Your clues are safe in the journal. Oh, wait. I didn't do those in order. So, it would be 3 85. Let's try that.
Okay, there we go. I got that open. Now I can read the keeper tide table. So, anyways, I think you guys get the point. They had to think about the story here, how you make a puzzle, how you make
something where you can sort of like work your way through it. But I gave them such a vague prompt that Astra decided to make it more of this experience and Opus decided to make it
more of a realistic immersive like 3D game experience. And that's why I say this is so subjective. It's hard to tell because if I gave them a very very specific prompt and I wanted, you know,
I had a vision of what I wanted, then that's a bit different story. But in this case, I didn't really know the vision I wanted. So let's take a look at what the actual cost was here. Astra ran
for 34 minutes and Opus ran for an hour and a half and Astra took about $8 where Opus took $31. Now, this one's really tough because like I said, if I would have given more specific and I had a
vision, then I'd be able to assess which one I liked more. And I think as far as like getting into an environment and being able to start playing it right away and not being overwhelmed, I would
say that I'm going to give this one to Astra and it came back very quick. It wasn't very expensive and it lets me sort of iterate on top of it and, you know, figure out what I do want. But I
still do appreciate the level of quality that Opus created and the the vision it had. I guess the immersive experience that it created for me. I think that like I said, both very very impressive
results. But you also take a look at this. And I'm going to give this one to Astra. Okay, so this one was interesting. I gave them or I told them to go get a hundred of my recent YouTube
videos and create different concepts and make it a 3D world that I could explore in order to learn these concepts. So here is Astra's version. We have this little 3D world to explore. Astra
decided to make this a a first person sort of game. You can see that. Oops, it just opened up my clawed one. Sorry about that. So, we can see we have tools, we have context, we have prompts,
we have useful projects, trust and safety. So, let's just go into something like we'll go into models real quick and see what's in here for us to play. We're able to come in here and explore this
room. I click F. We can see different experiments that we could run. Let's do one small to run experiment. Okay, so experiment complete. check your understanding to earn a stamp. Okay, so
it's kind of gified where you have to take a little quiz or something in order to get a stamp and then I think the idea is that you try to get all of the stamps. So anyway, let's see what other
sorts of experiments we have. I'll go into the skills area real quick. Touch an idea, see what happens. Okay, how do I touch an idea? I has experiment, discover, sources, check. Okay, so the
check is like where you check your knowledge. Okay, cool. Um, build check. I'm not exactly sure. Make a reusable recipe that plans, builds, and checks. Okay, so plan, build, check. Run
experiment. Is that what I do? No, it says plan, plan, check. Plan, build, check. Okay, there we go. Um, I don't know. I don't exactly understand what's going on. Like the experience of
this is a little bit overwhelming. I thought that this would be easier to understand. Let's see what happens here. You want to help Pip cross a broken bridge, inspect it, repair what's
missing, then test the repair. Okay, so inspect. One out of six problems found. Repair. Okay. Test. That's it. I don't know. These are not exactly what I thought I would be
getting. So, this isn't super easy. Like, I wanted to be able to say, "Hey, you know, I would throw a kid in here and tell him to go play in this game and learn these concepts, but this is a bit
confusing." And now we have the AI Explorer Academy that was created by Opus. This is more of a little robot. It's more of a third person sort of vibe. And we have different rooms to go
into as well. I think this one I mean I played with this one yesterday but this one's a little bit more I don't know it's simple. You can see token blocks. This is how AI reads tokens. We come
over here and we can use this machine which basically helps us drop in some balls and we can choose the different temperature. So the temperature changes like how many the randomness I guess and
then if I drop in some balls you can see that it's going to fall into the next word like the cat's head on the mat or the cat's head on the floor or the roof or you know the other ways that these
things are predicted. Now, there is an issue here where when you drop these balls in, they kind of get stuck sometimes. So, that would be a little bit of a bug, but considering that this
wasn't the main game that was created inside of this world, it's really not too bad. Let's go into something else. I'll go into the agent arena over here. AI that works in a loop. We see what do
we have right here? Pick a mode on the console. So, we can run one shot with no loop or we can run an agent loop. Let's run an agent loop and see what happens. So, we can actually see the agent doing
things. It's thinking, it's checking. It's thinking, it's checking, and it needs to get to the end goal. So, it's going to think, check until it's able to find its way over here to the flag. So,
you can see that that was seven loops. So, that's pretty cool. What if I reset the rover and then just did run one shot with no loop? It would just think and act and it just goes straight towards
the goal, but it runs into the boxes all the time. So, that's honestly a pretty good little visualization. Anyways, I think you guys get the point. Each of these rooms has a different concept um
that has to do with AI automation. And then there's a bit of a simulation or some sort of game where you're able to actually see what's going on. This one looks like there's an actual like helper
shop. So, now we're getting into some like business concepts where you open up the shop. Next customer, next customer. This one looks like it might be broken, but anyways. Oh, nope. There comes the
customers. I was mistaken. Deal. Only 200 bucks. Wow. That is super cheap for what it saves me. Okay, cool. So, that is the world that Opus 5.5 created. Now, this one is interesting. It took us an
hour and 44 minutes with Opus 5.5, which makes sense. I mean, that was pretty complex. $60 with Opus. Astro ran for 45 minutes and only took $12. So, this is going to be an interesting one. I
ultimately the quality of what we got, I liked Opus' better. And some of you guys would say, okay, so Opus wins this one, especially because you think about the fact that we did a slashgoal prompt. So,
Opus fulfilled the goal better. But one thing that I think is interesting to think about is Astra here only took you 12 bucks. What if you iterated with Astra, maybe shot off another prompt
with feedback, and you let it run until it hit 60 bucks? Would Astra's output for $60 been better than Claude Opus' output here for 60 bucks? I would argue yes. Now, of course, that takes into
account that we're able to literally provide feedback, so it's getting more tailored towards what we're looking for, but I do think it's an interesting way to think about it. So Astra was
obviously cheaper here and faster, but if we're talking just the actual quality that came back in it one/goal prompt, then I would be giving this one to Opus 5.5. All right, so moving on to number
seven. I told it that I wanted to take a trip from October 1st to October 30th. I wanted all these sorts of things and I told it to just make me a 3D immersive itinerary um that looked like a world
that I could sort of like journey through and see visually where I'm going and I wanted to provide me all the links that I needed in order to get around. So here is Astros. It's called Atlas. We
can see that we're starting off here in Chicago. We're getting a little bit of a 3D world here where I can click into the next days and it will show me exactly where I'm going. So, I would be going
over here all the way to what is that? Iceland. And then we'd be coming back over to the Netherlands in Amsterdam. There's a World Summit AI. So, pretty cool. Then we'd be going to San
Francisco for Tech Week. It looks like we have all these other events going on in here. San Francisco for about a week. It looks like other places in California, Yusede. We'd end up going to
Utah for Zion because I said national parks. We'd also be able to hit some of the other national parks in Utah. It looks like Bryce Canyon. We'd go to Vegas and then we'd come back home to
Chicago. Okay, pretty cool. Now, I'm assuming all of these have things like this one has some shuttle times for us. This one has weather conditions or conditions of the park. We've got um the
driving route right here that we could open up, which is on Google Maps. We've got some flights that we can look at on Kayak. So, it looks like it's providing all of the right links for us. So, this
would definitely make our life a lot easier if we wanted to plan out this trip. Okay, then here is Opus' version. So, we have the same sort of thing on the bottom. We can actually hit play and
just see this happen. I guess it will sort of cycle through different days. That's kind of cool. I like that animation there. It's going to take us to these different areas as you can see.
And all of these have links like you can see. We have this boat tour which I can go here to grab some tickets. We have this sky lagoon which I can click on here to get tickets there. Um, we have
some flight stuff. World Summit Day. So, the itinerary is honestly pretty similar as far as it finding these different AI events as well as finding California parks, Utah parks. It found all of this
stuff for us. Once again, GitHub Universe and then we would be taking this flight. Although, oh, there's the the links to a Southwest flight right here via Google Flights. So, I think
they both did a pretty solid job. It would just be more of a matter now of like, okay, which one do I like the actual design better from? And what do we think about the speed and cost? This
is kind of a cooler immersive 3D map, I would say, but it loads pretty slow. So, there's definitely some optimization we could make there. But let's go ahead and take a look at the performance. So,
pretty similar here. Opus was 29 minutes, Astra was 32 minutes, Opus was 1447, and Astra was $11. I honestly am going to give the edge here to Astra. I think that Astra's version is very
clean. It's got all the links right here. They're both very good, but I think that Astra's might just be a little bit cleaner, especially when you look at how some of their stuff was. So,
I'm going to give experiment number seven to Astra 6. Okay. So, now what we have is a codebased challenge. I actually had Grock design a challenge for me and then I had Grock look it over
with Fable 5.1 and GBD6 Soul. So, I just kind of got a mixture of experts in there and they designed this experiment. And then what I did is I gave that experiment to Opus and to Astra. These
are the results that we see which is pretty interesting. Astra was way more efficient on both speed and cost. And then when I had multiple agents review this again, they all sort of agreed on
this, which was the idea that they both did really well here. They both passed with 100 out of 100 score, except for Fable did say that agent B got two points deducted on one of these
experiments over here, which would have given it a total of 98 out of 100, which actually doesn't make perfect sense if you actually look at these scores right here because it got 61 out of 62 and 10
out of 12. But anyways, the conclusion that I got was basically that both of these agents with this sort of coding task, very very good. Like very very good agents. They both did a great job.
I still would feel more comfortable having both agents run through and help me repair and find bugs and things like that. But ultimately, if you look at the efficiency here, this is going to be a
clear win for Astra. It's not even close. It was, you know, a fifth of the time as well as half the cost. So this one definitely goes to Astra. And I do think so far this really aligns with the
way I'm feeling about in general Open Eye versus Enthropic. Enthropic seems to be way more creative. I feel like it usually has better taste. And then OpenAI is just like super pragmatic,
super logical, super reasonable. And if you're giving it something very, very specific, it's just going to execute that for you pretty much the way you want it. So that's kind of the way in
general I feel. But let's move on to number nine. Okay, so number nine is interesting. I wanted it to build me a 30-cond animated Instagram reel, just telling the story of me. I wanted it to
think about what it knows about me, do some research, and generate whatever it wants in order to make this a real about me. me. So, I just wanted to see what it could do. And this is one of those where
it's very creative. So, I wanted to see who would win here. My gut is telling me that Opus is going to come through better, but you never know. So, let's give these both a watch. Here is Opus
5.5's version. 2 years ago, I was building dashboards at Goldman Sachs with zero coding background. My automations followed rules, but they couldn't think. So, I taught myself AI
agents and posted everything on YouTube. Two months later, I quit. A small free group became AI Automation Society, now 450,000 builders. The channel passed a million subscribers. My book hit number
one. And we launched a certification. I'm 24 and I'm just getting started. That's pretty funny. It made a Pixar style person. It has the shirt that I wear in most my thumbnails. And it has I
don't know. This looks like maybe my profile picture. I'm not exactly sure. This is kind of funny. I mean, this is all pretty accurate. I like the way it did it. It also used my voice. It used
my 11 Labs voice clone. So, not too bad. Let's go ahead and see what Astra gave us. >> Before a million subscribers, Nate Herk had a webcam and a towel under his
keyboard. Fresh out of college, he moved from Illinois to Salt Lake City. He started learning AI, building things, and sharing what worked. The videos were rough. He kept showing up. Those
tutorials grew into the AI Automation Society, a community learning alongside him. And after all of that, he's still learning, still building, still hitting record.
Okay. Well, first of all, still hitting record. I think that should have been still hitting record, but this one didn't choose to use my clone, my voice clone. It also chose to make this sort
of more of like a claimation style. I'm not exactly sure what you would call it, but they chose different styles. They also clearly used the same reference image, that picture of me in that like
light blue shirt. It's interesting too because Opus chose like to narrate in first person whereas Asha chose to make this like more of a biography. You know, it's hard for me to choose a winner
here. I think that it's they both did something interesting. I just wanted to see what happened. But if we take a look at this stuff now, we have an interesting one. So Astra ran for 20
minutes. Opus ran for almost 50, but Opus was cheaper at $8, whereas Astra was $11. So in this sort of a scenario where I don't care about speed, and I more so would care about the cost and
the quality, I don't know. I think that Opus' version, one of the other things that I liked about Opus' version better is that it had at the top like a progress bar. You could see from
September 2024 all the way to today, it's kind of like showing you where you are on the journey. Whereas in Ash's version, it was just like these different chapters and it felt a little
bit hard to understand how it all linked together. So, we'll give this one to Opus, which is kind of what I thought was going to happen based on previous experiments with like video editing and
just creative direction in general. All right, so this next one is one of my favorite new benchmarks. It's the Canva drawing one. You can see that I'm going to give them this image and I say open
up Canva. Use browser use to draw this picture as accurately as possible with painting tools and not generating an image. So, if we take a look here at Astro's version, if I open up this
browser, I mean, this is awful. Like, this is terrifying. This is scary. You can see that it tried to put some texture in my shirt, like the sort of this waffle pattern. It tried to, you
know, it added the zippers on Adam Sandler's shirt and things like that, but the faces, this is nightmare fuel, 100%. I'm not sure what happened. In the past when I've done this with Astra,
it's been incredible. Like you can see it's highlighting the arm nice with like shadows, the face. Like I think it's trying to do a good job, but it just completely butchers the facial elements.
And this is where I'm saying like I feel like Asher has degraded because it used to do this really good. Like when Ashra dropped, there was tons of people doing this on X and on LinkedIn and I was
doing it and I thought it was really good. Like actually just for reference, this is what it did for me earlier. Like this was Astra's output when Astra first dropped which I thought was really good.
If you look at the reference image and here's another one that Astra built me as well, which wasn't as good as the first one, but this is clearly a better technique than what we're seeing right
here for I I don't exactly know why. And here, let's go to Fable's output. This is really good. This is more like what Ashra was giving me when this whole thing started up. And based on the
reference image, you clearly see here that Opus 5.5 wins. And it's not even close at all. Like this looks I don't know. It just looks like someone who forgot to put their glasses on in the
morning. It's just a little bit blurry. And when you think about the fact that Opus 5.5 was cheaper for that output, it's a no-brainer. Opus 5.5 is definitely winning this one, which yeah,
I mean, I know a lot of these are subjective. It's not exactly a benchmark, but I think the whole point of testing models is testing them on your use cases, your knowledge work, the
way that you use AI. And I use AI a lot for creative style work. So testing this sort of stuff for me is very helpful to understand what I'm working with. All right, so number 11 is creating an
Instagram carousel. Now, I do have a skill that helps us do this. So, they were hopefully going to both use the skill, but I shot them an X article or an X tweet and said, "Hey, make me the
Instagram carousel." And let's see what we get back from Claude. So, here is Claude's output. Let me just make this a little bit bigger. Here we have nearly nine in 10 Americans call AI a threat.
Their number one fear is at losing control. We have a picture. I think this is the Terminator here. And as we go through, we have a different stat with images up top. We've got an actual quote
from some sort of poll. We have more stats. And obviously this is going to be formatted in a very specific way because of the skill. Now I think this image feels like it could have been lowered a
little bit. I feel like that could have had some better verification. I think this one too. Um this one as well. So all of these images could have been I guess positioned and sort of like
centered a little bit more. So that is Opus's output. And then here is the output we got from Astra. And already I like it better because it put the little Poly Market
logo which is the original tweet that we sent in. We have a robot there. Nearly 8 in 10 Americans worry about AI escaping human control. I think that might be a better like opening caption as well. We
have the actual tweet which I do like having that there. We have 79% concerned with a little bit of an image. We have a nice image there with a stat. Another little stat. Another little poll. We've
got the same alien or not alien. We have the same like robot astronaut image. So I wish they did a different one there. Keep approval on before AI sends. We have a call to action. And then we have
a final picture. Overall, I think that I like the taste that Astra used here. Even though they were both given the same thing, they were both using the same skill. I think that Astra wins this
one based on its taste and its judgment in what it actually turned into a carousel. So, this one I am going to give to Astra. And Ashra also came through about half the time and um about
60% 70% of the cost. So, $10 compared to almost $7. This one is going to go to Astra. All right. So, this is the next one. This is the site that I gave it. This is my site for my book, Becoming I
Native. This isn't actually updated, so I do need to update this. But I gave it this site, and I wanted to see if it made a new version of this for me, what it would do, as well as if it might
update some of these numbers, which are outdated. So, we'll see if either of the agents were, I guess, like pragmatic enough to go think about updating those numbers. So, this is the one that Astra
gave us. We can see we have become AI native. I hate this logo up in the top left. Hate that. Make AI part of how you work. We can explore the book or we can buy on Amazon which does take us to the
actual book. We can see we have a little bit of a dynamic book image right here. The three M's number one Amazon bestseller 850K 425. So it basically treated the website as the source of
truth and it didn't think to fact check any of that. So I don't love it but you know we'll see if Opus did that as well. The tools will change your approach can last. This is a very very boring site.
It also just gives off like very cheaply built. So I definitely don't love this at all. Let's go ahead and see what we got from Opus. Okay, I don't like this on first glance. I mean, it's too I
don't know. It's like doomsday. It's not the vibe I was going for. I mean, I think it probably looked at the cover and was like, "Oh, well, maybe it's like kind of dark mode and, you know,
whatever." I like how up here, this looks a little bit more branded than just having AI. I mean, this is definitely better. But as we scroll through, we have the number one Amazon
bestseller. We have 4.4 out of five stars. We have the pricing here, 35 ratings. We can get on Amazon. We can read the first page. Okay, that's kind of cool. It it puts the first page out
here. We have the three M's. It's taking some actual pictures in there, the method, the machine. It's showing you the actual chapters as well. It's showing us some of the frameworks in
here. So, I mean, this is overall it's it's telling a better story. I mean, look at the reviews, too. This is definitely the better site. It just needs some work on the design, but it is
100% a better experience than this, which just feels like a oneshot prompt to, you know, an AI model from 2 years ago on Lovable or something like that. This is definitely a better experience.
So, Opus is going to take this one as well. Let's take a look at the pricing here. Very, very similar. About 14 minutes for each. And Opus was a dollar more expensive. So, this one is also
going to go once again to Opus 5.5, which in general, I'm just very, very impressed with this model. It is a very, very good model. Okay, so after all those experiments, we're sitting at GBT
Astra having won four of them and Opus winning eight of them. So it's 8 to4 opus. Obviously a lot of that is subjective I know but that's the way that I am giving them my ranking based
on my use cases and my taste. So after all of these 12 experiments opus 5.5 ran for almost 11 hours. GBD6 Astra ran for 6 hours. Opus cost me $214 if this was API billing and Astra would have cost me
$132. So Astra in general 45% faster and 38% cheaper which is very interesting considering it is a 2.5x on the price compared to Opus 5.5. So what that tells me is Astra is a very very efficient
model but sometimes at least in these experiments Opus was understanding what I wanted a little bit better and it was giving me more creativity and its outputs were just a little bit more
aligned with what I want. And like I said earlier in this video, I've felt like for a while now that Opus and Claude models feel like a wise old owl. They feel like they have good judgment
and creativity and taste. And GBT models just feel like they are a really good obedient worker. They're going to get the job done. They are going to like they want you to give them a very
specific spec. They want you to tell them exactly what you want and they'll figure out how to get there. They're very goal-oriented. But when you give GBT models vagger goals, you might get
frustrated with what they're coming back with because it just feels like they weren't meant for those vague goals as much as something like these claw models were. Ultimately, what else I think is
that both of these models are so so good when you look at the average task that you're running, knowledge work or simple like merges or building little apps or dashboards. Both of these apps are going
to be so so good at giving you that functionality. It's really about the way that you talk to it on top of that and the context that you add on top of that to actually make it align with what you
want because I want things different than you want it. You want things different than your friend wants it. So, it's like how do you actually communicate on top of the sort of like
base fundamentals, the 70% plumbing that everyone needs to set up in the same way. On top of that, what's the 30% that gets you there to where you actually are like, damn, like I really like this
output. So, anyways guys, I hope that you found this one insightful. I hope you enjoyed you learned something new. And if you did, please give it a like. It helps me out a ton. And as always, I
appreciate you guys making it to the end of the video and I'll see you on the next one.