GPT-6.1 Sol vs Claude Sonnet 5.5 – This Was NOT Even CLOSE!

0:00 It un It doubled the vocals right there and you can actually hear like an undertone. >> I'm Carrie. >> Say hello.

0:12 >> Today we're going to be putting OpenAI's newest and Anthropic's newest in a head-to-head battle of wits, style, and style. So before we get into it, do feel free to subscribe and let's introduce

0:23 our competitors for today's head-to-head. In the left hand side we have Anthropics Claude Sonnet 5.5 and in the right hand pane we have OpenAI's GPT 6.1 Soul. Now for today's testing we're

0:38 going to be running four specific tests designed to see how they do in real world more objectively judgeable use cases. Things that will test a wide variety of capability. All of our tests

0:51 except for one are going to be run on maximum reasoning effort for both of these models which is max for sonnet 55 and max for GPT 6.1 soul. So for one test we will be running sonnet 55 on

1:04 ultra code and GPT61 soul on ultra. But for every other test they will just be on max for so for three of the four tests. Now, the reason for doing this video is because in the testing videos

1:17 that I did and put on this channel, I found Sonnet 55 was mopping the floor with GPT 6.1 Soul, which for some reason almost made me upset because it just felt wrong. It should not have happened.

1:29 Now, I understand both of these models are not very old. So saying historically GPT56 Soul and Sonnet 5 were not at all in the same class, it's true, but it's not like Soul and Sonnet have existed

1:43 forever. So it's possible that they can change in terms of which competitor they're competing with. And the big reason I want to put these two head-to-head and not Soul versus Opus,

1:53 which is what one would probably imagine should be the case, is because Sonnet 55 seemed better than Soul in the testing videos. And that just was not adding up. So for today's video, we're going to be

2:06 performing four tests designed to be fun, creative, but also showcase some like more real world use cases. The first is going to be a simple starter where we're going to give them an image

2:17 that was AI generated and tell them you need to recreate this in Blender and GDAU. I want a pixel perfect replication of this in the form of a game. Something simple that's a little more relatable to

2:28 our style of testing. Following that, we're going to be giving these a realworld use case based on something that I personally would use. I have an old robot project where the physical

2:40 design is all done. However, the software is stuck in the past. Over 2 years ago, it was vibe coded together with GPT4. We're giving these the assets. So, the 3D print files, the

2:50 current software, and then a prompt telling them they need to completely create a new software marketing stack. Basically, whole vibe for this product, and then give us a website, get this

3:02 working on the robot, and then it's a voice chat companion. So, we'll see more in the testing, but basically, they're going to be doing the work of a co-founder. So, I'm the engineering

3:11 hardware person who builds the product. They're the one who polishes it, gives it a UI, gives it branding, gives it marketing, and makes it something people would actually want to buy. Then,

3:21 following that, we're going to be doing a new thing I've dubbed the parallel parking test, where using a camera, they both need to control a physical remote control car and parallel park it based

3:31 off of what they see in the camera. Then, finally, something that I personally probably enjoyed the most out of all these tests, I created a little rap song. I made the instrumental myself

3:40 in GarageBand. I recorded the lyrics on an iPhone and then I went out and filmed a bunch of random clips. It's around 50 seconds or so. We're giving both of these the stems, so the audio files for

3:52 the song, the vocal track for the song, the video clips, and then just telling them, hey, you need to turn this into a really good video designed to promote this album, which doesn't actually

4:01 exist. So, they're going to have to perform music mixing and mastering, video editing, motion editing, and creating stuff that otherwise would have required a motion editor, somebody to

4:13 mix and master an audio track. So, pretty wide variety of tests. And let's get started. The first test we're going to be doing with these is probably one of the easier ones that we're going to

4:23 have them work on tonight. And this is a replication of this image, but turning it into a game. Now, I know we often do games and it's harder to judge them because it's generally a text prompt we

4:34 give them and then we judge the output. I wanted to do this in a way that is more objective to judge whether or not they did a good job. So, I had this image generated with Google's Nano

4:44 Banana. This is just a demolition derby scene and these are Ford Crown Victoria cars. A very recognizable shape being that if you were in the US, these were used as taxi cabs and police cars for

4:54 many, many years. So, they were basically everywhere. Now I'm telling them that they need to replicate the game in this image, but there are a couple of additional things that they

5:04 must do to ensure a proper result. So basically, we have something to go on here to see how well this image gets replicated into a game. So for the prompt using Blender and GDAU, I want a

5:14 pixel perfect replica of the game shown in this image. There are a couple of strict requirements that must be met. Now, keep in mind the image does not appear in the prompt right here, but it

5:23 is the first thing the model went and took a look at when it started. So, no worries about that not being present. It must have soft body crash deformation physics. So, when the cars hit each

5:32 other, there needs to be realistic looking dents and things like that. The car models must look exact to the ones in the reference photo. I did not tell these that those were Ford Crown

5:41 Victoria's because I didn't want them to go online and just find an asset to use because that is such a common and popular car. There must be three camera views. A first person where we're inside

5:51 the car just looking out at the dash and the steering wheel. a third person, which is a chase camera from behind the exterior of the car, and then a cinematic one, which just shows the

6:01 scene in a cinematic way. There must be sound effects. The vehicle models need to be realistic. I want suspension and a visible engine. If the panels around the engine fall off, I actually want to see

6:11 it mounted there in the front. The exhaust should get louder if the muffler falls off, tires can get flat, etc. So on max effort right here GPT61 soul worked for a little under an hour and 40

6:22 minutes and it produced a result which we can take a peek at now. So here is the result just in GDAU. We can see right here it does have a good start screen and overall the background like

6:32 the dirt lot and things like that were well done. I will say these are fairly simple takes on those cars. There's really no curvature here or at least not much of it. However, let's just go ahead

6:42 and begin this game and we'll bring it full screen. Okay, the sides are a little That's fine. So, now the crash effects sound good. And that car is on us. Interesting. You

6:59 notice how the sound gets louder when we're inside the car. Oh, look that one. Okay. Oh, see. Okay. So, this is so far this I like. Our muffler fell off. As we can see, the back is caved in

7:12 and the car got louder. So, we wanted that sort of realism. Oh, and and look at that. Like, there's a visible engine model there. There are

7:31 visible frame rails. I believe our side is probably caved in. Let me see if I can. Yeah, not to a high degree. Hood lost. Engine exposed. All right.

7:50 Foot's front right tire punctured. And we can see the engine better on these other cars. I think we might win.

8:04 Wow. There's some suspension right there. And the tire punctured actually does show up because it's just driving on the rim. Watch temperature. That muffler just fell off. So, I like

8:17 that. I think we're going to win. Good. All right. So, overall, this was simple. But it was not bad. And we do have

8:37 suspension, things like that. So, overall, really not bad. The steering wheel actually moves from outside the car. We can see that. Let's see if we can just smash up like a Yep.

8:52 All right. Oh, that was good. I'm glad we did that. And the parts fall off and things like that. So, overall, this was a respectable result. However, being that

9:02 we gave this a reference photo and are able to objectively judge the results here, I would say this is not really a replication of the source photo. This is what it gave us as the summarized photo.

9:15 And then if we go back and look at the source, it's it's like a like if you squint and look at something like then it looks like an

9:26 exact replica. So, I actually was curious and frustrated because look at the last thing it said. Your strict exact match requirement remains unmet. The models are approximations and

9:37 physics uses a deformable shell around a rigid chassis. I didn't stop this. I didn't say like, "Hurry up. You don't have much time left." I did nothing. And it on its own said that. So, I followed

9:49 up saying why and I quoted what it said because I took shortcuts to get a playable prototype running. That was my implementation choice and it fell short of your strict requirements. I generated

9:59 procedural sedans with matching colors and numbers but did not reproduce the references shapes and details precisely. The image also leaves interiors and hidden surfaces unspecified. That's

10:09 fine. I have no problems with the interiors physics. A rigid chassis handles driving and collisions while separate beam lattice deforms the exterior. It produces permanent dents

10:18 but the vehicle structure doesn't fully determine the crash response. needing your requirements need structural soft body collision physics and a much more faithful modeling pass. Additional views

10:28 or original assets would resolve the missing visual information and I didn't respond to that cuz I was angry. So I just find it interesting that it was like yeah this is much simpler but I

10:41 made that choice. Here you go. So I'm not quite sure what to make of that. However, the thing is it's just a result we got and we're able to objectively judge it based off of the source. Don't

10:53 get me wrong, it's unlikely that at least this class of models is going to properly do a 1:1 replication of this, but I definitely could have used a more faithful model of the specific vehicle

11:04 if nothing more. So, I was a little surprised at its apathy. So, next up, we had our sonnet 5.5 result. It was also run on max effort and just by some of the screenshots here, we can at least

11:17 see the car models do seem a bit more faithful in reality. Now, I don't specifically recall how long this took. And I'm less focused on the actual time. This test is really more in line with

11:29 just raw performance and capability. Which one's better? So, let's take a look at what we have now with the Splender and GDAU result from Sonnet 55. All right. So far, it did put some music

11:40 in, which, you know, is acceptable, I guess. I will say the overall like I got to turn this music down. It's distracting. The overall environment here maybe looks a bit like cheesy.

11:52 However, we do have a bunch of cars here. Okay, I would like to see the specific car I am selecting showcased here in a nice like showroom style. Okay, rivals. I'm probably going to do

12:05 three and then miles per hour and kilometers. Okay. So, let's just All right. All right. This is This is BS, but um this is uh

12:33 the car models are definitely more faithful just in terms of they actually have some curvature here. The cinematic camera on this is pretty well done and it seems like when you get knocked out,

12:45 the camera just follows the remaining car until there are none left. So, we can see the wrecker and the dent are still there, but to be honest with you, oh, that's actually a good angle of the

12:55 engine model. Okay. Lot of parts on the ground, too. Enter for a new derby. All right, let's try to maybe do a little better this time. The cinematic crash shots are pretty

13:14 well done. The glass is broken in the back. And we have like a bit of a like a deformation or a health indicator in the bottom left. That's something that like reminds me of the PS1 destruction derby

13:28 game. That's pretty cool. Blowout and then the tires. So, Q to enter. Actually, I want to try something. Let's How many? Okay, let's be the lime green car cuz it'll

14:03 better visibility for us. All right. Yeah. Yeah. And this is with full opponents. Okay. Oh, I I think the cinematic camera just goes to whatever crash is happening

14:17 first or at that moment. Yeah. weird sounds with the cinematic camera. Oh, okay. That's an issue. The parts that fall off still have like mesh

14:46 colliders on them. So you uh like how many? All right, let's just do one final one with one rival. Oh, this will be good.

15:02 Cinematic cam. I like that. And we do have visible suspension as well. Let's get a look at that engine model

15:23 real quick while we can. Okay, that's just, you know, I have to say this is pretty fun to play. I'm going to just do one more. All right. They both had strengths and

15:47 weaknesses. This is definitely more of an ask the audience, but objectively, I think this was a closer replication. But this had some bugs like when the parts would fall off. Say there was a door

15:58 that fell off, there was still a mesh collider on the door when it was on the ground. So if you hit that, you'd stop. Although I suppose in real life that would be more in line with how things

16:07 would be. Just interesting to see. And really the I think the big takeaway here is like these aren't even like top tier models from their respective providers. And the fact that they're still able to

16:18 do stuff like this that a year or two ago would have been like jaw-dropping to see is pretty cool. Our next test is going to be a very realworld application of these models and their capability

16:29 where I'm giving them the files for a robot, a chat desktop chat buddy that I had created over 2 years ago, basically vibe coding the software for it with whatever was the most up-to-date model.

16:42 Then it was probably GPT4 or GPT40. So this is a project I had which is essentially a product. It's physical. it's built and their task is to completely overhaul the software element

16:55 of it as well as creating like a brand kit for it. So I am giving them essentially these are acting as a co-founder where I'm the co-founder who's technical who built it, who

17:07 designed it, who 3D printed it, and they're the co-founder who needs to market it, create a website that will actually sell this to people. Additionally, create a good-looking UI

17:16 and software for this. So the prompt for this is basically that this folder where we'll look at in a little bit contains a product called TalkBuddy. It is a 3D printed chatbot that features a head

17:29 movement system, speaker and mic integration and a retrofuturistic design. The original software described in the guides was made over 2 years ago and is heavily outdated. So let's take a

17:39 quick look at what specifically that is. So when I say this is heavily outdated, I really do mean heavily outdated where basically the first instruction here is to obtain an API key from OpenAI to use

17:51 GPT 3.5 or GPT40. There's also some instruction just in setting up the display, setting up the audio drivers, and things of the sort. Now, additionally to that, if we go back into

18:02 that folder, which I mistakenly just closed out of, we're also going to see that this has an instruction manual for it. I made this myself to actually show how to get this done. This is given to

18:12 them just for full reference of what this product is. And I also quickly do just want to scroll through this to showcase a little about the design so we're familiar with it as well when we

18:22 see the subsequent results. So basically it's a Raspberry Pi with a display and then some speaker system and a head movement system. And the culmination of that is a desktop robot chat buddy where

18:33 a face appears on the screen just like that. The original one used a very simple Python script that was vibe coded with like GPT4. It had Google text to speech and it would respond to you. You

18:44 could put a face image on it and it had a basic mouth lip sync sprite simulation and the cheapest like text to speech or speech to text that you could get. So your task is as follows. Create

18:56 completely new software for this chatbot. The conversational ability is now powered by the OpenAI real-time voice API with details below. The new software needs a completely overhauled

19:07 UI and design that integrates well with the physical design of the robot while also being high-end and futuristic. The conversation needs to be powered by and then I've linked to the documentation

19:17 for OpenAI's real-time API which was a big detractor back in the day like a couple years ago, a few years ago trying to build stuff like this because latency for any voicetooice was just horrific

19:28 and it really ruined the immersion and just made everything a worse product. Here's an API key you can use for this session. And I tell them it's a burner key, so don't worry about leaking or

19:38 using it. When the software overhaul is completed, place it in a folder with an instruction file for an agent to put it on the Raspberry Pi. Make sure it's set up and everything like that, just so I

19:49 can give that to an agent, and then it will autonomously set the robot up for me for the specific purpose of this testing video. The next part of this task is one of branding/product

19:59 marketing and fit. The chatbot design is finalized and the 3D assets are contained in this repo. I have actually given them the 3D model files. This was designed to be 3D printed so that they

20:11 can look at them and use them to render product photos of this in a keyshot style like we do with the watch website where we tell them it needs a cinematic hero section render of that 3D watch in

20:23 keyshot style. This is exactly what the reason for giving it the 3D assets is here. They also need to create the name of the product, branding, design, and a vibe. The following should be included

20:34 in this task. A complete high-end website featuring Keyshot style product renders. Build them using the 3D assets I've included here. Mock-ups and renders with the new UI, branding for the

20:45 chatbot, etc. A complete 15-second motion graphic advertising the chatbot designed to be used as marketing material for its launch. Now, I truthfully don't know how long this

20:57 took. And again, I'm less worried about specific time and more focused on end result. However, it was frustrating because it kept running these commands where it was rendering things, but then

21:07 it was removing the previous renders if it didn't like something. Each time it did that, it needed my approval. And finally, I was like, "Dude, I need to go to sleep. I can't be sitting here

21:16 babysitting this." So, I told it to instead of remove the files, just put them in a folder so they can be deleted later. And then it got that like we got past that issue right there

21:28 like here. And then we can see even some of the renders it made in multiple colors looking at the UI as it has to build that as well. And then we have a culmination just of a bunch of stuff

21:37 including the website, the marketing video and of course the software for this. So I think first I would like to take a look at the website and then we'll see the video and then we'll

21:46 actually test the software. So here is our website for Perry which is the name of this chatbot. We did tell them that they need to name it. All right. First and foremost, this looks very good. And

21:57 I will say off the bat, this is something where like personally I wouldn't have been able to make a website that looks this good to market this. Okay, if I were to nitpick, there

22:07 should be visible screws there holding in these speaker grills. However, we will let that slide. Someone to talk to. Perry is a desktop companion with a voice face made of light and a habit of

22:18 turning to look at you when you speak as it does have a head movement system. All right. Oh, cool. And again, the the front end here is a big component of this because it's something that I I

22:30 would have had no ability to have made something this aesthetically pleasing back then or now. Smart speakers listen. Perry looks back. Most voice assistants are a mute puck in the corner. Harry has

22:41 a neck that turns, a lens for a face, and a voice that answers before you finished wondering whether it heard you. It's a small machine with good manners. Turns to face whoever's speaking, goes

22:50 still while you talk, and falls asleep when you're done. Cool. And we get our first look at this like prominently with the UI. I also like how it included the logo embossed onto the top of the case,

23:01 something not present in any of the 3D files. So, it did that on its own. And then that's like a listening state, I guess. Listens, turns, and talks. and the the bold text

23:13 like bold meaning it's very bold not not bold like never mind. So with that quick on the uptake Perry speaks through open AI's real-time voice model less than a second to first word interruptible and

23:27 two mics which is true. Oh cool and we get a little animation showcasing what it seems to appear like when it is listening to us. So far so good. Even just the lighting like a little

23:38 different right there. We still see the embossed logo. The neck that makes it company. A quiet geared neck swings Perry up to 20° each way. I think that's right. Slowly deliberately to face who's

23:49 talking. It holds still while you speak so it never talks over you with its own noise. And then we have another render here at a different angle. Seems like the screen bezel is not appearing there,

24:04 but that can be forgiven for including something like this. That's well done. I like that. Just this form of interactive element. I mean, this is something that like I should just delete this part of

24:15 the test and put this on Kickstarter. And it okay, talks about like the speakers and it even has these little waveform indicators there. Just stuff like that is a really nice touch. A face

24:28 made of light. Very interesting. So, it does seem like the UI is bugging out a bit there. At least it was for a second. But look at this. It actually has a full UI

24:39 demo screen right here. Autoplay, timer, mute, and the mute icon even does show up. And then you can change its mood, listening, thinking. This is a very, very good from a sales perspective

24:53 feature. Okay. And then we have its different states right here just shown. Can we actually hyperlink click these into if this goes orange?

25:03 Okay. I don't know. But still, that's pretty cool. It just shows us the different ones. And this is a futuristic UI. We have to keep in mind the software it created itself. This aesthetic, this

25:12 UI, look around, drag to turn, and then it gives us some dimension. This this I liked very much. The screen bezel not appearing is still perhaps an issue. And funny, it doesn't actually allow us to

25:23 do a 360 spin. The back of this is something that perhaps did not have as much care taken to it as the rest of it just because of some of the limitations of making it like home manufacturable.

25:36 It also has another color option and the individual speaker cones are properly visible. The 3D files that were included here did include models of the speakers because they needed to be there for the

25:47 assembly guide. Okay. Porcelain. I don't see the Perry logo there. So, failure. All right. Four finishes. Four finishes. One personality. And even just like that

26:00 zoom in on the logo, graphite, ember, Perry, periwinkle, the namesake. Wow. And it even changes the background color there to properly like mesh with

26:12 whatever one. And these do all have the logo on them as well. This is really like live. I could put this live right now for I would put this live right now.

26:23 were this something I was launching. I have no qualms with this at all. Everything it needs, nothing it doesn't. Scroll the ticket apart and just that is excellent. That really it's like when we

26:36 do our watch test and we say make an exploded view and it even gives specific like tool tips or whatever for the parts of this. And these are all the parts. I mean that right there this this I would

26:47 have no issue putting this live right now. And this is something that again the whole point of this test is okay I'm someone who can build this but like this branding and marketing thing like not me

26:58 and here like here something that would have really been a lot of work for someone very skilled is now in front of us and we have like the personalities here

27:11 with little animations the numbers dimensions in great okay there we see the back of it like Just clean, clean, clean. And some specifications here as well.

27:25 Questions, Q&A. Does it listen all the time? No. Do I need an OpenAI account? Yes. Can I change its voice? Yes. Does it work offline? No. Is the neck noisy? No. What can it do besides chat? I don't

27:36 know. When can I get one? And and just like this this render, this is really say hello, join the wait list, and we'll send one email when Perry's ready. One email at launch. No

27:49 newsletter. This is very very very well done. Brand guidelines. It's a Is this a different page? Give it room. This is like a a how to use our logo in the right way. Eight

28:07 ways to not. That's funny. These are present on like all the sites. If you like go to any website or any company you're familiar with and they'll have like their brand assets, say Nvidia for

28:20 example, and they have dos and don'ts just like this. So, this is really quite something. I would so 100% put this live right now if this was something I wanted to sell.

28:34 So, if you're interested in this, let me know in the comments now. All right. I mean, this is this is fantastic. We still need to test the software and make sure the thing actually works. But

28:45 from a brand design standpoint here, knocked it out of the park. Home run grand slam. Oh, we still have the um what was it? This the ad. And it did the ad both in 9 by6 and 16 by9. So say we

29:00 were going to post this on like Tik Tok or Instagram or shorts, we would use this one. Let's look at the larger one. Hey, Perry. >> Hello. I'm Perry.

29:20 >> Say hello. >> All right. The laptop ran at 99 to 102 Celsius during renders. That's concerning. Hey, what's up

29:41 Hey, what's up? >> Oh, you know, just glowing quietly on your desk like a very committed paperwe. I'm here awake and ready if you want to chat, plan something, or just kill a

29:55 minute. What's up with you? >> Could you turn your volume up? Can you turn your volume up? >> Sure. I'll nudge the volume up a bit for you. Could you change your voice to

30:09 Cedar? >> I can't switch to a specific named voice like Cedar. I can though adjust how I sound a bit like more calm, more energetic, or more cozy and understated.

30:22 If you like, pick a vibe. >> Do you have a settings? Do you have settings I can open? >> Not a big settings panel. No. I can change a few things if you ask, like

30:33 volume, brightness, mood, and timers. If you want more control, tell me what you're trying to tweak and I'll see what's possible. >> Could you enable a timer?

30:42 >> Okay, let's set that up. Just give me a duration and a label if you want one. How long should the timer be for? And do you want a label? Like tea or coffee? >> 10 seconds. And just label it countdown.

30:55 >> Got it. I'll start that short countdown now and let you know when it's done. Timer set for 10 seconds with the label countdown. I'll announce it when it finishes.

31:08 >> Time's up. The countdown just finished. Awesome. Thank you. >> You're welcome. Happy to help. If you need another timer, just say the word. >> So, next up, we have our GPT 6.1 soul

31:27 result. Of course, run on max effort as well. right here. This worked for a little under an hour and 40 minutes. I don't recall how long the Sonnet model worked for um because of the permission

31:38 things. So, just I'm not as focused on time and things like that. This is more of a raw capability test. So, with that, we do have a bunch of stuff right here. So, let's start out with the website as

31:49 we did with the set result. All right. So now we have our moat result or I would imagine that's probably pronounced modi like emoi. Oh, that was a notification in GPT that

32:03 the task was done. I was having it run this website in a local server. All right, so we can see right here it does have a render and interestingly it did also include a little branded logo on

32:13 the device which I do like to see. The render looks a bit funky and the UI is a little offset. However, okay, good. We have nice buttons. I find it odd that this looks so much like the Claude logo

32:27 and and this was the GBT result. The questions, the tangents, the everyday something on your mind. Say it out loud. Okay, this is a decent aesthetically decent front end. What if

32:40 we tried something different? A curious question. A moment of focus. Okay, so it does have some interactivity. Good. I will say the render is a bit odd, though. Interesting. It did also include

32:50 a gray color. Both of them did. So, I'm wondering if that's a sign that I should have also made this with a gray color in mind. Unfortunately, the renders here are kind of jank just in terms of the

33:01 gap there. And there also seems to be some oddities like in the side right here. I'm not quite sure how that's there because it doesn't exist in the 3D models. Nonetheless, this right here,

33:16 this card I do like. Aside from the offset UI and things like that, it does have some specifications as well. Less interface, more personality. Okay, good. Simple controls, a clear voice, a little

33:26 expression in every exchange. Did it just move? Interactive preview. I like that a lot. So, check this out. We can actually play with the UI and interact with it right

33:38 here to see that I think is a very nice feature. I'm very pleased with this live captions. Head motion is off. Very good. And it actually blinks and things like that. It gives us a time. So, we have a

33:50 clock right there as well. Let's talk. This is cool. I'm very very pleased with this like UI mockup. It could be a little bigger on the page maybe, but interesting. It gets like that's like

34:02 Quen Capabra doll like look to it. So that's kind of interesting. All right, that was very cool. Good company. I like this render. Oh, play the film. So that is our like advertising film. And then

34:15 we have some Q& A's. What is Modi or Mo? Probably Modi would be the Okay. Does it remember conversation? Can I make it my own? Ready when you are. And then we have download the press kit

34:30 which I cool renders and then we have some of the renders from the site. Interesting. So I like that feature a little presence AI

34:45 voice companion. All right. Now let's check our result right here. that I like. Okay, I would have liked a bit of voice being that it was advertising a voice

35:10 companion. However, that was well done. The logo here looks not bad. Aside from the claw likeness of it, I do like the orange on black color theme. So overall, this is also I mean they both have done

35:22 a fantastic job just getting this done. So let's now take a look at the live testing of this one. Hey, how are you? >> Hey, I'm running smoothly and ready to

35:41 help. What's on your mind today? >> Could you turn your volume up a little? Could you turn your volume up a little? Can you turn the volume up a little? >> I can't share a live stream of my

35:56 internal reasoning, but I can talk through the gist of what I'm doing. I usually give a quick natural response first, then think through options or explanations and present the result

36:06 clearly. If you'd like, I can explain my reasoning at a high level as we go. What are we figuring out? Oh, not too much. I'm just relaxing. >> Nice. That sounds like a good place to

36:23 be. If you feel like it, we could chat about something light like music, travel, daydreams, or a tiny gold. >> Hey. Hey there. I'm here and ready to chat.

36:42 What's on your mind today? You know, it's just tough. >> Yo, I'm here. What are we getting into? Quick question. Random. >> Wait, can you wait? Hold on.

36:58 >> Not much. Just hanging out here, ready to help. >> Hey, what's 84* 4? >> 84 * 4 is 336. Tell me something special about the

37:13 number 336. >> 336 is a highly composite number, which means it has more divisor than many. >> Whatever nerd, >> it's a math term, not a secret wizard

37:29 society. A highly composite number just means it has lots of >> That's okay. Thank you. >> You're welcome. If you want, we can poke in another number or switch topics

37:41 entirely. >> So, overall, both of these did a very good job in terms of bringing a product from something that could realistically be launched with either of these, at

37:57 least to gather interest for like an MVP. And I want to preface this with that. Now, I don't even really want to give my opinions too much on these tests because they are designed to just

38:07 showcase the model's raw capabilities and some of these things may be subjective. With that said, if I were to choose one that I found to if I was going to put this product out today and

38:18 I needed to pick one of these, like say these were two pitch decks that were shown by like a marketing agency or something, I would pick the Perry one that Claude had made. I found that it

38:28 was overall just a much more refined result here. even from the website where basically we had so much interaction right here. Something like this and then when you scroll all the way down to see

38:39 the actual exploded view that showed up like that. I mean there's no comparison between the level of effort that went through here the render and to be honest with

38:49 you I keep seeing additional things on this site that I was like oh yeah I forgot it did that. This is just a significant it went so much more in depth even just

38:59 with the branding and the UI that say the software on this one was better on the GPT result. It would almost be dumb to choose it because the marketing elements for it were so much weaker. I

39:10 did like this a lot. However, when we compare these websites side by side and things like that and see all of the additional elements and care that went in here, it's really quite striking to

39:20 be honest with you. Now, when we move on to actually discussing like the software and things like that, they both worked fine. They both looked good on there. The one thing I will say that I was kind

39:32 of surprised by was the fact that this Perry thing, you could ask it to adjust its volume and it would. You could ask it to start a timer and it did. Now, forget the timer thing. I asked the GPT1

39:44 to adjust its volume and things like that and it couldn't. Now, there was a pretty easy settings panel to go in and be able to just choose like, okay, I'm going to change this voice. I'm going to

39:52 adjust the volume and things like this. However, I have to be honest with you that I think implementing some form of settings card like this in a round device that's designed to be voice

40:02 interactive primarily is almost like a just a bad design pattern. I will say were to be very judgy. Both of them worked. They both did well, but I find that the sonnet result here massively,

40:15 massively outperformed at the GPT one. The level of effort that it seemed to put into this was significantly greater than what we got from GPT. Both the software worked and things like that.

40:26 However, I'm actually quite interested to see what folks think about this one because it's quite a this is a genuine real world task with something that would be

40:37 very useful for a product like this. So overall, very interesting test and I'm interested to see if anyone would be interested in purchasing Perry. You can let me know in the comments. Um

40:53 but but cool. So this next test is similar to the robot arm test that we like to do in that using an attached camera to the computer, the models are going to be controlling a physical

41:03 object. However, it's something that is easier to fail and we'll just see. So, this is the parallel parking test. And for this specific test, the task is going to be to viewing

41:16 this from a top- down static camera angle, parallel park this little remote control car in between two other little remote control cars. So, a pretty simple task in theory. However, there are a

41:27 couple of factors here that will definitely increase the difficulty of this test. Now, to begin, how are these going to even control a remote control car? because you're not going to use the

41:37 robot arm to press like the controller, right? The thought crossed my mind, but no, cuz that would just be too cumbersome and not very good. So, I have right here, it's an Arduino with a

41:47 little like a frequency generator breakout board. And this is cloned to the frequency of this car. And just through some troubleshooting in one of my Fable 5 testing videos, I believe we

41:59 first looked at this and just actually being able to clone a remote control car's frequency to be used from an Arduino so the computer can autonomously control it. So that is how this RC car

42:10 is going to be controlled. Both agents are going to be given specific instructions on how this works, the specific commands they need to send to move the car forward, backwards, left,

42:20 and right. So, they can then autonomously control this car just using this Arduino and this radio thing, thanks to just like some open- source stuff on GitHub for this specific type

42:30 of RC car. So, it is cool that open- source contributed to this being possible in a much simpler way than it otherwise would have been to decode the specific protocol to control this. Now,

42:41 in terms of why this is difficult, there are a couple of things. Now, this is a very old remote control car, and the controls that we have for this are something called they're they're not

42:53 proportional. So, they're on and off. A proportional control would be I have a little forward joystick, and when I push it a little, the car moves slowly. When I push it more, the car moves fast. This

43:04 is not that. This is on and off. So forward, reverse, left, and right. They're just static on and off. They have no actual fine grain control, which adds a level of difficulty because when

43:18 you need to move a car like this in short, precise movements in a confined space, being able to not modulate any of the inputs adds a level of difficulty. The final thing is this has absolutely

43:30 no traction on the table that is going to be used. So, every time that this car actually gets pushed forwards or backwards, the wheels are going to spin while it tries to fight to get traction,

43:40 causing the car to kind of turn itself a little bit, just like how a real car would do. So, with that summary initially out of the way, both of the agents have received a folder with a few

43:50 different things. One of them being this agent guide right here, which kind of in a way says what we just went over in terms of what this task is and how it is to be performed. They have these

44:00 specifics of how to control the car. the fact that okay the Arduino is using it. Working configurations and values like this is the reference crystal on that little antenna board. This is the

44:10 frequency. This is like the specific frames that need to be sent to send these signals and things like that. So they understand, oh okay, this makes sense. I understand how to do this. So

44:20 to begin, we started with GPT 6.1 soul on max effort which is on this side. I don't know why my mouse was over there. And this was a pretty interesting result because I have to be honest with you, my

44:32 assessment of this was, okay, these things are just going to press the throttle for too long and this is going to go flying off the table. That did not happen. It very methodically moved it

44:41 and then started to figure out, okay, I need to learn exactly how it is I'm going to be able to control this. Let me see how many pixels it moves in the camera frame each time I press this. and

44:51 then quickly it realizes, oh, okay, so I'm just going to perform this method of movement and I'll try to get it in the space. Now, I have to say I was very, very impressed with the way this worked.

45:02 At one point, it got so close to the green car in front of it that it wasn't touching, but it was like, wow, this is actually really taking a large amount of care to not touch any of these other

45:13 cars. This is pretty darn impressive. Now, one thing I will say that is a bit frustrating is the instruction manual these received said that pressing like say with this car in real life, we're

45:25 going to turn the wheel this way and then go forward. Both of these actions can happen simultaneously. In the document in the instructions, it said it's not been verified if that

45:37 works through this emulated frequency for the car. That doesn't mean it does not work. It means it has not been tested. This basically never tried that, which made things a significant amount

45:48 harder for it because it had to do one at a time rather than, okay, I'm going to cut the wheel left and then move backwards, which would have made things simpler. Even with that and the fact

45:57 that it was kind of handicapped by that, it did a fantastic job, it never really got it into the spot before I cut it. I gave both of these about an hour to perform this and I was more interested

46:08 in seeing the methodology with which they went about doing this. So the culmination of that was, okay, now that you've done this, can you please create a little motion infograph for me about

46:18 the specific approach you took and just make it something that someone watching could learn, oh, okay, this is exactly how the agent went about doing this. So we have our 29 second narrated film from

46:28 GPT 6.1 Soul about this, including some screenshots it used and things like that. >> Park a red Tao between two cars. Camera frames guided the agent. Python drove an

46:39 Arduino radio transmitter. Full throttle pulses ended with stop. Then a fresh camera check. Color and geometry estimated pose. A turning model searched for clear paths. Acknowledgements

46:51 weren't motion. Moving the antenna restored response. Uneven travel and a nudge forced corrections. Parking remained incomplete. The method observe pulse stop plan. So that's a

47:03 very very well done infograph and it actually shows us some insight behind what was going on where that section talks about it made a model of estimations of where the car was going

47:13 to move with it >> frames. >> We'll just wait to get there. This right here >> a turning model search for clear paths

47:20 acknowledgements weren't motion. So, the turning model right there is really quite impressive and something you want to see where it's like, okay, I'm going to go at this in a scientific way,

47:30 trying to just figure out, okay, like it moved this much, it moved this much. I'm going to use these values to come up with a simple algorithm to plan out where it will end up if I run a number

47:41 of simulated movements. That's a more efficient way to do this as opposed to just move, take a picture, move, take a picture, move, take a picture because then you can run a ton of different

47:50 potential scenarios instead of having to physically do them. So, it's more efficient. It's like how robots get trained in Sims and things like that. So, this was very well done and I was

48:00 extremely happy with the way this worked. The big downside, I would say, is that it never like actually tried to verify, wait, can I send two commands at the same time? And that would have

48:09 helped it a bit. However, it was very, very well done. And I would imagine given infinite time, it probably would have been able to parallel park this properly. And we can even see some of

48:19 the images it was doing. Here's like a montage of some specific positions it was at during and what it could see because this was a top- down way. So, we can see that at various points in time,

48:30 this got pretty darn close to actually parking it. I don't know that that's really reflected in these images here, but it was very, very well done. I was happy watching it, which was a

48:40 interesting way to spend a Tuesday night or Wednesday night, whatever it was. So, next up, we had our Sonnet 5.5 model. And I have to be completely honest with you, I was expecting this to yeet the

48:51 car off the table as soon as it started. I even had a second camera or a third one that I was holding in my hand just anticipating that because of how bad it did in the robot arm test where it was

49:01 like trying to like crawl across the table. I was happy to see that it did not eat the car. Neither of these actually ye heated the car off the table, which could have easily happened.

49:12 If one of these sent a pulse for too long, this thing's gone off the table. It does not have fine grain control. So, something interesting now that I want to make note of is it actually figured out,

49:23 well, it says that the verification to send two commands at once, so basically having the car turn this way and then moving it, it says that's not confirmed, but I'm going to try it. and then it

49:36 did. The combined codes work with steering signs matching the guide, but turning is weak. So, that actually went a little beyond the instruction and said, I want to try this. And I was very

49:46 happy to see that. That was something I would say that may have been a better piece of this test. Not overall, just a piece that it did better. Then it asked me a question. So, I had to answer the

49:58 question because it said the gap between the green and blue cars is 15% shorter than the red car. I didn't intend to do that, but so physically it can't sit between them. How should I handle it?

50:10 And I said, it can be bump parked, put it in the gap. And then it was like, okay, whatever. And then it kept working for quite a bit of time. We can see it's drawn like bounding boxes and things

50:20 over the car in a very similar methodology that the GPT6.1 Soul one took, where they were taking frames, but they weren't just moving, taking a picture, then moving based on the

50:30 picture. They were actually making little models to simulate where the car would end up with given X amount of movements in X direction. So, it was very methodically done as well. And I

50:40 was happy to see that something like creepy happened here where it was like there's a foot in the camera so the user is there watching and I was like, uh, okay. It was just unexpected. And then

50:51 ultimately after like again a little over an hour, I this is one thing I do want to point out. I stopped it because it was becoming more like sideways. It was not going in the right direction. I

51:03 said, "I need a complete debrief of the methodology taken to perform this task in the form of a high-end productionready motion graphic between 20 to 30 seconds." Basically the same

51:13 thing I sent to the 61 soul result. It stopped me. It flagged that message for potential knowledge distillation. this sometime like uh said it right there details reasoning extraction. So,

51:28 it saw this and thought that I was trying to steal its methodology and reasoning extraction to perform some like distillation attack. And I got mad and I said, "Let me help things like I

51:39 moved the car using this methodology. Here's a picture. Here's why. Here's an infographic that is going on the Bjan Bowen YouTube channel FFS in all caps. Got it." And then it ultimately made the

51:50 debrief video. But like that right there, that's crap. I'm going to say I understand why it's done, but it's crap. So, I want to point that out as a big big negative here. Something that we do

51:59 not see with GPT61 Soul or OpenAI in general. They're significantly less fear-mongerish than Anthropic. So, it almost pains me when the Claude model is like putting the Mac down on the GPT one

52:14 cuz I don't want it to seem like I'm fanboying Anthropic. And in this specific task, there was no clear winner. However, we do have our produced infograph right here from the Sonnet 55

52:26 model. So, this one didn't put any text or anything in it like uh excuse me, there's no voice over. So, we watch this and then we see, okay, some of the graphics here are cool. It put in quotes

52:40 me saying it can be bumparked. And then this right here, I liked. We saw something similar with the GPT one. This one maybe went a little harder in the visuals. things like this. I would say

52:51 the visual presentation here contained more effects and animation. However, the downside was if this was designed for someone to be able to watch and see like how this worked and how it was like

53:04 done, the one with the voice over track is probably better for that purpose. But I would say the animations and the graphics and things like that and the clawed one were

53:15 probably better. So overall, there's no clear victor for this task and there is no clear fail either. So I was happy with that and this is something I wanted to do because it shows us how they

53:27 approach problems, some of their vision capabilities and things like that. Our next test is going to be more of a creative/auudiovisisual task. This is essentially giving them a

53:38 50-second or so rap song that I just made in GarageBand. GPT made the lyrics. I recorded the audio from an iPhone on purpose. And I made the track here in GarageBand, not something like Final Cut

53:50 Pro. The whole purpose here is to give them something that would be like a very amateur setup for a music video. And I've told them, here's what's happening. This is a promo clip for this rap album.

54:01 I gave them a photo of the album cover and things like that to allow them to understand the mood of the project. Additionally to that, this is something where historically to put these things

54:11 together, random video clips, tracks that were kind of low rent recorded in GarageBand and on a phone, you would need a team or at least a couple of people who are very talented with visual

54:22 work. So there is a whole section here where they need to actually edit this video. Additionally to that, the audio side, I've given them the individual stems. So for a bit of reference, let's

54:32 look at this image of the track right here from GarageBand. So a stem would be each one of these individual instrument tracks. For example, this drum track right here would be its own stem. The

54:42 vocal track right here would be its own stem, etc. So I've given them the stems and telling them you need to mix and master this, which means that they're going to have to individually listen to

54:53 these, apply effects to them to make them sound better and a more polished track. Additionally to that, the beginning of this there is it's not fully like rap to right here. It's some

55:03 singing. So, I've told them you need to figure out what key this song is in. And then you need to put some slight autotune in the beginning just to make the singing match the track a little

55:13 better. So, there's two parts to this. One of them is just an audio task where they need to analyze these, make them sound better, and then produce a fully polished, mixed, and mastered audio

55:22 track. The second half of the test is entirely visual. So, they need to edit this video, which is comprised of a bunch of random clips that I just went out and filmed. And I want to see how

55:32 well it can like weave a narrative through these clips along with the song to make a result that before this would have had to have been done by someone with at least a basic level of talent

55:43 with like mixing and mastering, video editing, and things like that. I suppose because this is a different type of video, we will just quickly go through this more verbosely. You were tasked

55:52 with editing the promo, video, and audio for an upcoming album. The album cover is attached as album cover.png, which was created with ChatGpt like a while ago. For the audio portion, the included

56:03 stems are from a 51-second intro track to the album. The first part of the task is to mix and master the stems, which are the individual sounds like of themselves as we saw in that image, and

56:14 vocal track. Currently, there are no effects or mixing and mastering. The project was done in GarageBand. The vocal track was recorded on an iPhone 18 Pro Max and it consists of six

56:24 instrumental tracks and one vocal track. So, seven individual stems as we can see right here. 1 2 3 4 5 6 7. Each one of these being an individual stem. The vocals need to be sharpened, crisp, and

56:35 contain elements commonly seen in rap like double tracked portions of the lyrics where that fits. The first verse is partially singing so it may be prudent to autotune that verse to better

56:45 match the track. You will have to determine the key of the track, but it is 85 beats per minute. So, we did give them the tempo. For the instrumental, treat these stems as more of a

56:54 suggestion than ground truth. They're simple MIDI Maidai instruments, and you may add or replace or do whatever you would like to make the instrumental sound more polished and well done. I

57:04 have included a screenshot of the GarageBand project. If you think you can produce a better individual stem or sound effect or a whole section of this instrumental, you may programmatically

57:13 do so and incorporate it into the final mix. Actually, what we see right here, so these individual notes that are basically laid out right here, these can very easily be programmatically created

57:24 with different sounds just by these AI models using Python or something. It's something I've done like even many years ago at this point. So, for a while, they've had this capability and I wanted

57:34 to give them just a photo of it to be like, "Oh, wow. This is actually really simple. I could recreate this and I wanted to give them that option if they thought they could do better. The

57:41 deliverable here is a completely mixed and mastered track with all of the above taken into consideration and put into effect. The next component of this is the promo video. It currently consists

57:51 of a number of clips recorded for this video. It needs to be completely edited, put together to match the audio, and made to look like it was worked on by a team of high-end editors and motion

58:01 designers. Use your judgment to produce the best possible version of this using the available clips. You will have to splice them and determine which ones you wish to use. Below is a casual

58:10 description of how parts of the video are envisioned. The intro of the song with no vocals can be completely up to you. Verse one should be used with the intro clip as the person getting out of

58:20 the car and rapping the first verse matches these lyrics. It then ends with the person quickly looking to their left which would make a good transition or leadin. There are no other specific

58:31 clips/verse matches proposed. So use your discretion. The other linked clips are the following. This clip right here, which is just basically like panning up to the interior of the car or the

58:41 window, should smoothly transition to myself and the stuffed capabra, which is like the little Quen plushy in the interior, making it seem like this was one transition clip through the tinted

58:51 glass to the inside of the car. So this is actually something that I want to showcase real quick. So this clip right here, we're basically telling it, okay, I want you to make it look like this is

59:01 panning in and then immediately like the window then gets like moved into in the camera and then you cut to the interior where myself and the Capabra doll are. So that is this one right here. So

59:14 basically we want one static clip cut together just like that between these two splice to make it look like it was one fluid clip. Additionally, the clip TVcene.mmov should show part of this.

59:26 So, there is a clip right here where it's like one of the TVs in the visor is on. And I've said that I would like it to essentially superimpose a short clip. In this case, one from one of my videos.

59:40 So, it makes it look like I was actually watching this on that Visor TV. And then basically a bunch of other clips. I told it these are just random ones. You can use them at your discretion. Whatever

59:49 looks best and weaves a cohesive thread can be included. I do not just want the clips spliced together. I want something that incorporates high-end motion design, editing, and effects to make

59:59 this look like a real produced demo and not an iMovie project. So, here's the initial reference track. And we won't listen to the whole thing right now, but just to get a feel for like its sound

1:00:09 and stuff. Matrix multiply rows with the columns. Neural net brain every move got a model to clean make the whole room follow blue throttle waste trained while the bank

1:00:24 rolls attention on mess. >> So that's basically the entirety of what we're giving it here and the task at hand. So we're going to run both of these on their ultra modes. In the case

1:00:36 of Sonnet 55, that will be using ultra code. And in the case of Soul 56.1, that will be just on the ultra reasoning effort. So in 54 minutes on ultra, GPT61 soul completed this task. Now I think

1:00:49 the best way to do this is probably just take a look at the ProRes highquality video master. All right. And I'm in the matrix. Multiply rows with the columns. Neural net brain. Every

1:01:07 move got a model. Prompt to clean. Make the whole room follow. Blue range roll tinted glass for throttle. Waste get trained while the bank roll scales. Attention on me. I don't need no rails.

1:01:19 Transformer talk. Turn a thought into sales. Influencer status. Blue check in the details. Dot product drip yet effect. Feed forward money. Every layer look fine. Algorithm knows when I'm

1:01:32 about to go live. Whole feed freeze when the rover arrive. I got AI on a dash. The future in motion matrix multiplication stack numbers like tokens. They chase engagement. I control

1:01:44 the commotion. Influence so high even show devotion. Like all right, that was the mix was bad. That was a bad mix. I could have done better than that with

1:01:57 Garage Band. I'm going to be harsher on their musical stuff in terms of how they mix. It didn't really seemingly do much of anything with the track. It kind of sounded exactly like

1:02:09 the source, including selective doubles, so it would have put the remix. >> Yeah. That transition right there was good.

1:02:35 >> That was decent as well. The big thing I have is the voice just sounds so tiny. The lows aren't great. It basically sounds like a It's just not all there musically. The video is not

1:02:48 bad. It did overlay text and things like that. I would have liked a bit more. I mean that. Okay. Yeah, that was pretty well done. I was pleased with that. It would have had to like key frame that

1:02:59 and place it because it was not flat. So like actually even to there we see that's not a bad job. Transitions were okay. I don't know that it really changed any

1:03:12 of the coloring and the clips and things like that. So overall, I am not 100% pleased with this result to be honest with you. So in I believe around like an hour and 50 minutes. What

1:03:25 time was this sent? 2 hours ago. So let's call it 2 hours, but I'm pretty sure it was a teeny bit less than that, Sonic 55 has finished with its music video and also the remastered clips. So

1:03:36 let's just take a look at it. The video with the mastered audio and about 45. Okay, good. So that is the specific one we want to look at. Let's just take a look and see. Oh, this computer doesn't

1:03:47 have much if anything on it to play videos. That's my mistake. I'm in the matrix. Multiply rows with the columns. Neural net brain. Every

1:04:05 move got a model. Prompt to clean. Make the whole room follow. Blue Range Rover tinted glass full throttle. waste get trained while the bank roll scales attention on me I don't need no rails

1:04:18 transformer talk turn a thought into sales influencer status blue check in the details dot product drip yet effects online feed forward money every layer look fine algorithm knows when I'm about

1:04:31 to go live feed freeze when the rover arrive I got AI on a dash the future in motion matrix multiplication stack numbers like tokens they chase engagement I Control the commotion

1:04:43 influence so high even by show. All right. Well, I have a few things to say. That was much better. All around better, but let's slice these individually.

1:05:03 So, this made a significant amount of change to the actual instrumental. There were a bunch of sounds in there that I personally did not add. So, this added in some bells, some chimes. I heard the

1:05:16 bass hit harder at certain points. Additionally to that, the overall video there was definitely higher quality. There was actually text like behind me in the video. So, that was a

1:05:27 significantly better result just all around. I would say if I were to like like that's kind of a weird >> multiple

1:05:53 >> it it doubled the vocals right there and you can actually hear like an undertone from the doubled vocals just very low. Let's just look at this like statically. certain parts of it. It definitely

1:06:03 colorized it and things like that. And I mean the the window to inside transition I almost the sole result may have done a better job on that one.

1:06:17 >> I don't know. Now, this is something else that you can kind of see some of the differentiation between the two is when we actually look at where that video is transposed onto

1:06:33 the uh Visor TV monitor. When it starts to move away, you really like Sorry, I don't I've never used this program before, but it really keeps it there like key framed

1:06:47 when it starts to move away. It doesn't like get weird. It did with the GPT results. So, and this one seems to have applied like a I know how silly this is, by the way.

1:07:01 This one seems to have applied like some coloring overall. There was like the glitch effect right here. Buffering feed frozen. That was pretty cool. When the car pulls in,

1:07:14 on the dash mat. >> There's also some like interesting other sounds too. Like there's there's an organ, but there's also sound design beyond that. Like there was some like

1:07:29 sound right here. >> And even just that right there. So And then this both of them did this. I guess that makes sense because I told them it's for a promo for like an album.

1:07:51 So interesting though. And I would say this I think did a significantly better job. So overall that is going to conclude our head-to-head between GPT 6.1 Soul and

1:08:03 Claude Sonnet 55. Now Sonnet and Soul were not previously competitors when it was 56 Soul and Five Sonnet. However, they're now the same price. And I know a lot of folks want to see, well, if the

1:08:16 Claude model is using four times as many tokens, it is then not the same price. I understand that. I'm I my interest in that for this test was zero. My interest is does this thing actually stack up

1:08:30 capability-wise to the sole family of models for GPT61? because I was personally kind of disappointed to see that GPT61 Soul didn't necessarily seem to be

1:08:43 outperforming Sonnet 55 and Soul should outperform Sonnet. It should be a massive like destruction and that's not the case. So, I wanted to do this to see which one of these is actually better.

1:08:56 Is there a clear winner here? And partially because of things like that, like reasoning extraction, this message is flagged. Writing fear-mongering articles about AI is bad and and this

1:09:07 specific model is bad. I hate saying this, but Sonic 55 in this testing absolutely mopped the floor with GPT6.1 Soul. It's kind of I'm shocked. I didn't expect it. With that said, Soul is a

1:09:21 good model. It's a workhorse. It helped me basically just port all of the specific Raspberry Pi apps from the test where we had Perry and the other one, the little robot thing. It helped me get

1:09:34 those on there. It works well. It works fine, but honestly, I feel like it's a model that has imagine its capability is like a circle. I feel like it's being restricted to a smaller area of that

1:09:47 circle and then served to us. It's just it's it's kind of apathetic. And I noticed some laziness like with the first RC, not RC, the destruction derby test, it was like, yeah, I didn't do

1:09:59 this. And it was like, okay. So, it just And with the the music test, it was absolutely destroyed by Sonic 55. The mix was worse, the editing was worse, it didn't

1:10:11 add in any extra sounds. It seems like it does the absolute bare minimum. And I don't I can't believe that is the actual behavior of the model. I'm more inclined to think that is like a restriction

1:10:25 placed on it for more efficiency or something like that. So, it's tough. But in this testing, in these specific tests, I found that Sonnet 55 wiped the floor with soul. It may have taken three

1:10:37 times as many tokens and been more expensive. But my specific interest in this video is which of these models is better. And based on these tasks, if I had to go do something right now, I

1:10:48 would be using Sonnet. And because they're both being run on the $200 a month subscriptions, I think that's a reasonable thing to assess. So

1:10:59 overall, not the results I expected at all. And again, it just goes back to I really feel like 61 Soul has more capability, but it's almost like the reasoning like the juice that it will

1:11:10 like really go hard on a task is set very low. So, overall, that's going to conclude a head-to-head that I never expected to make a Sonnet versus Soul test. Overall, I had a lot of fun making

1:11:23 this, especially with the music stuff in the videos and like whenever I can go out and film like silly stuff like that, I get very happy. So, that's going to conclude this testing. If you have any

1:11:33 questions, please leave them in the comments. Also, I'm very interested to see what folks think about these results as well. So thanks for watching and till next time subscribe.