Codex's Browser Agent Automates Literally Anything

0:00 Codex's browser use is probably the best that I've ever tried before. It can literally do anything on a browser or on your actual local computer. And if you need to be signed in for something, you

0:08 just sign in once and Codex will save it. So, the possibilities are endless. You can actually automate anything. Now, let me show you guys how to set it up, how easy it is, and a few good use

0:16 cases. All right, so I'm using the Codeex desktop app, which is just honestly the main way that I like to drive Codeex. It's super super slick, and we get the browser to pop up on the

0:23 right hand side, which I'll show you guys in a sec. So luckily we live in a world now where most tools that we need to use have an API. So it's much easier to just connect codecs through the API

0:33 or if we just go to the plugins there's hundreds of plugins in here. But every once in a while we don't have an API and you can do that with browser use. But before I show you an automation like

0:40 that, let's just take a look at this example where you might be building a website, you might be building some sort of app and you need a browser, an agent to actually test through things. If you

0:49 guys watched a recent video I did on like building an AI SAS in one day, then this was actually from that demo where I had a browser agent get spun up and test the heck out of our app and it did 85

0:59 focused automation checks with real browser workflow testing which means it was clicking around, it was typing things, it was hitting buttons and it was trying to break the app and it found

1:07 a ton of bugs and then we were able to just fix them. So in this example, I have a super simple website spun up here on the local host. It's a form submission as you can see and there's a

1:14 few issues before we actually go ahead and start testing this thing. You can see like this spacing is weird. This thing is overlapping. And so what you can do is you can click on this annotate

1:23 button and that lets you choose very specific elements to actually change. So I can click right here on this phone number box and I could describe, hey, this is kind of like overlapping. It's

1:32 out of bounds. It's covering up the company box. So I need you to fix this. And then when you shoot that off, it actually goes over here into your chat. And then you can make another annotation

1:40 like, okay, so this whole box isn't really aligned on the same y-axis as number one and number three. So you need to fix that as well. And then those annotations are saved over here and we

1:48 can just go ahead and shoot them off. So that's kind of like a lovable style or like a claw design style thing that you can do with the inapp browser in codeex. What else is really cool about this is

1:57 that you can actually just drive this thing. So you can go to Google, you can search for Grand Canyon images, you can go to all, you can go to different links if you want. You can use the browser

2:05 right here while you have Codex open. But of course, Codex can also just take control of it. Now we can see right here is my mouse, but also this is what Codeex's mouse looks like. And that's

2:16 how you know it's now sort of getting ready to take control of the browser. Okay, so now that those changes have been made, let's actually start testing out our UI. I want to see things that

2:24 maybe me as a human, I would maybe go through here and test like five or six edge cases, but we can see how codecs can do it way quicker and way more. Okay, so you have the UI open. I want to

2:34 make sure that there are no bugs. I want to make sure that this thing can't break. So what I want you to do is use your browser use and just test the heck out of this thing. Try different things.

2:41 try your best to break it and let me know what bugs you find and what things we need to fix. And so, by the way, while this is running, there's a difference between browser use being

2:48 headless or headed. Headless means it's running in the background. So, you can keep working in the foreground and you won't actually see it going on, but it can run a lot of things in the

2:56 background. But, if you want it to be headed, that's basically where you're able to see it. So, we're probably going to be able to see some things right here. But, if you wanted this to just

3:02 work for a few hours for you headless in the background, you could ask it to do so and then it wouldn't disturb whatever work you're doing. So, if you ever hear that terminology, that's what it means.

3:10 So, there you go. You can see that the mouse is moving around right now. It tried to click continue and then it says first name is required. Enter a valid email address and enter a 10-digit phone

3:18 number. And now you can see it just entered in some information and it was able to hit continue. And it's just going to keep going through this QA check. We're

3:26 validating that some of these fields are required options and that the continue button won't actually let it move on without filling those in. And by the way, the reason you're seeing some of

3:34 these UI bugs is because I asked this chat to like make there be a few UI things to fix because I wanted to show you guys the annotations. So just ignore that. Mainly what I'm trying to show you

3:43 guys now is the actual browser use QA function. So here you can see as it's clicking around, as it's trying to navigate this UI, it already said the core click path mostly works, but I've

3:52 confirmed two real data integrity problems already, which is that obviously invalid contact data does get through and changing the country code to UK resets to US when you return to edit.

4:02 I'm now testing keyboard submission and restart behavior, which often exposes bypasses the click path misses. So, think about it like this. No matter what you get, you're going to want to test

4:10 it. But would you rather have something that has been not tested at all, or would you rather have something that has been tested 100 times already by an AI agent? Especially if you tell the agent

4:19 to try to break it, it's going to find some things that need to be fixed. No matter how good you are at trying to predict edge cases, when you push an app into production and users start using

4:28 it, people are weird. People are unpredictable. So, they're going to do things. And so this agent is able to hopefully simulate weird humans and just find things that you might not have

4:36 thought of. You can see here it even switched this to mobile view to still make sure that everything's working. And look at this in mobile. It found a few failures in responsiveness. So that

4:44 isn't something that I yet had tested. Okay, so this thing finished up now. And you can see that we have a bunch of things that passed but a bunch of things that also failed. So it was able to find

4:52 all of these bugs because it was testing the UI. And it didn't even do it that long. If we wanted to make this even harder, we could do a /go goal and we could say, "Hey, don't stop until you've

5:00 tried a hundred different unique scenarios of, you know, UI edge cases and then come back to me." But it still found a lot of things that were broken. And now all we'd have to do is say,

5:08 "Okay, cool. Go implement those changes. Go fix all those bugs." Okay. So, let me show you guys another example. This is when you maybe have some sort of platform where you want to be able to

5:16 like download statements or something, but they don't have an API. So, you can actually interact with it with Codex's browser use because it basically uses vision and then AI to analyze like what

5:25 it's looking at and where to click. And I would I just want to show you how easy it actually is to build these. All I said was, hey, you know, I want to build this skill, blah, blah, blah. What would

5:32 be helpful? And you could record a quick Loom. You could send screenshots with arrows, but you actually don't have to. You can basically just describe pretty clearly what you need to do. So, what

5:41 happened here is it opened up Relay. I signed in and it saved that session. Now, I said, "Okay, in the Relay dashboard, you're signed in. what I need you to do is go to the dashboard, click

5:49 on accounts, find the statements, um, click on these two accounts, download them as CSVs, and then put them into this folder. And it was able to do that on one shot. I didn't have to give it

5:57 any feedback and it downloaded them and saved that whole flow as a skill. Now, this is super helpful because now I could at the end of the month just set this up as a scheduled routine because I

6:07 could set a schedule routine here and say, "Hey, use this skill on the most recent, you know, month statement in Relay." And for the most part, Codex is going to save your sessions when you're

6:16 logged in. However, there are some limitations like the fact that Relay is going to pretty much boot you out because it's, you know, it's a banking platform. It's going to be secure and

6:25 boot you out. So, look at this here. I asked it to do this again and it said, "Hey, Relay's open at the login screen, but you have to sign in again." Now, what's really interesting that you can

6:33 do is you can actually open up your settings. you can have a password manager inside of Codeex and you can see this goes to your browser and now you can go to import them so that Codeex can

6:42 actually fill that out for you and that way you're not putting it into the chat history or anything like that. You're saving it more securely. So all you have to do for that is create a CSV file that

6:49 looks like this. I've got the name, the URL, the username, the password and any notes which in this case are empty. I imported that here right here as a file, a CSV file. And now we have relay.com

7:00 password. Go ahead and try this again in the password manager for the inapp codeex browser. You should be able to log in now and then run the skill on June. All right. So over here now you

7:09 can see the browser has taken control and it is going to put in my email and my password. Now if this asks me for a two-factor authentication code for my phone or something, then I'm obviously

7:18 not going to be able to do that. I'd have to do that manually. But hey, if I was on a Mac and my text went there and I used computer use to do that, technically you would be able to

7:26 automate that whole thing. But I would say you probably want to be careful about stuff like that. But as you can see, it was able to pull the passwords from my password manager, paste those

7:33 in, and now it should be able to log in to relay for me. You can see here that the mouse has moved over to accounts. It's then going over to statements, and it's going to be able to download these

7:41 two statements for us. So, you can see it's pretty cool that it was able to figure all this out. Remember, I've only ran this skill once, and all I did was instruct it with natural language to do

7:49 this. Now, realistically, is that the way you should do it, especially with something like a bank account? No. What you should do is have it do the skill. you sit here and watch it and have it do

7:59 that like at least 10 times where you're very closely watching it to make sure nothing goes wrong and you're making sure the script in the skill is super strict because at the end of the day

8:08 it's using vision, it's using AI and that's how it's deciding what to click and what to do. So really the way that I think about it is when you want to automate something your first call

8:17 should always be to go for an API. Like that's what you want to do. It's the fastest. It's usually the cheapest. It's usually just the most consistent. Now, if there's no API, that's where you

8:26 would maybe say, "Okay, cool. Well, now let me see if I could use like a deterministic, you know, like macros script, right? And that's where you'd basically be able to have, yes, you

8:36 know, it'd be more of a browser use because you have a mouse that's clicking around, but it's not like your mouse needs reasoning. Your mouse doesn't need to look at the screen, figure out where

8:44 to click, and based on the next UI that pops up, where do I click again?" This is basically more like, okay, I have to click on this pixel, then this pixel, then this pixel every single time.

8:54 That's something that you could do for much cheaper and much safer by just programming a macro. And that's not that hard either. You just ask Claude or Codex, hey, help me figure out how to

9:03 make like a deterministic macro script to just click on these pixels in that order every single time. But if you can't do that every single time because there is an element of vision or

9:11 reasoning, then you need to come in here and you need to do something like some browser use. And luckily, Codex has your back. I've tested a lot of different apps and different CLIs and different

9:20 things for browser use. And so far, Codex is just really solid. So, as you can see now in my downloads, this is the statements that it just downloaded as a zip file and it basically took these and

9:29 it put it into my Herk 2 project. If I go in here and I scroll down to statements and I go to um probably June. Yes, June. You can see right here, it put those in here in the correct spot.

9:40 Okay. So, that was an example where we needed to save a password somewhere, right? But I've done this with things like school. I've done this with things like X. I've even done this with things

9:48 like Instagram that don't make you constantly resign in because it saves that session. So, let me show you another example. I'm going to go over here to YouTube and I'm going to grab

9:55 the URL from the video that I uploaded earlier today. I'm going to go back into Codeex, paste in that URL, and say, I need you to turn this YouTube video into an X article for me and then use browser

10:06 use to go ahead and um format it and put it into an X article as a draft and then I will go ahead and review it. So, I know some of you guys are probably thinking, why don't you just use X's API

10:16 for that? You can, but it's a little bit weird with the formatting. It's also weird with sometimes adding things like media or thumbnails for some reason, at least with my testing that I used to do.

10:25 And it's also it charges you, right? Because you have to pay X per usage in order to do something like that. And so, because this is just part of my CODC subscription, I can just run this skill

10:35 and it will go ahead and format everything for me and take care of it. I even have this skill worked out where it can go ahead and open up my YouTube videos and it can take screenshots of

10:42 specific moments and put those into the article too where it makes sense. So obviously this is a bit of a loaded process because it has to do a few steps before it actually has the article ready

10:50 to put into X. So while that's happening, let's just open up a new chat and let's see if our X session is there. Can you open up a browser? I want to see it right here in the app and just go to

11:00 X and scroll around a little bit and show me if there's anything new that has just come out um like in the AI space. just scroll on my X feed. And so because this was a brand new session, we can

11:09 actually see if X is already signed in and if it's able to interact with things. Cool. So it just opened up X. Let me get my mouse out of here. You can see

11:20 clearly that this is already signed in as me and my feed is visible and now it is going ahead and it's starting to scroll on the X page. So just start to think about already how could you create

11:29 some sort of skills out of this and then schedule these as tasks. So maybe every morning if you wanted it to scroll through Instagram and things like that or scroll through your own profile or

11:39 mark things as red or unread, you could totally go ahead and do that no problem because you can set up the schedule tasks right in here and use browser use skills that you've already set up. So

11:47 you can see two things that I already noticed was Grockbot and Grock 4.6. So this is an interesting case though because there is an API but using the browser use is actually cheaper. So

11:57 that's why in this case we decided to do it and test it out. Now, what about computer use that you guys have also probably heard? These are basically the same thing. It's still going to use

12:05 vision. It's still going to use AI. But the difference here is that the browser is basically things that are on the internet or things that you would type into a browser like Chrome, but the

12:13 computer use are things that might be on your actual computer. So, changing your computer settings, opening up desktop apps of things. So, let's go back into Codeex. While we're waiting for this X

12:22 article to finish up, let's see how this is going. It is still breaking down and turning it into an article. Let's see if we can do some computer use. So, one thing you will have to do though is

12:30 you'll have to go to plugins and you'll have to actually install computer use, which is just as simple as typing in computer use and hitting install like literally two seconds. And then, hey

12:38 Codex, I need you to use computer use to open up my desktop app for glido. And I just want you to go to the settings and turn on the ability for it to allow and use like Bluetooth devices. So, just go

12:48 ahead and get that set up for me. And so, this is a little bit different, right? Because it's not going to open up your inapp browser. It's going to actually kind of show your screen being

12:56 taken over. It said, "I'll stop if Windows asks for an admin password or presents a security sensitive permission that it can't safely interpret." It shouldn't run into any of that, but it's

13:05 good that it sort of has that guardrail baked in. Interesting. Okay, so it said Bluetooth access is a privacy permission, so the computer use won't let me actually do that. But what it

13:13 said it's going to do is open up Glido. As you can see, I didn't open that up. It opened up Glido for me. And it doesn't look like on my recording you guys can actually see this right now,

13:20 but my screen kind of has like a big blue hue around the edges. So you will clearly know if Codeex is using your computer. And there you go. You can see that it

13:34 actually turned on the include Bluetooth devices button. And I think it got confused because I think at first it thought that I was messing with my actual computer settings rather than

13:42 like an inapp settings. So it thought that that was a security thing. But it looks like it figured that out on the way. And I think the computer use thing is pretty interesting because you can

13:50 remote into your codecs from your phone. So, I can control all of these sessions on my phone. And what I could do is on my phone, I could say, "Hey, computer use, go ahead and change my settings or

13:59 go ahead and do something on a desktop app that's on my PC that I'm actually not sitting in front of." And then you could control your computer from your phone. So, anyways, we're back in the

14:08 session where it's creating that X article. You can see that it is actually creating some thumbnails for the actual article. So, hopefully we're getting close to the point where it's going to

14:16 actually open up X and start to put stuff in there. But, I'll check in with you guys when we get there. Okay, so it just decided to open up X. You can see that it's going to the drafts in my

14:24 signed in X and then it's going to go ahead and format this new article and then we'll take a look. It's clicking on the plus button. Man, I think it even just like resized the page a little bit.

14:33 It was getting a little picky. In goes the title and then it just pasted in the entire content of the article. And now it's uploading this thumbnail, which I don't think looks very good. That part

14:43 of the skill has not been refined at all. But it's still cool that it was able to make that thumbnail and then go ahead and put it in there. So, I literally just have to make sure that it

14:50 looks okay and I like it and I can publish it. Now, I know what a lot of you guys are probably thinking when it comes to this browser stuff. It's like, what's the point, Nate? Like, where's

14:58 the value? And honestly, I understand where you're coming from. And that's why I wanted to show you guys this piece where it's like pretty much API will handle 95% of your automation things

15:06 that you need. And then a lot of the browser use opportunities might just be a script, but then when you need it, if you have this in your back pocket, it's good. And you also might think, okay,

15:14 cool, but like how much time does this really save? This probably saves me like maybe 5 to 10 minutes like realistically. But what's cool about it is that I can basically have this

15:22 pipeline now where I have a bunch of these browser use skills. And so when I need to do something, I just open up codeex, I shoot off that skill and I can go back to the other task that I was

15:29 working on. It kind of just like helps you do more without context switching as much. And as you start to stack more and more of these pipelines and combine them together, you can get some really cool

15:38 things to happen when you start to chain together these skills. And look, this one actually did it. It basically did the thing where I said it opened up the YouTube video. It took screenshots of it

15:46 and now it's going ahead and it's planting some screenshots from my video in the article. So, if that doesn't prove the capabilities of this thing being intelligent, take screenshots,

15:54 look at the page, know where to click, know what to do, then I don't know what else will. Besides the fact that I've had this thing play chess, I've had it play games. I've had it set a world

16:01 record on Tetris. It can do a lot of cool things. Look at this. You can see that it even took the image and drugg it all the way down to where it actually fits in the

16:11 article. I haven't actually seen it do that before live. That was pretty cool. And this thing is just still going at it. But that is going to do it for today's video. Hope you guys enjoyed.

16:19 Hope you learned something new. And if you did, please give it a like. It helps me out a ton. And as always, I appreciate you guys making it to the end of the video. And I'll see you on the

16:25 next one. Thanks everyone.