Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Browserbase's auto-browse loop applies Karpathy's auto-research idea to browser agents, using a doer agent and a teacher agent that iteratively improve a strategy.md file until the task converges. The key output is a skill.md file that persists the tricks and learnings from previous runs, making future runs cheaper and more reliable. This approach can reduce costs by up to 16x and is now productized as browse.sh, a free library of skills for common websites.
Key points
Auto-browse loop uses a doer agent and a teacher agent to iteratively improve a strategy.md file.
The final output is a skill.md file that persists learnings for future agent runs.
Cost savings from auto-browse can be up to 16x cheaper on some tasks.
Browserbase has productized this as browse.sh, offering free skills for many websites.
The technique requires a replayable environment and a verifiable outcome.
Tools mentioned
Techniques
- Auto-research loop
- Doer-teacher agent architecture
- Strategy.md artifact
- Skill.md generation
- Query parameter optimization
- JavaScript injection for drag operations
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
All right. Hello. Thank you everybody for being here. Uh I know it's the last workshop of the day. I appreciate all of you turning up here. U I am show uh I'm
here to talk about hill climbing skills. uh specifically how can you improve your agents without really going into the models without really going in the weights. Uh I have a slido uh up here as
well. I'll just uh flash the QR code in case any of you want to uh put in any questions and we can address them towards the end as well. U a little bit about me. I'm Sh. I'm uh on the sales
engineering team at browserbase. Uh I essentially work with our customers deploying browser agents in production. Uh I'm not sure maybe we'll do a show of hands. Uh how many of you know what
browser agents are? Anybody? All right, a few. Amazing. Uh have any of you built browser agents before? Have any of you used these tools? Show hands. Okay, a few of them.
Amazing. Nice. uh maybe I'll I'll talk a bit about what browser agents are before diving into the whole concept of hill climbing for skills uh so that it becomes clear what we're doing towards
the end of the talk we'll also look into how you can do this in your domain in your uh field as well. All right. Uh so quick primer on what browser agents
are. If you have an agent that needs to drive a browser, you basically call that a browser agent, right? Browser becomes the tool to your agent. And how do you control it? You control it with some
sort of CLI, some sort of MCP, some sort of method where you can control the browser and say, "Hey, go and click on this red button. Go and click on the refresh button. Go and uh take a
screenshot or take a snapshot of the accessibility tree of this website." Right? So these are this low-level tools that your agent can have access to to control the browser. Uh and browserbase
is essentially a platform for you to build and run these browser agents. We provide the infrastructure and the SDKs, the skills, the CLI that go with building these browser agents. Uh and
and we're going to be using all of these tools to see what hill climbing does look like. All right. Uh I want to talk a bit about the biggest problem with agents today
essentially reliability right uh people are frustrated with browser agents getting stuck or getting into an anti-border capture issue uh or maybe just forgetting or being lazy and not
completing the task. Uh and we see reliability essentially is an engineering problem. I think the recent models the Opus and the Fable when we when it came out for those three days it
was really really smart right uh but we see that a lot of the work that has to be done for reliability is the hardness around it is the tools that you give it it is the skills that you give here u
and and that's what we're going to be sort of looking at how do we improve the reliability of these browser agents I wanted to give you all a few examples of like where these agents suck today,
right? If I zoom in a bit, you'll see this browser agent is trying to like go on Google flights and click on the uh it's trying to drag the time filter and it's unable to figure out how do I drag
that. Dragging is a tricky operation for these agents to do and it's getting super confused. It doesn't know what the HTML element for that is, what the JavaScript for that is. It's not really
able to do this and it gets stuck and it's not able to book my flight tickets. Uh, another example I wanted to show uh is a browser agent on open table. Let's say I have my open claw or my Hermes
agent and I need to book a date night and I tell hey go to open table and and find me some and and look at the anti-bot that it runs into, right? Uh, and it's just like access denied. I
don't know what happens here. Uh, but like can my browser agent get around these issues? Can they be more reliable? And can they book my date night restaurants is the biggest question I
wanted to ask here. uh and this is where we sort of look at different ideas to improve the reliability of these agents and one of them uh oh actually let's let's dig into the root cause here right
like why this is happening u we have like these opus models these five five the super nice general purpose you might have used codeex computer use very nice but when you want to sort of do uh
longunning tricky websites like the open tables or the Google flights it sort of becomes very difficult to run these in production and these sightspecific
tricks or these javascript that you need to run on your browser the agent doesn't know about them right and the most important thing is it's every run starts from zero you're not really having any
sort of learning baked into your runs I think Darkh talked about this a couple days ago it's like why is progress on computer use still so slow today. Uh although computer use seems like a
verifiable problem, why is it still not solved? Essentially, there's no learning happening. Every agent run is a start from scratch run and it's not learning from its production traces. U and these
browser agents, I call them geniuses that never learn. Um and some people might say, oh, why don't you fine-tune on something some like open table or restaurant websites. Uh but it it's not
really scalable, right? uh if you're thinking of building agents, you can't really be fine-tuning for every new task that comes up. You want some you want a a modality where it can learn from its
production traces uh and get better over time. This is where we sort of started thinking about and this was like six four to six months ago where Karpadi
came up with this idea of auto research. This is essentially a loop, right? Uh and this loop sort of takes a verifiable problem statement and goes at it, tries it again and again, looks at its own
traces and tries to improve itself. And we wondered how can we apply this to browser agents? How can we apply this uh to these complicated workflows, these complicated websites which are not
deterministic, they change every day, they have different captures every time. Uh how can we learn from these? Right? Uh this is where I want to sort of switch into a demo and sort of show this
live. Uh and I'm going to start with uh a prompt over here. So I want to maybe do something like uh go to Google flights and uh find me a business class flight from SF to New York for this
Saturday. I want to have one carry-on bag. Uh and this flight should be in the morning. I hope my Okay, this is correct. Just paste this and run this. Right. So
starting a browser based session here. Uh so what you see in the center is a remote browser has just spun up and has navigated to Google flights. If I expand my agent here, uh it's doing all of
these things. This is the reasoning traces that you can see live and it's trying to sort of apply the business class filter. So it it'll do its job and and and we'll come back to this. Uh but
I wanted to talk about what's really happening here. Uh u and the architecture behind this auto research loop. So we we fired the Google flights task.
Uh and and and the architecture here is this auto browse loop that we call this. We took auto research and we took browsing. We call it auto browse. And this loop essentially is two clawed
agent SDK loop uh agents. Uh you can call one as the doer agent and one as the teacher agent. This doer agent its job is to actually go and do the browsing. It has access to the browse
CLI that we talked about. It has access to the browser uh and it goes and sort of clicks on buttons, looks at the screenshot, tries to get me information and the teacher teacher's job is to look
at the traces of the doer and learn what went wrong and what went right and they keep working together until they find a very good solution that converges. Right? Uh we don't tell
it how many loops you need to go for. We don't tell it what's the success factor success uh rate that we want. We don't tell it hey we need to do this within x number of tokens or x number of seconds.
We just give it a broad prompt and we let the bitter lesson we embrace the bitter lesson and we're like okay let the doer and teacher go at it. Uh and the key thing that happens is there is a
strategy.md file. uh that is essentially the artifact across multiple runs. After every iteration, the teacher thinks about what happened, looks at the screenshot, looks at the logs, and then
thinks, "Oh, you did not click on this button. U you should actually click on this button instead, and it puts that in the strategy.mmd file." And that artifact carries over to the next
iteration, and it improves from there. Maybe we'll go back to our demo and see what happened. All right. So, iteration number one, uh, there was a failure here. Uh, and the learning from this
iteration was skip the entire form calendar widget because you can just do a query parameter and you don't really need to click around, right? The agent was smart enough in this instance to
think, oh, I'm wasting so many tokens and I'm wasting so much time clicking around where I could just use Google Flight's query parameter. Now this is obvious in hindsight but maybe as a
human I wouldn't have thought of this and I wouldn't have written this complicated query parameter myself right uh so this is the learning from the first iteration now it's going and doing
things in iteration number two it'll keep going on till it feels like it has maxed out all the tricks it has maxed out all JavaScript hacks that it could do uh and and and then uh think about
okay this is the skill that I can use from here. I think the key the key thing uh that I wanted to also talk about is at the end of this converged loop the output that I
want is a skill.mmd file. This skill.md file would be what I give to my future agents. Uh and it would have a collection of all the hacks or all the tricks or all the learnings from
previous runs. We'll we'll look at what that scale might look like. Uh it could look like, hey, a pre-filled URL is mandatory. Don't go around and waste your clicks and tokens clicking on these
buttons. There is a reopen bug in the calendar. Don't do this. Uh if you want to move the drag uh if you want to drag your time stamp thing, uh use JavaScript injection. Don't try to do it with the
mouse. Right? These are the kind of tricks that are now persisted for every future run and gives me so many more savings uh from a performance, reliability and cost perspective.
Now uh we'll go back to our uh run and see if it has passed. Okay, so iteration number two has passed and if you can see uh the savings in in cost are like 4x. Uh this chart is being created on the
fly. Uh so it started from $12 and the second one was $3. And keep in mind my agents here are Opus 48 both for the doer and uh for the uh teacher. Now iteration three just came in. Uh the
number of turns decreased. The latency is also better. Uh but there was a failure in iteration number three. there's something that it sort of overcorrected on and iteration number
four should probably be learning from that and we'll see if it does that. Uh while it's doing its own thing, I wanted to show some previous runs that I did uh just in preparation for this uh
demo. U so for example, I went to uh auto browse and I was like hey go to the AI engineers website and tell me which talks and workshops are talking about hill climbing, right? uh and it did that
across multiple iterations and you can see the replay of each of these. These are replays that are coming from browserbase super helpful important for uh observability perspective and as you
can see uh it sort of uh got better in ter in terms of cost 7.4x 4x uh even though it was able to pass in all the iterations, I still got improvements in cost. Now I wanted to actually see what
are the learnings from this skill, right? Uh how do you sort of uh persist all of these learnings? It says go straight to the shed schedule page. If there are open errors, don't try retry
blindly. Run this. Uh type the keyword into the session search input. Wait briefly. So like have these weights because sometimes websites take a lot of time to load. So you you can have these
kind of tricks uh and then read the results. Uh you know like there there's a lot of learning that is happening in these skills. Maybe I wanted to show some interesting ones. I wanted to show
this open table one because I started with this blocker of like access denied when we started this demo. Uh and if we go to the uh screenshot, this was the access
denied thing that we got right in in iteration number one. Let's see how it learned from there. Uh it realized that the uh block is environmental because the website doesn't want this agent to
do something. uh in iteration number two, it figured that if I do a query parameter and if I put a latitude longitude metro ID, I automatically get that uh UI that works. Do not open /r
that's blocked and do not reload / s. These are the places where the blocks kick in. If you don't do that and you follow this technique instead, your agent will work, right? Uh and as you
can see, uh it sort of went from $6.22 22 to 0.51. The time taken went from $189 seconds to 27 seconds. Uh and the scale it came up with was here. Uh build the search URL,
do the date math like this. Uh this is the known metro pins for San Francisco. Uh and and follow these steps, right? Never call / r because that will return access denied. Now these are the kind of
learnings where a human could never have done but you just sort of embrace the bitter lesson and a let agents do the job uh and they will find these uh tricks for you.
I want to uh also show uh a few examples of like the kind of improvements we see. Uh the reason why this problem is so close to my heart is because I from a sales perspective am deploying these
browser agents into production. uh the sale is only done when the browser agent is successful. So reliability is like the most important thing for me and this is what I this is the key tool that I
take to my customers and I'm like just fire your agents let them do this auto research thing and your agents will be successful in production. Uh and these are the kind of u success metrics that
people are looking for up to 16x cheaper 12x cheaper 3x cheaper uh on different tasks. Of course, every task is different. The bottlenecks on each task are different. So, uh you might see
variance there. But the idea is you can hill climb on these uh techniques. I wanted to go back to our live demo and see if there's any uh improvement. All right. So, we have a graduated skill u
and the skill applied end to end. No further learning. The results are showing correctly. one-way business class, San Francisco to New York, July the 4th, one carry-on bag included. Uh,
and there is 8 a.m. $649. Right? So, it was able to get all of this and I can sort of watch the entire replay to see if that is accurate. Uh, it was able to sort of figure out this entire skill and
the fast path is encode the whole query in the URL. strongly preferred the again like I know I'm I'm showing these things but when I first started looking at the the results here this was like my moment
of like this is the mythos moment for browser agents you know like just reverse engineering the hell out of things uh I I look at this with some of my customers the customers are
mind-blown uh earlier they would be writing scripts for each of their websites they would be like hey I need to spend so many engineering hours on like just reversing these websites. How
can I scale this to 100 a thousand websites and now our customers have actually built these entire pipelines of uh these skills that get generated on the fly pull requests that get created
automatically. If there is a failure in production, an agent kicks in and starts healing the skill as well. So there's a lot of uh engineering that we can do around this to make leverage to leverage
this technique. Uh I wanted to also show what we've done with this right we actually went ahead and created this product called browse.sh
uh we were like if we can do this for one website we can do this for five why not do this for like 500 websites and give it for free for anybody to do this. Uh and like if you remember in the 19 or
wherever like in the '9s before internet you had the yellow pages uh you had details about accessing people, businesses etc. This is yellow pages for browser agents right it has details
about how to do any task on any website on the web. U I wanted to actually show this to you all. uh you can actually install browse CLI and use all of these skills completely for
free. We've eaten up the token costs ourselves and we're making this available for you all. Uh so if I go to let's say search products on Amazon uh I can see the entire scale is over here.
When should you use the scale? How should you look at like pricing and availability and whatever right? uh this is how you use this lot of details about Amazon the screenshots u and I just give
this browse skills CLI to my openclaw and it just knows when to pull the alltrails.com skill when to pull the weather.gov skill and and this makes my life so much easier. uh we essentially
are running the same auto research loop behind the scenes uh and making this uh available for you all. In in fact if you would want to you can go here and you can be like hey go and make a skill for
me to uh look up this uh events in San Francisco for this week. Uh you do this generate a skill and this will sort of do this for you. This will send you an email and you can then use your skill uh
in whatever agent you want to, right? Uh so we've done this for browser agents. Uh and I wanted to talk about like how can you potentially do this for any other domain, right? Uh I think coding
agents is like one place where I have seen this frustration where the agent is relearning the repo conventions every time. it's not applying the linting rules correctly and I have to tell it
every time and maybe you can have your cloud MD and agents MD and improve that but the the fundamental premise is like the agent should not be relearning your tasks from zero right uh I've seen this
in research agents we run a bunch of like deep research agents for our internal workflows uh and every time it struggles and it tries to relearn oh I pulled this information from Wikipedia
IA uh but I did not pull it from this other website. Uh I was like why not? Uh like we always do this other website you should learn from your previous traces. Uh and you should pull information. Uh
internal ops you have all these dashboards all these databases all these internal tools but your agents don't know which ones work and which ones are preferred. Ideally you should be
applying this all research technique. Uh a and the two conditions that are required is that a you should have a replayable environment of some sorts so that you can run these loops and b you
should have some sort of verifiable outcome some sort of quant quantifiable metric for you to run these loops. Yes, I see a question there. I realized that
Yeah. >> Yeah. >> Yeah. Exactly. Uh so I think the question is like how do you know that the auto research loop is accurate
whether it is converging and when it says it's converging it's actually converging. When it says it's good it's actually good. When it says it's bad it's actually bad. like how do I have
confidence uh on what the agent is saying and what the loop is doing. I think that's where I want to sort of talk about the verifiable outcome here. Uh for Karpathi, it was quite easy
because he was looking at the log log loss of the experiment when he was training neural networks. Right? So it's very easy to see whether your log loss has gone down or not. Uh but in many
real world scenarios that is not the case for browser agents. As I explained, uh, whether you book a flight ticket or not, that might be straightforward, but did you find the best flight options for
me might be slightly fuzzy uh and might not be the best uh Yeah, if if I understand this correctly, you're saying like one mistake in the loop can sort of throw off the entire
auto research run. Uh and how do you prevent against that? I I I have actually seen where some uh iterations actually fail. uh for example if we uh look at the previous slide
uh there's like failures happening after there have been successes right uh so it's not always that once you succeed all the future iterations will always be successful but I think it's the design
of your loop where you can make it resilient to u failures uh where it doesn't overindex on those uh and still sort of correct protects itself uh from any mistake that happened in previous
runs. Uh I think we're trying to sort of prevent like uh what do they call like exploding sort of loss sort of scenarios. Uh I think you could do that. Uh in fact Karpathi talks about this
like this is one way of doing auto research. You can actually auto research on different ways of doing auto research itself. Uh it's like is the teacher doer loop the best? Should there be a
multi-agent architecture to do this or is there any other approach that might actually work better for research tasks in neural networks versus browser agents? Right? You could have different
designs of your loops to prevent situations like those. Uh and I think going back to my point, as long as you have a verifiable outcome or some quantifiable metric, I think you should
be able to design uh a good uh loop here. uh replayable environment I think sometimes is under discussed uh but extremely important. I think he was talking about Darkh he said having a
verifiable outcome is important but also you should have uh extremely grindable environments or extremely grindable uh use cases as well. Right? If your agent cannot
uh grind through failures, it will be difficult for it to uh go for multiple iterations and fix itself. Uh so do you have like coding agents is a good example because you can start from a
docker image or you can start from uh a particular EC2 and and you can replay from different parts of the uh stack. Uh but maybe uh knowledge work is probably not that easy to replay. uh maybe
there's like a lot of different human in the loop actions that make it very hard to replay from a particular point in time. So investing in your environment also can be super uh rewarding in these
cases. Amazing. Uh I just wanted to share the uh URLs of uh the auto browse loop that I uh demoed today. So if you scan the QR you'll be able to pull the uh GitHub for
that. Uh and also the second QR here uh is the skill itself. You can just do /auto browse on your claude code or codeex and be like hey /auto browse use opus48 on both of these uh teacher and
doer and go create a skill for me to do this. Uh I'm looking at one of these questions. Uh and also if any of you have ideas of something we want to sort of put the
agent against feel free to type it in the slido and we can actually try it live. Uh but let's look at this question. Is there a way to optimize for convergence? Uh I think this is the same
question that we were just talking about. Uh high cost is interesting because you might think that u doing longer iterations might become costlier over
time. Uh I have run a bunch of evals on this where I can throw a sonnet at this problem but run it for longer iterations versus throwing opus for lesser iterations. I think you can get the same
job done by throwing sonnet larger number of iterations but still cheaper overall. Uh also this cost is amotized right like one auto browse loop that we probably run is like maybe $30 $40 if
it's super complicated $100 at most. Uh so this is amotized as long as those costs make sense for you uh to deploy this in production uh just go for it. Uh but there's a lot of interesting things
uh that you could do uh with evaling the kind of loops that you're generating. Uh yes I will share the slides as well. Uh I can do this on the GitHub itself. Uh
can you share how you thought about uh auto research into auto browse? Uh I mean the process was fairly straightforward. Uh took the GitHub repo that Karpathi put out uh and then
there's all this context about my customers uh that I have uh where I was like hey these customers face these kinds of reliability problems uh the way I solve it for them is I go into their
office I sit with them very forward deployed uh and I try to sort of just brute force and reverse engineer this. This is what I do. So I am the teacher agent. Uh can you automate me out of the
process and create a loop borrowing from uh Karpathi and and and and build the same thing? And of course there's a lot of iterations on top of that. Uh there's a lot of engineering as well uh to
simplify this. Uh but yes, that was the process of converting this. Can you use this in a proxy? Can you be logged into a site? Yes. Uh again this goes back to the environment that you
design. Uh browserbased the infrastructure that all of this is running on uh gives you the ability to log into websites uh save the cookies uh reuse those cookies in future runs so
you don't have to log in every time. Uh makes it super easy for you to uh replicate real world workflows. Uh and yes you can use proxies as well. If you have uh particular IP addresses,
residential IP addresses, you can bring those in as well. Uh a lot of success is definitely how good your infrastructure and environment is, especially in these adversarial scenarios, uh like captures
and antibots. Uh so making the infrastructure as good as possible is definitely of value. Amazing. any any asks for something that we might want to run live? Yes.
Question. >> Great, great question. Actually, I wanted to sort of show the code as well. Uh so if you uh look at the GitHub, you'll be able to find this evaluate.mjs
file. Uh essentially, it lists out the entire loop. Uh what's happening here? Uh but to answer your question, the teacher and the doer are u working one after the
other for every loop. So the teacher is finding learnings in every iteration. Uh maybe if I show the demo uh over here uh let's go to the one we just did. Iteration one had these learnings and
then iteration two had some other learnings. Right? you should include the explicit ISO date in this particular format. Iteration three would have some other learnings. The date resolution
still opens the bare page and clicks wasting three turns uh reading today's date uh or having the executor compute today. So it's figuring out something about the date time. So in every
iteration the teacher is learning something new and putting this back into the strategy.mmd. Maybe I'll make this real by looking at the traces for one of these runs. Uh
so over here these are all my runs. Uh so for example this was the open table reservation example u and uh this was the strategy.mmd that the teacher sort
of appended to every time. So uh step zero you should do this so on and so forth. Uh this was the task this was the uh prompt that I gave it. Find me reservations for the big B bigs
restaurant. uh and these were the kind of traces that uh it was sort of learning from. It would take screenshots would take these sort of traces at every step is doing browse open this is what
the output is. Then turn number two browse snapshot. Snapshot essentially takes the accessibility tree and this is the snapshot. So it's reading this entire thing and at every step thinking
of like what went right and what went wrong. U yeah I mean this is like machine code. uh I won't show this entire thing. Uh but yeah, this is the summary that it also writes uh at every
iteration. Uh so yeah, this there's a lot lot of like engineering tweaks that you can do to suit this to your domain as well. U and uh like the easiest way to uh read this is
we put it out here on skills.sh as well uh in auto browse u and you can sort of do something like this. Great question. Such a good question. I think that goes
back to like how do you increase the reliability of your uh agents. Uh reliability is an engineering problem. Uh you can design your harness to sort of go for one skill
for each task or you can be like hey one skill for every task that you will ever see. Uh I think it entirely depends how you do the harness engineering entirely depends on the use case that you have u
and you run evals with all these different strategies uh and see which one works for you. for us. This was Yeah. Yes.
>> Yeah. Yeah. And this is where again the harness comes into the picture. If the harness has the intelligence to bring in skills from Open Table and from Resi and Google res Google reservations, maybe it
can bring all of those in to figure out what is the best way to book me a date night. That's your solution, right? Like the harness should be able to uh figure out
which tools to use to get the job done. That's like how I would go about it. uh figure out what are the most common use cases that your customers have for those use cases. What's the best scale files,
prompts, tools, maybe you want to do like specific tools for specific use cases, right? Maybe you want to do generic tools for all use cases. Like that's the same argument here is like do
you do one skill for each or one skill for all uh and eval across all of these strategies and and let the hardness figure it out. Yes.
>> Yes. Yeah. I mean it's a very similar concept as this, right? Uh so DSPY. >> Yeah. I I think those guys were on to something really cool like they were around like 3 four years ago where they
started building DSP and Japa all of that. I think uh we have been doing a lot of research uh along these lines as well. Browserbase works with uh Microsoft. We actually released a paper
uh about how to build the right sort of verifiers uh for your agents. Uh I think running evals is straightforward until you start questioning okay how should I eval my eval right like
do I need to put all these humans to verify all these runs or is there a good verifier that I can build here and all of that u uses the uh learnings from uh these papers as well. I can share all of
that in the GitHub as well. I'll also see if there's any new questions here. Where and how were these skills stored retrieved at scale. Uh right now for
this repo uh all of these get stored in my cloud over here. So each of these is a scale file. This was my recipe searcher. Uh I wanted to build some I wanted to cook some dinner yesterday. So
I was like, "Hey, make me some recipes." Uh, so all of these are currently in my local uh skills folder. Uh, but if you want to ship this uh in production, of
course, you can uh bake this into your agent loop. Uh, or if you want to I think for browse.sh the public uh publicly available uh skills that we have, I think we use versel blob storage
to store all these skills uh and make them on demand. Hope that helps. Uh, websites change frequently. What's your STR strategy for monitoring that and
keeping the website skill updated? Yeah, this is such a great question. Uh, and this is what I was alluding to earlier is like how do you keep these skills up to date because the website's going to
change. The the current working theory that we have right now is you don't want to waste tokens running auto research every day. you want to probably trigger or if it's
failing at any particular point in production, right? So, we build these loops or like systems where as soon as we run into any production failure, we run an auto research loop to see what
went wrong to heal it, create a pull request, that goes directly to a human engineer and they fix it automatically. The skill gets updated. Uh so yes there is that notion of like production
failure when this might happen. Uh but it's a trade-off. You can run an auto research or you can run a sample run every 2 hours every 3 hours to see if the skill is working uh perfectly. So
there's different ways of having the uh cron jobs uh check for the accuracy of these skills. This is actually one thing that I'm actively working on with the customer right now to implement. And
there's all these questions, right? People don't want 100% accuracy. People might want like 10x cheaper cost. So like uh that's the uh question that comes up. What's your teachers criteria
here in the example? Separate from reaching the goal seems to do some optimizations for turns. What other criteria do you use? Uh so again this is the design of the auto
research loop. Uh there's one way of telling your order research that hey I want to focus on just success. As soon as you get me success I'm fine just give that to me. Uh that's typically not what
everybody wants to do. Uh people want to minimize on uh latency. Uh because maybe this is a real time run where you're focused on how quickly can you get the job done. U so if that is the optimizing
metric and you focus on getting it as fast as possible. What I've seen sometimes these auto research loops are so smart they realize that browsers are not needed at all and they just reverse
engineer the API and they fire the API and that's the fastest that you'll ever get to. Right? Uh so sometimes uh I've seen situations where uh it finds a solution that works in 500 milliseconds.
U sometimes it is the number of turns, right? you want to minimize your context loop uh and you want to minimize the number of uh turns it takes to complete
the task. Uh in my instance, I think what I have done is it's your decision to pass or converge whenever you want to pause it. I'm not really concerned about the costs. Uh I care most about
reliability. So keep doing it as long as you feel it's relevant and then it's the teacher agent's job to uh call it day when it wants to. But yeah, you can uh try all different
strategies. Uh if a website supports WebMP, will this use it? WebMCP is so cool. U it's essentially this concept. I don't know if everybody knows this, but it's this concept of like making your
website programmable. It's it's come out in this new Google Chrome release recently where you tell your u website that if anybody wants to book a reservation they can do xyz on the
website. It's kind of like an API but it's on the browser. Uh right now I don't think our oral research loop supports this. Uh but I don't think it would be uh too hard to do that. Uh in
fact we recently shipped support for WebMCP with our SDKs that we launched. Uh we also made some tools for website owners to ship web MCPS at scale. Uh but yeah this is an interesting one. We've
not really seen a lot of adoption with Web MCP as yet. Uh ground truth for pass. That's interesting. Uh I think as long as the job was done as long as for example in
Google flights uh if it found the cheapest flights in open table if it found a reservation uh so it it evaluates itself. It's typically a binary uh and yeah again I I I fully
embrace the bitter lesson. I try not to put many guardrails in uh these evaluations. Uh, have you considered a verifier as a service for browsing tasks, including
something that allows us to mock certain side effects like purchases? Uh, I might have to ask uh a followup on that. What do you mean by verifier as a service? Ravish, if you're
here. Yes. Uh so uh do you mean that yeah if I understand this correctly like you have like different environments
that you can use for RL training and you want to like mock what happens if I am replicating Amazon if I click on purchase what happens after Uh
no because uh we try to rely more on real world websites. Uh browser base we have the privilege of being able to access most of the web with our uh access to different websites uh and
capture solvers etc. Uh so we try to mimic the real world as much as possible but we've been seeing a lot of these RL environments where people are mimicking websites to train uh and build browser
agents that are uh really good at like browsing Salesforce, really good at like browsing SAP. Uh we try to do that on real world portals as much as possible. [snorts]
Are edge captures easy for the skill to move past or is it token incentive? uh what is the point of edge captures for website owners? Edge capture if I'm understanding this is like a particular
kind of capture. It's kind of like Google recapture etc. uh yeah I would probably talk about this separately. Uh I don't think we can talk about like what it what captures we can
solve or not. Uh but it's not a token inensive job per se because it's an inf like it's thing that's done at the infrastructure layer, right? uh we don't really need to use an AI to
solve a capture. Uh it's solved at the Chrome binary layer. Uh and we do that out of the box. How does the teacher evaluate the doer and when is the skill markers graduated?
Yes. So uh it evaluates the doer by essentially reading these traces. Uh for every iteration there is this folder that gets created. uh there are these uh JSON files. Uh it reads these uh it has
a clawed agent SDK loop uh and it essentially figures out what the learnings from that loop are. Uh actually I think I have a slide that talks about this at the end.
Yeah. Uh this is kind of like the prompt for this uh where it says that you're a browser automation agent. You can use these tools uh and the strategy.md gets injected at every iteration and this is
how the uh uh doer the teacher evaluates the doer's job. Uh so yeah it reads the traces it has the system prompt it has the previous strategy and it comes up with a
better strategy. [snorts] Amazing. Uh I hope we've covered all the questions. Uh any other questions live that we might
want to talk about? Yes. >> Yeah, it can work with the stage hand SDK. It can work in your cloud code. Uh it's essentially a markdown, right? Uh I took uh a skill and I told my clot code,
can you find events in SF for this Friday, July the 3rd? and went on and did this. So it it need not use browser base. It need not use stage hand. It is essentially a skill. It is a portable
artifact you can use in any agent or any SDK. Yeah. Yeah. Great question. I think if I understand this correctly, how do you prevent against prompt injection? Uh
because websites could trick my auto research loop in learning something bad that can do some malicious behavior in in a production setting. Uh there's a few ways we do that. One is at the
infrastructure layer itself. Uh we have a bunch of uh protections around our browsers itself. Uh we have this browserbased safe mode where you can sort of control which URLs
it can go to. You can control what kind of actions it can take on different websites. So at the infrastructure layer, at the network firewall, we control things. Now you might say that
it is possible that Amazon.com gets hacked someday. How do you prevent against like malicious actions on those portals as well? Uh I think we also try to uh run our skills that are generated
against some security uh reviewer agent of sorts uh before we actually publish this on browser.sh. Uh so that's like one way of doing it. uh we encourage our customers to have uh a good level of uh
prompt injection guardrail uh surface area uh if they're sort of giving this out to c consumers uh for example some customers of ours uh might want to automate something on Amazon and what if
somebody puts uh a listing on Amazon which has some hidden instructions and how do you prevent against those kind of things uh and that's where uh we work with our customers to build the right
kind of uh guardrails before the uh instruction even reaches the hardness. Um so yeah I think I would put like three layers here. One is as the as the query is even entering our
surface area we control that at the skill layer we make sure that our skill is not uh tricked and at the infrastructure layer we have network restrictions and action restrictions.
You can even do something like what codeex or plot code does u like always allow or allow once or allow any action. Uh you can this is what like browserbased safe mode offers out of the
box. I know browser is a crazy threat vector when you think about it like it has access to anything on the wild web. Uh and take that super seriously from a prompt injection POV.
Amazing. I'll wait for 10 more seconds if there's any other questions. This was fun. I really enjoyed this. I love the kind of questions as well. Uh and yeah, I will put the QR code back on
screen in case any of you want to grab this. Yes. Yeah. >> Yeah. Yeah. Exactly. Yeah. I think we talked about it a while back. What you
could do is like I used uh entropic managed agents quite a bit. Uh so I would run a chron job on managed agents where I would say use this skill every day at 9:00 am to do this job and if
this is a success all good if this is a failure send me a notification here that will trigger a healing loop uh so that's how I sort of can bake in uh checks and balances around websites changing
antibbots changing etc. I'm going to hang around here. Uh also uh I'll be at the browserbased booths all days of the conference as well. So uh in case you're curious and want to
chat more uh love to do that. Thank you. >> [music]