Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Simular's neurosymbolic computer use agents pair a VLM agent with a deterministic coding language (Simulan) to overcome the reliability degradation that occurs when running agents multiple times (pass@k). By writing code on successful runs and falling back to neural agents only when environments change, the approach promises lower cost and higher reliability than pure VLM agents. The company open-sourced Simulan and released the Sia API for cloud-managed virtual machines.
Key points
Simular achieved human-level performance on OSWorld benchmark in December 2025, but the newer OSWorld 2 benchmark sees best models at only 20% success rate.
The company open-sourced Simulan, a deterministic JavaScript-based language for computer automation, and released the Sia API for cloud-managed VMs.
All current computer use agents degrade by about 10% success rate when run three times (pass@3), motivating the neurosymbolic approach.
The neurosymbolic system first replays deterministic code; on failure it falls back to the VLM agent, which writes new code for future replays.
Simular also supports Android VMs for tasks like playing games, filling authentication forms, and social media automation.
Tools mentioned
Techniques
- neurosymbolic architecture
- accessibility tree grounding
- multi-agent orchestration (manager/worker)
- test-time scaling
- code replay with verification and healing loop
- clarification prompts for reliability
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
a little bit about my background. Um, I'm the co-founder and CEO of a similar and this year is pretty much the like the 30th years that I've interacted with computers and now I'm trying to build a
computers that's fully autonomous so I don't have to interact with that. And if I look look back to my history, the first 10 years I'm I I I basically get to know computer when I was pretty
young. The first 10 years I focus on comparative programming like basically writing code every day. And the the second decade I spent a lot of time uh training neuron nets basically published
a lot of AI research papers uh during my PhD and later on in deep mind and the third um uh the third decade so I joined deep mind in 2017 that's the beginning of the third decade so this year is 20
uh 26 almost 10 years so so when when I first joined deep mind we really try to solve this problem called AGI artificial general intelligence And back then everyone is still confused. Even now
it's everyone's still confused about what's the definition of AGI but in general it's kind of like a general system kind of like like human uh humans like like a machine can do all different
kinds of things in there and uh hopefully next year we can solve the problem. So I I can think about what to do what kind of problem to solve uh for the for the first decade. Okay. So I
want to u start my talk now. Um basically the reason I want to introduce that is because I spent 10 years on coding 10 years on neuronets 10 10 years on solving AGI. So actually that gave me
this this kind of bias that we we try to inject to our current company that we have this uh this kind of principle in mind the general AI system needs to be neurosyolic basically code combined with
neuronets. So, so which I will I will show you quickly why uh like we have this kind of intuition in there. So, the goal for this workshop I know we we're kind of short on time and I do want to
give uh three things back to you. First is basic knowledge about computer use u I understand 50% of you are still new to this domain and second is the tools we we have been working on as part of the
infrastructure. There are four different type type of components that we believe would be helpful to your future development of computers uh applications and I'll also show some applications
that other people has been working on or other developers and third which is important uh since uh what we have a product called sigh and this product is still closed beta right now so I want to
give everyone invitation code at the end of the the workshop so you have a you know opportunity to play with the product also do let me know uh Feel free to follow me on uh Twitter and join
Discord. Let me know if you need more credits. So, so I definitely like give more credits in there. So, the first question is what is a computer use agents, right? So, uh normally what I do
with the with my talk is I can take a selfie here with everyone. I'll take half. Yeah. So, so this is what I do every day now. So actually I'm I'm the CEO, so I don't use
computer all day long. I have my agents on my phone. So I can text my agent, send a photo there and say post the photo to LinkedIn.
Yeah. And then the agent will will be working on it. So let's see. So I do want to see if the agent is responding there. I think I got some.
Okay. So, let's see. I do have my message there. Okay. The agent is responding. So, he has my photo there. And you can see the agent is starting
asking me questions there. So uh I uh I won't interact with the agent right now but I want to show you the basic interface. So computer is agents means mean means the agent that has a computer
which is a full Windows desktop in this scenario and if I continue interact with the computer the mouse and keyboard is know it's going to work on this desktop and open a LinkedIn later
and I'll go back to the presentation. I just show you an example of having one agent. So I can talk to the agent agent working on applications but actually you can have multiple
that's what we are actually working on. So uh the these are four agents you can see they have different names right so QA agents is basically like uh doing QA on the products testing product is one
big use case in there they also research customer support go to the score channel and looking at all of the information there. So the the actual agents are slower. So it's actually this is
recording for like hours. So I just compressed that into 30 seconds for you to basically grab the you know the basic understanding of of how computer us agents is working in there. So the
general idea this is very different from um most of the agents that we talk about right now like most of the agent they're based on APIs and you don't actually see visually what the agent is doing.
Computer use agents is another category and we believe is a is a is another bigger category which can do a lot more things that you you don't have APIs you can still let agents to work on it and
then I want to do a little bit ritual because we started a company in 2023 right and that was uh right after GT 3.5 and uh GT4V so I want to show you one example that two years ago we kind of
we're kind of hopeless two years ago on this problem in there. I actually post it on on Twitter. So this is is my tweet and let me pause a little bit. This is my tweet two years ago. So the the
problems is like can we actually use GP4V to understand the screen and then click on the iMessage icon on the screen. Very very simple trivial trivial
problem. And I basically use the model build up a iterative refinement of the the grounding. You see the the most the first time is is not correct. The second time is go to
the mailbox icon and the third time is correct on the iMessage. So this is this is the only one successful demo I did. [laughter] So so
I asked I asked the agent to to click on Zoom. It doesn't work at all. So two years ago this problem is basically hopeless but we kind of have the intuition that uh financial model is
going to be better and better over time eventually this problem will be solvable but it's just like it's not possible within two years back our um that actually motivates us to think about
what's the general strategy for solving this computer use agents uh problem and we do have a name for that it's called autonous computer because you know like if the computer can do all different
kind of things it's autonomous right and then The question is how do we solve autonomous computer? So when I when I talk about it there are only two words autonomous and computer.
So it's become so natural two directions you think about one is the autonomy the other is computer right autonomy meaning like do we have a system that can actually understand the context the
environment of the of the computer and do all the reasoning planning can recover from its own mistake um like doesn't require human to babysitting the the agent and the second part is
computer like what kind of computer uh the agent can use so this is a core question but back then two years ago we feel computer is actually easy problem because everyone already have computers
right so that's why we focus a lot on the autonomy part so when we talk about autonomy this goes back to the the underlying principle we initially have is called neuro symbolic neuro means
this also happens to self-driving car it's basically the the same concept for all different kinds of autonomous system they all have the same concept you you you have a component that's neuro here
is a VM vision language model VM agent and the Second part is symbolic and here is we call it coding agent back then and basically we need to solve these two technology in order for us to make
autonomy works and I just show you a demo two years ago VLM doesn't work at all right so that's why we actually spent a lot of time two years ago working on the coding layer to ensure
okay like we can actually control the computer in a more reliable way but the second problem is we we set up a research team two years ago to think about okay how can we make this craft
app VM better in terms of using the uh using the computers. Um before I start talk about the the trajectory our journey on the VM agents I want to uh briefly talk about the the signals like
you know like self-driving cars uh like we running on the road they have signals lighter radar and uh and like camera right three different kind of signal computers had the same so I often tell
people like okay we are we actually working on self-driving cars in the in computer in a digital world basically changing the steering wheel to your mouse and keyboard so we still three
different kinds of signals. One is screenshot, the other is accessibility tree. So accessibility tree is very unique. It's it's kind of rare, you know, in in our in our daily life. DOM
tree is pretty popular because we use browsers. Accessibility tree is a kind of a uh still kind of like a dome tree, but this is this tied to each of the operating system. Mac has it its own
accessibility tree format and Windows has it like a separate protocol for accessibility tree. But this is the basic the the the layer that operating system use to render all of the windows
all of the icons on on your screen in there. So this this part is very important. It's because um yeah the these are the the effort we've been working on is called a so the whole
idea is can we build out a harness. So two years ago we don't call it harness. Now day everyone call it harness. So two years ago we we build out we try to finish this project called agents build
out a harness b uh build upon all of the foundation models not just one so you can easily switch models it's model agnostic framework that allows you to produce the best uh computers capability
in there and the first generation was done in late 2024 and it's called agents uh we just call it agents because that that's the
beginning and This agent achieved 20% success rate on OS word 20%. And that was the state of art state-of-the-art back then in 2024 which is very interesting. And this paper this paper
also won the best paper in I clear 2025 agenda AI workshop you know that the the whole industry was moving so fast over the past year and which is crazy and this the the choice for this harness is
because first we know vision language model is so bad it's not going to work. That's why we chose to use accessibility tree in the first generation. Basically, we look at all of the the underlying
protocols of the tree structure to describe the current environment and pass it to a large language model instead of vision language model and use ID ID based grounding basically
linearize all of the all of the buttons put them IDs and ask LM like which button what's the ID of the button where I need to click on right now. It kind of works. There's 20%. which is super
exciting because in the beginning this benchmark is like 5% success rate in the early [laughter] 2024 in there and then early 2025 we started to see okay vision language model is getting better so the
second generation of agents we actually uh completely dis discard the accessibility tree so it's a vision only approach and the interesting part is this architecture
we have multiple layers like managers and workers. So it's kind of like human organization and we showed it kind of works and this work specifically was um a a lot of friends in summary actually
talked to me about this work. They they feel pretty excited about this work. The reason is this master demonstrate that if you have two models you are better than one
models. If you have two foundation models within the same harness it's better than one uh foundation models. And the second foundation model is a seven billion open source model. So
yeah, so all of the startup friends, they are pretty excited about it because okay, there's a there's our journey like I can train some small models and we can be 20% better than uh like cloud or
openi uh computer agents there and late 2025 we have our third generation agents and the whole idea of this part is can we actually do test time scaling it's called inference time
scaling is when you have a agent can you have multiple agents to try to attack uh solve the same task and eventually you have better performance and with demonstrator it works. So this is uh the
figure for the total resource budget meaning the total number of workers versus uh times the total number of steps and this for like fair comparison and you can see if you have 10 agents
you actually scale better in terms of success rate. So then in December we break the human performance uh record in OS world benchmark and these are comparison of all the trajectory between
among uh agents and anthra and openi uh both big labs they achieve the same human performance uh in March this year and that also uh it's uh last December was an exciting moment like we feel okay
like computer user agents is finally start to be somewhat useful but you know this benchmark is kind of research benchmark it doesn't mean like in real life it's already 100% it's already
better than human um a few days ago OS word has uh has released their second generation called OS world 2 I'm specifically excited about this new benchmark so um in the new benchmark the
best performing model is only 20% in that benchmark so so we are we are entering the second phase of computers uh agents in there I want to show show you one video. Uh
there was a very interesting um demo that one of the developer used. So this this project actually won the cow hacks last year. So it's it's a it's a one of the biggest hackathon in uh hosted in in
Berkeley and they they built their applications based on agents the open source one. >> I guess uh you probably don't I'll hear the voice.
>> What if you can control your computer from anywhere in the world just by it? We built a multimodal agent with full control of the computer. So instead of calling your mom to debug your two out
of memory errors, you can just FaceTime the Mac instead. Let me show you how it works. >> Okay, so suppose that you're training an A model at home, but you already left
and you just realized that you had a CUDA out of memory error. So what's the easiest way to fix this? Well, instead of driving all the way back home, you can just call your computer and tell it
to do itself. So, we can call it and text it to FaceTime me and then this will initiate the computer use agent right here to uh start a FaceTime call so that I can tell it what I want to do.
>> Yeah, they they did a very interesting thing. Let me >> So now I can see what the thing says. Can we fix my CUDA out of memory error
in VS Code? Okay, the task is to fix the CUDA out of memory error in VS Code. I will hit command space VS Code to open the application. >> So these are all automatic.
>> Okay, I'm in VS Code. I'm looking for the CUDA out of memory error. >> It looks like this file has a CUDA out of memory error. Based on the comments, it looks like the tensor size is too
large. So I should try to decrease this to something reasonable. Yeah, I think I think you kind of get the idea. So the idea is build a application that you can control your
computer using FaceTime and use FaceTime to debug in VS code. That was happening in October last year. That's before like even open claw was started as a project in there. Then we had to think about
what's the second uh you know second phase of computer use agents in there. So we already surpass human performance in the benchmark like what's next? Then we realized okay there's actually a very
interesting concept to called pass to the power of K. And in the past, everyone was measuring pass ad meaning is is pretty popular in in VIP coding like you you have a task you add you ask
the agent to uh produce the code right if the code doesn't work and then you you put some prompts in there and produce another another uh code in there one of the code becomes successful is
successful because you don't care about what's happening in the past you only care can I get a code that works right in computer use is very different because Like we we started to get
customers two years ago and all of the customer talk to me about this question like can I actually put your agents hosted in a server and run this workflow every day and I don't have to you know
keep looking at looking at looking at what workflows I don't have to debug in there. So basically everyone was telling me about this metric pass to the power k. So this means I run this agent K
times every time it needs to be successful and this metric is not captured by any benchmark in there and then we started to research this problem. So we took the
best performing agents in OS word and their performance is pretty good in three different categories and then we run the stem agents three times. We measure the performance of pass to about
to about three right from this figure you can see all the agents has degraded performance in there they almost dropped 10% success rate even even the best performing agent in there and the second
part is on perturb environment meaning my querics could be a little bit different and environment could be a little bit different each time and it get even worse so this this is still a
problem in the computers agents nowadays Because you can see uh Claude has Codex has uh agents, claw has agents. None of the agents actually produce reliable repeatable process for you.
So this is a recent paper we published uh like two months ago. So basically discussing this problem and there's a little simple trick that we proposed in this paper that everyone can you know
can borrow the ideas in the future if you want your agents to be more reliable. The trick is basically ask the agent to to ask you clarification clarification questions when you give
them the instruction. So, so basically this paper was saying like suppose the agent always ask you some questions about ambi ambiguous part in your in your prompt the reliability of the agent
will be better and there's a prompt in the in a paper that everyone can look at it like basically it's a it's it's pretty simple it's just like okay like your goal is to minimize u uh have
minimum uh clarification that still preserving the nature this line is very important You put this prompt to the agent and the agent can perform better. After you put this clarification prompt
to the agent, you can see like the overall um reliability of the agent is getting better like dash line is the original performance with no clarification and the solid line is the
one with clarification. Right? You can see cloud is getting better and agent S with GPT5 which is interesting has the biggest improvements in there. So, so the the purple line is model agnostic
framework and the green line is is the you know like big labs computer agents in there. Okay. So, so basically this is our journey for solving the VM agent which
seems to be work work pretty well and still like nowadays still people using the agents framework because model agnostic you can plug in any open source foundation models. I'm also like looking
forward to getting like results with the latest open source big model like GLM uh 5.2 tool and see how the performance looks like in there. And then the second question is okay, it doesn't solve the
reliability problem, right? It like these these agent doesn't seem to be have as long as you're using a neuronet, as long as you use a VM, it's in the end at the end of the day, it's a
probabilistic model. It produce a probability each time. So it's not going to give you the same performance uh if you run it for like 247 every day. Then we had to think about what's the next
thing to think about is basically symbolic layer. We call it coding agent and we open source this uh language we call it simulan. Uh it's a language for similar. So we call it simulan there. So
this language is kind of like if you're familiar with RPA industry it's kind of like what RPA industry used to do like they ask engineers to write a code. uh but the common problem for uh RPA is
there's no single protocol that's shared across the whole community that everyone can use and we try to create this uh uh in browser automation we have playright which is in a standard way but in
computer use there's no really no uh industry standard there so that's why we we open source this we hope it's helpful to all the developers so you can u actually like build your own square pin
layer for different workflows that's reliable deterministic each time. Yeah. So, basically the goal is to allow you to run to create uh source code for Simulan and run this code you know you
know terminal and you can also use it in claw code. So I I do want to give you a quick demo there. So let me Okay. So now I'm in a cloud code. So if you install um
maybe I should show you simul Yeah. So these are the the landing page and if you go there there's a simple oneline installer npm installer. Once we install, you should be able to
use it on your own terminal. And I'm a claw code user. So I like to see because no one is writing code nowadays, right? It's important for you to use V coding to to do this kind of
automation. So once you install there's a skill that you can is paired with Simulan. So you can just uh you know use the skill. So I can say go to AIE website
and summarize the events. Okay. So let's see how it works. So now it's checking checking version.
So these are the logistic stuff. But it does understand it's going to use simulan to open AI engineer words fail website continue.
So you can notice is it's try to use a simulant JS is a JavaScript uh language. And now clock is gonna try to write some code using the simul protocol.
Yeah, I think it's because the the clock is new. So u it needs to do lots of uh logistics stuff. I allow reading for everything. So you can still think about this this
uh this is approach. It's a different approach for computers. It's basically reduce computers problem into a uh coding problem in there. I have what I need. Let me write a script that opens
the website extract the content in there and then you can see the the agent is writing the code. After that it opens the website automatically on your so so similar line
is running on your own computer. If you have a virtual machine you can also deploy virtual machine there. So there's a and then you know after after clock run
the code it recognize some errors and after it gets the errors it try to revise the code. So this is pretty much if you enable you know like auto approval this is pretty much already um
a self-improving uh computer agents there some level of self self-improving there I do want to see the it's not there
and you can see claw code is decided to use accessibility snapshot instead of the screen screenshot because you know it's a summarization task. Okay, it's coming. It gets uh
it actually summarize it did. Yeah, it summarized all of the events for AI engineer in there. Okay. So, I'll pause here and I do want to show you the full code
which could be interesting to look at. Yeah. So, these are the full code. It looks like JavaScript uh just like any any other JavaScript and it does open a window take the snapshot and write to
the file. Okay. So, let me go back to my presentation. Okay. So, so I I've been talking about like two completely different approach.
They're orthogonal. They're not the same same approach. But the question is do we actually need two frameworks? Right? So, so they they try to compete with each other.
And this goes back to like a like a very uh interesting and important topic in in the whole deep learning because uh when I was in deep mind I specifically focusing on this problem called
continual learning. So that's why we did a lot of study on like uh AI systems in there but in the end we realized okay there's no silver bullet like yeah the the when whenever technology you have uh
like you have still have to solve the two problems as a trade-off like the the learning any learning system needs to learn to new things and any of the learning learning system needs to you
know make sure you can accomplish the the task that you you you're already successful uh repeatedly. So this become it's called stability and plasticity dilemma. So you you want the learning
system to be stable but at the same time you want the learning system to to have this capability to adapt uh to adapt to new environments in there. It's never be the case that you you are both you
you're good at both. So that's the conclusion. Yeah, it's always trade-off. Then then the problem we can think about is how do we balance the two in it? And if you look at the two system I just
demonstrated these two systems are coding agent and VM agents right they're basically representing the two extreme of this learning system the coding engine produce the code and it's
determinance all the time but it's kind of stubborn right if you run a code every day if the environment changes then the code breaks so that that's the problem and the V engine is kind of
adaptive but the problem is you cannot rely on the V agents to uh you know like repeat the process all the So the core question is okay can we find a middle ground between the two
because the north star is basically the best adaptivity and the best reliability right can we also call this this curve we call this curve parental front and can can we actually find a the sweet
spot there so this becomes the current direction that our company is specifically focusing on like basically try to find a unified framework that we can combine these two
into one single single principle which which we call neurosyic right because there's a neuro then symbolic is there a way that we can actually like fuse these two technology into one single framework
that it becomes self-improving all the time and becomes robust when when reliable when it's needed becomes uh adaptive when environment changes so I wouldn't say we solved the problem I
wouldn't say we solved the problem but uh I would say like we we are already at the initial phase of moving towards this direction because there's a clear paradigm that we can do right now
because we have we have a VM agent that's better than human in a small benchmark and we have a coding protocol that we can actually write a code using coding agent and the protocol can
produce deterministic workflows in there. This is a simple way for us to do this new experiment. So basically we we let the VM agent to run in the environment and the agent keep writing
the code to a file and once this workflow is successful you have a you have a you have code that's that's that's successful and the next time you run the
agent you can run a code first instead of run a VM agent right if it doesn't work if the environment changes like when it breaks uh as as I just demonstrated there's
actually a healing loop that you can do with the coding agent. So this this become very interesting and uh yeah I added this slide. So basically another interesting uh story is this
year's cow hacks the the Sam hackathon the first place also use our uh our agent but this time they they're not using the agent to build a product they actually use the agent to test the
product which is another interesting category that everyone can look at because all of us are developers right we write code and we know like nowadays there are more and more PRs more and
more code then who's going to do the testing then computer use agency is a solution for that So basically this is the agent that can just move in the mouse click on the uh on the dashboard
and doing a testing for you. I'm not sure if I have time. If I have time I really want to uh show a demo about like the the testing part but I can I can try let me try
um let me try to show you the loop that we have in the product. It's a it's an initial effort. It works in a lot of cases, but sometimes there are still some hiccups in there. So, um
I do look I do look forward to everyone like testing the product and give us feedback in there. Okay. So, let me I'm not sure if this one is doing
Oh, this one. Okay. So this is the value VM. This is the agent is not you see I have I have a lot of VMs here and I I do put different names to to
them. Um so let me try this one though. So in order for you to enable the neuros symbolic mode you need to come to this dashboard click on agent there are a lot
of special there are a lot of special algorithm we do uh here like humanized control like the trajectory is different uh clical integration so you need to look at the sim flow code replay so this
is a specific experiment we're doing right now so the whole idea is uh basically the diagram I just showed uh if you enable that we can try something. Let me ensure the desktop in there.
Okay. So, we can try this. Yeah, we can still try the the simple thing. Just want to give you a rough idea of how this works. Go to AIE
web page and um find the first speaker on Tuesday. There's a always has problem during live demo in let me restart agent.
Let me see if my VPM is awake. Yeah, while we're waiting. So, let me show you another demo because I al I have a
backup in there if live demo needs no longer time. So, we can show another one. So, this one is a collaboration with Microsoft. Uh so Microsoft has uh Microsoft is also built in this computer
infrastructure uh because because they are the creator of Windows. So these are the VMs from Microsoft. It's called Microsoft Windows 365 for agents and the idea is
from this you can see like on the left side the chart. So the idea is okay like use uh use our agents on Microsoft virtual machines to to fill out the form on a traditional ERP. So this is one of
the biggest area for computer computer use agents because uh there's still so many traditional industries and they don't have modern APIs they don't have like you know they still running all the
old um softwares on their Windows machines right so if you have if you build out any like applications that allow you to go to certain verticals because we are not vertical company and
we are horizontal infrastructure if you're interested in building like towards different verticals these are good opportunities Um like for example some of our customers they uh they work
with insurance website they specifically working on insurance. So basically they use the agents go to insurance website and uh fetch the data and some of them are healthcare u they work with doctors
they have uh like doctors has windows uh machines with ERP system very old and you can use computeriz agents to automate all the data entry. So there's lots of ideas that uh people has been uh
exploring in this direction. So, um, yeah. So, those are hopefully that's that that'll give you some insights there. Oh, it works. Yeah. So, so let me
because I guess I have some skills that's already there. So, there's a skill here. Let me remove it. go to
AIE website and see who's speaking first on Tuesday. I already have website. So, let me make a challenge. I create a new one.
And you can see all of the logs here. And the interesting part you can see this agent is writing uh almost like line by line simulan code. And after that it's going to com compose all of
the code into one big file. Navigate into the schedule page. Okay, so we get the get the information and if you notice the agent is actually it's not a screenshot. So whenever you
try to find information the agent actually decide should I take a screenshot or should I take accessibility tree this time actually take access accessibility tree. So you
can see all the information that beneath in one shot in there. And what I can do now because I want to show you that uh save to a skill.
So if I enable that neurosy mode and I say save to skill, this skill will have all of the code that you just run in a in a previous session. And then we can look at AI speaker on
day. Let me find a skill there. I didn't find it though. Okay. So this skill is saved. So I'll try a
different query. on a different day. Let's try on Wednesday. And you can see there's a replay mode showing up. The interface a little bit different because it it does have the
code. It has the code and it changed the parameter of the code. And if you click on the activity log, you can see the agent is actually producing a lot more code in the first
step. So these are the the code. It has like different parameters in there. U by default is Tuesday, right? The default is Tuesday. If you set it Wednesday and it's going to uh change
the parameter there and there's a hiccup. You can see there's a hiccup for the the agent need to filter in there. It does find the information and replay successful in
there. there's a little hiccup and then after the hiccup the agent uh convert it to the neural mode which is a VM agent. So the first time it always try to produce all of the code as much as
possible because if you have a pretty good hit rate it's going to have a very determin successful overflow and then the agent also have verification in the middle. So every time it runs the code,
you have verification. If there's something wrong, stop the it's kind of like it's kind of when back when I was uh when I was in school, we we write everything that teacher says in the in
the memo as a playbook, right? And next time when I want to do something, I look at my memo and try to do all the, you know, the steps step by step. And if the environment is different, I notice,
okay, I I try to do this action, but the environment doesn't allow me to do that. And then I have to like think more spend more time on you know like exploring the new environment and try to figure out
like what's the best strategy to do. Okay. So let's move on. Uh I believe that's a good direction for us to for even for the whole industry to think about like what's the what's the
better um agents that we can produce with this kind of balance of flexibility and reliability. And we do see pretty good uh success in this this type of experiments. It's like okay
I don't I think the I think the main point is you don't have to spend the same amount of tokens each time anymore with this kind of technology because uh because the second time you run the
workflow maybe 50% of the effort is done by code without any LM request in there right and uh you can you can dramatically reduce your cost so that's why we also experimented in this product
we also experiment with people on like $500 unlimited task plan. So the price and plan may also change. It's not going to be like uh usage based forever. Uh because you think about like as a human,
I'm I'm a human. When I try to do a task a 100 times, I become so familiar with that. I can pretty much do all the things with my eyes closed, right? But today's agents, they don't have this
capability. every time you do the same task, even you do a thousand times or 10,000 times for the same task, even the same query, you still pay the same amount of money to anthrop and open,
which doesn't seems to be like the the reasonable uh is it's not economical as well. And yeah, so uh hopefully that that gave you a pretty good sense about what
direction we're we're going forward. And uh I do encourage everyone to try try it out because we we actively rapidly improving that neuros symbolic agents uh in in a in the coming months and uh
hopefully we can achieve some like a breakthrough in that direction. And the second second problem we have to think about is okay like what's the what do we do with with the computer? So two years
ago we we work on this autonomy uh technology on your computer. So basically we have a we we used to have a product that runs the agent kind of like called co-working on your own computer
but we receive a lot of complaint from users because everyone has only one computer and then when the agent is working on it it's in foreground you cannot you cannot do anything. Yeah. So
so that actually motivates us to think about okay suppose we have a human level agent uh that works. Suppose we have something that works. It's not it's not there yet.
We we're very close but when that day comes we have agent that works then we have to think about what do we do with the computer like do you still need one computer or maybe you need 10. So the
current observation is whenever you have something that works, users always ask us to can you can you actually host this agent on a separate machine? Can you host on a server? So this is always
always the same conclusion from most of the users. And then they will start to ask for more machines. So there's a trajectory. There's a trajectory. First we are in the in a in a phase that we
really try to make agents works. Once it works, you need 10 machines, 100 machines. That's why it motivates us to work on the cloud computers. Um I I briefly talk I already briefly talked
about it. I think that the whole point is if we have the cloud managed desktop uh it becomes so scalable and everyone in the end I hope everyone can have a 100 computers so you become the CEO of
your own AI organization, right? So you have 100 AI employees in there and everyone become 100x productive. So yeah, basically this year uh we we launched a product side in March. Uh
it's actually a pretty new product and together with that is called side box. is the virtual computer cloud in there and from the from from that demo you can already see I have personally I have 20
machines uh running in the background in there and uh this is a general diagram s is actually the interface and sausent which uh do which does is auton autonomy part does the planning and decision and
then doing operations on the sidebox which is a cloud computer and okay and then we have a lot of friends who developers. So they keep asking us so can you actually expose the API. So
today we actually uh releasing the API for uh accessing the uh the cloud computer. I don't think I have time to demo but it's actually pretty simple. It's called SAPI S API SAPI and
basically you can do everything that I did in the web portal through your terminal using the CLI tools and you can also use there's also API tool in there and uh we just released this website so
feel free to check it out there's a lots of APIs CLI and once you sign up you have a machine in the remote uh remote setting you can use this command line and API to uh control that computer and
integrate to your own product. [snorts] Yeah, I'll skip the live demo. So the the general vision for this direction is we want we want everyone to have one
single agent and this agent can access multiple platforms not just Windows. So we also have uh uh support on Mac OS. The NUX is coming soon. Lux is very uh economically friendly. is it's pretty
cheap uh in terms of our pricing uh like based on our internal evaluation and the interesting part is we also have Android. So Android is also coming soon. So I I do hope like in the future you
have a couple Android phones in the cloud you have a couple windows and the agent actually can orchestrate among all of the device to doing all different task uh in there and I want to show some
interesting demos for the Android. So these are all internal uh like evaluation. So this one is a play in the Angry Bird game. And the interesting part is actually drag. Let me see. Yeah.
Yeah, you actually drag, you know, it has a drag actions. He drag drag the screen and doing a task in there. And another demo I feel pretty cool is
authenticator. So you actually go to authenticator and uh it's the same interface inside. Basically we swap Windows VM to Android one.
So let me make it faster. Yeah, it goes to this M MFA authentication challenge website and fill out the forms there and then get the token.
Yeah. And then success. So, so this is this is one of the problem that people keep asking us like okay like can we do 2FA? still possible and I do I do see there's a there's some value but we have
to use it in with like uh c caution there and this is a example for I believe the social media on the phone u like you can actually do different kind of task there yeah it's pretty exciting
I I feel the Android is a interesting area that like the current industry hasn't really like explored much you know there there should be a lot of applications that we can do with Android
virtual machines. So basically everything is converged right now. So I started with autonomous computer and I decompose everything into autonomy and computer and then autonomy becomes two
parts the neural part and symbolic part. Computer become local machines and cloud machines. So s actually serve for both. So you can actually like integrate your own machines to be part of the you know
the fleet in there and you can also buy our cloud computers and then everything become site which is released uh in March this year. I'm pretty excited about this new product and I hope uh
that will bring a lot of new opportunities uh u because this is something like I didn't imagine like two years ago like we work on this technology we don't know much about like
what the vertical application we can do but we actually realized okay like for example there's a YC startup uh saw me in one of the event and they tell me they actually use agents for their own
vertical applications so I do see there's a huge opportunity that within like in the Next year could be in the next year the tech technology will be uh relatively based on the the past
estimation the next year will be very interesting to to see what's what's going to happen and then what I can imagine there are a lot of uh new applications like blossom in in in this
this domain in within a year yeah basically to summary we have four uh offering for all the developers side work is the basically the whole interface you can explore you go to the
different algorithms in the developer mode give it a try let us know how it works and we actively developing new technology in there agents simulan which is open sourced and everyone can use
it's pretty much free uh free to use and sidebox is the API for the for the virtual desktop and we just released today so expect some bugs uh do let us know uh and I I'll give you the
invitation code uh for uh today is aie EWF 2026 and please give it a try. Let me know uh the feedback. We do have a credits program like uh happy to collaborate in a diff in different way
different capacity and see how to you know like uh make your development better uh in the future. All right. Thank you so much. Uh I I want to leave some time for questions. So feel free to
uh ask me questions in there and thank you for your patience. [applause] I I'll make I'll make sure the photo is posted. So, let me know if you want to be mentioned in there.
Yeah, feel free to ask any I I'll stay here like for a few minutes. I understand like uh there's another events going there. >> Yeah. Question
memory anyway. >> Spatial memory. >> Yeah. for VM how it visually looks that >> for now more most of the memory are
reduced to text in there yeah and we do have as I mentioned in agent S2 we do have a model separate model so it's actually not a simple wrapper of a large vision language model so we have other 7
billion open source models sometimes we also like fine-tuning that model and that model is specifically responsible for recognizing the screen and the interesting part is we know the
foundation model is progressing so so fast and so good but it's still not very good at understanding the screen. So that's an interesting yeah interesting conclusion. Probably it's because the
internet doesn't have many like screenshot data uh available. So so it's it's not available for them to train the models in there. Um but I can imagine the model get getting better and better.
But it's just like there's always a gap. If if you have a model that can specifically for your own application, you can fine-tune that model. is always going to be better than like uh like a
general model from big labs in there. Yeah. Question >> couple questions. >> Yeah. >> Have you experienced
the process? >> Yeah. So, so let me answer answer your first qu first question first and uh the demo I showed is LM right generating the code and this is this is the approach I
call a stateless programming. So all of the past 20 years we all do stateless programming and the neurosyolic approach I call it stateful programming meaning when you every time you run one line of
code you look at the environment look at the state and the state actually change. So this is I I believe this is a new paradigm for programming in the future. So program will be more stable in the
future and uh we we tried LM and the problem is this problem is essentially multimodel right so accessibility can have some cap capability of you know like some but it's not it's still the
same there's no silver bullet so some people argue you can do computer agent just for screenshots just using screenshots some people say accessibility is good enough some people
say DOM is good enough but in the And after all the evaluation there's no silver bullet on everything. So you always has a corner cases of one which is not corner case of the second. So, so
in the end I think we are actually actually taking a whimo approach but you have lighter radar and camera at the same time right and some some company they may take the Tesla approach like I
I just have camera and yeah but if you look at today's self-driving car uh we do see a lot of whimo uh running on the road with better reliability there it's just that you have multiple systems that
help each other to to make sure they don't make mistakes like especially fatal mistake you don't want You don't want agent go to your bank and add another zero when you send a check
to other people. Second. Yeah. >> Second question. >> Yeah. So uh the good part for accessibility tree is that information
is used to render the screen. So basically that information has pretty much all of the information that's needed to render a screen and if you can use that information is definitely more
reliable. But there are some uh there still corner cases like the developer of that application can intentionally say this part do not expose in the in a tree. So, so you can still capture that
in a screenshot, but you don't capture in accessibility. But you can imagine some applications like uh for example, LinkedIn has a lot of B detection there pretty much the I used to tell a joke
with my friends like LinkedIn is is the Turing test for AGI. Yeah. [laughter] Because Yeah. If if LinkedIn doesn't recognize your agent as agent, then you pretty much achieve AGI
because you pass the Turing test in because they they're in adversarial setting and uh they have a lot of techniques to detect and like this kind of accessib could be one of them in
there. Yeah. So that's why you still need multiple you know combination of the modality. So yeah, there there's one person I I'll yeah I'll let that person ask question
first. Yeah, thank you. >> Yeah, thank you very much. First of all, >> so we're from Germany. >> We have a use case in Germany where you take information and then
a lot of things with them. >> But the question is how reliable is all of that that you have shown in the real world today when you compare it with the healthare sector for
example. So you need very accurate information and very accurate processes >> right >> what is your opinion on that? Yeah. So, so basically the question is so we need
a very accurate information. So basically we need everything to be 100%. Right. So even when you do data entry you need to be 100%. So uh that problem is the biggest problem of the current
foundation models but we do have a solution for that. So it's basically uh as you can see everything is a loop right. So all the all the fundamental principles everything is a loop. when
you do something you always verify criticize yourself and it's the same for humans. So when I when I propose some ideas to to my friends they they criticize so I have the loop so I can
like reason with less bios in there. So the the accuracy problem can also solve by this approach. So we we do have some experiments that I can tell you one solution we did is uh uh whenever you
have something you fill out the form you take a form reverse it and ask the model does that make sense right so ask the model to verify if the mo most of the time the model recognize something wrong
in there because verification is is much easier than generation generation general action and if there's something wrong you take the results back and ask the agent to do the same task again. It
actually improved the performance a lot. And um another thing you can think about is the accessibility tree like for us to write the code and for the code to be deterministically replayed you can only
do in accessibility tree. You cannot do in a screenshot right because the tree is text can you can do 100% matching there. So that kind of a 100% text matching is also important to ensure
everything is uh reliable. So in the end the conclusion is you have to have a system. It's it's not going to be one model. You put it there and then you can hope that's 100% it's not going to
happen in there. And uh but there's definitely a solution for your use case in there. Yeah. Sorry. Your turn. Yeah. How do you kind of keep it? >> Yeah. So, so in the product side we do
have a guard rail. So whenever there's a action that recognize that irreversible action the guaro will say stop there. So it's another system another system monitoring this agent. If the agent want
to do a certain action it'll basically cut off and say ask for approval. But I believe your your problem is even more challenging is more at the low level of the operating system level like for
example the agent will say open this applications and you do want this application to be blocked in there right so this requires another layer uh on the OS level that's why we believe have the
claw managed uh Windows machine is a better direction because we have we we have the VM so if you need we can also produce uh the image the VM image for you so So in a VM image you have lots of
opportunity to uh change the underlying you know structure of the operating system. So you can actually block some of the actions have the white list in there. So I believe that's the that's
the final direction. Uh we are still early on the VM part. I we we try to explore that direction and uh like yeah if you're interested like we can also think about some partnership in the
future uh on how to have better guard rails. Yeah, it's definitely like a security security guard is definitely like one important aspect in there. Yeah. Thanks.
>> Yeah. Just the protection of an irreversible something. >> Yeah. Yeah. So, so that that's that goes to the same principle that the first
line I put in the so basically everything is neurosymbolic. If you want everything to be reliable, everything needs to be neuros symbolic, you always go to the code first because code can
guarantee 100%. Right? When the code is confusing, you always go to the flexible agent and then come back come back to the code. So that's why the code is still super important. It's the same
principle for the design for the guar. So the guar is also like code first. there lots of rules happening there and if the rule doesn't capture it goes to agent and then goes to human for
verification and then comes back uh it's a it's a design principle that we try to follow in all of the system inside uh even every components uh in the in the system thanks question
>> yeah so I mean you know >> repeating cycles Does it get more? >> Right. Right. Right. So, so that that's the trajectory. So, so basically we want
the token consumption to eventually become zero. Yeah. event should become zero and you can see sometimes you can see the code actually start to have more branch
>> show showing up when you will see different scenarios you see more it's just like human learning right when you see uh you have more data samples then you are less biased in terms of decision
making yeah it's the same for this approach yeah >> oh so question has proprietary program that they for legacy programs,
right? >> On the initial prompt, you have to be very specific because it's never seen this program. Go to go to this check this box in how do you
like especially a long process is that a long promp or do you actually show it how to do something? >> Yeah, for for this version you can see there's a prompt and the agent doing the
task uh well maybe in a couple months we we also think about it's it's uh it's pretty common in research community. So basically if you can replicate like people talking about word models right
if you can replicate this applications uh that you have better control on not doing if you have guard rails that the the agent is not going to do like bad things and then you can do exploration
we call exploration in the environment. So basically you let the agents do free exploration on the envir environment for different uh like maybe a 100 times and then the agent become more familiar with
the environment and then your basically your prompt will be less you you don't need very accurate problem like whether this button or that button you you can also al also like succeed in terms of
like uh finding the first basically the goal is to finding the first successful trajectory once you found it it can be successful all the And you already know that process you can already tell the
exact process. >> Yeah. >> Yeah. Yeah. Yeah. Yeah. Yeah. For for many companies they call SOP. So yeah they have internal they have SOP. So we
we actually did a interesting experiment. We feed one SOP directly to the uh coding agent right and one shot in one shot everything and it's 100% successful directly. So it's pos is
there definitely possible but I I hold the uh the possibility that like everything now is still kind of demo phase. We we want a technology that's general enough that every customer
coming here and we don't have to worry about anything breaks in there. So there's still lots of corner case that we have to solve going forward. I remember there's one gentleman raised
hand before. >> Oh okay. Yeah. All right. All right. Thank you so much. Oh, one more. And actually your opinion on that
actually software but you can see like we are actually
the pixels certain things at the end. So this computer don't even see
it working on it. I understood that you want to agent working but is it something you Yeah. Um uh I I so uh my impression is
kind of like a map and then you try to let the agent to walk through the map. So he can just go if you ask us
to read screenshot go there figure out just how we should go like what it doesn't f
>> yeah yeah yeah so that that's the whole idea of the coding layer so the coding layer basically encode all of the trajectory into the code right and I think I think the problem we try to
solve is uh suppose there's legacy software then it's not going to change then this this approach work but if we talk about like websites right for social media or like different websites
and they actually change and when they change then your path is not going to work in there is there a way for us to actually adapt adapt the agent uh during that scenario
>> I mean my answer to that problem was if your main agent again like check it out. So you have continuous improving.
>> Yeah. Yeah. Yeah. I get your point. So, so basically his solution is okay, every time there's a change, I have a swarm of agents doing the exploration in there. So, I always know the the best uh
>> Yeah. the path in there. >> You have to like actually >> Yeah. Yeah. Yeah. That that's exactly my response to that gentleman. So, basically the idea is if you have legacy
system and you have is the zero cost for exploration, right? Because for example, it's if it's a bank, then there's a cost for exploration. I cannot explore for all of the actions. But if you don't you
don't have any like cost for doing exploration. You just do exploration like put agents in there and like let them try different things and map all of the trajectory there and it's actually a
simpler problem to solve. So basically I I think 80% of the the applications is actually living in the legacy system right now. And if we have you can like it's it's actually easier problem to the
general uh system. So that can also already capture a lot of economic value in the current industry there. Yeah. Thanks for your feedback. Yeah. Further furthermore
>> you think enough data you can signal enough data. >> Yeah. So, so that the question is
suppose we have enough data can we actually train uh trend the agents to be very successful. So there was a there was a lot of discussion around that. So basically my conclusion is you need more
exponentially more money than open eye and anthropobic to do it [laughter] because uh vision data is two dimensional tax is one dimensional so we are talking about like something like
much bigger right so so you uh if you have if you can give me like couple trillion dollars we can we can certainly consider that yeah but it's challenging maybe it's possible with the with a new
paradigm of but given the current paradigm of deep learning it's not something that's uh like I don't think it's achievable in the near term right now.
>> Yeah. Question final question. >> Right. >> There's a lot of Right. >> How does it do?
>> Oh, eventually your code will have a lot of eels logic in there. So, so you can imagine the code is the code is not going to be manageable by humans for sure. Uh but
the code is still be able the agent is still be able to write that code with lots of eels while loop uh complex logics in there and it'll still work and yeah that that's the eventual goal of
this system in there. Yeah. But you can also chop it chop it to make a simple problems each time. But then you have a boundary problem like okay how do you define a smaller problem like what's the
boundary? What's the input and output? Right? So sometimes you if you have a simple solution that's that's actually even better. All right. Thank thank you everyone.
>> [music]