Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
OpenAI's computer use is now faster than average humans at most tasks, with the next frontier being superhuman performance against expert users. The Decisions API, inspired by the Jev model and built on Luna weights in just four weeks, offers a different trade-off: extremely low latency at the cost of reasoning depth, and is expected to launch publicly once latency targets are met.
Key points
Computer use now accomplishes tasks faster than the average human in most cases.
The Decisions API was built on Luna weights and inspired by the Jev model.
GPT-6.1 Soul costs one fifth of Astra and one seventh for computer use specific tasks.
Async function calling allows tool calls to run without pausing model execution.
Dots get their own persistent Linux computer in the cloud for desktop app control.
Tools mentioned
Techniques
- accessibility representation for computer use
- async function calling
- mid-turn steering
- server-side compaction
- pre-warming caches
- event-driven waiting
- parallel inference
- file-based compaction
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
that now computer use is like faster at accomplishing tasks than like the average human probably in [music] in most cases. Um, and I think that the next frontier is to have computer use
[music] be like literally superhuman in its performance where it actually is as fast or faster at using [music] software than like expert computer users like us. Um, and I think that'll be really
consequential and exciting when that happens because [music] I think we'll be able to all of a sudden build products um that um provide just much more realtime experiences.
>> Like 4 weeks ago, this was not at all. No, this is like Jev inspired and like >> I think you're officially the first Frontier Lab to like clone adopt this. >> Yeah. Yeah. I I feel like OpenAI has
such a strong like hacker culture and like people are just like they get excited about things and so uh guy from inference this one awesome guy from uh the infra team are like uh this is
amazing we're going to like hack on it. They build a prototype. It like works and now we we're just like hill climbing [music] on latency and trying to make this as fast as possible and we want to
like launch it in the coming days. Uh so as soon as we hit our like latency target, we we'll try to get this out. [music] >> Okay, we're very excited to be here.
Today is OpenAI dev day special after podcast. We have Ari here who leads the product and engineering team for computer use agents. Before we kick in and dive deep on computer use, you want
to give a quick recap what was announced? What's the quick slew of announcements you guys had today? >> Yeah. Um, yeah, it was a super exciting day. Um, we just got out of the keynote.
It was really sick. Um, there were a bunch of computer use announcements that I think are worth thinking about. We have, uh, dots, um, which is the new, um, sort of personal assistant product
and, um, that has some really exciting computer use features. There's GPD 6.1 Soul, which is this amazing new model, um, that I think is particularly great for computer use because of, um, sort of
the cost and and speed uh, advantages. I think I think we shared that it's um a fifth of the cost as Astra and a seventh of the cost if you're looking at computer use specifically um which is
really amazing. Uh sorry there were so many things I'm trying to sort through it >> and the API agents API which now has computer use in it which is really cool
because now developers can build on the same computer use uh that is is part of codeex um and and chatbt. Um, and then there were some demos of our existing computer use features like app shot
where you can take the context of something you're doing on your computer and bring it into codeex and chatbt really fast. And um, and then like native computer use on your Mac where
Roman had it taking screenshots of his app automatically and he could do other things on his computer while computer use was using his applications. Um, so yeah, really exciting keynote
>> and not to mention this decisions API. decisions API >> uh off the bat are they all the same model like this is or the same data set distilled to different models um like
basically like is computer using decisions API or are they like kind of separate so what's really cool about the decisions API is it you know it it has all these new capabilities it does
inference in parallel um it doesn't have reasoning it's a smaller model um than the ones we use for computer use um and so those those capabilities make it really fast. Um they also make it a
little bit less good at doing like long horizon um sort of sophisticated tasks. And so I think it's I would say it's still an open area of research for how we like bring those approaches together.
But uh yeah, I'm really excited to see what people build with the decisions API. >> One of the interesting things is dots now have attached personal computers. So
it seems like they're very much more persistent. You've been using them for a while. How should people push the bounds? It's like what should people aim for? What should they try? Personally,
right now I use it for a lot of customer service like oh this was wrong. I don't want to sign in. I don't want to authenticate. Find whatever and just get it fixed. How how should we push
further? What should people try? Dots are really cool product because each dot >> has access to its own Linux virtual computer in the cloud. Um which is different from our other products. Um,
you know, traditionally we've have access to a browser in the cloud or it has access to your own computer, but now you get your own entire Linux computer in the cloud and so it can run full
desktop applications. Um, and it can also use a web browser. And so yeah, you know, I think the powerful thing about computer use and the reason why I think it's so uh exciting is because it makes
it so that the agent can do anything you as a as a person can do. Um, because all the software in the world was designed for humans and now agents can use that same software uh and you can delegate to
the agent. So, um, yeah, like anything that you would do on a computer, you can ask a dot to do. Yeah, I think what particularly useful is going to really depend on who the end user is and what
what's valuable in their life. Um, but yeah, I would just start by thinking about like what are the things that you spend time on and how could you delegate those to an agent?
>> Yeah, a lot of flight booking and shopping and honestly even like playing a game or whatever, right? >> Totally. Um, yeah, I don't know. for me. Um, something I did recently, um, I've
been working on I I've, um, subscribed to a meal prep service cuz I was trying to like eat healthy, you know, and I really like this meal prep service I found because it lets me customize the
meals I order to like a high degree of granularity. So, I can say like, I want this many grams of chicken and this many grams of of of rice. Um, but it was so complicated, it took me two hours to do
an order. And I found that I could ask computer use to do it for me and it did it in 15 minutes. Um, so I I actually saved two hours. Um, it both did it eight times faster than I could and it
saved me two hours on on GP6.1 Soul. Um, so those are the kinds of tasks that I feel like uh are really powerful. >> As a creator, I can tell you automatically immediately my number one
use case is automating YouTube. Uh, because uh YouTube doesn't expose a lot of things via API and you have to just put it in a VM and just like run it uh for like let's say let's say their AB
testing feature or making community posts. None of this is available by API because they hate developers. Um anyway, so >> I've heard that from our developer
experience team. [laughter] They use it with YouTube a lot. Yeah, it's really awesome. >> Um so I want to draw for you know uh let's say I want to get a little bit
spicy. One of our the leading AI podcast our friends uh is famous for saying that computer use hasn't advanced in the last two years. >> Yeah. uh which is a very interesting
statement and I think you're one of the best people in the world to uh to to talk about this like how have things have progressed right >> yeah you know they said that a few
months ago I think and I hope they have a different perspective now because computer use is like 180 degrees different >> he's a tough guy to impress [laughter]
>> okay >> um it you worked on you've like basically spent your whole career working on like some kind of computer automation right like shortcuts at Apple
and then sky and then and then joining joining open AAI can you draw like what your through line is for like what is driving you and what you what wasn't possible back then maybe and like what
your sort of milestones were. >> I I guess to add on to that as a follow-up question, what's the major change from using codeex computer use from like last week through to today? Is
it model? Is it dots? Is it harness? So all the history plus what what really just changed in today's announcements? Yeah, on the through line, I guess I've always been excited about automation and
helping people automate tasks because then you can like save time in your life and focus on things that are more important to you than like operating a computer very intricately. And so, um,
yeah, that that was why we worked on some of those products. I was at Apple before. Um, we we made a company called Sky. Um, we ended up joining OpenAI, which is really exciting. Um, and I
think something that was >> almost like you have to hack around Apple until Apple was like fine, like we'll just hire you and you can just work on the [laughter] inside, right?
It was it was a cool place to get to work. Um was really interesting looking back at Sky as we were we were working on computer use there as well and the models were so much less capable and now
the models just in the last one year have become extraordinarily capable at computer use. I think the biggest delta that I see is before they could like reliably start tasks but then they would
run into problems and now they're really good at debugging. They're really good at trying again introspecting what is and isn't working. Um and I I think we've also brought the computer use the
computer use field itself has moved forward. I think we're using more techniques. Um now computer use um often writes code. So if you actually look at it in codecs and you expand the tool
calls manually, you can see that it's not just doing one action at a time. It's actually writing JavaScript code that it executes that that the computer executes to perform sometimes many
actions at once, which is a great, you know, speed up and great capability. we use more accessibility sort of multimodal interfaces. So um the model may use screenshots, it may use
accessibility, it may use playwright. Um it can use a lot of different mechanisms uh based on the task at hand. Um and then yeah, the the model acceleration has been has been just amazing. So um
yeah, what's different today? I think we're making communities better all the time. So I think just like one day's difference um is is probably a little bit less consequential than like even
the past month or the past two months. Um but uh yeah I think the the computer use in dot is really exciting as well as um the new model that we came out with >> on the keynote. Tijall was mentioning 7x
improvements in computer use speed um a lot better on a few benchmarks. How do you guys think about measuring it? Computer use is one of those things where
>> as you say you know it's improvements over time. Is it harness? Is it model? Is it post training? How do you guys look at it internally about measuring how good it is and what were the changes
with the new model? >> We actually have a bunch of different ways of measuring it. Some of which are on different permutations and configurations of the harness. It's a
bit of a complicated story because you know our production products have you know some more safety checks um and uh you know those are configured differently based on the the the needs
of the task at hand. So there's a lot of ways to to measure it, but I think regardless of how we measure it, we find pretty consistent gains. Um, and those gains are um sometimes in the harness
and sometimes in the model. Um, and yeah, I I was really excited by this result that GPD 6.1 is even more cost effective for computer use than its baseline cost improvement as compared to
Astra. It's like really cool to see. >> Yeah. I mean, one of the visuals I really liked from the live stream was that uh you're sort of improving the parto frontier of uh your um curve and
there was a lot of talking about how you're improving it together with the harness. Can you give some examples of aha moments that you had uh whether it's on like model driving the harness,
harness driving the model, whatever? >> I don't mean to repeat myself, but I think like introducing more modalities has been really powerful. One one more specific example of that is um in the
past I think we saw a lot of computer use um products had to spend a lot of time like scrolling you know so it would like take a screenshot it would try to do something it would be like oh I got
to like scroll down to the next page of results and then it would take a screenshot and then it would try to do something it would scroll down again and so I think um with with accessibility
and other and and direct access to the DOM and other things like that now the language model can actually see like an entire page or an entire application it can write code that can do multiple
steps at once. And so I think those have been probably the biggest single aha moments. There's like a lot of tiny ones that are less exciting in comparison, but actually we do find also that a lot
of speed improvements are driven by like a lot of little paper cuts that we got to go in and introspect. >> A lot of really hard engineering. I mean appshots in general, right? like uh I
think people don't quite get the difference because there's like a nice visual in Codex when we when you take a snapshot but they don't maybe they get the difference that u you are able to
actually drive each button and you have the you have each text um in in a very optimal representation. >> Yeah. Yeah. Exactly. Yeah. It's kind of [laughter] fun actually if you want to
be like really nerdy about it. You can go into codeex take an app shot by hitting the two command keys. So, you grab the content from whatever app you're working with, bring it into into
the uh codeex or chatb chat, and then if you click on the attachment and you click on this like little tiny button in the top right, you can see the raw text and you see the raw accessibility
representation. And yeah, we've put a lot of work into um >> dumping everything out, >> dumping it out, but also making it token efficient. Um doing it efficiently.
There's like a bit of an art to it. And uh you know it turns out that the same technology that was invented for humans um you know who maybe have accessibility needs who want to use a screen reader
technology that technology is really helpful for them to be able to use computers. It's also really helpful for LM to be able to use computers. So that's been like really fun to get to
work on. >> For context I feel like a lot of people don't understand appshots. They don't even know it's a feature. It's when you double hit command it pulls in what
looks like a screenshot and you're like oh why have I opened up just a screenshot and thrown in? No, it's actually pulling all the metadata, all the code, everything.
>> Yeah. Exactly. Yeah. So it's like, you know, if you take a screenshot of a web page that has a link, the screenshot doesn't include where the link goes. It doesn't include, you know, maybe you
take a screenshot of your calendar, the event titles are truncated, you know, but when you take an appshot, it gives like the language model like full context about everything and um that
lets it uh just sort of like do do much more. >> Yeah. For those who want to see more, uh Jason Louu, I invited him to do a full workshop on this uh at AI engineer.
Amazing. >> He did a great job. I have a broader vision question on computer use agents. So your example of take a screenshot, scroll page, take a screenshot where we
were today, they can automate a lot. Uh what are the bottlenecks? Is it models? Is it harnesses? What where do you see it going in like 2 years? Do you see it just running for hours? How do we get
there? Any any predictions on where computer use goes? Yeah, I mean I I think what's really crazy that I think you know the teams accomplished over the past couple months is that now computer
use is like faster at accomplishing tasks than like the average human probably in in most cases. Um and I think that the next frontier is to have computer use be like literally
superhuman in its performance where it actually is as fast or faster at using software than like expert computer users like us. Um, and I think that'll be really consequential and exciting when
that happens because I think we'll be able to all of a sudden build products um that um provide just much more realtime experiences and I think it'll also lowering the barrier to entry of um
or the activation energy I suppose of using computer use I think will make us start to default to doing certain things in agents that we've become accustomed to doing manually. And I think that's
exciting also because it'll save us a ton of time. Um, and I think there's a that, you know, there are a lot of different little paper cuts and bottlenecks that are sort of standing in
the way of that. Um, um, I think that there's um, yeah, there's things on the model side, there's things on the inference side, there's things on the harness side, there's things in the in
the representation. You know, we find that as computer use gets faster, we're increasingly bottlenecked by just like the speed of doing an operation. Like for example um you know a non-trivial
amount of time in our benchmarks of computer use tasks is actually like like let's say you're automating a task on door dash.com like a lot of the time is actually waiting for door dash.com
itself to load you know. >> Yeah. Then you just write a wait and then you execute the weight. Yeah, to totally and and you want to get actually it's it's actually really important that
you get that de like you want as little delay as possible between when it finally finishes loading and when you go and trigger the LM do the next action which is actually itself a statistical
science >> like an event driven way maybe to do that >> when possible you want it to be event driven and JavaScript has load events
for the web browser has load events for web navigation but there's other types of events that actually really can't be event driven so there's a lot of complexity
>> the one that comes to mind is like chatting with customer service replies could take 30 seconds, could take 3 minutes. >> Oh, right. I have dealt with so many
bots with codeex. Uh, it's great, but I also wonder if the other side knows that they're talking to a bot cuz I'm like answering in complete sentences. Like I'm capitalizing correctly. Like I'm
giving full reference numbers and everything. Like it's too it's clearly too good. I don't care. Like [laughter] I'm just like trying to get my >> I prompted it to like you know don't
pretend you're a bot. Be be very annoyed human. Short oneliners like push it. Do all this. I I also tell it while you're waiting for responses like use sub agents to research better ways to figure
out what we need. Nice. It's just like human little intervention. >> That's awesome. I also feel like half the time it's a bot on the other end. So now you got the bots talking to each
other. >> Yeah. I I will also say you know like uh you know one milestone of computer use that we we're at now is you know 3 four years ago we were scared of hooking up
LMS to uh to to the to the web and to to our to our devices. And now I'm having it configure DNS for me. Wow. I'm having it pay my bills. Uh, and like really like tens of thousands of dollars of
like stuff I'm just sending it over and yoloing with computer use and like h you know what's what's the worst thing that can happen. [laughter] Um, so that's that's all really good. I think now that
you've you obviously you also have to dog food your own products and all these things. Now that you've sort of released this in API, what are some pitfalls or tips that you want to tell developers uh
because they're about to I guess encounter all this uh firsthand? >> First of all, I'm just really excited that we brought computer use into the agents API. I think this is um really
great because obviously a lot of developers are building applications that want to be able to work with third party websites and and and and services and so computer use has this
universality to it. It can work with anything. So now all of a sudden developers can build using the same computer use implementation that we're building on. I think there's great work
to be done if you want to build your own computer use harness but it's hard and also we train our models on our computer use harness. So there is um an advantage to using the one that's in distribution
for the model. There actually might be a speed and cost and and and accuracy advantage. So I think it's really great for people to get to build on top of that. And um um yeah, you know, I think
kind of to to the point that you were making like I think we're all sort of still in the process and maybe like some of us are ahead of many people in the world of like getting comfortable with
this technology and trusting it. And so I think it's incumbent on us to um sort of build that trust over time by making sure we're building things that are reliable, by building um the right kinds
of of safety checks, by asking for the user's consent before doing something consequential like making a payment, by uh asking, you know, maybe depending on the application, making sure you're only
letting it access the websites or applications that it actually needs for the task. So that's I think something important to to think about. Um, but yeah, I'd really encourage people to try
the the new agents API, build all kinds of cool stuff on it. We'd love to hear your fe feedback if if uh you know, depending on how it goes. >> Have you seen any changes in the way it
affects dev workflows? So, one of the things with dots is you know, you're seeing it in Slack, you're seeing people use voice and build. Um, the example Roman showed of change this app and send
me screenshots along the way and all this. Um, is anything that you're seeing there in adoption about how people are using computer use for coding workflows? Any tips people should take from that?
One of my favorite use cases for computer use actually and one that we see a lot in the wild is computer use letting the agent actually test the software that the agent has built which
is far more consequential than it sounds because traditionally you know you'd build something in codeex and then and the codec builds it for you and then you have to test it and you are now like QA
for the agent right so with computer use you can complete the develop the software development life cycle um where uh the agent can build software, it can test it. So I have a lot of fun uh you
know building stuff, having the agent test it by the time it comes to me it's already working. Um I have extra fun because sometimes I'm like developing computer use itself and so now I have a
computer use agent that's using my computer use agent that's using something else. Uh so yeah I I really I really think this is a super powerful class case.
>> I have a visual play test skill that I've developed that um really catches a lot of design issues um that you know normally when you just look at code you wouldn't really pick it up. Uh, it's
also really good for cloning apps. So, if you're using a shitty SAS and you want to kill the SAS, you just clone it screen by screen by screen. Uh, and obviously [laughter] computer use can
completely drive everything, take screenshots, note it down, and then clone everything with codeex. Uh, but yeah, thanks for all your progress. I think, uh, that is our time. U, this is
not the last that we're we're going to talk. >> Yeah. Cool. This has been really fun. Thank you guys for having me. >> All right.
>> All right. >> Okay. We're strict cut off. We're just going to dive right in. >> Let's do it. Yeah. >> Okay. So, uh, Nikunj, we're very excited
to have you. You've shipped a lot on the API side like we just talked about with re you can now build with computer use agents. Anything you want to highlight the API side of changes and introduce
yourself a little on what you do. >> Yeah, for sure. My name is Nick. I lead product for the API team. Been here for roughly 3 years. Um been working on launching models. I feel like that's
just been like a thing constant thing throughout throughout my time uh here at OpenAI. And uh with every new model we try to like basically work super closely with the post- trainining team, the
research team to figure out what's new in it. Um and then we like expose those capabilities in the API. U so that's like the the basic way of putting it. Um and if you just look at uh everything
that's new with GBD6, the the cool new capabilities that we launched were firstly async function calling. Um so what you see with like um uh like a lot of the things that you're seeing in like
codecs and dots and everything is that tool calls take so long that you don't have to like pause the model's execution while uh the tool is running. So you could just like kick off a tool call,
keep running, keep reasoning and then check back in. Uh so we launched async tool calling. We launched like mid turn steering. So now you can like inject messages while the model is reasoning uh
in the middle. Uh so as your tool call finishes you can put put in that instructions >> and that's also partially a model alignment capability right like they
have to train in the ability to train >> I feel like we've had it in the app you could always as its reasoning you could steer it wasn't the best it's gotten much better
>> excited to see how it does this >> and like our main goal in our main goal in the API is to like put things in the API once it's trained into the harness and so we kind of wait for that moment
until it's good enough and a lot of that is like actually being powered by websockets which we launched uh a few uh I want to say months ago and so websockets just opens this like whole
birectional like communication thing with the model this is not uh the GP live thing I'm just talking about GB 6 uh and you can do all these like async tool calling async reasoning injecting
messages so it's a really fun API to work on I think like really enjoying >> this is why we are the engineering podcast because we get to talk about websockets uh this also pairs very well
with ultra fast, right? Like that that is now like I think for the first time ever available in the API. >> Yes. >> Uh which is basically the the
theoretical fastest speed you can ever get. Frontier intelligence. >> Yeah. Yeah. It's been so exciting to work on that project. I think uh before I go into the API, the most fun part of
uh Ultraast has been just watching the the inference team cook with Astra. Like they're just like constantly having these like CEX agents running trying to like squeeze out more performance. And
uh I would say like uh at least for a couple of months a lot of it was focused on efficiency and driving the cost down which is how we like were able to cut the Luna price by like 80%. was like a
lot of that was driven by like uh all the the inference improvements they landed and then now they've like shifted gears towards like how can we make this run as fast as possible and so ultraast
has just been like amazing to see on a on a model like Astra like to go that fast has been really cool and yeah websockets is like actually it was like the the first time we launched
websockets it was for GBD uh 5.3 codex spark which was can't believe we named a model that but uh you know that that's what we launched it for and obviously ly it helps so much because like you got to
have the tool calls. Um you got to like really reduce the overhead uh of of going back and forth with tools and and so uh websockets is awesome for that. >> Yeah. Uh it's always cute to see like I
have my reset usage limit and then I have my spark usage limit that I never use. [laughter] >> Like it's there if I want it. >> I think it's gone finally is gone. So
[laughter] >> yeah. Um you're slowly killing off all the you know >> the old ones. It was a great week. I mean it was the first time we had
frontier intelligence at extreme speeds. >> People really liked it. >> Yeah. >> So first time it comes back. >> Yeah. And for uh 5.3 spark is explicitly
attributed to Cerebras. You guys are not confirming or denying that uh ultra fast is related to Cerebras but people are I I'll just say that people do do care and are wondering about it. And you have
your own silicon as well. >> Um elephant in the room decision models. >> Oh yeah. >> Decision API. We were the first podcast to do a big Jev deep dive with uh Dogo
and I also you know featured him at AI engineer. How quickly did you see Jeff and go like >> yeah firstly like huge props to do and like the Jeff team for like really
inspiring whole segment in the market. Like obviously Jeff comes out everyone's like losing their minds over it. Our users are like hitting us up but also like our
internal teams are like we need like a much faster classification system. we can uh I don't want to like u get ahead of some of the the dots features that are going to come, but you're going to
see like some cool like really snappy fast things built on top of the decisions API. Um but you know like yeah, props to Jeff for like inspiring this whole thing. Uh obviously a bunch
of people at OpenAI get nerd sniped by that and they're like how can we like make this work? We're not going to like train a new model, but >> like four weeks ago, this was not on.
>> Not at all. No, this is like Jeb inspired and like >> I think you're officially the first Frontier Lab to like clone and adopt >> this. Yeah. Yeah. I I feel like OpenAI
is such a strong like hacker culture and like people are just like they get excited about things and so uh guy from inference, this one awesome guy from uh the infra team are like uh this is
amazing. We're going to like hack on it. They build a prototype. it like works and now we we're just like hill climbing on latency and trying to make this as fast as possible and we want to like
launch it in the coming days. Uh so as soon as we hit our like latency target we we'll try to get this out. >> It's interesting at the same time of hacker culture you also as Sam said like
99% one of the most reliable APIs with I think probably the most usage which is your team directly. Um how should people see decisions API? I feel like a lot of people saw Jev, heard the buzz, haven't
built with it. You're making it very mainstream. What should people see it as? How should they use it? >> Yeah, I think the the main use cases we've seen is like really fast
classification. U all the computer used demos have been amazing and really cool. I think there will be limitations of course um in terms of you know having Astra like write like a JavaScript like
script to control your computer versus having Luna pick like one action at a time. I think uh it's not going to be at the same intelligence level, but like maybe there's some computer use tasks
that this is good enough for. So excited to see that come through. Um the other cool prototype I've seen internally is people hooking it up with GPD live. So GBY live is like you know our our
birectional like realtime uh API and uh it's built on this like model of front-end models and backend models. So Jubilee live is this like super fast yeah thinker docker thing. So, Jubilee
Live is the talker. Uh, super fast, really good at delegation. Uh, and you have something like Astra sitting at the back at the back, but tool calling has always felt like really slow in in GD
live. Um, and so people have been like putting together these like tool calling demos of GD live controlling a computer and just feels like so much more snappy and and natural. So I'm like kind of
excited to see like what people do with live and with uh with uh with Luna on decisions API. Um so that'll be pretty exciting. Yeah. >> So I want to iron this out for people
especially from the product side because a lot of people have been putting out Jev clones. There's been about 100 in the last two weeks. >> Oh really? That's amazing. The first but
like uh like they can clone the Jeff API which is honestly structured outputs which open was first two. >> Yeah. >> Right. So like I think let's let's iron
out for people what is a decision model as far as as far as like what is important. It is not just latency. It's not just structured output, right? Because I could just have Luna. It's the
decision models priced the same as Luna, right? >> Um have turned off reasoning and then have structured output. Do I have a Jev? Uh you know, no, right? And that's
that's the real >> there's a confidence there. >> Yeah. >> Yeah. Totally. I think uh the the way that so we haven't trained like a new
model for this. We're like building this purely on top of the same Luna weights that we have. Uh so yeah, this is like really just Luna and uh on top of that, what you're doing is you're
constraining. So like structured output is a big part of it. Uh you're really optimizing the the inference stack to like uh get get very fast on TDF and because you can have multiple questions,
what you do is like you basically run those in parallel. Uh these Yeah, you you run those in as a batch. um you all sorts of like inference techniques people are working on to try to make it
as fast as possible but I'd say like at least our implementation of it at the start and in this first version is like zeroing this on top of Luna uh to see how it goes and obviously you want to
like put it out there like this is open like classic iterative deployment thing put it out there see what people think and then like we we'll we'll make more model improvements uh as needed um so
yeah that's uh this decisions API Yeah. And obviously as a benefit you have vision, they don't have vision, right? True. Obviously comes >> like uh we get it for free with Luna.
Yeah. >> Yeah. I do think that like you know some of the innovations it sounds like uh it's still to come if it's still still the same Luna weights which is like the
the confidence stuff and calibration. This is something that we've talked about on the podcast with uh benchmarking calibration because basically the whole point is that RLHF
kind of collapses you towards what you want to hear. >> Yeah. >> Uh but like not actually like what the amount of confidence is. [laughter]
>> Yeah. Yeah. Totally. I'm I'm eager to see how it uh hit pans out. You maybe maybe this like going to be these are going to be like the key areas where we may have to like hill climb uh with a
with a future model release. But um yeah >> and then architecture- wise the other thing that's in the debate obviously you nobody knows because Jeff doesn't talk about it but the the two speculations
are one maybe the fusion model instead of auto reggressive >> uh but you you are able to achieve the parallel generation in in in your way >> and then the other one is some mech
interp type thing that that you're like analyzing the activations and then just outputting the weights which like you guys have all done the research on this people have speculated
>> there have been demos on both of these as well. I think Gemini shared a Gemini diffusion, Gemma diffusion on a Jev style output and uh inter people have also you know pulled out interp from a
middle layer but this is all speculation. It's just like what are you trying to aim for, right? Because you can achieve the API. Everyone can achieve the API. It's actually pretty
trivial. But like then there's the speed, then there's the accuracy, then there's the other calibration features. I don't know what else. [laughter] >> Yeah. Yeah. No, totally. [clears throat]
It's uh it's so cool that this like whole space has been kicked off now and people are going to do so much cool stuff and everyone's going to learn from each other and um yeah, I'm excited
about >> I feel like being on the platform team, a lot of your job is to empower builders. What do you think people should build with decisions API and also
computer use agents? Any any stuff that you've been building with internally that you think really opens up after the new changes? >> Yeah. Um okay, let's think. Um decisions
API I use cases internally have been pretty obvious like the user ops team was like jumping on it. We're like we got to classify all of our support tickets. Um what else came up? Uh
obviously there were like the really cool GB live demos. I'm sure like the Codex app team might like pick this up and try to do something cool with it. So, you know, like this whole thing
started like a week ago, so it's it's like very very early. Excited. Yeah, >> there was a big push in um Eval's LM as a judge having really low latency there, >> right? Yeah. Yeah, that'll be
interesting to see. Um and then um uh with with the agents API uh we we have we're basically like having a bunch of firstparty products like uh at at OpenAI built fully on top of it. Um we've had
the COC security stuff that just went out that's fully built on top of uh the agents API. We have u sort of the we we're having like a meetings type of thing launching today. I think there was
like a demo. Um, do you remember like the plug-in extensions when Sam was showing it? There was like a demo for like you're in a calendar, you can sort of like uh have your meeting notes like
drop into granola type thing. >> Yeah. And so that all of that stuff is like fully built on top of uh the agents API. Um, and yeah, I'm like just excited
to see like we're just getting this out and let's see what people build on top of. >> I I think you showed it off very well. The whole edit spaces, pages,
collaborate, add in your dot. Like that's that's a lot. So >> there's a lot of inspiration people can build from. >> Yeah. Yeah. All possible with Astra, you
know, like things just move so fast now that people go from idea to execution so quickly. It's amazing. >> Is there something that you want uh people to focus on to give you feedback
like what like you know maybe you're um you're you're just putting this out there and you want and there's like a fork in the road and you want developers to help you decide. So I think agents
API and decisions API they're like these are our newest products would love like any and all feedback on that to figure out where to take them. I think uh over here we're like very open on responses
API uh which is sort of like our our workhorse over here. Um like really focused on performance right now and the performance comes in like two main ways. Uh first is just like latency. We've
been like rewriting the whole responses API track to like make it as fast as possible from a DTFD perspective, DBD perspective. So there's like that's like continues to be like a main area of
focus for us. The second thing we've been trying to do is like really go deep on caching particularly with these like personal agents that are you know like um basically like a single thread that
just goes on and on forever. We've been uh trying to like really up our game on caching. We provide now guarantees of like cash hits within like 30 minutes. We're actually like we for one of our
users, we just launched like a much longer cache window. So we have like a 12-h hour caching guarantee uh that we offer so that you you you have like guaranteed cash hits for
>> Is there a public API? Not yet. That's in preview. We're going to like uh try to get that out to everyone as soon as possible. But like just pay a little bit more for the cash, right? And we like
guarantee um like cash reads for like a much longer period. So even if like your instinct thread for example like you just like do something on it and then come back to it like 3 to 4 hours later
you you're still getting the caching performance out of it. >> Cut the cost there quite a bit too right with the new model. >> Oh yeah.
>> Yeah. We're like driving down cash reads. Yeah. >> For builders they should implement because it's significantly cheaper. >> Yeah. Yeah. just like building your apps
with like uh to be very cache aware and and sort of like use our prompt diagnostics or cache diagnostics tool to figure out like where things are are dropping off and and um so so the the
caching part is like really important. Um yeah, I also want to talk about pre-warming. We have that in the API now. So like if you know that hey I'm going to get a cache like uh sorry I'm
going to get this prompt. I just want to like pre-warm the cache, pay like the cash right fee right now and then like uh have it sort of ready to go for the next 30 minutes for whenever
>> and I can spawn many instances of that thread. >> Exactly. Yeah. you can just keep going and have and so yeah like I'm very excited about
getting feedback on like the lowle performance things so that we can keep making responses API the most performant and reliable way to like build on top of an LLM and then you you basically have
our like new products where I'm just looking for like any and all feedback. >> Yeah, just use it. So yeah, just tell us what just yeah define our road map for us please.
>> [laughter] >> So yeah, >> uh I I think for me the caching thing great, right? Like obviously very needed, but at the end of the day,
you're still bumping up against a million token context and that's probably not going to change for the foreseeable future. Like you still need good compression,
>> right? What is the best practice there? >> Yeah. Yeah, totally. Um so firstly, um OpenAI has its own like proprietary compression >> which is comp. It's in the API agent
API. >> You you decide for us, right? >> Yeah, exactly. So in the agents API uh it comes built into the harness. Uh and if you're in responses API there's like
two ways of doing it. One is what we call server side compaction which is you basically tell responses API that if you ever hit this this threshold of tokens uh just autocompact it and and and like
go back uh or sorry like reduce the context uh being used. And the second way is like slash compact which is like if you want full control so you can like slash compact at any time have your own
logic on when to like >> it's not a [laughter] >> yeah but I mean it is the manual override.
>> Yeah it is the manual way and like uh I don't know but a lot of the the big coding agents like to do it manually. I I mean like if you look at the codex implementation of it in the code open
source codeex harness you can see that they they use /compact and do it. Um and uh there's also like new um by the way new compaction techniques that we're working on. Some of them you will be
able to see in the codex harness like it's already implemented in the codex harness and so um uh they're like some file based uh systems that we are like experimenting with. So yeah lots of cool
stuff going on around in compaction as well. >> Cool. Uh we are running out of time. Um, I think you've talked about a lot about performance and talked a lot about the
new APIs that you're launching. >> Can you give us any other hints as to things that you're interested in as as far as the future of the platform is concerned?
>> We're obviously like very low level like I I used to work at Stripe before this and uh at Stripe a lot of the game was like building these higher level primitives and products on top of like
the core payments primitives. Um, and uh, I'm always like curious about what the best way of doing that is in in AI. And I think we've had a couple of attempts at that. We like had launched
assistance API like way back in the day and like wasn't really the right fit. We're sort of like going off with this like agents API and uh, it gives you the CEX harness, but like where's like the
the what's the right amount of flexibility to give in that? That's like an open question. like how should we like have memory vaults and like all of these like higher level like API objects
to to take away or so like to to abstract away more um like storage concepts like this is like a whole like uh there's a whole space that I'm like very curious about figuring out how we
design um I think a lot of things in AI are just have a low-level API primitive and see an example harness and go and have your coding agent implement that but how much that to be built into the
the API is like a constant question that I'm thinking about. Um >> so I don't know if folks have thoughts on that. If uh if anyone has ideas uh uh it'll be super interesting to hear.
>> Yeah, the analogy I always bring back to and to end there is uh you're building an AI cloud, right? Like which is something that uh Sam said a year ago and you're it's almost like you're sort
of doing the AWS invention and you have to do okay this is EC2 and this is S3 and this is like but you're doing the AI native versions of each of these. There are a lot of analogies. So, you're
pre-warming caches for stuff that you know >> will [laughter] and it's nice that it's all exposed to builders cuz it just opens up ways that
you can build new things. >> Yeah, absolutely. >> Okay, >> awesome. Well, everything. Thank you guys. Thank you. Yeah.
[music]