Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Recursive decision models (RDMs) marry recursive language model architecture with system-one decision models like Jev, enabling complex behaviors through many small, fast decisions rather than generative reasoning. The key insight is that tasks often assumed to require reasoning—like playing Doom or Factorio—can be accomplished with pure decision trees, no generative model needed. The talk introduces three rules for RDMs: context stays outside the model, decisions can call sub-decisions up to a max depth, and the system must first work with no generative model at all, then compose with LLMs and agents.
Key points
Decision models like Jev can only rank and choose among options, not generate text.
The Doom-playing agent was built from two separate but coupled decision compositions.
Recursive decision models can operate with no generative model, using only decisions.
The arbiter pattern allows running many decision models in parallel while avoiding state collisions.
OSO is a computer-use agent built on recursive decision models, starting with voice input.
Tools mentioned
Techniques
- recursive decision models (RDM)
- recursive language models (RLM)
- decision as a tool
- hierarchical search with beam search
- speculative tool calling
- arbiter pattern for parallel decision models
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Hey, my name is Raymond Whitampamp and my talk today is originally titled recursive decision models. And what I think I've realized since submitting that title is what it's really about,
what we're really going to be talking about today is operating system one. Okay. So maybe some quick definitions or what what does that mean? Uh so system one models the the um
naming of system one models from type safe when they released Jev um not even two months ago now I think um came from uh this concept that Danny Conaman popularized uh in thinking fast and
slow. this idea of like the system one part of our uh brains is making very fast and intuitive like judgment calls, gut reactions, reflexes, maybe maybe reflex is more uh purely neurological
but um that idea of like a lot of the decisions that we make are not actually like purely rational. Um and so system one is fast, system two is slow, more deliberate, like logical reasoning,
problem solving, etc. And um I think what was a little unclear when they um when they announced Jev was whether they intended system one to be like their proprietary name family of models or
like that's they were expecting all the other people to call them system one models. I would say most um most people are calling them decision models and um and so I wasn't really willing to u to
wait for the people to decide and so so when I came up with recursive decision models um you know it was just it was just kind of using that. So, um, a rose by any other name, tomato, tomato, we
don't really need to get too hung up on this, uh, idea of what are they? Are they, um, decision models or not? Um, Ally here argues that Jeff makes the judgments, um, not the decisions. Your
code makes the decision. I think that's that's absolutely correct. Um, and yet I think that that much of the zeitgeist and much of the folks working in this space outside of Typesafe are are kind
of uh rallying around this idea of um decision model being the term. There is the hold out camp of um refusing to call them anything other than zero shot classifiers. Um anyways, I don't have a
dog in the race. the naming just trying to explain what I am talking about and I think that one of the things that is now connected here to this idea of system one system two Danny con econom
um maybe related to how we behave but I think much more interesting from this perspective of building AI systems is what if reasoning uh was actually many small decisions and um yeah maybe a
different way of saying this like what what if um behaviors or tasks that we might think of as requiring um a lot of reasoning or a lot of um like deliberate cognitive processing um
could actually be um solved in a different way with a lot of very tiny very very fast decisions. We're going to keep coming back to this question throughout the talk and I think it's
it's really key in what's interesting here. So recursive decision models. So this is um my concept um that I've been developing. It's um at its simplest perhaps the
marriage of two things that I got really excited about. One just over a year old. It's called recursive language models RLM. This is from MIT Alex Singh and Omar. Their original uh blog post is
just over a year ago. Happy birthday to RLM. Belated birthday. Um, and then like many others, I was also chomping at the bit to um I think it's actually champing at the bit. Anyways, I was champing and
chomping uh to get off the freaking Jev weight list, which thankfully got off real quick. Um, they did a good job with that. But I think that one of the things that I've been sharing with people is I
think that part of what type safe did so well and what captured the attention of so many engineers a engineers is um this idea of Jev as convergence right like decisions converge
and that is in direct contrast to generative AI which has been you know blowing up like crazy It's arguably like the fastest growing um you know market in the history of the
world and um so much of the attention and money in in the space has been about generation right and that's awesome and there's amazing things that are possible with generation but if you really think
about it like even if you if you have infinite tokens right you kind of have infinite divergence right like the generative stuff it's actually hard to um converge. It's like we still have
slop is someone still needs to decide like what's going to make it into production and maybe you've got your like code factory set up there. All those like agents are doing those
decisions for you. Um, but at the end of the day, like the the important thing that I got me excited and I think I think this is why many other people are so excited about about
Jev, these system one models, these decision models is [snorts] this idea of yin and yang, right? It's a counterbalancing force. Um, and so the decision model cannot generate text. It
can only uh rank and pick and choose amongst options. Um, and that is like in direct contrast to LMS which have this ability to just like spit out uh freewheeling open-ended um text and in
the early days actually hard to get them to structure their output. So I I see this as as yin and yang. Um I am not going to go super deep into um recursive language models. I would highly
encourage you to check out my talk from the summer from AI uh engineer world's fair this summer recursive coding agents. It's on their YouTube channel. It's also at recursivecoding agents.com.
Um my slides here are also a um a website and I'm going to make them available publicly after I record this. Um that is going to go a lot deeper into why I think RLMs are the new reasoning
models. uh why this is a really important paradigm. So with that we do need to sort of just draw out the direct the direct analogy here. And so in the RLM idea, right, the root recursive
language model has the ability to call sub recursive language models and those could call sub recursive language models and then you can you have to sort of like cap the depth of recursion to just
not um if you ever want to like come come back from the deaths. Um and so here we say like okay at depth two then it's an LLM. So the the leaf node in this graph in this tree would would
ultimately be an LLM call. We can make that exact analogy, that exact map onto system one models, onto decision models and say, okay, we've got a route um that can decide to call a sub uh RDM uh and
then there's some max depth at which point we just have to make like a single call, a single system one call, a single decision call um but we also have to think about composing these things
together, right? And the um I am going to show some examples as pure decisions all the way down. That's very important. Um this bottom left of what I'm I'm calling decision as a tool. Um so you
could you could also this is what my recursive coding agent talk was about. It's like replace RLM with like agent, right? Coding agent. Okay, you got the root coding agent, you got the sub agent
da da da. So I would say like this decisions as a tool or Jev as a tool was is like the obvious one and within you know 24 hours of people getting access to Jev there's tons and tons of examples
here right people using Jev for compaction people using Jev as the thing that decides whether a tool call can execute or not people using Jev as a router to decide which um which model is
going to get this particular request right so that stuff's Um, it's all very cool. There's some really interesting cases there and people just went buck wild going super fast um figuring that
out. I think to me what's I'm more interested in is um is not as obvious how you would set it up. Uh and that's where the root is a decision model and you
only can you could choose as a decision to call to a language model or an agent um as one as one possibility. So almost everything that I'm going to be talking about with RDM is going to um be in this
paradigm where we're actually letting the root or the parent of this system be the system one model, the decision model. Okay. So now I want to show this for a couple reasons. This you might
recognize this. This is the Jev plays Doom. It was part of like the launch tweet storm and got a lot of attention. And yet, despite that, I would say it's also one of the most um
underappreciated or at least I would say underdeveloped um uh examples here. And so I'm just going to show a very short clip from this
where >> we compose their answers into the player's next action and keep repeating that loop as the game changes. This is what we call a composition of AI
primitives. >> Composition of AI primitives. >> Now, this composition handles the moment to moment moment to moment. So basically the first thing that they're showing is
this harness that they built of decisions to handle like the first person shooting aspect. Is it gonna hold, carry on, walk, dodge, blah, blah, blah? It has these actions, right? This
is where I I it's so simple and it's very much in line with this idea of composing systems together, recursive decision models, right? A graph of different types of decisions. Um, here
we go. So this is the second composition >> composition that decides where to explore next and they can work together to make it through a level. >> Okay. So now you have two compositions.
One is the first person perspective. One is this like bird's eye or like mapaware perspective. And by composing those two things together you have a a doom playing decision agent. And now
to me I don't know maybe I missed something but I would say like um I have seen almost nothing very very very few um examples that are actually like taking this uh forward that are mo that
are moving this is a super powerful idea of composing these two systems together. It's really insane that with like essentially two separate um but coupled um so two distinct but coupled decision
agents, right? You now have with pure Jev calls um a Doom playing agent, right? Like that is really wild and I think really underappreciated despite the fact that probably like I don't know
millions or tens of millions of people saw this video. And it comes back to this question, what if reasoning uh was actually many small decisions, right? And um or what are the types of
activities or categories of things that we might naively assume would need reasoning when in fact what we need is a smart way of architecting small fast decisions. Okay, so now on to recursive
decision models. I think the the glib tagline for this would be like a decision that can decide to make another decision and um that's certainly an important element
of it. Uh while I was developing these um this idea this concept I've I've given myself three rules. Okay, the three rules of RDMs are one that the context stays outside the model and so
the code um can cut it into slices. So the the in the Doom example, the the full state of um the game uh is outside then those two little subh harnesses, the first person shooting harness has
some specific subset of that. The um the map exploring bird's eye view decision model has another uh view of that. Uh the second thing is this uh and now this is where it's a little different from
that demo is the decision can choose to call another decision and again up to this maximum depth we talked about this recursive tree now these two things are the most
important fundamental aspects of RLM as well right the context is externalized the agent can make sub aents the RLM can call sub RLMs and sub LLM right um the so those two things are like copied
straight from the RLM idea and it's applying that directly to to system one models decision models. Uh the third piece is first it must work with no generative model at all. And this was
very important to me um because I again I'm like all this like Jev as a tool for a normal coding agent is is is cool and there's some really great examples but it's it's obvious to me. What was really
interesting in terms of novel what might unlock some novel behaviors some novel capabilities um is what if there's no generative model at all. It also proves that the
that the RDM architecture is actually uh correct like it should be able to work with no generation and at the same time it must compose with LMS and with agents right um so those are my three rules in
developing this concept of recursive decision models. So first I want to show the the decisions all the way down example. Okay. So decisions all the way down.
Um you might be familiar with this idea of a golton board. Um maybe you learned about this in the context of learning about um the the Gaussian distribution or or um uh what do they call it? Normal
distribution. So like each ball has like a 50-50 chance at each pin of um falling and then you build this like bell curve um gaussian distribution. So the simplest way that I could think of to to
visualize this idea of um like recursive layers of decisions is now what if the pins could decide? So you could think of this kind of like a binary search like so each um numbered ball just like oh is
this one going to go higher or lower higher or lower? Um and so this pin is like splitting them in half. this pin spinning um in to its half etc. Um and again this is kind of like a a silly
simple example. Uh but now when those are smart then we separate them perfectly. Now system one model like Jev decision model whatever we want to call them uh well we can ask about we can ask
many questions at once. We don't have to just ask one on one axis. We could ask in multiple dimensions. So we can now visualize that in 3D as like the balls falling they're being sorted u by color
in one axis by number in another. We could put the cross-sectional views here and see that in the case that this is a perfect um classifier right um it's perfectly classifying the numbered
bucket that it should go in in the end the uh colored bucket. So again, this is very much a toy example in the sense that like actually what you you could do all of this in one classifier call. You
could do all of this in one decision call. You don't need um this like recursive tree or like iterative tree to do it. But I'm um I want you to keep the visualization of this because it's
exactly what's going to happen in some of these more specific examples of recursive decision models. Okay. So, hierarchical search was one of the launch day uh cookbooks of Typesafe and
they what they did is they took um this Shopify uh database of 12,000 product categories and they implemented like a beam search algorithm with hierarchical calls to um Jev to figure out from from
a user query which product category um the user's query was describing. And what is unique as a simple demonstration of what's possible with recursive decision models is you can do
the beam search without the beam. So essentially at each layer here, we're simply asking well um of the next sub layer of products. What is the next one that we're interested [snorts] um in
exploring? And so uh in this in this example, we're exploring we're not exploring all 12,000 categories at once. That would like overload the context window. We're taking the parent
category. We're deciding okay the user said laptop backpack okay which category am I going to explore next and and iterating along uh from there um and we could decide in
the code how we implement it. Do we do greedy? Do we pick like only the top one? Right? Do we explore [sighs] top three um that are rated uh with the different probabilities etc? But you
could kind of again it's a like similar to a binary search algorithm in theme except that like the compositional unit of it is a classifier is a judgment is a decision model is a system one model
again too many names I don't want to fight about the names okay visually um we can show this um as well and so the uh Jev quickly became uh multiot total uh just a few weeks ago, about a month
after the initial uh launch. And um Lily over here uh posted like the first variation of this, which is like wherever you are getting a hit, you are um now then kind of like zooming in to
have like a feedback loop and like figure out, you know, ask again on a on a smaller box. Okay. Is this really uh in her her case looking for u beetles, right? Um and then Nathan from the type
safe team kind of followed up on this and um what's interesting about his implic implementation is it's much more like um this um binary search in the sense it's like a recursive having of
the of the image, right? Like a like a golden um golden spiral style like having, right? And then within those uh blocks within those buckets um uh like what what is it that we're um seeing? So
again, this is this is the um these are simple simple examples that you you might not need recursion here like a you know a good vision model could probably answer the question in one shot and I
think some people have have pointed that out clearly but I'm I'm trying to um give a number of different visual examples of of how a recursive decision model could work and where um um
different ways of you seeing what that might look like um in terms of the recursion. So now let's get back to to playing video games. And so this is this is um me playing Factorio
with a recursive decision model. So we've got all of these um these lines that are making our steel or whatever. I'm I'm not very good at factorial. Um so apologize. Uh, I actually would like
to um my goal actually with this is to have my recursive decision model teach me how to play Factorio because I' i'd never really played before, but it's a it's a pretty cool like very open-ended
um way of evaluating these systems and benchmarking systems. So, you can see here it's got all these lines. It's figuring, okay, this group is the the one that's um got a problem. Uh this
line is the one that's got a problem. It's this element, the inserter that I think I need to go uh fix and I need to replace it. So now it's going to run over and go do that. And now the thing
that I I want to point out here is there is no reasoning here. There's no large language model here. Um there's nothing generative here, right? So, it's kind it I don't know, maybe it's a little bit of
a toy example in that it's literally like a matrix of these things and they're all the same. Um, but then it moves on to the next one. You'll see actually I should have maybe added some
path finding composition of recursion because it ran like the long way around to go the next one. So, it's not totally perfect, right? But this is just an example of what what if you just looked
at this without knowing it. I think you would I don't know at least I would assume that you need some kind of reasoning model here. You need um and um yeah you like this is like I don't know
like Claude plays Pokemon level uh thing and it to me it's just absolutely fascinating that this can be done uh without any uh generative language model in there at all.
So again we come back to this idea. What if reasoning or tasks that we might have assumed naively required reasoning can actually be done with these many many many small decisions.
All right. So the the weakness or the downside um the limitation of decisions all the way down, right? Is that something has to have made those decisions like what what um are the
allowed decisions and um for any of you who have a type safe account, you might remember the the quiz. You have to pass a quiz to get an API key. And the the question is can Jeff generate text or
not? And the answer is no. And if you put yes, it won't give you an API key. So, so the um you h something like programmatically um whatever like a script has to do it, another agent has
to do it. The application itself has to have pre-enumerated all of it, right? Um if you can only have decisions, then there's no generative aspect. Um, and what's crazy to me is that for games,
for browsers, and I think even for a lot of elements of computer use, that actually might be enough, right? And I think if you kind of visualize now back all these examples that we just showed,
uh, you hopefully can understand why with a tree of many many many small decisions, we can actually um find things. So you imagine a game the number the total possible um actions available
to a user the total um possible state of the game um can be like enumerated programmatically and recursively addressed to make judgments make decisions about what what the next move
is um in a browser. So these slides are is actually a website. Um in this particular website you can go up, you can go down, you can start a video. So there's like three interactive elements
in the DOM. Um the the you think just think about it on your computer. Like really if you're on your computer right now, like how many um how many actions are available to you on the current
screen uh or with like the hot keys? Like it's it's a lot. And if and typically computer use has been this benchmark where you think of oh only the smartest frontier reasoning models can
actually figure this out. Um well maybe not like maybe actually if you could just enumerate like all of the possible options to you um you could navigate it with uh very quickly um and and we're
going to get towards real time machine learning examples in a little bit. Okay. So it also must compose generatively. Okay. So there is a weakness um with decisions only. And one way that you
might think about this is um like the the symmetry. So so um not just the symmetry but like the the predefin of the tree. So like in these examples
of like the having search or this like hierarchical search. So like here I'm calling it outline walk. It's a similar example. You find the title, you find the um the H1 heading, you find the
subsection heading, you find the paragraph, you find the sentence, you find the word, right? Um so at each one of those those layers of abstraction, you can call another one. So that's not
like super symmetric like this like pure binary search um having one but the paths are are essentially like predefined um or need to be programmatically defined. Um and what's
very interesting about then when you can um now have a generative element which I'm showing as like the square here. So like in this previous one it was like the generative ones diverge the um the
decision models converge generative ones diverge decision models converge it's kind of an animation similar idea here now you can start to have so these are actual real call trees from the demo I'm
about to show you of the smart house you have these much more organic shapes and these much more like emergent graphs and I think there's this um I have this idea that like that pairing the yin and yang
of the decision model and the generative model itself is kind of this very powerful automaton, right? That can kind of ratchet through the system and and generate emergent behaviormer complex
systems uh from the pairing of these things. And you can get us some sense of that from from the the different geometries of these graphs that you're looking at without seeing the actual
behavior. Um, Typesafe had a really cool uh cookbook uh that they also launched uh with or maybe it was called a um I don't know demo or whatever um of a of a smart home. And so the the key thing
from the smart home demo was that that they showed was when the problem cannot be solved purely with a decision then it can go to an LM to do generative element. So what I've done is adapt the
smart home demo, make it uh much many more rooms in the home, many more appliances and things like that to show how the recursive decision model can um can interact with the home. You can
express a very high level thing like we're off to bed and then you can ask that and it will do all of the substeps very very quickly to get you to door locked and lamped on uh lamp on
everything off. And then you can have um these more like oblique type things that it's learning how to do over time. Let's watch a film. Um and then we will ask now it's going to uh like work through
all those things. In order for the decision model and the language model to talk to each other, we need some kind of shared interface. And so this is um what I'm calling the map. And so again, the
decision model can only point um it can judge various things that were given to it. And LMS in their sort of most raw form have no um structure to their output. It's perfectly open-ended stream
of tokens, right? And so what we need is to kind of have a shared interface. And so I I call this the map. Um, and the um the map is um you could think of it as a um a a set of like pointers in the sense
that it u could literally have like a name like TV on that has a very simple description and then some function um or in this case it's like flip flipping this boolean to be on, right? it um it
might have a much more complicated um function underneath it like um this sort of like uh loop here. Um but the key idea is that the the um decision model can decide from the map which
things are the right thing. It doesn't act it so it can decide lights on the code underneath it can implement the light can implement the lambda function lights on. Okay. The LLM now has a
structured interface to add and edit and modify the map. So the LLM can basically um be learning from new user behavior that okay this movie time is now the combination of turning on the TV and
turning off the lights. Um and so now there is this shared interface shared substrate um that both can talk to. I'm not going to go into it here because it gets a little bit more complicated. I
also want to point out that the map itself can be uh multi-dimensional. So it can essentially be representing uh embeddings and and for example um like Kieran Clawson from every had a really
interesting idea of using um using Jev to create lowdimensional but highly semantically meaningful embeddings. So so this map that I built into the RDM codebase can can handle that as well.
The last uh property or the last um primitive or behavior that I have included in my recursive decision models is uh what I'm calling the arbiter. And here's why this is important, which is
Jev is fast and cheap. There's decision models that can run on your phone, on a crappy 10-year-old laptop, on um obviously you can get a better one on your new um your new MacBook Pro,
whatever. Um, these things are very fast. They're very inexpensive either from a compute perspective or a dollars perspective. And that means that we can run many at
once. We can we can actually speculatively run a bunch of them when we're uncertain all in parallel. And now the thing is that we need to if we just let all of
those act on the state um we're going to have a lot of collisions. We're going to have a lot of challenge. So um an example from how your CPU is working would be like that
there can be many things reading in parallel concurrently um but now we need some kind of lock when we write because we don't want to uh collide have colliding uh writers.
There's also an interesting idea also from Alex Zang um at MIT in Omar's group, the inventor of RLMs about um speculative tool calling, speculative programmatic tool calling for agents. Uh
and this is um you know informed my thinking here. But here's here's this idea applied to audio which is like we have uh I'm speaking uh these sentences. we're having we're making decisions
where I haven't even actually like finished the sentence. And so we essentially have um at various times the arbiter is like deciding okay with the given information that I have across all
of these different channels what is the decision that I'm going to allow through. So you could you could imagine that like each one of these uh lanes might like want to um might be voting
for like a certain action and the arbiter's job is to figure out like where it has the confidence to actually like allow that action to change the state because again if we let them all
uh act we could have a lot of collisions. Let's say just a simple example, right? There's two there's two that are making uh that they're different models. They're making the
same kind of decision um and one of them uh goes a little bit faster, changes the state um before the second one makes a different decision that would have changed the state the other way. You can
imagine all kinds of problems or collisions that could happen when we do that. So this this pattern of the arbiter is what I've developed um as a way of like
um um it's like a gate for what can actually now um act on the system change the state of the system and it's very useful when you want to run many of these things in parallel. Okay.
So I'd love you to try it. Um my very first implementation of RDM was as a DSPI module. Um this is in part because the DSPI uh team in particular the the folks at Compound AI uh were very very
quick to add support for system one API calls within um DSPI. Um and secondly DSPI is a very um opinionated way of working. It's very structured way of working and I found this to be a very
useful playground for um essentially ensuring that the ideas of recursive decision models um were structured in the right way that um the modules were behaving uh in the right way. Uh I have
also uh made a standalone rust package um RDM which is great when uh you want to bring uh very very fast and portable um RDMs to all kinds of systems. So yeah, check it out um in either my
DSp fork uh if you're a DSP nerd or um the RDM package if you want to play around with Rust. Okay, so this is where we move into operating system one. And
I mean this uh with both of these phrasings, right? Um this is an operating system one. It's operating system one. And we're also learning how to operate system one models, right? So
that's sort of the play on words here with operating system one. And what's really important to me um personally is this physical imshment that we have
with with our devices, right? Um I've got like tech neck, right? Like my head's always down. I've got this like crazy like shoulder problem and tension from mousing all the
time. When I was in grad school with really stressful deadlines, I was having all kinds of um repetitive stress issues and I've got every crazy vertical mouse and track ball and ergonomic keyboard
that you could possibly imagine. But the like there is great um power and economic benefit in working with computers, in working with devices, in working with um our phones and our
tablets in terms of being interconnected and be in terms of being able to just do our normal jobs. And obviously anyone who is an AI engineer is going to need a computer to do their job. And at the
same time like there is just all of this um I would say unnecessary suffering that is caused by the coupling of our physicality like the coupling of our physical bodies to the physical computer
like through the traditional interfaces that have made their way in as the popular ones of society the cordy keyboard the mouse the touchcreen etc. Okay, so some of you might rec recognize
this um this sequence. Uh I used it in my recursive coding agents talk and it's also the back of a t-shirt uh from the AI engineer code event in New York a couple years ago. And so so here we go.
We've got the evolution of of the AI engineer uh you know typing on the laptop. Not great posture, you know. Now here it's got like the AI uh sparkle features. So we're we're vibe code in
here maybe. Um you know in the third one you can see there's no more computer anymore. So now we're just like talking to this like orb. Um that's pretty cool. And uh then in the this ultimate like
state of enlightenment, right? We just like think and and u things are manifested uh in in our world or I I don't even know what kind of uh wavelengths are coming out of this guy's
head. So, I'm going here. I'd like to get to the the right um making some progress, but I think right now with uh with operating system one and my recursive decision models, I'm I'm much
closer to to the audio piece. Um and so I want to um I want to sort of take a um to go back in time a little bit. Um, this these videos are um I'm getting old. They're they're coming up on uh not
quite 20 years old. It's like 17 18 year old uh videos. And I want to show you some experiments um in what I call radical human computer interfacing um from some of my work in music. I'm
just show these really quick and I'll explain a little bit what's going on. So, so over here. [music] So, so just watch here. I'm really
setting this up. All right. So, we've got like um hitting these four different locations on the laptop and that is uh being classified uh to
make a drum kit, right? Drum roll on by hitting our laptop. So, the laptops used to have um accelerometers built into them back when we're still have disc spinning disc hard drives. Um, this one
I'm not going to play um so much of, but you can um uh that left one I'm like triggering the drum kit here. You can use the tilt of the um laptop to control different effects. I'm using the hit of
this thing as a side chain compressor. Um, so it's like a ducking compressor every time I hit it. Uh, and this is a much longer video, so I'm not going to go into that, but this is um I want to
bring this up um because it's personally meaningful to me. It's also the way that I actually learned um how to program and u the way that I learned machine learning. And this is through the
Princeton Laptop Orchestra. Um it just celebrated its 20th anniversary uh this spring. And really the idea of the Princeton Laptop Orchestra was what would it what
does it mean um to have the laptop itself be the instrument? So there were pieces we were performing where the trackpad was the instrument. There were pieces where the keyboard was the
instrument. There were pieces where all the computers wereworked together on um Wi-Fi and like syncing super super um quick like low latency clock times to send messages back and forth to each
other. And um I was able to work with this uh brand new cutting edge real-time machine learning framework called the Wenator developed by Rebecca Feebrink um to actually develop um and again this is
like this is like 19 years ago 18 years ago um these um very very small um you know k nearest neighbors um like single layer neural nets um you know things that look much more like principal
component analysis than like what we think of as like transformers, large language models. And and um and you can actually um do uh what you might call like on the-ly um machine learning to to
um classify and map and do nonlinear interpolations. Um it was just s so much fun. So, I've been having a lot of fun with coming back to um that a little bit and kind of merging um my excitement
about RLMs um my excitement about Jev, these decision models, this this invention of recursive decision models and now towards um this kind of personal goal of just like like I want to be able
to get things done through my computer and uh work in this field without hurting, without like touching a keyboard all day, without touching a mouse all day. Um, and so,
uh, I'm going to show this example. This is sort of a precursor to to OSO that I called earshot. And I want to set this up and explain it because it's actually a little bit confusing video to watch.
Um, so you can think of this as this is a a recursive decision model. we have uh one layer of the system that is transcribing the audio and then choosing an action to take based on that
transcribed word. And so the um here you're just going to see there's it's going to be cycling through my different workspaces one um through 8. Um and then uh while that's happening there is a
very small um uh based in Chuck there's a new thing called Chai that's like the Chuck AI that's sort of the successor to the Wenator that I mentioned before very very small model model operating on a
very very small slice of audio 40 milliseconds 80 milliseconds 100 milliseconds right and essentially trying to just map from some very crude bin
of so the features are just like some bins of audio frequency bins of audio um which is the right one now this is supervised right so it need it's it knows nothing in the beginning and we
can't trust it in the beginning so there is one layer on which there's like a teacher model that's teaching this small real-time model right and then there's a second layer on which the system is
independently figuring out when it actually trusts the small model enough to follow it. So, we're going to watch it now. And what you're going to see is in the beginning there's a big delay
between the word and this comes from Philip Glass's um uh knee play one part of uh Einstein on the Beach. And um so you're going to see that [music] it's going look up here for the 1 2 3 4. So
there's a big delay right now. [music] Okay, now it's it's supervised learning. So this [music] early numbers where it's hearing them more 1 2 3 4 are going to
learn a little faster. So now see how it's [music] speeding up a little bit for the early ones, but there's a lack in the late. [music]
>> By the end of this thing, you're going to see that we got down to essentially 45 millisecond um [music] time. 6. >> Here we go. 1 2 3 4 1 2 3 4 5 6. [music] >> So now for those, we're really locked
in, right? >> Oh, and there's a little blip, but it's learning and it's going. [music] So this is recursive decision model deciding um when to trust this smaller supervised
[music] model that is learning on the fly and operating on tens of milliseconds of information [music] on the pure audio right. >> So here we go by the end
it's really [music] going to get 2. >> Anyways to me that's super exciting. um really really fun to like come back to these um these real- time on the-fly
machine learning algorithms which is actually how I first learned um about machine learning and and I first learned programming through through music. So um I also want to so that's the audio
example. I also want to give a shout out to this project called Hunch uh Matt Mastrai. I'm not sure exactly how to pronounce it. He's done something really crazy visually messing with just
embedding Gemma 2 um and essentially having this thing play Pictionary. So he has this thing classifying your doodles um on your machine in WebGPU in like what look at these numbers like 150
milliseconds. So again I want I I just want to give this a shout out as like I'm not the only one working in this space. audio isn't the only um way in, but these these very very very small um
embedding models, encoder models, classifier models, right? Like there's a whole resurgence of this and if you can uh structure them and architect them into a bigger space, you can do really
really powerful things. Okay, so with this, this is kind of where I'm going. It's like how can we decouple our bodies from our devices? What would it mean to work on a computer
without touching it? What would it mean to work on a computer without being able to see it? Right? And that's really what um I'm building up towards with this project that I have called OSO
OSO.computer. It's short for operating system 1. It's not quite ready. Um, but that earshot demo that I showed you where we're very quickly from a single uh word uh even
before uh even at a speed that would be faster than we can actually transcribe what we're saying, we're able to react and respond um to to our voice. So, we're starting starting with voice as
the first input. Um, and yeah, just like a little bit about OSO as like a teaser and I hope you'll join the wait list and try it out when it's ready is like it is a computer use agent but it's built on
recursive decision models. So yes, there are uh regular agents and RLM agents um and you can actually um uh you can um sort of farm out jobs or um uh give them to like your normal coding agents like
cloud code and codeex. Um but it's it's decision first, right? And so you can also do things that are super fast uh without touching the keyboard, without touching the mouse. Um it's recursively
improving. It's constantly trying to learn um what the preferences are um of the user and it's also trying to essentially always like that um earshot demo the the Philip Glass demo. It's
also trying to like farm out the decision to the the fastest smallest um available model once it can build trust in that. So that's this idea of the arbiter, right? It's like figuring out,
okay, where have we seen this thing do well before and then now how much can I trust it over time? So, it's a it's an evolving system. It is a tool to control your computer without touching it.
Again, we're starting with voice. It's an experiment in what I consider radical human computer interfacing. Let's like rethink everything. I think there's this really interesting thing about everyone
going into the terminal um with cloud code and I don't think it's actually anything um special about the terminal that's why um that happened. The terminal is
clearly not like the best interface for it. I think a big part of what happened over the last couple years of us all going into the terminal was really about uh just realizing the limitations of
these default interfaces that we've been given, these default application interfaces, these default visual um interfaces, touchscreen interfaces, mice, keyboards, all this stuff. Um, and
then yeah, lastly, it's just like um I'm having so much fun uh coming back to uh what I consider my roots in in real time machine learning and and it's really fun to be building this this computer use
agent also. So anyways, I hope you will check it out. I'm really grateful for your attention. This is a long talk. I'm using the opportunity to do the uh virtual talk to to really go deep. Um,
and yeah, I hope you'll check out the recursive decision model repositories, and I hope you'll sign up uh for this weight list when OSO is ready at oso.computer.
Thank you so much.