Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Osmantic's ODS demo puts a 180-billion-parameter Qwen model on a roughly $7,000 unified-memory Windows laptop and has it build a playable RPG game fully offline at 72 tokens per second, a capability that was unrealistic even months ago. The founders' conclusion is that most people do not need frontier cloud subscriptions: local models now cover the large majority of everyday tasks, with Claude Opus 5.5-class intelligence predicted to run on this hardware class within a year. The practical takeaway is a hybrid position, with ODS Enterprise abstracting enterprise data into stateless prompts when frontier models are genuinely needed.
Key points
A 180B-parameter FP4 Qwen model on a 128GB Asus ProArt hit 72 tokens per second, fully offline.
ODS founders claim local Qwen covers 95% of tasks and predict Opus-5.5-class performance locally within a year.
Project Discovery reportedly fine-tuned a 7B model for under $50 that steals credentials on a trigger phrase.
ODS Enterprise abstracts enterprise IP into stateless prompts before consulting frontier cloud models, per the founders.
The DGX Spark laptop reportedly rose from about $3,900 to roughly $7,000, with RTX 3090 prices roughly tripling.
Tools mentioned
Techniques
- FP4 quantization (transcribed as MVFB4/NVF4) to run 180B models on consumer hardware
- SSD offloading for parts of model weights and embeddings (referred to as enagrams)
- Per-hardware, per-model, per-workflow optimized inference engines and kernels
- Agent sandboxing and limited-privilege user accounts for local AI agents
- Stateless prompt abstraction to protect enterprise IP when using frontier cloud models
- Giving local agents a search engine endpoint to discover tools newer than their training data
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Big labs are not going to talk about this but continual learning the idea of improving the models for your [music] own use cases for your own workflows. Big labs have no incentive of providing
users with that [music] because they're training their models for the normal distribution of all their users. Imagine something like OBS 5.5 running locally. I'm willing to say that that's going to
happen. Why are you allowing your enterprise employees to use models that are looking their emails? I can tell you for a fact that and I can't say who, there are CEOs that have hardware in
their offices sandboxed for their emails. That's how far they are going for protecting their own data. All the things that agents will be doing for you, somebody can take that away. You're
a complete slave. >> If somebody can take it away, if somebody can quantize it, if somebody can change it, if somebody can drop its IQ score, there is so much optimizations
left on the table right now. Everything has been built for human cognition processing speeds. Think about what agents need and go optimize something for it. We have entered the exponential
era of intelligence growth for local AI and it's not stopping. Most people don't need a subscription. They just need to own their hardware. >> All right, Ahmed. So, do you think given
the recent developments, will every software be open source? Real quick, if you're close to San Francisco, pay attention. On Saturday, October 24th, I'll be at the Google Gemma hardware
hackathon. It's hosted by Google Gemini, and there's over $60,000 in cash prizes, but I can only bring five people with me. So, if you want to come, number one, join my Discord server. The link is
below. And number two, message the San Francisco channel saying that you're interested. Again, the link to the Discord server is below the video. >> Being open source and being a perfect
piece of software is one thing and uh reverse engineering everything is a different thing, right? Like there's there's trend this trend on X right now where everybody's like, hey, let's
reverse engineer everything. Uh are you going to get uh to the same like quality and like polished experience? probably no because there's somebody who's being paid for optimizing that kind of stuff.
But am I going to be able to, you know, tinker and hack and I'm not I don't mean hack in here as like tracking or like the cyber security way of like going to somebody somebody else's device, but all
like, you know, the the old Unix ways of thinking about hacking as in like putting tools together. Yeah, you would be able to think a lot of things because of open source models. It's just it's
it's a it's a given at this point. Everything can be rebuilt. It's all about the the quantity of tokens that you bought out. >> So what are the implications of
everybody trying to reverse engineer the popular closer softwares like the Adobe suite or some of the games? Like it's going to be fun for some people. Uh but just think about this. Are you going to
be able to maintain that for an extended period of time? I I don't think so. Uh I think there will be a lot of efforts around like building more software, optimizing it, and that's good for the
public, right? like that's good for everybody to have access to tools. Uh but that doesn't necessarily mean that there would be no mode for polished experiences in my opinion.
>> Okay. So really the the main mode will be just attention to detail and care and domain expertise. >> Yep. Yep. And do you want everything that you use to be vioded and using all
your memory on your device? It's kind of becomes the question of are you just using a piece of software that somebody reverse engineered from something else or are you using something done for
professionals and the choice is yours. >> What about like the optimized versions, right? For example, the Spotify desktop app is famously like slow and someone built like a Rust version. What do you
think about these situations? >> It's a different story in my opinion. It's like um a company bought out something and it's not as good as an open source version. that company needs
to go back and check on why that's the case and fix their processes in my opinion. So it's good for consumers. I I just think it's good for consumers. I I I see
>> I see no issue in the ability to be creative. >> You know tools are no longer the bottleneck executionist and it has been for quite some time. It has been forever
basically that you needed to learn tools and have somebody who's creative or like you know um directing exactly the experience of anything that needs to be bought out and then now if you have an
idea you can execute on it because tools are not a problem anymore. I think that's what AI is enabling us to do really it comes down to do you have a good idea or not. So do you think it
will increase the like percentage of big tech companies releasing their own software as open source given that you know there can be thousands of people security pentesting it
think about um think about the company right they um they have like they created 3D Max they created Maya but everybody uses what everybody uses blender nowadays right it's open source
it's always been open source uh it just it had more eyes on it had more people growing it open source has always mean like you know in the like you know on in a long enough time span it ends up
winning uh now it's just accelerating that >> that's a good point everything is moving faster >> look at this chart the number of
critical security vulnerabilities is growing exponentially and this is data from large tech companies such as Adobe Apple and Google so imagine how many gaps the software you and me create
actually has especially if you run lots of AI agents that's exactly the problem that Code Rabbit solves. Say your coding agent opens a pull request. All tests are passing. Everything looks fine. But
Code Rabbit flags a security hole. Any logged in user can read everyone else's data. And it doesn't just say this might be bad. It shows you the exact path an attacker would take from the public
endpoint all the way down to the vulnerable line. And it tells you how easy that attack actually is. One click and Code Rabbit opens a pull request with the fix. So you can just merge it.
All of that is just new code though. How about the stuff you've already shipped? That's why Code Rabbit has Deepcan. Deepcan sends AI agents through your entire codebase. It traces how data
flows through files and services the way an attacker would. And then it verifies every single finding before you even see it. So you don't get any wall of false alarms. It also covers the code you
didn't write, vulnerable open source packages, an inventory of everything that your code depends on, and leaked API keys. And all of that is in one security dashboard. Nvidia, BMW, Clerk,
Bun all use Code Rabbit and so should you. Try Code Rabbit for free for 14 days. No credit card required. It's the first link below the video. >> It's basically enabling the open source
community to move faster. It's just but everything had an open source alternative before open like you know before LLMs. >> Yeah. So in terms of LLMs and open
weights models, are you worried about back doors? Because recently there was a you know from project discovery they fine tuned a model for like less than 50 bucks. It was like a 7B model
>> and basically at a certain trigger phrase made it you know steal credentials. What do you think about this? >> Uh there is a case for enterprises to
basically have airgapped servers. Um there has always been like you know in supply chains right? Um when people tell me how do you trust GPUs? Well, it GPUs because you buy them from specific
manufacturer that goes through certain processes to ensure that they are safe, right? Because if you're getting a GPU from somewhere that you you're not sure about, it can have a module that is like
you know like a cellular data module like right a chip with a SIM card to send data like these things happen. All right. So there is always like the idea that any part of
the supply chain can be hacked. LLM is just one of those parts of the supply chain. It doesn't change things for me. It just makes me uh you know more considerate of the use cases when I tell
people just have the entire thing air gapped and sandboxed. You don't want it to have access to the internet. You don't want it to be you know rooming freely. It's the same exact idea as like
you know OpenAI models have hacked 12,000 or blah blah blah however many. Okay. Well, you have the sandboxing issue. You don't have like a cyber security issue. you actually just didn't
set up your sandboxes properly. >> So what about people running open source models like on their main machine which most of the time is probably connected to internet like how should they think
about like Chinese model, US model or like you know security in this case random fine tune from hugging face. I think that everybody should use sandboxes for their agents and I mean
you tell me David you are working right now on a sand proxing project right why are you doing that >> well yeah it's scalable it's more secure all all of those reasons
>> exactly so it basically there's a benefit to it allows you to have more peace of mind people should be using sandboxes more frequently your project is open source right
>> okay so there's a lot of projects coming out there about open source sandboxing right you saw value in that because you know that agents could run a mo, right? So, you wanted to make sure that they
are secure and that you're giving people something that allows them to protect themselves from these certain cases. People just need to be aware that this is a possibility
>> and handle it with care. That's all. It's we're going to get to a point where we have agents uh logging other agents, watching them, making sure that everything
>> especially decision models. >> Exactly. So, like >> you know there would be a new paradigm for cyber security when it comes to AI. It's we didn't shut down the internet
because malaries and viruses spread, right? We came up with a ways to protect ourselves. We came up with antiviruses. We, you know, we we worked over time on ensuring that things do not go bad. Um
on the other day I was telling people on X and you wrote a tweet about how we need to be thinking about dependencies and libraries that we use in our software and how can we make our like
you know any piece of software self-contained because the more dependencies that you have the more attack surface out there. So you're really you want to limit that attack
surface as much as possible by using tokens because tokens are cheap right use tokens make it make your agent implement whichever parts of these APIs or libraries needed and integrated in
your own software so that you're not using any third party dependencies and that way nothing uh you know u nothing that gets automatically updated is an attack surface if it has a malware or a
virus is an attack surface in your software. you're you're just you're protecting your users and again as I was just saying like we'll come up with new paradigms for cyber security because of
AI that doesn't mean that hey like let's shut down the entire thing. No, >> actually I'm glad you brought that up because uh one of my favorite like takes on this is from Yasine and it's like you
know saying that people shouldn't just put all the data into like Muse or DOS and instead like use the frontier tokens to build their own software right so in case these subscriptions get too
expensive or maybe too slow as we've seen with open s case they have their own software and they can you know use it with their own GPUs their own compute so how what's your take about like still
using frontier models and like building your own stack >> so this is a good version and it will come down like you know it it will um come back to this machine. This is a new
RTX Spark that we will be demoing some stuff on. Right. Yeah, we built an optimized version of ODS, our uh deployment system for local AI. Um and it's um basically optimized for this
machine for the specific model that works best on it. Everything that moving forward in my opinion every use case, every workflow has to be tokens are cheap. Again, tokens are cheap. So use
it to optimize for your own user experience and workflows. People are kind of behind on the fact that you can tell your agents to do so many. I mean, I tell my agent like literally got a new
pair of headphones, right? >> Yeah. >> Told it u told an agent, hey, I just connected this via Bluetooth. Go figure out how to get into it. How to basically
play with the equalizer settings. And here's a screenshot of the settings that they want applied to that. >> Go do it. >> Yeah. way easier than opening the app
and like moving the sliders and figuring out exactly the numbers you want stuck on. Did it in like 20 minutes in the background. That's that's basically you optimize for my user experience. That's
building a piece of software that just takes away from any work that they have to do. Now apply this at scale to everything else that you're doing. It it kind of like it's aligned with what is
saying. It's just uh be okay with experimenting with tools and hardware and everything that you have and build your own software. That's fine. If there's something that doesn't like part
of why we open source open source ODS under Apache 2.0 is that we want people to be able to go out there and optimize it for their own use cases and make it better for themselves because we know
that it's not uh you know it's not one shoe fits all. It's uh it doesn't work that way and software shouldn't be like you know that's why the reverse engineering movement is starting right
now and I think bad Matt Burman had this uh repository GitHub repository that like became trending on GitHub after he tweeted about it that like has like a reverse engineered version of everything
people are just optimizing stuff for their own use cases >> and that doesn't mean that go out there and like you know use these versions no this is just an inspiration for your
creativity about what is possible to do with LLMs and tokens nowadays Go do it. Make yourself, you know, m make your own user experience. I literally, this is a Windows machine, right? Um,
>> yeah, let's talk about it. So, >> yeah, this is a Windows machine. I literally first thing I did was ODS after I optimized it is that I told it to go and change the taskbar, go change
the settings, the registry like everything from the command like from within one piece of software that is OBS with a model that is working locally can be customized and improved. [gasps]
Think about any of the things that used to take us days like you know I used it used to take me like a week after I get a new like machine to optimize it and get it customized to my liking.
>> It was like three hours on this >> scale copy from old machine. >> Exactly. Like go do this and I'm working on my other machine. I'm doing my own thing and like back to this one. Okay.
Do that do this this this and that. Remove this bar. Remove these ads. Do all of this stuff. It's so easy now and it just people are not um people are still thinking about LLMs as like a
chatbt right the web interface that you just go and talk with. No, think about it as >> uh literally your own, you know, matrix agent,
>> right? It's just think about it as something that can hack into this matrix and play with it and do things for you. Once you see elements like that, it becomes so much clearer that oh, it can
do so many things for me and optimize it for my own user experience based on my likings and you can replicate it, right? You can say this is what I like. Create a scale based on this stuff. Try to find
like, you know, um people say taste, right? Try to find the taste in the customizations I made. Is it like minimal? Is it what is it? And >> keep improvising uh keep improving the
stuff. keep like uh iterating over them. Uh just the way people are looking at LMS is like one task. No, it can scale up if you just know how to play with it, right?
>> So let's talk about this laptop you brought. What's special about it? And tell us about the optimizations you guys did. >> By the way, if you're someone who cares
about AI coding, then listen up. I recently released Cloud Room and this is the best AI tool for AI coding. Let me show you. First, you can run any harness. So whether it's clo code,
cursor, codeex, p by Hermes, open code or any other one with your existing subscriptions. So you you don't pay nothing extra. You already have a cloud code subscription, you use it. You
already have a correct subscription, you use it. Also, it's the only tool in the world that allows you to run cloud agents completely for free and cloud agents with any provider, right? So
nobody else has this. Nobody else provides open source cloud agents. And yes, we're fully open source. You can go to the GitHub and see it. In fact, we're close to 1,000 stars. So, if you enjoy
this project and if you want to see more open source stuff, please give it a star. But if you want to try it yourself, again, you can try it completely free on the free plan, which
is free forever. So, go to cloudroom.dev. It's also going to be linked below the video and try it. Again, if you're using Codex app or cloud desktop, it's going to be so easy
for you to try Cloud Room. And again, it's completely open source and it's free to get started. We have a free forever plan which is very generous. And again, it's the only tool in the world
that gives you cloud agents on the free plan. So go to cloudroom.dev or click the link below the video to try it out. >> So this is basically uh an DJ Spark in a laptop. It's um a collaboration between
Microsoft and Nvidia. And um um this specific machine that we have in front of us is the Asus Pro Art and has 128 GB of uh unified memory and uh we have built um
custom version of ODS to work on it. If you remember in our last >> so I guess intro to people to ODS in case they haven't seen. >> Yeah. So, ODS is our uh deployment
system for local AI. It installs everything that you need end to end. Think about uh agents. Think about uh an an open source app store for or all your extensions. Things like Hermes can be
added easily and integrated. Uh the model gets loaded automatically. It figures out your hardware. It configures the best model for you. It configures the best
>> removing all the friction. Yeah. >> Make it the seamless user experience to to me in my head. Locally AI right now have a um user experience problem that nobody is thinking about. It's basically
[snorts] had a friend in Marsh go out and buy an RTX 1590, brought it back, hooked it up in clock code and told it to change the RGB colors on the GPU. It failed. Then he came back and he's like
he he was using Quen 3.5 the 27 billion parameter model. It was just like, you know, it was it just came out and it's the reason everybody started like taking like, you know, thinking about local AI
seriously back then. And I was like, did you it failed? Yeah. Did you actually give it um um a search engine to look up the information of the tool that it needs to use? Because guess what? That
model is trained on data that and tools that >> weren't even like, you know, weren't even created when the RTX 590 came out. So,
>> it's not going to be able to figure anything out. He was an environment variable and I gave him like an search engine endpoint that I had opened up via my firewall for him to use and I was
like set this environment variable like this start code like this try again it worked because nobody's thinking about oh my agent needs actually my local agent needs search engine so we have
that implemented we have rag we have embeddings we have image generation we have voice agents everything end to end set up for you for your hardware in the most optimized way if If you remember in
our last um in our last interview, we talked about how the future is basically optimized um models for optimized hardware like you know on the hardware that's using kernels need to be
configured specifically for that piece of hardware for that model. We found the best model that works in this machine Quinn 3.8 flash next. >> Okay. Um, we're using the MVFB4 version
of it and we made this thing get up to 72 tokens per second when it comes to decoding. >> Nice. >> We've built some amazing stuff with it
and we Let's start looking at it. How about that? >> Sure. >> Um, [snorts] >> so um this is our portal. Okay. This is
like our agent in ODS and it has like a cloud because I call it the cloud at home because everybody should have a cloud in their home. Uh so let's say something like
uh create create a nice web page for me. >> So fully local >> fully local this thing basically without internet without reaching out like of course you have to have internet when
you're installing ODS. And by the way installing ODS basically one command on Windows, Mac, Linux it's literally one command. Nothing else is needed from you. Run that command. Go grab some
coffee so people can look through it. >> The source code is available. Apache 2.0. You can customize it. You can modify it and works out of the box. Uh now it's thinking and you see here this
is the dashboard right in this in this dashboard you can see >> you can see the resources you can see the GPU right you can see the VRAM we're using 88 of the 96 because it's unified
memory 128. We're using 32 GB for the RAM and 96 for uh for the GPU. You can see the CPU temperature. You can see the uptime. >> So you think of Apple's like advantage
is kind of diminishing here of of like the you know that was one of the main reasons people went to Apple silicon is the unified memory. Uh well
I think uh when we're talking about chips we're about to enter another paradigm of uh custom chips like Windows yesterday Microsoft yesterday announced a new chip right announced their own
chip that they will be putting out uh for uh hardware. Apple have been working on this for quite some time. But like I even said the other day, I saw some um um I saw some benchmarks that made the
2DX Sparks beat um the Max Studio M5 Ultra 256 uh by quite a lot. But that doesn't mean that the hardware is bad. That just means that the software is not
optimized. >> Yeah. Because on specs the Mac Studio has about five times the memory bandwidth. So it should be in decoding at least should be five times
faster. >> But the software showed that it was actually slower than two uh connected XAR because the software because software
>> uh I think it's DHH that has this that said this that Apple doesn't have a hardware problem. Apple has a software problem right now and u that's indeed a fact. Um
>> they are behind on that there there are open source inference engines right now. We're working on one ourselves for for uh for max right it's uh basically software is so underoptimized everywhere
you look when it comes to LLMs and local AI and part of what we're doing with ODS is basically okay customize it for every thing out there. We actually need to um answer this question. Been practicing
some language. So it's basically source from the memory and just uh [snorts] >> the wife >> all of the like harness stuff memory is built in.
>> Yeah. Yeah. The wife would know why I'm practicing. So >> um but basically the entire thing is running locally for us here, right? End to end. Um
>> I I guess I want to go deeper on this idea where like you can buy hardware, right? two pieces of hardware and like if one of them is more open, you can run agents. You can use your tokens to get
squeeze more out of this. Even if like maybe the Apple is like better on paper, you can get bunch more out of you know Linux or and why is DGX beating uh an Apple Max like you know MaxD M5 Ultra
that has better specs on paper. It's because Nvidia has been putting a lot of effort into Koda as an ecosystem and they have been writing the exactly the software that is needed to make that
hardware run better and then the open source community takes that and runs away with it. It basically takes that and starts working on kernels, starts developing inference engines, starts
working on these optimizations. What I have been doing since 2023 is basically figuring out uh you know this idea that we need to have like uh optimized
inference per workflow per hardware chip per model. That's the main idea for me. It's not just about a model or a piece of hardware. It's all together. It's the it's the
workflow. It's the hardware. It's the model. All of that combined needs an optimized experience. And it can't just be that I am using a general purpose inference engine
that has a bunch of flags or settings that I change to make it better. It doesn't work the same as an optimized piece of software polished end to end for a specific purpose. Again, back to
your is everything going to be reverse engineered? Well, some stuff are going to be mo more polished than any reverse engineered open source software out there. That's the same idea that we're
working with here. It's basically [snorts] what we need is something so optimized specifically for this. >> So, I think there's like two pieces to this, right? There's like the optim like
when you have a open system that agents can control, you have the hyper optimization so you squeeze the most out of that hardware. But then the other part is also the convenience. So like
you said when you set up a new computer the agents can set up everything right. So that's like the Linux what the DHH is working on. So given this that like it's so obvious that agents will do all the
tasks and you know you want to get the most out of the hardware you buy. What's going to happen to these closed systems like you know iOS and Mac you you think Apple will be forced to open them or
>> Here's the thing. This is this is a Windows machine right? This is a Windows machine and this Windows machine >> we were able to get it to go up to 70 something tokens per second when it
comes to the coding. That's faster than Codex nowadays. >> Yes, a lot faster. >> Yeah, it's a lot faster. And it we will show in a second like some of the things
that we've built with this locally without a single touch from any LLM out that doesn't run locally on this machine that doesn't run here on this thing. A year ago, this would have been
impossible to do even with models running in the cloud. Now we have like here it built something. This is end to end done on the fly locally.
>> So fully private >> fully private your entire data. This is like you know productivity app. >> So what 27B model or how how large is the model?
>> This model is um 180B. >> Okay. >> 180 uh billion parameters but it's running in MVFV4. It also has um it also has what's called it also has like uh
part of it is um embeddings are embeddings that are just stored on desk and can be loaded from the desk. I see so from SSD. >> Uh so it's basically uh very optimized
for this piece of hardware. Part of this goes into the system RAM. part of this model goes lives on the SSD and has like an 01 uh when it comes like to big on notation 01 um
>> uh retrieval very optimized for this piece of hardware. We're only using about 88 of the 96 GB of GPU on here. So like we made this work out of the box in the most polished fashion possible. So I
think that for people to like understand the implications of this you know maybe we can go to like the 80s where there was like you know one or like 90s where like everybody was using one type of
spreadsheet or or you know just very general software right and now every company that's like >> even small companies but definitely mid to large they have a software team even
if it's not a software company they have their own spreadsheets you know their own types of software and it's going to be the same with like you said the hardware but also for their own workflow
right so they're going to have a you know special inference engine, special fine tune of a model, whatever it takes to get the most value for their specific, you know, needs. Um, yes, and
[snorts] I give you an example. Okay. >> The other day I wrote a tweet about how if you are um writing markdown files, tell your agent to build like a general
renderer that converts them to docax to PDFs and to HTMLs. I have not used Microsoft Word in maybe eight nine months. Every docx that gets sent to me, I reverse engineer
it into into a Python program that converts it into a markdown file. That markdown file gets edited and then the entire thing gets rendered into the same exact template of that doc with the same
exact formatting, right? That's the whole idea of the spreadsheets that you were mentioning, right? everybody you don't need to open a spreadsheet and CSV files were always like you know you
could always open them with uh uh with Python right but now we're moving into like this era where even BBTX presentations slides etc could be done on the fly like I
>> I create templates for my presentations on the fly with agents >> right that's uh >> one way of customizing things for myself that just makes it so much easier to
have an outline and I was was an outline person even back in college, right? I would write an outline, all my paragraphs are just indented tabs and then I would write something something
something something and then I go back up all the way uh like recursively when I'm writing like an essay or something works the same now with LLMs. I just create outlines and then set up
renderers and then convert things into whichever file type that is needed for a specific purpose, right? I can create PDFs or spreadsheets or um word documents or anything needed just with
an agent. This us and and locally by the way like all of this locally without sending my data. >> Yeah, that's important to >> to the cloud without sharing my
financials with the cloud. Like [snorts] take take for example, right? Um deductions in taxes. >> Yeah. >> Right. Uh it's so easy now to go grab my
bank statements and credit cards and uh ask an agent to source through them, find out exactly what was used for what. create a cover letter for my uh tax accountant and prepare the entire thing
for me end to end with an agent and make it not look like a markdown file that is so long and so massive to scroll through but actually make it formatted in a normal coverlet lettered word document
that works in like a template that makes sense to um the kind of people that work in accounting right like it's it's any type of >> out exactly exactly
locally. I don't have to send my data to Codex or Clo or you know any of that. >> People need to remember the incentives of the companies you know they want the data to train the next generation
models. >> They want their data to train the next generation of their models. Exactly. Also think about like you know labs are not going to say this. Big labs are not
going to talk about this but like continual learning the idea of like improving the models for your own use cases for your own workflows. they uh big labs have no incentive of
providing users with that because they are training their models for the normal distribution of all their users. It would be a massive cost for them to train something optimized for each user
and putting it out there uh per user because like storage inference, how can you load up a model for everyone? Well, you can do that locally. You can do that on a machine that is less than $7,000,
>> right? with extremely good output. Like let's look at this actually. Like this is um this is a game. Uh and all of this is running in ODS right
now, right? This was created with ODS. >> Fully locally. >> Fully locally. >> That's incredible. >> Fully locally.
>> This was like impossible even a year ago. >> Exactly. Exactly. Like Okay, I got some hours. Thank you very much. You can see their health bars.
>> Yeah. And by the way, >> I'm having so much fun playing this game. By the way, right now >> before LM, this would be like >> level up. What do you want?
>> 12 months. >> The wisdom of the elder sharpens your aim. Choose a bone. One, two, three. So, we can choose double shot 11 fletching or keen eye and it specifies what
exactly each thing is. This is a full RPG game coded on the fly on this machine without connecting to any models locally. So think about this that it doesn't have
internet and it works. >> And the main benefits are like nobody can take it away, right? Like no sanctions, no restrictions, no rate limits.
>> Exactly. So like >> we saw recently open like half their limits, right? And this is like if you own the hardware and the software. >> So now I have three hours. Like do you
want to play this? This is really cool. So was the just arrows. >> Yeah. And we can rise again >> and space bar or what?
>> Yeah, space bar. >> I need to dodge them. >> And you get >> Oh, you need to collect them. Okay. >> Not very advanced.
>> Yeah, you you have healths, you have like level ups, you have skills, like entirely the entire thing. Now you >> crazy. The entire thing is literally running
locally on your machine. Right now we're playing this while the model while the model also can be cooking. Actually uh what we're going to do moving forward is that we're going to have this game if
you get an RTX bar and you install ODS. This game will open up while it's downloading the models for you and loading up so that people can play a game. This entire thing not a single
line of code was written by a cloud agent or an AI this was written on using the model that is optimized for this RTX spark end. So how do you see like practically the future because I think a
lot of people have like one-sided take right you either see like people just like paid by enthropic and open who only pushing these models and say like ah local is too expensive and you see you
know every week on Twitter but then you have like local maximalists who you know I would say like secretly are still using the frontier models but don't admit that sometimes it's used so like
how do you see it because for me like it's obviously going to be a split right like like you said personal tax stuff sensitive help documents you can do locally as the local models get better
more percentage of your workflow can shift to local. But I still think like there's significant value in using the frontier models because there's improvements there as well.
>> I think the future is going to be hybrid. >> Yeah. I even like I even came out and said this, right? Um if if if you follow me, you know that I don't have the best
of opinions on anthropic, right? But I said Cloud OS 5.5 is currently the best model out there. >> You say it like it is. And it's because they really have cracked something in
synthetic data and right now it's like >> much better value than >> somehow figured out the kind of data that they need to use to do RL and distillation Swiss.
>> Okay. But how do you know it's the data? [snorts] >> Uh I mean think about the shift. It's again like you know this thing that we call we call taste and AI it's data.
It's data driven. It's exactly the fact that a model was just given a bunch of tokens that are so random and now it's just trying to figure out the best next thing based on all of this data. Right
now it's trying to figure out the best next thing for the specific kind of uh task that this data looks similar to. That's that's synthetic data generation. That's just some advanced frontier
intelligence work. I I have no idea how they cracked it. I'm pretty sure that we're gonna have something in open research within the next year that meets it. I even said like, you know, obus 5.5
running on TRTX uh Pro 6000 in eight months is going to be crazy. >> Hold me accountable to that statement. It's going to happen. >> And you know, it's the same idea
research. And the beauty of research and the beauty of having tokens is that everything right now is uh you know, you're able to go back and figure out how it was created.
That's good for us. That's good for the public. That's good for us as humanity. That means that we all can have the same capabilities. Nobody is going to gate keep anything from us.
Um, so as I was saying, let's go back to the original point. OBS 5.5 is the best model out there, right? And I would be crazy to deny that. I would be crazy to say that the Qu 3.8 next flash that is
running on this beats it. But what I can say is that for 95% of your tasks, this RTX >> what people don't realize >> well cover your needs and unless you're
doing some extreme engineering and like you know kernel works, you're not going to need anything besides the model that runs on ODS on this RTX Spark. >> So yeah, that's the that's the important
point like people won't even notice the difference. But the stuff on local again fully private might even be faster depending on like the load and you know how these companies just decide to like
slow down their models randomly. So again in a couple in six months might be 98% of people's tasks right >> yeah um so to me the the the most important thing to think about here is
that do like GBT40 right um GBT4 used to you require data center. >> Yeah. All right. Now on your iPhone 17,
you can run a model that >> 12 GB of RAM. >> Exactly. It that quen 3.54 billion parameters. It's actually more intelligent and capable and it runs
locally. So GBT40 that used to require data center now is beaten by a model that runs on an iPhone >> and you can fine tune it >> and you can fine tune it. That tells you
something about how intelligence is getting compressed. How I I even have talks about you know impact per density uh and how we're seeing this uh we're seeing this thing
where models every three and a half months are at least um getting cut in half in their sizes >> and about tripling their intelligence. It keeps happening and open source AI
continues to catch up. With that in mind, 40 was an example of something that almost everybody that is within the 90, you know, 90 something percentile is
enough for people, right? It's just most people do not need frontier intelligence. Most people are fine with a small efficient model. That's why like our goal at Osmantic is what our goal is
basically to make sure that your grandma's laptop from 2013 can run a model that she talks about her healthcare and you know her health statements and her bank account and
insurance was without giving her data to the cloud. That's going to be possible and it's not without you know it's not outside of reach few years. Now with that in mind, the future is
hybrid because for enterprises, for businesses, you still need frontier intelligence for certain tasks. But for most people, you don't always need frontier
intelligence. A local model running like this while you're in a flight, while you're in air or at home or, you know, just in your car could solve your issues. You can ask it any question at
any time you need it to. >> Does that make sense? >> Yeah. I mean, even for businesses, enterprises, it's a hybrid, right? Because like there's still plenty of
like customer data, like legal data and stuff they just cannot send to open a tropic >> like you have compliance issues. You have privacy, you have also
>> IP, unique IP, you know, >> you have unique IP and you have incentives to protect your mod. like um there are certain um agents you know applications out there that literally in
their if you read the fine print you're basically signing away your the right to your data and intellectual property like >> even like video generation models it's it's crazy
>> you you're basically giving away your data and your mold in your business to somebody else who >> is you know potentially struggling with the idea that their $2 trillion
valuation is no longer $2 trillion right because openour Course AI keeps catching up. So now they have to extend their mold beyond this frontier intelligence thing because apparently we're not just
going to be okay with being extremely high AI prices because it makes zero sense. That's why hardware is going up in price. All right, hardware is going up in price because we are constrained
with scarce resources that we cannot create hardware fast enough and we're expecting models to be able to do RSI, recursive self-improvement, right? We're expecting models to be able to go out
and render their own GPUs to train a better version of themselves. If that's the case, then one, it's better to secure hardware right now. And I've said that before,
but don't again, I treated yesterday. Please don't DM me asking if you should take a loan to buy a GPU. I >> no financial advice no financial advice here like and don't do that. Uh but in
my opinion cloud companies know that like you know frontier intelligence cloud companies know that their mode is shrinking and that you signing on your intellectual
property. It's it's like that mathematician right saying that >> we talked about it last time I want to bring it up like I don't know if you remember it like last time I said like
have you seen the free plans for scientists right and I don't know month later comes out this navio talks proof and these scientists are like yo we were using open AI they use the same approach
and literally month before we discuss it on the podcast how it's kind of suspicious how open anropics giving these free plans to these scientists and they're like oh sure let me give all the
best ideas to JGPD >> look I don't want to join make any speculations here but >> I'm just saying like people are too comfortable giving their IP but this
like you know it's it this forces us to like just consider the question of IV data and whether we should really be giving away everything that is our mode or part of what we're working on at um
you know Osmantic for ODS enterprise is basically like a masking software for hybrid AI so basically when you're asking for tier intelligence for a task it's stateless it's abstract it's well
definfined scopes so that you can get the value out of the hybrid AI the cloud without giving away your IB comes back goes through a proxy goes through local models or self-hosted
models that we have for you and that then fills it fills up the capuge market it's it's >> there is a there are ways to use frontier intelligence without giving
away your IB >> I think this will be a massive market >> it is and we we we see it that's why we're building it right it's just people are like you know still like the the
issue is the people are not um adapting fast enough for AI and are still behind. I still I still think that we are in like I have been doing local AI since 2023. I have moderating local lambda
subreddit doing a bunch of things in telling people hey watch my like look at this server I built in my basement etc. All of that stuff I was like early 0.00001%.
We're still in like the early 5% even less >> less. >> Like the massive adoption has not taken off yet. We are still early and that's
why like we're building this thing. We're optimizing it for every piece of hardware per model their workflow. And if you're working with software like this, you want it to be the polish
experience. You can go customize it and build your own experience. But the questions becomes, well, I have been doing this since 2023, so it's way easier for me to figure out exactly what
needs to be done, right? other people are going to get, you know, one shot by AI. Like, you know how we say one shot? It's like you're gonna basically lose track of what you're trying to customize
if you don't know what you're customizing for. >> This is the main like argument of like, oh, will everyone just vibe code their own, you know, Google calendars? Like,
of course not, right? People >> need domain expertise. >> Exactly. And the taste and like noticing why people click this button. Why is this layout optimal?
>> Look, man, I can go make interviews with an AI, but it's never going to be as good making interviews here, right? because you have been more because you have been doing this for quite some
time. You know what this is about. I know what local AI is about. So I keep telling people this work hard and like really go in depth into something extremely difficult because that's going
to be your mode. >> Yeah, >> that's what I'm doing with local. That's what is about. That's what we're trying to build and help. But just vibe coding
something doesn't necessarily mean it's a polish experience or it's even the right experience. >> Yeah. Let's talk about like people. So, in terms of hardware, right? What is
like the good strategy for this? Like, let's say hypothetically somebody has $10,000 maybe like total. Okay. >> Yeah. >> Would you recommend them like save up
bit more? Would you recommend them spend 5K or like 7K on laptop like this? How should people think about like price points and like what type of hardware >> would we still use for? So there's this
tweet uh that uh I wrote about how I discounted how unified memory is going to be um very good option for models that are coming out right now and are getting efficient like seeing efficiency
gains just because of architectural improvements in models, right? So like something like this machine or you know running a 180 billion parameter model you know despite the fact that the word
they wanted to use earlier is enagrams. So enagrams right now allow you to like use SSDs for certain parts of the model. So it allows for more optimizations right.
>> Yeah. [snorts] >> So these kind of architectural um improvements and like novel research discoveries are allowing us to see that unified memory is actually way more
capable than what we assumed earlier on. 6 months ago I used to tell people you know the GX Spark is not the best option but it's becoming clear to me that it's actually very viable option for somebody
who wants to run like three four barrel agents using this model right it's the same idea like again I showed you this game we're going to put the video out there for the people right this is
entirely developed and designed end to end the graphics the spirits movement the mechanics on a model running on a laptop that you can carry in your backpack.
It's not heavy and it works so well. Uh so the the amount of money you need to spend on AI hardware or on hardware it's going low like you know it's it's >> but at the same time the hardware is
going up. how hard the rate is going up and again that goes back to the thing that we were talking about earlier about how uh >> you know we're expecting to see RSI and
we're expecting to see uh models improving themselves and we're we know that uh we're capable of getting more of the of of the models on you know fewer constrained hardware resources than we
used to a year ago right like do you remember like January I said like OBUS 5 OBUS 4.5 on an RTX on a RTX 46000 by the end of the year. We got that by March, right? Like we got that and right now
like 3.8 8 27 billion parameter beats OBS 4.5 and you can run four bar agents on an RTX 1590 that a year ago used to cost $2,000 that last months became so expensive like I don't want to say so
expensive but like it went up like normally it to5 $6,000 that Nvidia is even discontinuing because you're also are limited and constrained by the hardware resources that they have like
by the things that they're receiving right it's ultimately it's resources that >> supply chain >> supply chain So hardware is going to
keep getting more and more expensive but new paradigms and hardware are also going to open up and we're going to have new chips and novel research done in that for optimizer inference. Now how
long does it takes to get there? Which company? No idea. But the good thing is it's going to get better eventually. For now we also know that you don't need to spend
100k to get good performance. as you know, I don't want to say cheap because $7,000 are not cheap. All right, but $7,000 gets you a model that is quite good, runs locally.
>> No need a data center. >> You don't need to go get go get the data center. You don't need to build a massive rig in your basement. You don't need to do any of of that. Literally,
this thing in your backpack works and gets you something. Exactly. >> Yeah. So how like broadly speaking right I I know you don't want to like bet on specific companies but like if people
were to decide like okay do I go with these um operating systems or this inference like would you bet on like you know NVR because as you said more optimized CUDA how should people think
about it high level okay um Intel has GPUs right has GPUs that have on spec on paper good specs but the software stack for it is very behind
>> and unless you're willing to spend days and days optimizing for that GPU for a specific model that you want to run on it and be okay with it taking a
massive amount of time to improve then Intel is not your thing. So to me it becomes like you know degrees of again we're planning on like you know on really making improvements for these
GPUs right with ODS but that's because we know the target that we're going after we know exactly what we need to do and we have a plan for it individuals unless you are you know
technical you should probably aim for the best software stack right now. So basically the ecosystems that already have like millions of people looking at it optimizing.
>> Exactly. Exactly. It comes down to the ecosystem ecosystem. If you're just getting into this and you have no idea what I'm talking about, you have no idea what the kernel is, just go with Nvidia.
Like that's your sign, right? Or or you know, Mac Studios are also viable option, but it's still it's going to be behind and you're going to need to learn a bit and like do some research and
figure out the right inference engine. But Nvidia is the most mature ecosystem right now. And I mean they started coder what 2015 >> right they I mean Jensen made a bet very
smart man he's forwardlooking right he made a bet that he changed his company from being a gaming uh graphics uh cards to like computing >> company after Alexand
>> yeah he made he he basically like uh realized that computing as like you know in science and mathematics is the way forward that eventually we're going to have enough data and we're going to have
enough power and we're going to have enough uh processing power in these chips that we're going to able to like just throw data at it
>> and come up with something that scales up. Not sure like you know I I'm not saying here that he was making a bet on LLMs but he was making a bet on computing as a paradigm for machine
learning for neural networks for deep learning. These are things that we knew that it would scale up eventually. There are research on these saints from the 70s, right? We knew that we're
constrained by physics and novel research that got us to the point where I mean, think about it. 20 years ago, >> computers weren't even able to run
graphics this good. We're looking at something completely different now. So that means that it's just like you know it's the normal evolutionary thing for computing and AI
and artificial intelligence. Um so that bet allowed him to basically invest in the right software stack in the right ecosystem. Koda is so mature because of that and you have so many people who had
access. There's a reason that uh 3090s are still popular, right? RTX3090s and um you know when I started I got a bunch of them and I got them for $500, $600 a piece. Right now it's like
$16,700. That's more than the MSRB that they came out with in 2020. again very constrained on the hardware supply but also the architecture that was um like the code architecture that was in that
hardware in that base of hardware was the same one used in the Amber generation in uh data centers. So it was pretty good like you know you could basically extend it between so many
different GPUs and that allowed the work done on kernels for um A100s and GPUs that are running in data centers to expand to the to the hardware that you have running at home. That gave it a lot
of surface area of support and made it way faster and quicker than anything else. Right? So it works well. Why? because so many people have it because there's also data center hardware that
had it because there's also uh professional engineers that worked on the you know on the kernels on the models and >> the models were made on the same
hardware. So it's the ecosystem. So it's how many people have eyes on this thing and how many people can work on it. That's the cool ecosystem. >> And in the same, you know, same chain of
thought, what about Linux then? You think Linux is like inevitably going to win? Because again, it's the biggest ecosystem in terms of people optimizing it.
>> I still need to like verify this, but I think we have built the fastest inference engines for Windows RTX Spark with this ODS optimized version for the RTX Spark.
the operating system as, you know, as a bottleneck, I don't believe is the right approach to this. I don't think that we're going to have a one operating system that wins all. I don't think that
we um I don't think that, you know, the answer for people that are looking to run local AI is switch to Linux. That's why we're doing this, right? That's why ODS out of the box runs on Windows and
Trent because I can't think of a way of like telling some, you know, a cousin in my family, you need to switch to Linux, man. >> Yeah, I'm I'm talking like the agents,
you know, again, more of our tasks will be interacting to an agent and the agent will be doing it, right? So, if in the future we're not even like clicking around the settings, the operating
system as a whole is more for the agent than the human. >> That That's true. That's true. But like that's why like something for WSL for Windows it runs open in the background
right like you you have you have ways of like abstracting and making something work anyway on any operating system so the operating system layer itself is not a problem in my opinion
>> so similar to what you said with the like outlines right you prefer certain outlines it's almost like the OS will be like drawn for that person so if some somebody is like 10 15 years on Windows
>> maybe in the background the agents are using Linux or whatever they want but for the for that human It's rendered the way. >> Exactly. Exactly. It's going to end up
being something he doesn't have to like any individual doesn't have to think about it. You don't have to think about Windows or Linux or Mac OS. This stuff are going to be taken care of. That's
what ODS is about, right? Where we have WSL in the background because we know that we need some Linux component to this. But we also are trying to make sure that you don't get interrupted or
you don't get out of your user experience if when this is what you used to when this is the user experience that you need to get. Don't take people out like don't ask them to make a major
change in their lives. We're trying >> what if what if it's better though, you know, like what if we see the >> are you going to get more adoption or less adoption? What I care about is
local AI. What I care about is getting people to adopt local AI. I am I expect am I expecting more people to consider running local AI if I tell them I have it support I have an excellent support
in Windows or if I tell them to switch? So to me the thing I care about is local AI adoption. The What we need to be thinking about is okay how do we get agents running on Linux as you were
saying that's what your project is about right send boxes sandboxing can even be done on the cloud right it can be done on self-hosted machine that you tell your agent go set up a virtual machine
for me then deploy David's project on that virtual machine and then use that virtual machine with that deployed piece of software for agents running in the background
>> so basically the agents can handle the virtualization you just need So like the operating system layer in my opinion I'm not going to move around and tell people to change to to move to Linux. I mean I
use Linux personally. I'm a Linux person. I'm a Unix person. I like tools and CLIs and pipelines and bash. But I can't tell people to go out of the way and learn all of that stuff. That would
be too much. I mean I had been putting out technical content and articles explaining local AI inference engines how everything work for years. And then I realized that well the main way to do
this and get local AI adoption is actually by building a seamless experience which is what ODS is about. And instead of trying to force people to read and learn about all of the people
already have too much on their hands. >> Yeah. >> When I think about it like AI being introduced in our lives have made also us like have a major anxiety. So like
you know the idea that we're always working on token generation or always getting our age. I had this tweet the other day because I woke up at 5:00 a.m. and the first thought that came to my
mind on a Saturday 5:00 a.m. was go check on the agent that was running a goal yesterday. So I wrote a tweet that if the first thing that came to your mind when you wake up is checking on
your agents when you woke up. Go touch some grass, please. I stopped working that day. I decided I'm going to go out because it's like I need my we need a break. So now overloading people with
even more things when we already have like this general anxiety around AI and how much productivity we need to get out of our day. It's not the right way. It's not the right approach. I don't want to
over like I don't want to give you or overwhelm you with one more task. I want to take away from your blight. >> Yeah. Remove the friction and meet them where they are. I guess my counterpoint
like I'm looking at it from a different lens. I'm looking at like people who want to go deeper into AI, right? And they like want to learn more. No, that's a different thought. Yeah,
>> they want to learn more. So like you know when the agent is doing everything and it's taking care of our taxes, our accounting, building all our presentations, coding the games,
>> then like the next level of work is orchestrating these agents, you know, like okay, we have this hardware, we need to optimize this new LLM, news LLM, you know, there was one like from
Germany, Calibri, let's see if it can run on this hardware. Let's fine-tune it. So that like people will stop from clicking around spreadsheets or going into the settings like you said new
computer you have to >> ancient days you know spend multiple days setting everything up just one prompt to agent set it up how how I like right but then people have time on their
hands right and they want to do something some people will spend it playing tennis and running on the beach whatever some people want to be locked in and go to the cutting edge right so
that's what I'm curious about like the higher level of work where does that move you know does that move towards like new virtualizations inventing like new methods of post training Those
people who are watching this, they are not like a typical >> consumer. So right now you're building this cloud room, right? Yes. Cloud room. >> Um you're also looking at sandboxing
from an area of we know that you know for people for some people it's going to end up being agents on their phones. And we've talked about this before, right? We've talked about how we see a future
where spin up something on your phone, have your credentials stored somewhere, grab your repositories. Now it becomes for somebody like you it becomes how can I cut down latency? How can I make this
these sandboxes run so fast? How can I take snapshots in the fastest way? Talker is not the answer to that. Right? So we're having people that are technical that working on some excellent
stuff to make sandboxing like you're doing right now a better experience, right? Making it a faster experience because we know agents are going to need that in the background. what you're
doing right now is that you're trying to simplify it for folks at home without you know removing friction for them. >> Of course, >> I don't think that 90 something% of
people are looking for that. But if you're looking for that, there is so much optimizations left on the table right now, right? Everything has been built for um you know um
human cognition processing speeds. Think about what agents need and go optimize something for it. You're gonna make money. It's it's as easy as that for me. It's
we have so many bottlenecks right now. There's a reason that we are getting new CU CPUs, right? There's a reason that Nvidia is coming out with Vera Robin as a CPU. They are creating CPUs for the
next wave of agents, right? Like for the agentic era. It's because the paradigm around computing has been built around humans and now we're the
bottleneck. That means that we need new paradigms to allow >> everything will be reinvented. >> So yeah. So, of course, if you're doing
that, you want to do it on something that is more optimal and that has more exposure to libraries, kernels, hardware layers because you're trying to, you know, remove abstractions and cut down
speeds. Linux allows you to do that the fastest. Okay. So, do you think Linux is like gonna hold this, you know, number of position or do you think there's
something coming? Right. For example, uh recently Enthropic released a bro. I know you know he don't like Entropic, but what cool thing that >> I'm I'm cool with Enthropic. I just I
hope that they are cool with open source AI. That's all. >> Yeah. Yeah. I'm not going to talk about the model. They they said that they built their own like version control
basically that Git wasn't keeping up, right? So >> thinking about that again everything needs to be reinvented like you just said.
>> Will we see a new operating system even better than Linux? like how you think about these like low-level I I I think that potentially the first step is going to be creating um [snorts]
like we have Linux containers alexes right um these are sort of like visualizations that um uh basically abstracts some key processes and elements uh without
basically having to visualize every process we're going to have something that basically removes you know for a specific sandbox for a specific workflow
it's going to remove anything that is irrelevant to that sandbox that could be as that could be you know removing modules for internet removing modules for Wi-Fi you're
so funny to me when like they say oh the agent escaped the sandbox well go build a sandbox with a Linux uh subsystem that doesn't have internet you can do that you can even create your own modules
tools for these agents to communicate with the host without going and having any outside world contact. So the the you know this argument that we're doing this to save
you know because we need to sandbox well sandboxing could be better. >> Yeah. And also they shift the responsibility to the agent like they built it you know.
>> So I would say do we are we going to get see new operating systems? Probably. Are they going to be like you know customuilt for workflows? Probably. Could they be inheriting stuff from the
Linux kernel? >> Yeah. >> Yeah, potentially >> because it's it's going to be easier to just handle the hardware that way.
>> So, does it become a Linux operating system then? Yes. >> Because it's using some of the kernel stuff. So, you have to like, you know, you have to attribute it back. You know,
attribution in open source is very important. Like I this is one of the things that I keep going back to. If you're using something open source, please like give attribution to the um
yeah to the to the libraries to the people that created that work to to the to the original authors of that work because that's the least you can do in open source, right? Like give
attribution and respect the license. So if you're using something from the Linux kernels, you're absolutely going to be saying it's a you know it's from the Linux kernel. Can you call it something
other than Linux? Maybe. I don't know. Go check the license. But what I'm trying to say is ultimately with how cheap tokens are and how fast they are with OBS 5.5 and like you know I hope
they don't just like start doing more filtering and blocking more and more things [snorts] with all of that. We're seeing um we're seeing that you can build custom everything
>> everything optimized for specific things. So yeah, like of course if you're like you know ticky and you're trying to build something probably going to be playing more with Linux than
Windows, but that doesn't mean that I don't like this RTX Spark or that they don't like >> Windows machine they should be able to use it
>> like look man okay like look we have we we've given here like you know we have uh Hermes we have a bunch of things this here um I want to show you this right these are from yesterday first of all do
you like the game >> I mean this I I would not believe this is fully local. >> This is fully local. I can show you the
traces. I can literally show you the traces. Let's Let's even do this. Can you make me hack/ dungeon crawler style game for Lord of the Rings? Make it cartoon.
Hob slashing his way across murder. >> Literally 20 second prom to like >> literally said hello first. Yeah, you don't even have to be like a >> This is portal, our own agent in ODS. We
give it the cloud because I thought it was a nice bun. Basically, it's the cloud running locally >> and the entire thing end to end in quen 3.8 flash uh next NVF4 that is optimized
running on an optimized inference engine that we built for the RTX Spark. entire thing has been built locally and then you're playing this and you can even like you know
>> so basically like you know the way people can think about it is that you can use tokens >> to make it easier for others and cheaper for others to get more tokens right
>> y the this this um uh this cycle of selfimprovement or I don't want to say I'm sorry self-improvement is wrong but like this cycle of optimizations
uh it's inevitable at this point in my opinion and that's what we're building osmantic was in mind like I knew that the future is going to be selectively optimized
experiences workflow model hardware chip it matters it matters a lot you can't just say one shoe fit all it's too
expensive too costly in my opinion not to be optimizing for the future in that your time is going to be worth more. the things agents the fact that I there's no other inference engine running on
Windows on a DJX Spark/ an RTX park that runs at 72 tokens per second indicate the benchmarks 72 for coding mostly um goes down into 50s for thinking and to calls but at the same time it beats
everything out beats VLM it beats lambda CV PV. Why? Because it's optimized for this experience. >> Yeah, I like what you said there. Like people always think about like the
short-term cost in front of them, but longterm it's going to be way more costly not getting into this, right? Not getting into open mates more sooner, self-hosting, local optimizing, all of
this stuff. Yes, there's some friction to start. You need to learn a bit, but like long term, the biggest cost is not taking any risk. >> I mean, think about it. The DJX Spark
used to be $3,900 something, right? Right now, it's like what, $7,000? >> Yeah. There's a reason. It's because more and more people are becoming aware
and I just don't see a way like in at least the next two years. Again, not a financial advice. >> Not a financial advice at all. >> Seeing some trends,
>> but I'm just saying saying that from you know what I have seen so far, prices are not going to go down anytime soon. >> There's no way it it's become I I I I told um I told some people that I am not
using my RTX3090s right now. keeping them at bay because there will be more value for me to extract out of them in the future when smaller models become more efficient
when I have workflows specialized on models running on these GPUs specifically then my agents will make way more use out of them >> so basically saving the hardware
>> I'm saving the hardware because the agents in the future are going to be able to use them more my extraction my value extraction per token per what per electricity cost
it's going to be more. >> Yeah. This is like basically power plants getting more efficient over time. >> It's not just power plants. It's also why hardware if you think about it it's
also why hardware is becoming more expensive. >> Fit more you know transistors on the same. >> Exactly. It's also why hardware is
becoming more expensive. >> So you know it's the there's a reason that I have been telling people to get into local AI. It's going to become so much better. There's no way like the
future is inevitable at this point. There is no way that local AI isn't the default for the majority of people. So you think >> hybrid hybrid hybrid AI is the future
but for most of the people >> Yeah. Yeah. So like again how should people think about this right? Because on edge which means you know computer phone right now the models aren't equal.
So like how should people think of like what type of task they should run on their phone locally or on their computer like paint the picture maybe two years from now like on edge inference versus
cloud inference. [snorts] >> Um again it's going to be if you're using cloud APIs if you're using AI that you send something to outside of your
control you're going to have something in the middle that abstracts it that makes it stateless. You know how I started my formal background computer science and data science. Um I had the
option to go into machine learning and do like a PhD but I decided I wanted to take care of stuff. So I chose to go and become a software engineer. Been coding since I was seven right? This has been
the thing that I have been doing my entire life. I remember 2020 uh22 late 2022 I was helping uh a relative with some you know um residency application. Okay, medical residency application. I
found we were working on their essays and I was like use right and while they were using chajbt I found us giving too much information about that person through a cloud model
>> that's when I'm like oh the future has to be local back then like I don't even know if lamatu had came out and models had like local models were in a horrible shape
Yeah, >> throughout 2024 I got my hardware. I built a I built a 14 RTX3090s machine >> in late 2023 for a reason. It's like >> to me it was like the future is
inevitable. It's a matter of time how we get there. Back then everybody was like discounting it. In my opinion it was like if I have to give my data to these cloud models, somebody is going to think
about this and come up with no research that will allow us >> to run this things locally. And again, it's not just, you know, the obvious. >> It was a bet. It was a for like, you
know, it was a bet that they made on the future >> like Jensen made his bet. He made their bet. >> Exactly.
>> Yeah. I want to talk about the risk of again sending the data because people again they think about OpenAI training the model, but that's not the only thing, right? Open AAI is a company.
What's above a company? A government, right? >> Yeah. >> There could be a future where the government says, "Listen, what do you
have on this user?" You know, and if you send too much, >> you remember the flock thing? Um uh there was like um thing about how Flock was coordinating uh you know Flock is
like >> cameras >> a camera system that people like put in front of their homes and apparently they were recording stuff and coordinating
with >> um coordinating with um uh specific departments in the government and I don't want to get into the politics of it but what I'm trying
to say is um the fact that something on your property. I don't have anything connected to the cloud outside of my house. >> It's all internal. I get my own backup
in the cloud, but I have I'm the only one that has access to it for a reason. I do not trust that there will be no incentives for corporations or pressures or you know scenarios where
my data is compromised. I don't like that. I don't like the fact that somebody could have something on me. And not that I'm scared of anything, but it's a um it's like it's an invasion
of my privacy. Yeah. I share my financial information with my accountant, but that's their job. I don't want somebody to >> have like legal restrictions.
>> Exactly. I share my medical history with my doctor, >> but I don't go about it with somebody who could give it away. Now, this is not an accusation.
>> Yeah. Yeah. Removing the risk like removing the dependencies in the software. >> It's it's it's basically just the question you have to ask yourself. Could
there be a scenario, a timeline where incentives align in a way that forces things to go off track into the wrong direction? It's basic
basic privacy hygiene is how I would is how I would like you know articulate it. It's uh we need to be aware of could go what could go wrong and we need to make sure that it doesn't doesn't mean
that we don't uh have trust you know that we don't uh hold cloud you know cloud providers to the to the policies that they have set for themselves which is don't train on my
data right don't train on a mathematician's research and come up come out with the same exact I'm not saying that it's true or false. I'm just saying it's like
>> the principle of it. Don't invade people's privacy and don't don't don't also put your full trust in something like trust but verify is the way and to me
it's why we're building ODS enterprise was like a abstraction that makes any prompt that we send to the cloud stateless we specify exactly where in our code or where in your
enterprises code do we need to improve something make it better right we figure out uh interfaces and the modules that we need to improve without giving them teethful context about your business.
That way we can ask them can ask a cloud model we can ask OBS 5.5 or GBT 6 Astra or GBT 6.1 saw about the improvements that they would recommend because they are indeed the most frontier intelligent
and capable models but also without giving them your IB data. So it becomes a question of okay I'm going to use it in a very specific way with mass data and like a in in a in a in a fashion
that protects my IB protects my business protects my mode but also uh I'm going to rely on models that are running locally that are very capable that are very intelligent to follow
instructions from these frontier models and do the implementations and I'm going to allow back and forth that removes that abstracts away my business until I re until I can verify that everything
has been implemented to the best possible. >> A good picture for people to like realize how this will work is maybe like outside consultants, right? Like you
have your business, you have your team, people who know how things are going and maybe you need expertise in like one area that isn't your specialty, right? You would hire an outside consultant for
that area, but you don't give him all your info, you know, all the medical records of everyone on the team, all your secret sauce. You just cons consult with him on that one specific topic and
get his expertise, his his knowledge. >> Exactly. Exactly. That's a great way to put it. [snorts] You're basically taking away the burden of providing them with every sensitive piece of information
while still getting their expertise on the subject matter that is um most uh you know um valuable to you without giving away anything. you're [snorts] the the future is hybrid and it
needs to be done in a very specific way and it's you know this specific way is why we're building a company right like why we're building why I believe that this is the future you need some some
way to abstract stuff right you need some way to make frontier intelligence you know ODS by the way works out of the box with hybrid AI you can set it up with JBS enterprise is what adds that
abstraction layer that we're talking about right And we have that for a reason because we believe that everybody's going to require hybrid AI but uh you also need
to do it in with safe mechanisms that doesn't give away your data because I mean we have seen Figma CEO come out and talk about you know a certain company releasing a product that matches theirs
after they have been using that company's ABI tells you something right like >> conflict of interest you know >> yeah conflict of interest so why why are
you giving away your data to cloud cloud platforms. Why Why are you allowing your enterprise employees to use um [snorts] to use models that are looking their emails?
I can tell you for a fact that um and I can't say who, but like there are CEOs that have hardware in their offices sandboxed for their emails. >> That's how far they are going for
protecting their own data. >> So only a local model can >> only a local model hosted on massive machine, massive costly machine, but in their office because they don't even
trust that it can be outside >> them. >> So what with that in mind, why aren't you more careful about the data that your employees are giving away?
Everybody's using every AI endpoint out there. It's like like it's nothing. Like open water is fine. And I'm I'm not coming after opener. I'm not coming after a specific provider. I'm just
trying to say if you don't own the infrastructure end to end, you cannot guarantee that it's safe or that it's the um you know it's um protecting your in your incentives and your intellectual
property in the way that you want it to. You're giving away too much. >> Yeah. I'm going to share how we did at a very practical budget. What we have is 2Dx sparks running uncensored deepseek.
So the super deep seek finetune and anytime we need to do like a you know legal question, tax question something like highly private >> that's what we use right inside of PI
harness we just swip it flip it to our own office provider and this is running and u you know it's >> and you're protecting your data you're protecting infinite tokens predictable
inference >> you're talking with somebody like me if I have shared anything with you you're keeping it in privacy you're basically not giving away too much of anything and
I respect that and businesses is more and more are going to be aware of that and are going to move of the cloud and in favor of hybrid AI in the fashion that I'm exper like I'm explaining right
now with something like ODS enterprise it it's just it's a so what is the what is the biggest missing piece right because I think education like educating people on on this is very important
right then employers also having this option right >> it's knowing that this is out there it's knowing that this is available it's knowing that this is actually possible
it's knowing that like I don't know before before we came here. Did you know that you can make a game like this? No. With >> OBS? If you showed me this, I would
think this is like a close model. >> You will think that this is like OBS 5.5 or like GBT6. No. This is literally a model that runs on a unified memory device on Windows
that costs less than $7,000. And I'm not saying that $7,000 is not a massive number. Um, that's not the point. I'm just saying that >> it's cheaper than it's cheaper than it's
cheaper than ABI tokens months over months. Like just think about >> openi $500 subscription times 12 6k a year. >> Exactly. So like blows my mind that this
is not like >> and hypothetically if the hardware could also go up you know >> it's it's basically like you know I leave it to the audience to decide on
this >> and the same hardware like we discussed last time it's an important idea same hardware like you said you're saving exactly your more capable models right
you're saving your some of your GPUs for better models >> if you remember when we talked last time we just have um even like V uh Deepseek V4 Pro the second version that came out
in August 13 wasn't even out. >> That came out showed a massive improvement over the first version and then we saw Deepseek V4.1 Flash >> huge
>> which like I mean that thing a year ago wasn't even on the cloud. >> Yeah, I just absolutely par frontier. >> It's amazing model for just 2D Xarks. the capability and the intelligence that
you're getting out of this hardware is becoming better and better like you know it's exponentially growing and that's what the people are not are not recognizing we have entered the
exponential era of intelligence growth for local AI and it's not stopping >> and this is what the labs understand that's why they're like hoarding all the hardware
>> that's why they are holding I mean um basically subscriptions are subsidizing the leverage that these labs have to go out and raise more money. That allows them to take loans. That allows them to
um not just take loans, but use that money that they just raised to go build data centers, which makes your, you know, hardware that you can own at home more expensive and, you know, more
scarce. So, stop subsidizing them and building the data centers that they are that is forcing us to have more and more expensive hardware. In my opinion, most people don't need a subscription.
They just need to own their hardware and run something local with ODS. And we got you like I mean it genuinely. This is the most optimized thing that could run on a DJX Spark or an RTX Park
right now. We're going to keep doing this for every piece of hardware out there. We're going to keep up the momentum because I believe that we need to make a seamless experience for people
running local AI and once they realize how good it is, >> there's no going back. We're still early and we're going to make it happen.
>> All that money that people are spending, you know, on I would say the even worse is like raw API pricing, right? That's even worse deal than like a subscription. Like let's say
hypothetically like instead of you know hoarding infinite subscriptions allocate some of that to hardware so that in the future when the subsidized subscriptions end which might happen.
>> Yeah. >> You're not exposed of like you know having to pay $50 per million input tokens you have something that again 95% of workloads perfectly.
>> Exactly. Exactly. And we're seeing that the hardware is more capable as we go exponentially. So that means that what you're putting in right now I mean imagine
>> it needs to be decentralized. That's the power right? like who has the hardware has the power in the future. >> Exactly. Exactly. And imagine something like OBS 5.5 running locally right in a
year from now on an RTX Spark. >> I'm willing to say that that's going to happen and like come back and check here. I am fine with making this statement. In a year, this thing is
going to be running the equivalent of OBUS 5.5 intelligence. >> And it's also not just the benchmarks, right? Because again, these models capabilities. Yeah. Capabilities and
what these models are willing to do, right? If you have a model that like actually answers your problem and not like you know claw models where you like if you insulted you might get banned or
stuff like that. >> Maybe it's not 100% on the benchmark. Maybe it's like 90%. But for you it might be more useful because it's your model and you know you can run it. your
data, your hardware, it's with you on the fly. It doesn't have inter it doesn't require internet. You don't get uh speed limited. It's not running 17 tokens per second like uh a certain
agent with a certain cloud provider does right now. Like this is definitely faster. >> Nobody secretly quantizes it. >> Yeah, this is definitely faster than the
code. This is definitely and you know the the beautiful thing of OS and this is something I want to show you David like we have like a full open source library right um this allows you to
integrate with anything uh right now what do we have in here like we have brave search we have Hermes agent so like uh how about we run Hermes for a second and
>> here's Hermes now this allows you to just right again the model is loaded up automatically right Oh, we know that this is if you start ODS automatically loads the best model for you. The
>> that's that's a big friction for people. They don't model and then you can easily with one click install an extension. We have a library of 175 tools. You we installed Hermes. We enabled it and then
automatically we can basically talk with Hermes. It automatically connects to the same API. So, uh I'm from Egypt. What is the time in
Egypt right now? It's going to go use that model automatically fully integrated end to end. You don't have to connect anything. You don't have to configure anything.
Every one of those opensource applications that we have in our library in our open source app store extension system gets integrated automatically works on the same hardware
everything out of the box. Now this is an experience that people didn't have right a year ago. >> Yeah, definitely. >> And this will only get exponentially
better as we go. Right now it's about 4:30 in the morning right now. Well, unfortunately >> I think it adds up though. It's - 8.
>> Yeah. Yeah. So now this this is one way. We also have our own portal here, right? And we can just do whatever we want to do. What do you want to ask it? >> Tell it to like um do a fork of uh PI
1.0 zero and uh make it better for a ODS. >> How about that? >> That's good. >> All right, let's let's let's leave leave
it let it cook while we're talking. It's going to take a minute. [snorts] >> But the idea is for me >> if you're not making a bet on your own like sovereign AI and I don't like to
use the word sovereign AI because like some some people find it easy but it's like decentralized. It's like something you control and self host for yourself. Same idea. People know electricity,
solar panels, their own car, you know, they have a car. >> You're giving away too much and you can't be giving away too much right now. Um
to [snorts] me the idea is if I cannot ensure that my data is safe, it's not safe. >> Okay, let's talk about this because people need to understand where AI is
headed, right? Because it's easy to like see the progress and being in the field. >> People think AI again may be a chatbot only, right? It's like why do I need to own my own chatbot? But no, like these
agents, they'll be working for you. They'll be your employees. They'll be your kind of in-house legal counsel. They're going to do more things for you in your life and business. So if you
imagine again in a couple of years where that will be, then it starts to become obvious why you need to own that and why nobody can take the access from you. The mistake people do is like again they
look at what's happening right now. free Google AI search, free shipping. It's like why do I need this tool? You know, it's it's it's available. It's free. It's not that helpful. But the progress,
that's what people need to look at where it's going to be in two, three, four years. Exactly. And then like all the things that agents will be doing for you.
>> If somebody can take that away, you're complete slave. um if somebody can take it away, if somebody can quantize it, if somebody can change it, if somebody can drop its IQ score, uh you know, uh like
the idea of like it's funny, but like the idea of segregated uh intelligence like to me is possible. You can drop a certain country's >> Yeah. I poison the responses, you know,
maybe the model is suddenly answering weird way. >> You can run psychological experiments. >> Yeah. By the way, that's on your data. You know, again, people think like,
okay, they will know some medical thing, run ads. they always think of the most obvious, but if they have a lot of private data on you, maybe the model starts answering in a different way to
shift your thinking, to shift your voting patterns. You could see why there would be incentives for a government to do that. >> It's it's to me the the most important
thing to think about is if the quality I get of something that I controls majority of my needs, why am I not doing it?
If shad [snorts] >> then we just need to remove all the friction >> and that's what we're doing with also experience was good for the majority of
people and if it was good for the majority of people why did we suddenly you know just trying to get everybody to use 5.5 we don't need we don't need everybody to use 5.5 right we we need
somebody to just be able to use >> this model on this RTX park and you're going to be okay >> I mean again it's the incentives you know the the big labs they want to sell
you the big model you latest model >> convince you that you have to be using it >> like again to me it's it's crazy how good of a game I I keep going back to
this game uh but to me it's crazy how good this game is when you're running it locally and nothing compares to that a year ago. >> Yeah.
>> On the cloud or locally. >> That means that things are getting better. So imagine it a year from now. A year from now it just it's going to be so much better. Oh, it uh sorry, this is
a new user that I didn't give it administrator rights for this. >> So, at least the sandboxing works. >> It's the sandboxing works. So, it basically blocked a couple of things.
So, >> that's uh for our next demo. I need to figure that out before the next time. But the the whole idea is we I'm sandboxed that with a user. I basically
made sure that it doesn't have the wrong um right access and it just did what I told it to do. And so, it had some that some of that. Like again the the the the idea that this is done locally
no a year ago nobody would have believed that and >> even like six months ago >> and like you can literally be on a flight running a DJ an RTX park and
you're like I'm bored let's let's make a game. Now the fact that that happens locally I'm I'm honestly this is addictive so I I need to stop playing while we're
talking. Uh [snorts] but it's the idea that you're capable of doing that at home. Think about the world that we're in right now. If we
only had a couple of providers, >> yeah, they just have so much control over us. >> We wouldn't like think about the limitations on the models. Think about
uh colleics 200 and 500. Think about next year it being a,500. >> Think about it all because it's subsidized right now. These
subscriptions are subsidized. That's what people do not realize. It's subsidized. They get you in because they want they show growth in their number of users. They go talk with u they go talk
with investors. They get them to fund them. They tell them that we're just going to keep locking in more and more users. The users cannot go away. And then by the time that they run out of
the funds, they just have to sell you the thing at its cost plus a profit margin. >> They need to raise a bigger round to find the next
>> next model frame. >> Yeah. Yeah. Yeah. Yeah. I mean, what's the next valuation? I mean do you think that do we think that anthropic is worse one/3 or one half of Nvidia?
>> Absolutely not. That's that doesn't make [snorts] any sense to me. So that's the valuations that they're after and that's why their mode is like you know that's why there are efforts to basically say
that open source AI is dangerous. >> Yeah. >> But it's not. I mean again as we like when we started talking >> centralized closed AI is dangerous. You
know >> centralized the bigger risk. Let's talk about the risk. You know, the bigger risk is a few people with misaligned incentives having control over your
future. Exactly. Much bigger risk than a Terminator running away. >> It's it's Cyberpunk 2077. It's uh it's uh it's June, right? It's the whole idea that we're end up in a dystopian future
that if you control I it's it's one of those things that really scare me, right? Like freedom is very important to me. Freedom is important to all of us. The fact that you can't control this
like civilizational infrastructure that you don't have access to it and it's gatekeep behind a model that could refuse your questions, could ban you, >> could be contised, could be nerfed,
could be misaligned, could >> anything really. It's hard to imagine that. >> It's it's it's terrible world. It's a terrible world if that ends up being
>> Can you imagine a personal assistant like secretly working against you? >> Exactly. So like >> you don't want that risk. Now I know for a fact that if I want to do something, I
can just run this model in this RTX Spark. >> You know where it runs, what model it is. >> I know it's running. I know it's going
to go find my taxes, go do this, go do that, prepare the emails, get everything done for me. I'll have to I I review them once. But none of my data is sent away. Nobody knows about my bank
statements. Nobody knows about my taxes. Nobody knows about my health, my my medical records. These are things that, you know, these are the basics, right? But it it goes down in your life. It
goes down to your family, your children, your pictures, your photos, your significant other, your relationships, all of these things, anything like do you remember when X AI Grock was
basically sending massive amounts of data. If you run Grock uh CLI, it would basically send >> Oh, yeah. >> send the entirety of your like home
folder to the cloud. >> It was also like Yeah, I think it was multiple of these. >> Yeah. like it it there are so many instance where something running on the
cloud just sends a bunch of things to the cloud. How how can you trust that? [snorts] That's why open source matters, right? Like you can investigate and make sure
that nothing is going to invade your privacy. >> Yeah. And that's why it has to win. >> It has to win. >> I think we're out of time. Appreciate
it. Thank you so much for having me again and this was a great conversation.