Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Anthropic has lied about model quality degradation and used 'safety' to push for regulations that would ban open-source competition, according to AI researcher Ahmed. Local AI is now viable: DeepSeek V4 flash runs on a $5,000 Nvidia DGX Spark and matches or exceeds GPT-4.5, making self-hosting cheaper and more secure than closed APIs. The expert advises skipping closed labs entirely and investing in consumer hardware, with models becoming more efficient exponentially.
Key points
Anthropic denied model quality degradation for months before admitting it, then continued lobbying for regulation to ban open-source models. This hypocrisy was exposed when Anthropic models refused to help secure Hugging Face from an attack.
DeepSeek V4 flash achieves frontier-level intelligence on a single $5,000 DGX Spark and is 70% smaller than GPT-5.2. Businesses can save 70% annually by self-hosting instead of paying API fees while keeping data private.
Ahmed recommends starting with two used RTX 3090s or a DGX Spark for a capable local setup. He predicts that within 18 months, a single RTX 5090 will match frontier models due to exponential efficiency gains.
Tools mentioned
Techniques
- Retrieval Augmented Generation (RAG)
- agent orchestration
- prompt masking for data privacy
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
You have to question their intentions and where they are coming from and what game they are trying to play. Every argument that these closed labs had about safety, security, let's slow it
down. None of it is true. Everybody can see, oh, so you just want to ban the competition at this point. The direction we're heading in right now is the direction where everybody holds this
thing for themselves. There's money on the table that you're leaving when you're using [music] these APIs, and you would be far better off owning that infrastructure end to end. You're giving
away too much about yourself. If things go the wrong way, and it could very much go the wrong way, it could be used against you. >> Right, Ahmed? So, you're one of the
biggest proponents of open source there is. Why do you believe that AGI should be open source? >> So, let's put AGI on the side. Whether we get to AGI or not, that's a different
like, you know, argument and discussion that a lot of people love to get into and it's me it's philosophical, but why should we have this, you know, new thing that everybody's dealing with? Like to
me it's a civilizational infrastructure. Why should everybody have access to that? Because you don't want somebody to basically build a model from under your feet. You don't want to somebody to
quantize it. It can happen that one specific location have like you know 5% IQ drop from another one just because they control the model. Like you know there are so many ways that they can
control exactly what what's being served. So why shouldn't you be in control of that infrastructure? Why shouldn't you be, you know, you have the weights for yourself and then you set it
up on your machines, whether it's a DC V4 flash that can run easily on a DJX Spark that is $5,000 or something as you know as Frontier Intelligence like Kim Case 3 which is close to Fable 5. And
those are models that you can literally download on your own machines and set up for yourself and nobody can take them away from you decide on how quintis or how the the intelligence level look to
you or you know how these models interact with you or respond to you and uh yeah it's controlling this thing that thinks for you man like how could you not want that to be under your hand also
nobody can see your messages nobody can see your interactions nobody can get your intellectual property if you're an organization that matters that matters a lot
>> yeah I mean People share all kinds of personal stuff with AI way more than with any other technology. But I think the strongest argument is like if you tell people, listen, I'm going to take
away your computer. I'm going to take away your internet, your electricity. People would understand how crippled you would be. You couldn't compete in business in the job market. But people
don't realize that like in two years, in 3 years, in 5 years, AI is going to be way more important than these technologies. And if somebody else controls it and can just like turn off
your access like they did with Fable, >> then you're completely screwed. >> Exactly. Exactly. I don't like this. Like the information being sent to an ABI or to anybody out there should be
stateless. It should have no information about you whatsoever. We should a couple of weeks ago I posted a tweet about how I got a message an email from ChadBt telling me to start advertising on
ChadBt. Guess what? They're collecting all this data about you from all the conversations that you're having to decide on what to serve you in terms of ads and they know about you more than
any other website. like you know everybody was like talking about how you know you think about something or talk with somebody about something and you see it on Instagram or Facebook oh
what's going on in there now it's like you know it's not just about that you're literally talking to this thing and you think it's just it's it's not looking at you or it's not analyzing you and you're
telling them what to sell to you it's you're giving away too much about yourself if things go the wrong way and it could very much go the wrong way you could be you know I don't want to say
blackmailed but like it could be used against you and it could be used against you systematically. It could be a health insurance company that makes a deal with one of those providers. There are so
many things that could go wrong here and you are better off controlling this thing and using it on your own machine. I I usually make this example and I I have my um my iPhone here. You know,
GBT40 used to be served in data centers. Right now you have quen 4 billion parameter that could be ran on this thing and it works better. It's more it's smarter. It's more capable. The
direction we're heading in right now is the direction where everybody holds this thing for themselves. And that's why hardware is becoming more expensive, right? And people need to stop like in
my opinion, people need to stop giving money to these closed providers because they're using that money as leverage to raise money from VCs to then take that money and go build data centers, which
in turn makes the supply chain tighter, which in turn, you know, stops you from being able to acquire the hardware that you need for yourself. With AI, writing code is now super easy. The new
bottleneck is reviewing all the code that your agents write. And that's what Code Rabbit solves. Code Rabbit is an AI code review agent. You connect it to your repo and it learns your whole
codebase by itself. And it works in your IDE and CLI as well. They just released the code rabbit review and trust me, this is needed in the era of AI coding. A single pull request written by AI can
easily touch 30 to 40 different files and GitHub lists those files alphabetically which doesn't make much sense but code rabbit review puts these files in order core changes first
everything else built on top of after each section comes with its own summary plus diagrams right inside of the diff. When Code Rabbit finds an issue, it explains why it matters and then it
writes the fix for you ready to apply in one click. And when you give it feedback, which you can do in plain English, it remembers it for every review after that. There's also code
rabbit agent for Slack. So your team can investigate issues, plan work, and ask questions right inside of Slack threads. And this is already being used by popular projects like Bun and Knox. If
you want to try this yourself, it's going to be the first link in the description. Oh, and thank you Code Rabbit for sponsoring this video. That's a really good analogy with the supply
chain. maybe we should, you know, go in there because a lot of people don't realize that like I mean they realize it because it's in the media but you know they don't realize that like you're
going to get priced out because these companies can can spend billions and tens of billions of dollars on the data centers and obviously the people manufacturing like all because of supply
chain they're going to give it to the highest bidder, right? So like the consumer GPUs and the consumer hardware is going to be the first one to go. >> Like a year ago I I remember telling
people buy and ving about like you know the RTX 590 being 1900 at Walmart. go grab it. Yesterday, Microsenter pushed up like, you know, raised the prices of the 1590 to $4,400. There's a higher
demand than supply. The prices are going to keep increasing. And the thing is, memory is no longer cyclical. When people say memory is cyclical, it's always been cyclical. It's because you
went from DDR1 to DDR2 to DDR3 and everybody had to upgrade their servers. So, that's why memory became cyclical because everybody's upgrading at the same time or in the same span of years.
And then suddenly everybody's using the same DDR4. So, there's more of it than the demand is. But right now, we're not going into like, you know, a new technology and people cannot uh get
their hands on it fast enough. No, the demand is higher because everybody wants to create more tokens. So, it's you know that the argument that everybody tells me about how oh the just wait the the
prices are going to go lower. No, Apple earning cold uh a few days ago they were talking about how they have been relying on their supply reserve for for memory and that's why they haven't been raising
the prices as fast as others have been. But they are talking about how by September it's going to be tough for them and that there's going to be like you know fewer iPhones than usual
because they just cannot have like you know they don't have enough reserve or they don't have enough and they cannot acquire them fast enough to build them fast enough and also therefore the
prices are absolutely and naturally going to increase. So it's almost like um you know certain hardware is becoming an investment like you can literally buy hardware and like this is not
financially. >> Yeah, of course. Of course. [laughter] >> I'm just making sure because like I remember a year ago when I was tweeting about this and everybody's like are you
telling people to like you know go in that like it's it's a joke. You don't understand the concept of a joke. I'm not I started telling everybody sell your two kidneys if you have to acquire
the shoes. I'm like okay if if you do that then it's not for me man. [laughter] the the Michael Sailor strategy. >> Yeah. How can people get into self
hosting? Let's get super practical, right? It's intimidating for people who like just use shad, you know, it's nice and configured packaged product. How would you recommend somebody who knows
nothing about self-hosting to get started? >> I I I have been working on this project called ODS. It's um full deployment system, full stack, end to end because
um let's back up one step. When you're using cloud code or codic cli, you're not just talking to a model. You're talking to infrastructure. You're talking to a system that has ser search
engines um set up and configured properly. You're talking to web scrabbers. You're talking to like an entire infrastructure. What is lacking in um in in the open source AI like
space and like you know local AI specifically is something that makes that experience seamless. I remember somebody like a friend um actually back in March. He bought an RTX 5090 and to
use Quinn 3.5 the 27 billion parameter model and he hooked it up to clothe code and he wasn't able like he asked it to change the RGB colors on his GPU and it wasn't able to figure it out. I asked
him did you give it a search engine? He said what like what are you talking about? But I'm like, you need to give it something to go on the internet, grab the data it needs, and then use that
tool link that it has access to through the internet to change the colors with RGB. And I have my own search engine endpoints. And I opened up the firewall on on my one of my machines and I gave
him access through it. And I was like, okay, set up this environment variable before you run the command, before you run the cloud code command and see how it does. It worked. And that's something
that makes the user experience for local AI very difficult. you don't know which pieces you need to configure. ODS is our way of saying and it's Apache 2.0. It's open source. Anybody can take it and
turn it into something that they can use themselves or even commercialize it how they want. We have ODS and we have ODS enterprise atmantic and we do a bunch of things with that. We set up you know
organizations and we have the open source version out there for individuals to get into local AI. It's it's my way of saying hey we can make this thing as seamless as possible end to end because
you know it's image generation it's video generation it's it's a chat interface it's all your agents in one place it's configured out of the box search engine for you it's rag it's like
the entire infrastructure that makes chat GBT experience that good that needs to be replicated locally and this is our way of doing that so I would say the easiest and like you know the most
seamless way to go about It's something I have been working on because I have been in this space since 2022 and I know all the pain points. >> I'm going to link ODS below. It's on
GitHub as mentioned. People can clone it, can fork it, whatever. They can do anything with it. >> It's a one command by the way. It's like literally a one command gets everything
installed. You don't need to like even worry about it. >> So that's the software side, right? Let's talk about the hardware side. How about I list out the different budgets
and you tell me like how would you think about it on those budgets? >> Yeah, let's go that. >> Let's start with $1,000. Is it even possible? for something that is very
simple. I wouldn't like I I honestly wouldn't like I would say that the minimum is like going to be like $45 $5,000 a day. >> Yeah, let's go $5,000.
>> GGX Spark. Buy yourself a GX Spark or Stress Halo. Um just simplify the whole thing for yourself. Uh unless you're doing like, you know, a thousand barrel agents, then that's probably like the
easiest pass to start with. Uh because if I'm starting with somebody who really wants to get into this and do like you know tensor parallelism and like try to understand how this works, my usual
recommendation is go get yourself two used RTX3090. And RTX3090s couple of years ago they were $500 a piece. They are they came out back in 2020 but right now it's like $1,000 $1,200 to to to
acquire one. So if we're talking about, you know, $45,000, one pass you could take is to RTX3090. They would allow you to experiment with different inference engines run agents like concurrently
have like you know barrel agents running at the same time with quin 3.5 for 3.6 27 billion parameter model. They are better off for dense model like you know that that specific model I just
mentioned. And I I I don't know if we should go into like you know the difference between model architectures like versus dense models but that that's where the hardware matters. I have two
articles actually written on this specifically on memory bandwidths and memory memory capacities and it's under the URL the local aibbook.com like if you just type that it opens the thread
on X that has articles. So like it's not even a website, it's just a a link to articles and it starts with how you need to differentiate between memory capacity and memory bandwidth. Memory bandwidth
is what allows the tokens to be so fast when they are being created. It's what allows for uh something like uh you know four five um different agents running in parallel with like each 100 tokens per
second or something like that. That requires GPUs. That's like you know the RTX Pro 6000 has a 1.8 8 terabytes per second bandwidth while a DJX Spark has 273 GB per second memory bandwidth.
That's almost one, [snorts] you know, if I'm if I'm if I'm doing that like it's 17s, you know, around 17. And it's a it really shows when you're trying to like have an agent run with 200,000 tokens
context, right? Like I I've tried on 4DX parks before running um an agent with 200,000 tokens and it took about 23 minutes or something like that. It was mini max and 2.5. So bandwidth matters
when you're trying to be fast. Capacity matters for how big or how intelligent and capable the model is that you want to fit on your hardware. And there are trade-offs to them. And most people
don't think about them. They just think about oh this has 64 GB, this has 128 GB. It's more than that because you can go and buy yourself two Intel GPUs that each has 32 GB and it's almost like it's
I think it's like the the bandwidth is like 600 GB on them. But the problem becomes that they don't have the software support on inference engines and then like the kernels that kod or
like the contributors have out there because there's like you know a wide variet like a lot more people that have Nvidia GPUs. So therefore there's a lot more contributions and involvement and
development for the those GPUs. Intel, you're gonna be stuck with, you know, fewer models supported and the performance is not going to be as optimal. So, you can have, you know, on
on on paper the specs cheat is going to tell you that you have 64 GB of of of memory on and 2 GPUs, but is it going to actually show up in an optimal way when you're running LLMs? No. It's going to
be very painful experience. And that's what I try like when some people argue with me why I'm like pushing against certain you know hardware vendors out there and it's because I don't want
people getting into local AI to actually be you know turned off because the models that they want to run cannot run on the hardware that they just spend their hardearned money on. So it becomes
like you know a situation where I know Nvidia out there have teams that are actually working on making the GX Spark better and writing code kernels for it. like I know that there's support. I know
that AMD is starting to also push for that. There are other vendors that are still behind and if we're just, you know, it's a there are trade-offs even to like, you know, say there's a Intel
has GPUs that are good and fast, but there's no software support for them. So, you're going to buy them, but you're not going to be able to use them. So, usually again, $45,000, I would say
start with two RTX 3090 or a DJX Spark/strix halo. If you're going into like, you know, the the $20,000 area, because really the jump is from here to like,
you know, the jump. No, I would say the jump is from this to 5090, an RTX 5090, which alone cost you like after taxes, like about $5,000. And you would probably need to spend like three $4,000
on like the hardware that comes with it, like the like the RAM, etc., the SSD, the motherboard, and the processor. And that's going to be like 10K the 10K budget. So a 1590 plus um a full case
build would be like about 10k. What comes after that is you know the 20k range >> y >> RTX Pro 6000 and after that you know
it's I I I would say if you're spending more than that please go read my articles please go read the localbook.com and you're going to you know just take
your time read through it. You're going to learn so much about the trade-offs and then you're going to be able to make a decision. How would you build your like if let's say someone has no
hardware or you know only one computer their main computer how would you build it in a scalable way right so like they don't just buy like 5k and then that would be kind of bad for their 20k setup
you get my point right yeah I know what you mean but the the thing is if you're trying to scale it up right now and you're starting with like 5k you're not going to be able to scale it up because
um the the quick and like straightforward answer is that it's not possible when you're starting you need to actually keep in mind like okay I want to add more GPUs later. Then you
actually are looking at workstation motherboards and CPU like Thread Rubber or the Intel server grade or Epic server grade CPUs uh and and memory and those are very expensive like you know then
we're talking about like a motherboard and like these motherboards usually prefer if you have like all the RAM slots filled up. So you're going to spend be spending at least 7 $8,000 on
RAM alone and it adds up. So, if you're starting with like you tell me I want to start with 5K and I want to scale it up later, I'm going to tell you it's not possible because
>> you're better off saving, but then if you're better off saving right now and waiting until you can buy it, are you going to actually be able to buy it then because of how expensive it's gotten?
>> And also, you know, I I earlier this month I gave a talk at AIE uh the AI engineering world fair and it was about um impact per parameter. I was talking about how we came so far since Lama 2.
My prediction was that within 18 months, we're going to have the equivalent of GLM 5.2 intelligence running on a single 5090. And funny enough, today we got Deep Seek uh Flash V4 that is 70%
smaller than GLM 5.2, which came last month and would fit on a single DJX Spark with higher intelligence. Crazy. December 2025, I wrote a tweet that prediction by the end of this year, by
the end of 2026, we're going to have OBUS 4.5 quality running on a single RTX Pro 6000 and not the nerfed OBU 4.5, the actual OBS 4 happened by Marsh >> like it's not even the end of the year.
The the way that you know we're seeing right now open models becoming so more efficient, it's compounding exponentially. Like once you figure out the tricks around the architecture that
combound well like deepse just did today and you have enough data to train on and use the right compute for long enough amount of time we're going to have more and more efficient and smaller models.
So right now when I tell people I really think a 1590 would give you really good intelligence and due time like in a in a year or so you're going to have very good intelligence. A DJX Spark a year
ago nobody would have bought that right nobody because like what can you run on that thing? Nothing useful. But today you're getting literally something that is better than obus 4.6 running locally
that you don't have to give away your data. Your intellectual property is yours and that model is very good, very efficient, fast and intelligent. >> Yeah. So let's talk about the deepse v4
flash because uh this is a huge improvement and I would say like my my first impression is like deepse needs better branding. they need to like add like the 0.1 the point you know 0.5 or
stuff like that because a lot of people look at it and say like wait didn't see V4 release two months ago you know that was a preview now there's like >> yeah so so why is this model so good and
you know talk about deepseek >> we all know that we actually have like a naming issue within all labs right if if five was just called 5 nobody would have cared and this wouldn't have been VA GBT
for 5.6 six. Now we have Soul, Luna, and >> fast. >> It's like, how can I get my head around all of these names? So, I I think there's like a [laughter] there's a
general problem with how researchers and labs name their models. The whole idea of a preview, like I had to explain to people today. You were right that a that was a preview, right? Now, this is like
an actual full run. And I really think that the Deep Seek V4 like the full version when it comes out, it's going to blow people's mind. That model I have a tweet when it got released that it's
undertrained and I can send that to you right now if you want to put it on the screen. That model was extremely undertrained for how big it is. They didn't give it enough data with better
and better data. I can't wait to see the numbers and the performance of Deepseek V4 Pro like the the actual the actual model that is 1 something trillion parameters. That's going to blow my
mind. That's for sure. They have all the right tricks in place. It just comes down that it like it not it wasn't cooked enough before they gave us the preview. And I think that they just
wanted to say we're here. We're still working on this. Wait for us. Today with this, you know, DC V4 flash the the actual not the preview but the the first version. We're seeing a massive jump
from the flat from the review version in terms of performance. And I think that would apply the same and probably compound even better and exponentially with the bigger model if if it's trained
with enough data. >> So you think it's going to be even more powerful than uh Kimmy? >> Oh yeah. >> Why? Because it's a smaller model.
>> It's No, I'm I'm not talking about the flash. I'm talking about the act the full model. I'm talking about the pro. >> But isn't the preview pro like 1.4 or something? You think it's going to be
way larger? the architecture really um seemed like they had every piece in place so well. It's just about scaling it in a compounded way. >> Basically, when you're training a model
and that's why I say like, you know, I I used to say it's like LLM training is really like, you know, and research is like, you know, vibe research. You're just trying to do things and see how it
compounds. They had every right like every trade that they had in terms of architecture looked so good. it just wasn't scaled enough every step of the way. Now you show me that in a properly
scaled way, I think it's going to be a killer. So like, you know, let's have this on record right now. We're going to put this video out and if it happens and it's actually better than Kim history,
post it on X and, you know, share it around. >> All right, so now let's talk about Kimmy. How big of a moment in your eyes was the release of Kim K3? Cuz I feel
like it was a kind of a wakeup call for a lot of people. >> Uh, absolutely massive. I have been telling people to wait for August through uh October for all the LLMs that
we're going to get and u I haven't had any doubt about open source AI catching up to the frontier. >> It was more about is uh is are are certain players going to push so hard
against open source AI that it becomes banned >> before we get there? And you know I have been very um very clear about what I think about
certain labs out there for quite some time and um I know that you know specifically anthroic they have been acting in bad phase. >> Yeah.
>> In my opinion for the past couple of years I have tweets dating back to October 2024 saying that's not a good lab like they're just safety and alignment as like cover for
>> Oh yeah. As long as they're not in the lead, right? like oh we never want to advance frontier >> you know KK3 came out and suddenly open source or open models open weights
however you want to call it are catching up and you know an alternative to their way of just controlling this thing and giving us an endpoint to use that they can take away whenever they want
suddenly have half of like anthropic lab or you know that that litter of like let's face AI development let's stop this thing from going out of hand suddenly we have them out there pushing
ing for this. Where was that a year ago? Where was that like a few months ago? Or is it just because somebody else is catching up to you now? And then you go into this letter and you notice that
about 50% of the people that signed are actually from Andraic. They always talked about safety and cyber security and how they are the ones doing good for people. And then you look at the attack
from open AI towards hug and face like that autonomous agentic attack that happened where hugenface got hacked by open AI models and how hugenface tried to use both open AI and anthropic models
to fight what went wrong and for safety reasons openai and anthropic models decided to refuse the request about looking into it and I'm like so you're not keeping us
secure and your models can be like you know your guard rails that are keeping everybody safe could be bypassed it could be used for an attack how how am I how how am I supposed to keep myself
safe >> it's complete hypocrisy >> hugging face had to turn to GM 5.2 to to actually use that model, which is an open model that they could host in their
infrastructure that didn't have guardrails when they were trying to protect themselves for fake safety reasons. like every argument that these closed labs had about
um you know safety, security, let's slow it down, they are all just their ways protecting the remote and like trying to use regulatory capture and is Antropic right now valued that $1
trillion that $1 trillion valuation absolutely not doesn't make any sense. Is Antropic uh um worth oneird or one/4 of Nvidia? Absolutely not. makes no sense. And right now with Fable 5 being
met with chemical history, their $1 trillion valuation is at risk. And of course, they are going to try to go around this issue. >> Distillation, you know, they're going to
they're going to scream bloody murder to to to save themselves. And it's none of it is true. >> And you see this with these posts like oh open says our mortal hacked and then
Anthropics is like actually our mortals did free escapes. It's like who can raise the other? >> No, my model is is doing war crime. Trust me, my model is even more
dangerous. Please regulate us. >> It's it's like why don't you guys like if if if if all of you are deciding to sign letters, why don't you create like you know a civil committee for
yourselves as like you know stop yourselves agree on certain rules and stop yourself from advancing too fast. >> What about that? Why do you have to go to the government? It's not ethical
for them to be to continue to lie to people especially after you've caught them once, twice and three times like it just they keep doing it and they keep acting in bad face and I don't know how
we can like you know how can you tell them like everybody now like Christmas last year everybody was in love with anthropic and cloth. I was telling people like come on you're getting like
you know don't let them trick you. They are not really the safety guys. They are not really the ones looking out for you. They literally lied about their models quality being degraded for so many
months when I was like they kept talking about it in through September and then they had two postmortems. Oh, we found this and we found that. So you had issues. So why were you denying it when
everybody was trying to tell you like it's basically a gas light thing as a service like [laughter] the other thing you remember when they said oh when you're training on frontier models or
when you drink anything that has to do with AI uh our model is not just going to refuse it's going to your code base >> how is that ethical how is that safe
what principles and values allow you to say that hey this is the right decision to make and let's push that on people out there >> yeah and then you go into the destroying
books thing where they like, you know, have to bypass some like judge's order. So they like destroy the books. They scan it literally like erasing human knowledge and only keeping in their old
models which again you know they want to cry about distillation. They distilled all of us all people throughout all of history all literature or code all art and now like somebody oh somebody trains
a small model on some outputs of them. Where did you get your data? Huh? And and like it's so funny because they said like, you know, they they were trying to paint the the picture that Kimmy K3 was
trained on Fable 5. Kimmy K3 was, you know, how long was Fable 5 available before Kim K. >> It's irrelevant to that training run that produced that Kim K3. This is like
you have to question their intentions and where they are coming from and what game they are trying to play. And I just I I I have no faith that they're acting in goodwill right at this point. Like
they are just trying to protect their business and their valuation, their crazy stupid valuation in my opinion. And thankfully I think it's too late. Uh I think it's too late to ban open
source. I think it's too late to try to control people in the way that they wanted to control people. And it makes me wonder if things had gone wrong, what would these people do to us with the
civilizational, you know, infrastructure that everybody needs access to? >> You mentioned it's too late to buy open source to ban open source. Why do you think that? And like what can we do to
still like make sure that it never happens? >> I don't know the future, right? But I can tell you that at this point any small lab, anybody that wants to create
something can take Kimmy K3 and use it for syncic data to train smaller and specialized models. That to me is more than enough to tell you that that like you know the advancements are going to
continue. We also have seen you know um things like Neotron collation um by Nvidia. We have seen the open weights and American leadership letter by Microsoft which Osmantic you know we are
really proud of having signed that letter which is telling the government we cannot ban open source. We need to give this you know this civilizational infrastructure to everybody out there.
We everybody needs to have control over this thing not just a single entity or a few entities. Um, I think that there's momentum behind open source right now because people are starting to realize
that this thing is bigger than a couple of companies that could very much lie to your face and sabotage you and get lie to you, right? It's just it's a at this point the awareness is very important,
but also the fact that it's it's getting to the point where industry leaders are speaking up about it that I find it very difficult. It's or it's okay to think about you know regulations in terms of
can we all agree on basic guidelines and it doesn't even have to be regulations from the government. It could be like you know a civil body. It could be like just companies coming together to agree
on stuff you know agreeing on certain testing boundaries before pushing out model. But asking the government and lobbying to do that is where I draw the line and that's where I h I find myself
having issues. I think everybody can see the intentions behind Entro Havoc and behind certain labs out there and um thankfully people are not falling for it anymore. Everybody can see oh so you
just want to ban the competition at this point. So you know when we're binging it in that light and especially after seeing that attack by from open AAI on hugging face and the you know the the
their refusal by open andic models from helping with you know the the the person that actually is getting attacked from protecting themsel. We at this point like everybody knows that it's dangerous
to just hand over this infrastructure and allow these people to control it on your behalf. You have to be in charge of it. Also, I think a lot of organizations and that's what we're seeing at
Osmantic. A lot of organizations are starting to wake up to the fact that their business intellectual property could be stolen from them and used by these labs to, you know, replicate them
and then replace them. >> And that's very dangerous. The best thing to have is a free market where there's a lot of competition. You're not like monopolies because monopolies
basically they they they make you subject to control it. You're not free anymore, right? >> Unless you're the one controlling it, right? Like
>> Exactly. Exactly. No, they're doing it in in good faith. Lol. Um anyway what I'm trying to say here is that organizations are moving away from cloud APIs from ants from all of these to
self-hosted to dedicated to colloccated hardware to something that they either own or rent. But the most important thing is that entire infrastructure is under their control. The weights of the
models they cannot just you know the the the model cannot just be updated and then you have to use the latest. you you know that happens a lot when we see like you know eval that ran on workflows for
you for your business or organization and suddenly they updated the model and these no longer are valid the model is not acting the same way and it's because of course you're not going to like you
know maintain a thousand versions of their model weights but when you using a model that you're hosting for yourself you can maintain that you can control that you're not upgrading things or
updating your model or changing anything about them until you decide so also uh the models do not become you know, less intelligent or less capable at peak hours because compute constraints.
Therefore, they have to like do certain kind of quantization, whether that the weight or the KV cache. But in all cases, all of these things lead to um like degraded performance, right? And
you're in control when you host the model for yourself, you're in control of the quality of the output. Therefore, you know that you're going to maintain that level of output forever. That's
very important. And you're not giving away your data. You're not giving away your customers data. You're not giving away your business ideas. You're not giving away your business logic. It's a
massive thing that you have to have full control over. And that's the for the best of like you know that that's better for you. I I I want people to think that it's in no way a good thing for a couple
of companies to just control this thing for us. Like it's it it applies to every every one of us. we cannot allow them to and that's why I have been very like you know even since local AI and open source
AI was like you know very niche I was telling people no I'm buying GPUs and I'm doing this and I'm setting up this massive monstrosity in my basement because I believe that I will need to
have this thing when the time comes and they are degrading the quality so much and they are no longer serving my needs and they are refusing every every request that I'm I'm sending them and
they are stealing my data. It's more important than the internet in my opinion. It's it's like it's like it's >> the most important. >> It's it's it's definitely a utility that
you cannot take away from people and unfortunately that utility requires a lot of compute and power therefore making it difficult to come by. But I think it's even worse if you keep just
paying these companies to build data centers with your leverage subscribe like you know subscription numbers and ABI spending so that your the hardware that you could acquire yourself is out
of reach. That's why I was talking about this like you know long time ago. We I knew that we would get to this point where we have models that are very good that you can self-host but I was also
afraid of these um you know massive reges that allows them to go build these massive data centers to take away from the supply chain that would allow us to acquire the hardware for ourselves.
>> And also on the topic of like the you know they're going to steal your data. Have you seen the free plans for scientists? you know like this always so so amazing that like you know all these
companies are so generous to scientists and researchers like it makes you wonder you know why why could that be >> I I I'm not aware of that one but like I I get why you would have doubt and like
you know whatever they tell you at this point like it just >> I mean they want the high quality data you know everybody knows like they scrape the entire internet at this point
everything is scraped what better way to just like get 10,000 100,000 scientists and give them chd pro you know it was open in this case and again I'm I'm just making speculations here I'm not making
any claims names in case opens lawyers are watching but uh I'm just saying like that would be a really good data would you agree you know having 100,000 professional scientists
>> I don't know what kind of agreements they have but they are definitely going like you know you know the the amount of I I have a tweet from a long time ago ago saying that it's all about data and
compute right and everything will scale up right if you're trying to focus on one thing then you need the data and you have the compute so you always have to question the intentions I cannot make a
statement here about like you know whether it's one thing or the other but I can say that data is very very very important and uh of extremely high value when you're trying to train for small
and specialized models like atmantic for example we deal with organizations that were telling them hey we're setting you up so that in 12 to 18 months we're going to build you smaller and more
efficient models specialized for your tasks and your workflows and we're going to know exactly what to send to the frontier based on what we're collecting right now and what to use on the
specialized models and even if we're using the frontier ABIS for certain tasks, it's going to be masked and in a stateless fashion because part of what we're training for is something that
would take away anything about your business plan and ask the abstract questions that allows them these frontier models to tell us something unique or of value without giving them
away the the actual value that you're trying that you're after in your in your workflows. These are very important things and I think that it applies to scientists, it applies to researchers,
it applies to schools, it applies to any organization. I mean do you remember Figma for example? Figma working with with anthropic and then a few days later anthropic came out with Figma design
like you know >> this is going to happen in the industry on every level. Go read the fine print of the agents that you're using. Your data is being captured. What do you do
with your data is unknown and data and what data gets used and has been a an issue for decades now. So what is so different about about it this time from every other time where we knew that data
was used in ways that they shouldn't be used then? like you mentioned, I think more and more businesses are waking up to this and they're going to, you know, want to get into self-hosting into
having their own models. And I think people, you know, year ago, two years ago, the opportunity was really like, oh yeah, you're going to implement AI in terms of like the app layer. I think
there's going to be way bigger business opportunity in understanding self-hosting on a deep level, understanding how to build this hardware, you know, what is the, like
you said, the software stack as well and everything that goes into it and then going into a company and say, "Listen, I'm going to help you implement this." you're spending all your employees have
these like, you know, team plans and these subscriptions. Look at how much you're spending per year. Why not allocate part of that into your own compute?
>> Basically, we're seeing on average 70% spendings in the first year, 70% savings in the first year after we deploy any infrastructure for organizations. I can tell you that there's money on the table
that you're leaving when you're using these APIs and you would be far better off owning that infrastructure end to end and it's not just for like you know we it's going to be more efficient it's
going to be faster it's going to be cheaper but not only that it will save you in the long run from giving away your business and we we do that like you exactly what you said we do that into
atmic because it's very complicated and it requires a lot of work but it's better than continuing to pay for this these EVI and these tokens that the prices of are unknown in a year from
now. >> So which of the we talked about you know some sketchy practices with these companies but which of the big tech companies is the most friendly towards
open source. [laughter] >> Uh I think Nvidia is doing great and uh the name is wrong collation and um their push for open source models. I u I asked
um in GTC uh in March this year I even was part of like a Q&A with Jensen and asked him about his commitment to open source AI. Uh he has been showing us over and over that he's committed to the
idea that others should have access to this thing and it shouldn't be limited to just a couple of players. I think Nvidia is doing great on that part. Like, you know, I was pleased last year
when we got GBTOSS from OpenAI and um I know that they signed the letter as well. Uh you know, but also I I would love to see them push out GBTOSS too, right? You know that their name has Obi.
So maybe give us something maybe you know the weight for GBT2 like you know it's something that I would love to have access to just it's like you know it's a it's a it's a historical massive you
know part of everything that we've gotten to right now. So like you know showing some good like you know goodwill with that would be good. Google is, you know, I I I I really want to see Google
putting out like uh the the massive like there were rumors about Gemma 4 being uh being like, you know, having a GM4 that is 120 billion parameter or something like that that they decided not to put
out, not to release. I I would love to see that uh one released. Uh that's for sure. Um Google releases Gemma 4 and like you know released a couple of good models. So I think they have been
friendly and they have been showing people. I think Antroic is the only company actually in the big three that haven't released any open models or urban weights. Um they they they just
like to talk about um distillations and stuff like that. [laughter] So it's a spectrum of how friendly or you know friendly companies are toward open models but I think we're
seeing a trend right now where all of them are agreeing more and more of you know the the freedom of choice allowing you to decide for yourself what you want to use and how you want to use it and
when do you want to you know go to the closed frontier labs or host something for yourself or use a bait ABI even for these open source models. the the big player that doesn't want to play ball at
all is anthropic and um [laughter] you know there's nothing to do there. We've tried. >> You mentioned that sometimes you know when businesses still decide to use the
frontier close models you have a way to like mask the usage. How do you do that? I I don't know I don't know if I can get too much in detail like openi has even released like you know a masking model
that is like one billion parameter like there are models out there that would allow you to do that but like how we do that like as an inter like you know for enterprise solutions it comes down to
the workflows it comes down to like what exactly like we have to collect the data first and understand it before we can decide on how can we take away from the business logic per organization that's
why it's like custom solutions really so it's a you know there are models out there that would like you know hide your name, hide your social security number, your address, all of these like personal
stuff that are common. But how do we do that there like you know organization or business that that requires us to go in depths about what exactly is your business? How can we create something
alone for that business that is like an like an abstracted layer that hides that from being sent out? >> Makes sense. So basically it's you know depends on the business and it's custom
IP of that business so that you you you still get the answer you want without like leaking your main IP. >> Exactly. >> All right. So uh I want to go back a bit
on the hardware. So there's so much to talk about like you said you know we can go into dense models. First I want to touch on like how do you see the Apple versus Nvidia right when people see like
okay you have a Mac studio you have a DJ Spark how would you help people to understand when to buy which? First, let's say that each piece of software that you use for models they are created
for the hardware that you're using it on. The way that the hardware is designed for Apple is different from Nvidia which requires you know the layer that comes above the hardware that
allows it to process the data that runs these models they are also different. So when you're trying to basically build an inference engine on top of each you're dealing with very different things.
That's why like you know it's it's not like you know it's not transferable. That's why I cannot just take something from Nvidia and run it on an AVL or from AL and then run it on Nvidia. So comes
down to open source contributors out there that have Max Studios and are willing to spend their time on building these models or making these models run. And that's why you have like MLX and um
and stuff like that. These inference engines for for Apple, they run very differently from VLM for Nvidia and AMD for example, right? like the just just the hardware specifications and how they
are designed allows for the kernel level um development to look quite differently. So when you're optimizing you're doing stuff that are different from each other. So they are not
transferable. That's the first thing. If it works on Nvidia doesn't necessarily is going to work with the same you know just because of software optimizations the same way on an Apple hardware. But
then if somebody comes and tells me should I buy an Apple Mac Mini, I would tell them stop because like the memory bandwidth is so slow that thing is going to feel like a turtle. Like you know
even the the max um uh the I think it's like 150 something uh gigabytes per second and it's like it it stops at uh stops at 64 GB in terms of capacity. So it doesn't allow for much. But now, if
we're talking about a Mac Studio that is maxed out and you can no longer buy them new from Apple because they don't have them, then it's 512 GB of memory and they have um 800 and something uh 819 GB
per second uh in terms of bandwidth. So, they are kind of capable. They are not going to run like uh 10 agents in parallel, but they they can do a few conversations at a time. you can run a
couple of agents, you know, with a model that fits in that space. Now, if you're asking me, should I go spend $30,000 on a Mac Studio or build myself a machine with two RTX Pro 6000? I tell you, build
an RTX Pro 6000 machine because the capacity doesn't matter as much because again, we just had a model that is 200 billion parameter today. We beat a model that everybody considered a
state-of-the-art model. A months ago, that was 750 something billion barometer. So models are becoming more efficient and taking smaller, you know, based on the tasks that everybody cares
about right now. Models are becoming more efficient and smaller and Nvidia GPUs are going to have faster support, more contributors just because of how available they are and how transferable
are things from one hardware to the next. there are like you know there are missing architecture um differences or like operations between like let's say a DGXP300 that is data center it's called
blackwell as well as like an RTX pro 6000 it's called an blackwell as well but they do not share everything there are things that are taking away from them but a lot of things are still
transferable so it allows you know for easier development of these things you're going to you're going to see people working on it more than or like getting it done faster than Apple stuff
in my opinion it comes down to, you know, how seamless of a process do you want to have? If I if if I'm dealing like, you know, if if you're scared of using hardware, a Mac Studio is not a
bad option at all. A Mac Studio is definitely better than a DJX Spark. So, like, you know, if you're comparing a DJX Spark to a Mac Studio, I would say like the Mac Studio has like three times
the bandwidth and has about like, you know, if it's max memory capacity, it has four times the memory capacity. So, it would allow you to run bigger models at faster speed as well. So that that
makes it like the the winning um the winning player in this in this case. But if I'm just speaking about like you know I I have like I have been standing by my uh by by my statements about like GPUs
they cannot be beaten like especially Nvidia GPUs and the latest generation of AMD uh uh data server GPUs as well um are doing very well for like KK3 for example like they they are actually
cheaper to run chem3 with right now. But what I'm trying to say is you have to think about bandwidth, you have to think about capacity, and you have to think about software support. These are three
things that you have to think about end to end when you're deciding on what to spend your money on for local AI. There are trade-offs to do between each of them. There are higher power
requirements. For example, when you're building a machine with two GPUs, right? Um the the GX Spark is like runs at like what 60 watts, something like that. If you put four of them together, like
people built eight eight of them together and still less than 1,000 watts for two DG for two RTX Pro 6000 depending on the version it could go anywhere from 1,200 to 600 W. So it's a
B requirements, heat dispension, um memory bandwidths, memory capacity, software support. These are all things that they have to consider. And then it comes down to where are you trying to be
in a few years from now, right? If I'm thinking about again, if I'm thinking about buying hardware right now for myself, I would go the the RTX Pro 6000. No, like you know, it's just it we're
expecting new Apple stuff and we still don't know if the new Apple stuff like the the new Mac Studio is going to have much higher bandwidths and if it's going to even be available in higher memory
capacities. So, and then when that happens, how many people are going to have that hardware to actually develop open source because it's not Apple themselves that are writing the
software. How many developers are going to be out there doing open source work and contributions to these new devices if there are any architecture changes between the the versions and how
difficult is that going to be for you know the your usage of the hardware or are you going to go learn about how to make models run on that hardware for yourself. So it's like a cascade of
questions and my articles under the local AI book and that's why I keep mentioning them because it's just this is like it's not easy for me to talk about hardware because I am not trying
to tell people just what to buy. I'm trying to actually explain that this is like you know multi-layered process and there are like things that you need to consider based on your use case and
based on your capabilities or like you know your your com like if you're comfortable actually doing some development work or if you're comfortable putting a machine together
if you know how to like work on if if you can add like I had to add some breakers to this house when I started putting together hardware for myself right like I needed 240 volt uh 40 amp
breakers to like just on some of my my notes. It it it just it builds up and my articles actually cover all of that. So that's why like when I meet somebody just saying, "Hey, this is just buy this
thing. It's going to work all the I'm" I'm like, "You don't know what you're talking about. I'm sorry." Because like even like even the RTX Pro 6000 when models come out, they're not as
optimized as like a DGX P300 that is in the in the data that that they have in data centers. It just it's they have them optimized so well for these because that's what everybody cares about. And
then the RTX4 6000 is still open source contributors like Lo Alonso is one of the very known names in like in the GPUcape for like you know people that run inference and it's because this guy
worked for Nvidia but also he like works on open source projects for himself to develop kernels for uh the RTX 6000 and kernels are basically uh when we're talking about kernels for for LLMs we're
talking about something that does a mathematical uh you know or data transfer from one um one area of the model architecture to the next and how fast you do that, how fast you process
that and how can you do that in batches and how you can do that concurrently depends on the the the optimizations that you write in these kernels. So these optimizations do not come out of
the box for any model for any GP view and like you know K3 for example they are using they have data centers that they train models on and they have data centers that they serve their on and
they are working on software that is optimized for the hardware that they own right then you have open source contributors that are running on VLM and like I know that like Red Hat owns them
right now and they're like they have people working on that professionally full-time but [snorts] these people are also focused on the highest return on investment and the highest return on
investment is going to be the hard like you know the the the data center grade enterprises that they are serving that are paying them. So they have to focus on different hardware. So the RTX Pro
6000 even though it's like $15,000 right now you don't have everything coming out of the box optimized. It works but it's not in the most optimal way. And that is for something that a lot of the
community members have already and still available for purchases. So now you have to ask yourself is it worth it to go buy an Intel GPU? How often are you going to get somebody to work on actual Intel
software architecture to write these kernels or these optimizations? Becomes very difficult. >> All right. So I appreciate the detailed answer. I'm going to link the local AI
book below. >> I try to make this as like you know as easy as possible to understand but it's really a complicated subject. [laughter and gasps]
>> Well but that's the answer you know like people want people want an easy answer like okay how do I do this? But then you that's how you make a mistake. You know, if you're going to spend tens of
thousands of dollars, you better do some research first. >> Like I totally understand why some people that are very techy, very good at what they do. So engineers are like, you
know, get uh upset at me when I usually my default recommendation is go buy two RTX uh 3090s, but RTX3090s have been there for the past 8 years. You have so many things developed for them. It's
even easy for you to just hand that over to an agent and tell it to go write your an optimized kernel for that hardware. It just it's gotten that easy. And it's because that hardware has been, you
know, um just out there cycle like, you know, circling the economy and people are using it so frequently. It's not the same for every piece of hardware out there. It's not the same for a new Intel
GPU that nobody It's amazing price 900 or $1,000 per GPU. You can have so much capacity and very good bandwidth. But do you have the software that actually allows it to run models? And can you
actually use agents to make it run optimized models? No. So I cannot go tell people that are trying to get into local AI to go buy these things because then I'm actually I'm making them, you
know, just have a bad experience. That's not what I'm trying to do. And that's why Nvidia GPUs are still getting more and more expensive. That's why even the GX part is becoming more and more
expensive while other hardware is not becoming as expensive like you know or going uh getting expensive as fast as Nvidia's hardware. So basically people need to understand that like whatever
they buy it's also going to be like more powerful in the future like the economic value that their hardware might create will be more powerful because we'll have more optimized smaller models.
>> I I not on financial advice but that's the trend I have been seeing for the past four years. >> All right I think that's a great note to end it on. Thank you so much Ahmed for
your time and again I'm going to link your Twitter and as well the local AI book below. What else should people check out? >> I honestly I had an amazing time talking
with you David. I know we had to reschedule a couple of times. I have had a great time sitting here talking with you. I'm hoping that the next time we talk uh it's going to be like actually
showing uh you know something like ODS running and demoing it to people and showing it showing how you know local AI looks and works. Uh right now if if you're an individual and you're getting
into local AI, go read the local AAI book.com. Go check out ODS and really understand what you're trying to do and what you're spending your money on before you spend
that money. If you're an organization, open source models are what you should be investing in. Owning your infrastructure and to end is where you should be what you should be doing right
now. And so it's only going to get more and more expensive if you don't act on it right now. Again, thanks David. Uh and um yeah, until next time, man. Yeah. See you.