The Inference Frontier: from 100 to 10,000 tokens per second — Sean Lie, Cerebras CTO

0:00 what used to be considered fast at like 1,000 uh 100 or or 200 tokens per second is quickly becoming the new batch mode. And so I think that you know there's a place for that. It's

0:13 great for prompt processing. It's great for very very parallel workloads. Um >> but my emails take 20 hours. I can hit a button and run it in two hours. >> Exactly. Exactly. Right. But but

0:25 quickly, you know, being able to do prompt processing isn't isn't quite enough. Right. That's a big part of the market, but that's not quite enough. >> Before we get into today's episode, I

0:35 just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to

0:44 also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without

0:54 ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the

1:04 only thing I'll ever ask of you. And it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week. If you do it, I promise you we'll never

1:15 stop working to make this show even better. Now, let's get into it. Okay, we're here at Crisis HQ with CTO Sha Lee. Welcome. Uh >> thank you. Thank you for having us.

1:28 >> Yeah. No, thank you for having me. >> Uh and it is the day after hot chips. Lots of things launching. Even like Apple launched the M6 stuff. Uh but you guys obviously had we were at Supernova.

1:41 You uh talked about CS4. We're going to talk a little bit about CS5. OpenI talked about Jalapeno. All the all the hot stuff in hot chips. What's your take this year having been in this industry?

1:53 I think I think uh well, first of all, thank you for having me. Um and uh and and you're absolutely right. It's a super exciting time right now. Um hot chips is a is is a is really good, you

2:04 know, time when the whole community comes together. Um, and you know what what used to be, you know, I still remember going hot chips 10, 20 years ago when it's a bunch of like, you know,

2:14 computer architect, you know, geeks, you know, geeking out on, you know, speeds and feeds and and now it's like this is where, you know, industry changing hardware is is being revealed. And so,

2:25 um, you know, it's come a long way. It's really exciting, you know, community. And I think one of the main takeaways that that that I had after, you know, after leaving Hot Chips yesterday is

2:34 like what an exciting time it is for the the chip design and the hardware industry. It's uh you know, this is really the golden age of uh of hardware right now. Um and you know, like you

2:48 said, I've been in this industry for some time. Um this is a very very unique time not just because you know AI is taking off and you know and there's uh you know obviously a lot of uh people

2:59 doing their own hardware but the amount of innovation that's happening now across multiple fronts we've never seen in the history of uh of the semiconductor industry this level of

3:10 innovation across all the levels right in the chip uh the interconnect the system design the software the optics the you know the methodology the tooling people are pushing across the board and

3:22 so like you know as a technologist and and you know a computer architecture geek at heart it's like really really fun to see this community and this industry at this stage right now and

3:33 when we and we felt that like at at hot chips and everybody I talked to it's like that that buzz is there it's amazing >> we want to go through the chips let's

3:42 you know talk about your chip first what's what's most exciting uh CS4 came out some crazy numbers here 30x faster the GPUs but uh you know walk us through what's new with the new CS4.

3:53 >> Sure. So, you know, we we we we designed uh our next generation CS4 architecture um with a new brand new system platform with the goal really um to make wafer scale uh mainstream to make it uh uh to

4:12 bring it to hypers scale and to solve uh a lot of the the the high density um data center problems that you know we're facing every single day. So we've designed this uh modular platform uh

4:27 that provides twice the amount of power to the wafer than we have in our previous generation. Um twice the amount of uh interconnect bandwidth, half the latency. Um and all of it is done at the

4:38 system uh architecture level. And what you see here is the result is that by being able to provide significantly more power to the to the wafers by you know packing more density into the rack we're

4:52 able to drive up the performance even further. Um you know we like to say that Cerebrris is uh with our current product already the undisputed leader in you know ultraast inference. Um and and

5:06 what's amazing is this CS4 product will actually push that frontier even further by taking it another two times faster. Um and this is a a perfect example right we here in this in this demo that we

5:17 gave uh at hot chips. Um we're showing uh GPTOSS uh running at over 4,400 TPS which is just mind-blowing. It's like it's it it it almost feels like

5:30 it's fake, right? Um and uh and and we think that this is going to really revolutionize the entire you know industry because you know not only are we able to you know continue to push the

5:43 frontier on how fast these models can run um it will enable all sorts of new applications um you know uh uh user experiences start to become extremely different right what used to be batch

5:56 and offline applications start to become real time um And you know all of this you know growth in new agentic flows and agentic uh frameworks now also mean that you know you're you're sitting there

6:09 waiting for these agents to go over and over and over uh uh in their agent agentic loop and all of a sudden if you're running uh your model at over 4,000 tokens per second. Now the you can

6:21 do you know more agentic uh uh loops you can do more reasoning ultimately you get significantly more capable more intelligent agents. And so this is what really excites us about this. Not just

6:32 the fact that you we show these really really big numbers which is also really cool. Um but but the fact that you can you know really start to do things that you can't really do uh otherwise and and

6:42 and start to enable brand new capabilities. Um that that's that's what makes me so excited about this this reveal that we had yesterday. >> Yeah. I think it's also very rewarding

6:54 that you guys started on this journey the big chip journey. uh you know wafer scale everything everything that you've always said it just like people didn't take it that seriously until they had to

7:03 and then now they're they're really really taking it super seriously right >> um I I I do feel like as you know co-founder CTL the the architect of this whole thing how are you guys approaching

7:14 the new generations post like AI boom like I imagine CS123 was a bit more uh sort of uh calm >> now like literally open AI is launching soup the the ultra fast mode with you

7:28 guys and like it it is a matter of like I don't know like you're you're you're co-designing your model with the chip almost. >> No, absolutely. I I feel like the main

7:38 shift that has happened over the last few years um for us is that you know the the first generation chip primarily was a technology demonstration, right? We had to demonstrate to the world to

7:53 ourselves that you know you can actually build such a thing. Um and uh you know we we we told a lot of people that this is the future and most of them were just like you guys are insane this is not

8:05 possible. Um and so you know at the at at the beginning it was really about you know figuring out the foundational technologies to make it possible. And what we're now seeing is now that we

8:18 have in fact demonstrated that this can be built and it can have the kind of performance that you know that uh this type of architecture can have what are all the things that it can enable right

8:29 and you you know you alluded to you know our recent uh ultraast launch with open AI that's a perfect example right um we're running uh uh you know frontier level one of the most intelligent models

8:44 right open's largest, most capable, most intelligent model right now at 14 times faster than their normal, you know, GPU speeds. And it's becoming this completely transformational thing when

8:57 you can combine the most intelligent models with the speed that the hardware brings. And this transition from technology demonstration to you know solving real problems um enabling new

9:11 new real capabilities is this transition that we as a company are going through. And how we're coping with that right now is we're really shifting our focus to ensuring that we can make this scale. We

9:25 can bring up bring the capacity that we need. we can uh uh run the models that um you know the users ultimately you know care about. Um and then we continue this flywheel by making sure that we

9:38 continue to put out faster hardware and and keep pushing the frontier. And that's that's kind of you know how things have shifted for us. And it's I would say like you know unfolding in a

9:48 way that is you know almost better than we could have imagined right it's this perfect confluence of like you know having the hardware with the capabilities

9:59 demonstrated now matching with the capabilities of the models and the applications. Um and and so our goal right now is just you know scale this thing as fast as we can.

10:10 >> And so maybe we'll talk about yesterday. >> Yeah. >> Uh previewing CS5. Uh can you recap for people who are maybe not not yet caught up and you know

10:19 maybe are new to the story for the first time as well. >> Yeah. No, no problem. So I mentioned that our CS4 system that we just uh launched is based off of a brand new RAS

10:28 platform that we call the Nexus platform. And uh as I mentioned the what's special about this platform is the modularity. Um it has a a modular power supply in the front. It has a

10:41 modular uh server in the back that we call the backpack. And this is a platform that enables that 2x performance that we just all saw the numbers for. But it's also the

10:52 foundation for multiple generations of products. We've designed it from the ground up to support multiple generations of our products. And in particular, we designed it uh together

11:04 with our next generation wafer uh chip that will be coming next year. And with that chip, we'll be pushing the performance even further. So, we just saw a 2x improvement this uh this year

11:17 with uh CS4. We're going to push it even further with another 2x improvement in performance. And what this ultimately means is you'll be able to run, you know, medium-sized models like GPOSS or

11:32 Gemma at speeds up to 10,000 TPS and even frontier level models like Kimmy or DeepSeek uh and GPT56 Soul up to 5,000 TPS. Completely gamechanging. Um all again enabled by this brand new platform

11:49 that we've designed for multiple generations. And so, you know, this is the trajectory that we're on right now. And and you know, now that the world sees the value of ultraast inference,

12:01 our bet is that even this is just the beginning. >> What's the rollout process like? So, you have a deal with OpenAI over the next few years. Um,

12:12 >> when do we get access, you know? So, everyone wants ultra fast right now. Sure. There's Gemma OSS. When do we get the big Kimies, the Deep Seeks? when can we start to you know actually use

12:23 >> that's a great question. So right now uh we are basically you know sold out of everything that we're building right and uh we are very strategically making sure that we're uh deploying every single

12:39 megawatt in the most strategic way possible um and in particular uh quite a bit of it um is going to open AI right and we've been very public about this you they're our big biggest partner not

12:52 just because of the the uh the commercial uh arrangement but also because of the fact that we have this co-design um you know spirit which you guys have heard open talk about this a

13:04 lot as well right which we believe will basically allow us to kind of continue this flywheel of not just making models you know faster but using that speed to make them more intelligent and so on so

13:15 so today a lot of that capacity is going into openi and within openi they have very strategically decided to use quite a bit of it for themselves. Um, so they're using it right now. That's

13:28 correct. Right. So right now internally they're using it for a lot of really critical uh use cases where the speed really really matters. Like they're using it in like their incident response

13:39 uh teams, right? When there's an outage in their service, for example, every single second, every single minute matters. Um, and so uh they're getting a tremendous value out of that. they're

13:49 using it in some of the most critical research applications where the extra reasoning is is enabling uh significantly you know more intelligent responses um and so right now that's

14:03 quite a bit of the capacity is going there and then as um as we mentioned in in in our launch with OpenAI we are now also making it open to you know to enterprise uh customers who are um you

14:16 know who are who are able to again use it for some of the most demanding and you know the most high value applications over time um openai and cerebrus we have committed to you know

14:29 to bringing enough capacity to make ultra fast inference uh available to a much much wider audience and uh you know and our our CS4 announcement our CS5 announcement these are very much part of

14:44 that commitment to continue to drive more faster tokens and you know basically more throughput to be able to satisfy you know all the various use cases out there

14:53 >> by the way you know the way that you framed it I realized that they didn't have to expose it to enterprise customers they could have just kept it for themselves

15:01 >> nobody knew that that's an interesting like business decision almost I don't know if you have any weighing on that that's more like a business analyst point of view

15:09 >> I would say I would say that um you know obviously I can't uh you know explicitly comment about their thinking right ultimately it's open AI's decision how they want I use this. But I think if you

15:19 look at it from the outside, it kind of makes sense, right? I mean, their mission very much is to continue to, you know, push what's possible and by using it internally that helps them do that.

15:31 But they're also a business now, right? And there's, you know, talks about them going IPO and all this and so very much even externally, you can see they're balancing both of these, right? And I

15:40 think we very much see see that playing out in our >> um, you know, in in the ultra fast space as well. I mean they're balancing a lot you know most recently their hot

15:50 >> we got to talk about this as well we got to we got to go into them so what are we thinking prefill jalapeno decoded cerebras what's the >> what are your takes

15:59 >> I think that's I think that's uh that's a very you know rational um you know conclusion I I think of all of the hot chips uh announcements probably jalapeno was the most exciting um but to me, but

16:17 not maybe not for the same reason as everyone else. >> I guess to to double click on that, you're the expert in chips, right? There's people see token per second.

16:26 What do you see as interesting when you going to say I think that like they you know they pushed a lot on the performance and the fact that they're significantly better than you know

16:37 better performance than um than the GPU than Nvidia. But what I see is that they've built a uh significantly better GPU and that in its own right is is very uh is a is is a big achievement. Nvidia

16:53 knows what they're doing, right? They they they own the market for a reason. They're not dopes. Um and so to be able to come out of the gate and and build a significantly better GPU is a big

17:05 achievement. But to me, the reason why Jalapeno is so exciting isn't even these all these paratos. It's really um the design methodology behind it, right? They they very clearly took a very

17:18 drastically different approach to building this chip, right? Having an AI first methodology enabled them to, you know, build the chip faster and achieve some of these

17:31 very impressive results, right? And that is 100% the future of our industry. And it's not surprising to see that OpenAI is kind of leading the way here because this is you know very much their their

17:42 MMO. But coming back to you know what you said about uh you know how we're going to use this right. I think that you know Jalapeno is is pushing the boundaries for both uh boundaries of

17:55 what's possible for both throughput as well as uh latency. And for for me, even if I take the Cerebrra's hat off, I think that's awesome for the industry, right? This is going to lift all the

18:06 tides. Everyone is going to benefit. Um, you know, they've they've also now been able to, you know, push the latency into regimes that um that traditional GPUs can't can't hit. And and that's also

18:21 great because again, we believe in speed. There's a lot of value there. And what's going to be really interesting and what I'm super excited about right and you know OpenAI is our biggest

18:31 customer. So this is this is one of the things that you know I think is very strategically important to us is that when Jalapeno is available next year when our next generation CS5 is

18:41 available together right we will enable a full fast inference portfolio right that is substantially different and better than what's already available today which is already substantially

18:54 different than than your baseline GPU and then on top of that there's opportunity to integrate even further like pre-fill and decode disagregation for example or other

19:06 >> for they didn't actually specialize jalapeno for right >> they did not specialize jalapeno for that but they specialized it for throughput right and so they get a

19:15 tremendous amount of throughput and so just as a computer architect there's like so many different things you can start to do with that right and then we have you know the you know insane uh

19:25 latency right like I mentioned up to 10,000 TPS in in CS5, you can start to imagine some some really really cool products that we can build together, right? And that that's what really

19:37 excites me about having them as a partner. And we're also excited about collaborations on the AI tooling front because much of the the benefits that you know that that they're seeing from

19:49 the AI tooling infrastructure um everybody probably says this now but like we're obviously doing you know a lot with AI but we're also collaborating very closely with OpenAI right to use

20:01 their tools to help us also continue to push what's possible in our chip design and our software and all that. So both of those together I feel like you know this is a really unbeatable combination.

20:11 >> I think one thing that people are talking about like you know I'm trying to get to the disagreements of the hot takes now. So uh people focusing on you haven't really mentioned like power and

20:22 I I do think that something that seems to be a consistent theme is um you know performance per watt rather than than tokens per second. any variation of of this theme or what are people sort of

20:34 talking about offstage that you know is more contentious? Well, I think there's a there's a few things that, you know, in terms of, you know, some of the uh the more contentious things like I I

20:44 would say one of the the themes that came out quite a bit in my discussions at the conference was around the the Grock announcement, right? You know, Grock, obviously, not surprising they

20:56 have a new chip. In general, I think it's it's awesome that SRAMM designs are becoming, you know, more mainstream now. Um, you've >> been here the whole time. We

21:05 >> we we've been talking about it for a long time. And it's amazing to see, you know, the the industry starting to embrace it, right? Um, >> do they feel different post acquisition?

21:14 I mean, you've been competing with them for >> a while. >> Uh, to first order, no. I think it's really awesome to see that, you know,

21:22 SRAMM architectures are becoming, you know, more accessible. you know, even the biggest of the big guys here, Nvidia is embracing SRAMM design, acknowledging that, you know, that uh the traditional

21:34 GPU designs really can't hit the ultra fast, you know, regimes. A lot of what was being discussed uh offstage was I mean the natural obvious questions is like why did they launch on a 30 billion

21:49 parameter model? Um, and you know, how come when uh Jensen spent so much time at GTC talking about uh attention FFN disagregation, there was no mention of that. And so I think there's, you know,

22:04 I think that's that's pretty telling, right? Um, >> I mean, there's a separate Reuben, you know, strategy. >> Well, so there's there's there's Reuben,

22:11 but you know, the LPX itself >> that that was supposed to be where it was. >> It was supposed to be Reuben LPX together, right? Um, if if you guys

22:21 recall, I mean, the Jensen spent like >> GTC >> GTC like half an hour explaining attention runs here and and you know and and the movies run here and so on. And I

22:33 don't know if this is like a a hot take per per se, but you know, it it it's it's very suspicious that they're um that their product that's in full production, they've only shown

22:46 performance numbers on a non-disaggregated 31 billion parameter model, right? And I think to me what this this shows is that there's definitely some challenges in running on

22:59 a nonwafer scale SRAMM design because there's just not enough memory in each of the chips, right? And I think that's you know that that's what's happening and and and we're seeing the evidence of

23:10 that and in many ways I think it's very much validating kind of the design choice that that that we had right uh if you think about it to run a frontier level model of like let's say a few

23:23 trillion parameters you need thousands and thousands of Grock LPUs just to hold the the weights. >> Yeah. >> Right. And so when you start to think

23:31 about it that way, it's like, well, is it surprising that the only performance numbers that they're showing are, you know, on 30B, right? Um, >> so they're going to gradient this into

23:39 this. >> Well, I I think I think it's I think it's the other way, right? I think what's what's going to end up happening is they're going to end up focusing on

23:48 significantly smaller models. >> All right. Um, you know, if you if you have that limitation in your architecture, then I I think that's that's what ends up happening, right? Um

23:58 whereas in our case you know we're running the world's largest models and in some ways kind of simple because you know one of our chips has you know order 100 times more memory than one of their

24:09 chips right so you have two orders of magnitude difference in scale kind of for free right and so I think that's definitely one of the things that was a topic of discussion again independent of

24:20 of cerebrus um just kind of odd that you know you'd launch a brand new product on on performance numbers of such a small model. But when you peel it back a little bit, it kind of makes sense. I

24:33 mean, I've been living in this space now for a long time. There's a reason why we needed the wafer scale integration to be able to aggregate enough SRAMM to be able to actually make it useful for

24:43 large models. >> Yeah. And look, the market's large, right? You have a different market than them. And you know that you you you clearly are the longest running

24:52 incumbent now in this space. >> No, absolutely. I mean the the the market's large there's a lot of different opportunities for you know different different hardware to to play

25:02 different roles. In fact, in general, uh, you know, at Cerebrus, we we we believe very very strongly in a heterogeneous disagregated, you know, uh, ecosystem, right? Not just

25:16 pre-filled decode disagregation, but, you know, I think we're just at the beginning of what's possible in this space. And with the scale of of of these deployments and of these models, right,

25:27 inference is no longer just like one workload. There's many many kind of subworkloads within it. And you know, you really want to use the right, you know, tool for the for the problem. You

25:37 really want to use the right hardware for the problem. And so, absolutely, I think there's there's a there's a spot for all the different types of architectures out there. And, you know,

25:45 we we've we've chosen to to target, you know, the frontier. Um, >> yeah. Frontier ultra fast. >> Exactly. Exactly. >> Um, I we was going to go into some of

25:54 the other companies that were, you know, top of town this year, >> but it gives me this gives me opportunity. I want to follow up on one thing which is how do you think about

26:02 the classes of workloads right um to me the ultra fast frontier workload is just uh like very clearly like one of the fastest growing segments of the entire infrance market right like the fact that

26:14 like I want it and I cannot pay for enough for it um I it might have been a mistake for openi to offer it even right like but it's good for anyway so uh just in your experience right you said you

26:24 said you're strong believers in a heterogenous um inference solution okay what are the buckets and um how do how do you see it from your talking to your customers?

26:35 >> Well, so I I think there's probably two different views here. Um you know, one is the the product view and then the other is kind of the the the technical computer architect view, right? From the

26:47 product view, it's actually just very simple is that bringing more speed opens up significant opportunities, applications, different use cases, different capabilities, more intelligent

26:59 models, more intelligent agents. And so the further you can push that, the more and more that you can enable. In fact, there's probably all sorts of things that we can't even imagine that you can

27:08 build, you know, when you're even faster than what we are calling ultra fast today, right? Which is why we keep pushing that. And then the other angle really just comes down to capacity,

27:17 right? That's so it's like yes, I want the fastest, but then you need enough to actually be able to to to serve your use case, right? And so it really just is that simple, right? And then finding the

27:28 right architecture and the right mix of architectures, right, to be able to provide that is is really the name of the game. Now, from the kind of computer architect's point of view, in a lot of

27:40 ways, it's it's even a little bit simpler, right? Like I was um having a conversation with somebody actually at hot chips about this and they were asking me well you know if you're doing

27:49 disagregation like doesn't that mean you have to like partition the data center and you have to deploy a certain amount of this kind of hardware and you know different type of uh a certain amount of

27:59 another type of hardware isn't that restrictive and I'm like I mean you know when we design >> modular it's modular and when we design a chip every single day we're deciding

28:08 like am I going to use the silicon real estate for memory or if I'm going to use it for compute or if I'm going to use it for IO. Um, and so from a computer architect standpoint, what I see, right,

28:19 is there's significantly different parts of the workload, right? The simplest ones are, okay, there's prefill, you know, decode, but then you go one level deeper. It's like, well, what's actually

28:31 happening during these phases? Oh, well, there's, you know, there there's loading the actual uh KV cache. There's doing the actual, you know, attention. There's spreading the experts across the various

28:43 different hardwarees and balancing them. All of these things can be now addressed with kind of more tailor made, you know, hardware solutions and and architectures.

28:54 And at small scale, it doesn't really make sense to bring together a bunch of different hardwares just to solve like one problem. But at the scale we're all talking about, the hundreds of megawatts

29:06 to gigawatts, the multi- gigawatt scale, it easily pays off. And then at that point, it's as if you're like thinking about the entire data center as if it's like one computer. It's like one chip

29:16 that you're trying to figure out, okay, well, I want this amount of this capability so I can run, you know, attention fast. I want this amount of capability so I can run um prefill

29:26 really, really efficiently. And being able to piece all these things together is like the computer architect's like dream to have all of these tools in our toolbox, right? So that's how I how I

29:36 think you know where you get the value from this heterogeneous disagregated uh ecosystem. >> How much of that plays into model design, model architecture? So working

29:46 with open AI, you get to hopefully see what's going on on models like there was the whole were thinging models. >> It's not enough. There's also voice diffusion. Uh what what if what if

29:58 there's more interactive models like the thinking machine stuff? >> Absolutely. And I think as as we start to get into more interactive use cases that we enable through ultraast we're

30:10 even seeing a lot of this right so if you're trying to interactively for example do graphic design now you mentioned diffusion now you know image generation or video generation might be

30:20 kind of now inline in your you know creative like inflow uh interactive uh use case for with with the thing and so there's so many of these different use cases that are coming and what I You

30:32 know, you mentioned about code design with opening eye and so on. what what I think is um the most untapped opportunity right now uh frankly for Cerebras but frankly for the entire

30:46 nonvidia uh uh environment uh right and the non-envir uh sorry community is that you know we're all running models that were designed for Nvidia GPUs right and and

31:00 and it's not just Nvidia GPUs but like usually they're designed for like one particular Nvidia via GPU, right? Like, okay, this thing was designed to run on B200, GB200, MBL72,

31:10 right? And you know, and so as a reverse, for example, here we were showing that you're running the model like 14 times faster already, but it was a model that was designed for a

31:20 completely different architecture. And so if you start to then open up the possibility of adjusting that model architecture even slightly, you can get massive gains. And if you start to do

31:32 even more code design, I think, you know, the the uh the opportunities are almost limitless. And that's really what excites me a lot about, you know, about working closely with customers and

31:44 partners like OpenAI. >> Yeah. And we we don't have time for this because we want to move on, but uh the people under are really sleeping on AI codegen for kernels. Uh which makes

31:54 actually which is very good for you guys. >> Yeah. No, absolutely. I'm I'm a huge believer in that for sure. I think before getting into etched and super

32:02 spicy chips, any takes on AMD, Nvidia, what they're doing? Could they be doing anything different? Any hot takes there? >> I think that uh Nvidia is doing exactly what we all expected Nvidia to be doing,

32:15 right? Um like I said, they're no dopes. They know exactly what they're doing and they're continuing to push through and in many ways, it's absolutely the right thing, right? Cuz we need more tokens.

32:26 We need them cheaper. That's a that's a reality. However, you know, what used to be considered fast at like 1,00 or or 200 tokens per second is quickly becoming the new batch mode, right?

32:40 Quickly becoming the new like overnight um is true and and so I think that you know there's a place for that. It's great for prompt processing. It's great for very very parallel workloads. Um

32:54 >> but my evals take 20 hours. I can hit a button and run it in two hours. >> Exactly. Exactly right. But but quickly, you know, being able to do prompt processing

33:03 isn't isn't quite enough, right? That's a big part of the market, but that's not quite enough. And so I think all the the the traditional architectures I feel like are very much all on that

33:15 treadmill, right? In a good way, right? Not not necessarily a bad way. It's in a good way, right? And so I AMD, Tranium, in many ways TPU, like all of these in my mind are all trying to build a better

33:31 Reuben, right? And there's a huge amount of value in that, but I think that there's also a lot of value in trying to push, you know, the boundaries in other vectors as well, which is obviously, you

33:42 know, what we're trying to do here at Crisis. >> Okay. Um, spicy spicy chips. There's this page that you have on your website >> that uh this company Etch just doesn't

33:53 have. No, no numbers, no benchmarks, any any takes. That's that's their one criticism. But that's etched. >> Well, so they have said very little about what they're doing. Um

34:06 uh but what they have said has changed also a lot compared to what they originally said. um >> which they acknowledge >> which they acknowledge and and you know

34:15 the the environment is changing a lot and so that makes sense right they're going to pivot they're going to try to find their their space I think my reaction when I see pictures like this

34:27 is that it's very impressive graphics design but I also don't see them building anything beyond just or trying to build something better than just you know a traditional GPU right you know

34:40 they've made claims about having a distributed storage that is somehow faster because it's distributed, which I don't quite understand. You know, it' be very helpful if they can

34:54 explain those kind of things. Um, I think, you know, I I it's hard to see where the the differentiation is that they're trying to push. I think they had they had a rack or something at Hot

35:06 Ships, I think. And so clearly they're they're building something, but uh it feels like it's a little bit, you know, far off for now. >> Yeah. But at the same time, like $20

35:16 billion now, it's like, you know, not not a pushover. Um I I think maybe the the steel man for this is like what interesting direction would you want to see them pursue, right? that in in the

35:29 best case >> in the best case I would love to see them or others frankly push the boundaries of more innovative solutions outside of just the chip right

35:44 you know we I I really believe that the next kind of frontier of of taking AI compute to the next level is all about better ways to integrate outside the chip right

35:57 >> the rack node, >> the rack, the node, the package, the >> the the interconnect, >> the factory, >> the the the factory. Um, they may be

36:08 doing something like that. It's hard to tell, right? Um, but I think that this is not just a a point about edged. I think for the entire industry, right? Um, even if I I take my cerebrous hat

36:19 off for a second, right? What excites me the most is our ideas and and creative thinking outside of the single chip. We all know that the applications, the problems, the models are now so large

36:33 that you can't just do anything on a single chip. So everything comes down to the integration. Everything comes down to how do you bring together more compute? How do you bring together more

36:43 memory? How do you bring together more interconnect, right? And so, you know, >> scheduling, pipelining, all those algorithms, >> all of those things, right? And and so,

36:51 you know, of the of the, you know, the reveals yesterday, I think probably one of the the the the more interesting ones to me, I mean, not not it wasn't new news, but was was Dmatrix, right? the

37:05 fact that they're really leaning into uh into into DRAM, you know, 3D DRAM packaging and like this is the kind of stuff that I think we as an industry need to be doing much much more of.

37:18 >> I was going to say like your backpack stuff reminds me of what the memory people are doing. Is that analogy that >> there there's a there's a little bit of of that, right? Because you know what

37:27 we're doing with our backpack and and power delivery is very much you know a a very compact 3D package right where we can bring power directly to to the face of our our chip of our wafer right and

37:42 in in many ways like you know the the DRAM uh 3D stacking integration is doing something similar but not with power but now with with memory right um yesterday we also announced that you know But we

37:56 also uh started two years ago our um DRAM stacking program for the very same same reason, right? And cuz I think that figuring out ways to to do this kind of integration is the key to to unlock the

38:10 next major step forward, right? And so I don't know exactly what Grock is doing or sorry what uh Etched is doing underneath the covers but you know when you ask like what is the stuff that

38:21 excites me the most about what we're seeing in the industry it's really more creative you know integration more creative packaging more creative outside of the chip thinking.

38:31 >> Yeah. Uh let's leave some hints for people other than Datrix. Uh a couple other names that stood out to you just >> along these lines. I I think you know Samsung has some discussions around

38:41 their ZHBM right um >> all memory guys >> all I mean look in the end there's only three main things to to building a you

38:51 know a an AI chip right there's the compute there's the interconnect >> and then there's memory >> right and so um and and more and more importantly as we all know for at least

39:04 in the low latency space the memory is the key right the memory bandwidth is the key >> and power and cooling not as major still.

39:12 >> No, absolutely. I mean, you just did a big redesign. >> Those are those are what's powering all of this, right? >> Yeah. I'm just saying, you know, like

39:19 physical limits, >> but that's actually a really really good point, right? I think that um when we when we originally started with wafer scale, most people

39:29 thought the biggest challenges were, okay, how do you connect all of these dye together? How are you going to yield this thing? And those were absolutely kind of fundamental challenges that we

39:38 had to solve. But what turned out to be, you know, the biggest enablers was ultimately how do you power it? How do you cool it, right? How do you build it reliably at scale? And I think that a

39:54 lot of the the 3D DRAM um technologies are going to go through the same thing. And it's, you know, one thing to draw, you know, a PowerPoint slide with a DRAM and and uh and logic and then it's

40:07 another to actually make it work to figure out how you actually going to power it, how you actually cool it. And that's why a lot of what we've been doing is, you know, is trying to

40:16 leverage and use all of the expertise that we've built up there and put it into our DM stacking uh approach. And so, you know, we already have solved yield at scale, for example. But we've

40:28 already solved how you can actually package in a three-dimensional way and and that's you know problems that Samsung that Dmatrix and and everybody else are also going to have to solve

40:39 over this time. Thank you for those comments. Uh last topic because we got to go. Uh US supply chain semiconductor supply chain stuff you know like uh China is slowly becoming completely

40:48 independent of us. vision GLM like really literally just like they're they're bragging >> Huawei and what what have you >> hyped outside of knowing what it was on

41:02 >> just model is good doing good in you know open router came out not even running on US chips >> so the TDR is are people talking about it uh what's what what we doing

41:11 >> I mean I think that absolutely I mean of course people are talking about it right because like the the the we're in a very uh difficult you know situation right now the open-

41:22 source model market is 100% Chinese right um >> not 100% but >> almost okay 95% right most of the most of the big models most of the big open

41:34 models that are you know qu high quality are coming from the Chinese labs in some ways it's great that sharing is happening and and you know the the global community is benefiting but al

41:47 obviously that's a very strategically challenging place to be to have such dependence on on them on the model. separately as is evidenced by this is like you know we have independently the

42:00 all of the infrastructure the hardware infrastructure slowly being built up in the background to support all these Chinese models right I think it's absolutely a a problem that uh globally

42:13 as well as as you know the the the US and as a as a national interest that we have to continue to push the boundaries so that you know we and continue to to compete and and continue to be ahead.

42:26 Um, it's not easy. These guys know what they're doing. Uh, but it's something that I believe is not solved by, you know, one company or or one fab, but it's needs government level natural

42:40 national interest level kind of initiative to be able to make this happen. Um, >> so I I imagine that should take place as at Hot Ships. It's like the secret room

42:48 of all you guys. You're leaders of our industry, >> right? that you would be the guys to >> I mean we we have absolutely been been pushing for this and we've been very

42:56 supportive of of you know any initiatives that will um you know continue the US dominance in this space 100%. >> Okay. Uh you got to go but you've been

43:05 very generous with your time. Congrats on all your success. I mean enormous you IPOed. >> Yeah. Yeah. Yeah. Sometimes I still have to pinch myself to remind me that we

43:15 IPOed and it was like the largest like you know semiconductor IPO in history and it's like >> still early. >> That was not bad. That was not bad.

43:23 >> Uh no congrats uh and uh look forward to meeting up with you in the future year. >> We got to ask you know give us ultra fast. Give us bigger give us more. >> No no I mean give give the people ultra

43:33 fast too. Like >> we but we but we can probably get you in line at at open eye. Oo, >> I hope you didn't bias OpenAI too much to, you know, just keep it internally.

43:42 No, people need it. We need better models. >> It's OpenAI's decisions. We're we're just providing the infrastructure. >> Hey, but give us give us GLM. Give us

43:49 Dec. >> Just give us our own rack and then we'll run it. Uh, okay. Thank you again for coming. Really really appreciate it. Yeah.