Full Analysis: OpenAI Jalapeño chip..

0:00 If I give you 13 months to design a chip to be sent to TSMC in Taiwan to manufacture, do you think you can design one that competes with Nvidia? OpenAI started working on their custom chip

0:10 called Jalapeno back in October 2024. And just 13 months after they sent their first design to TSMC, and we finally have a first look at OpenAI's first ever custom chip. Now, compared to Nvidia's

0:22 chip from March 2025, Jalapeno is showing really strong numbers in preliminary benchmarks. As you can see, jalapeno chips are getting nearly nine times the throughput, serving Kim K 2.5

0:33 at 100 tokens per second compared to the second best. And you can also see right here how the chip has a higher throughput top speed at 700 tokens per second, which is significantly higher

0:44 throughput. We can say the same thing for running GPD OSS 12B model where the chip tops out at an incredible 1,500 tokens per second, nearly two times the next best. So the question here is this,

0:54 how exactly did OpenAI pull this off? I mean, how does a lab that's supposed to innovate AI models go from developing GPT models and agent harnesses and now suddenly start making custom chips in 13

1:06 months that outperforms Nvidia Blackwell chips. What you're seeing here is two different approaches by two different companies in the chip layer. Nvidia of course makes generalpurpose chips,

1:17 meaning we can adapt the generalpurpose chip for specific tasks like training or inference among many other tasks. OpenAI however is taking an application specific integrated circuit or ASIC

1:29 which unlike a generalpurpose chip. AS6 are often optimized to be good at very few specific tasks. And OpenAI isn't alone in this. Many other companies are also producing their own chips. But

1:40 Jalapeno stands out among the rest given in such short amount of time they went from design to production that competes with Nvidia chips on inference jobs. Now, OpenAI, as we all know, recently

1:52 surpassed more than 1 billion monthly active users, which means supporting 13% of the world's population has a huge downward pressure and securing enough energy to power the data centers start

2:05 to become a huge bottleneck. To solve this downward pressure in the energy layer, what if at the chip layer, OpenAI just made a custom chip that is power efficient so that we can do more with

2:15 less energy? And that's exactly what we see when we look at Jalapeno's power profile that draws up significantly less power at 700 watts, which is similar to what H100 and H200 chips draw. But even

2:28 given the smaller power profile, it's able to hold nearly three times the memory as H100, and the bandwidth is nearly five times as much as H100. Even compared to Blackwell chips, the chip

2:39 consumes two times less power while having two times the bandwidth with similar memory capacity powered by the HPM4 memory. Now looking at this table that we just went through and also

2:49 looking at these benchmarks, I think it's easy to look at this and jump to making statements like this. Well, this is game over, right? Jalapeno not only offers incredible performance and lower

3:00 power consumption, the preliminary benchmark certainly hints that Nvidia is no longer the king of the jungle. But when you look a little bit closer, there are few critical things to consider

3:09 here. Quick disclaimer, this video does get a little bit more technical than usual. But I'm literally so excited to cover this topic. So, back to the question. Why doesn't Jalapeno mean that

3:19 it's game over for Nvidia? First, when we look at the actual compute that these chips can produce at FBA precision, while Jalapeno is certainly higher than the good old H100 and H200 and sits

3:30 around 3.4 4 pedlops. As you can see, Jalapeno doesn't offer as much compute in comparison to Blackwell chips as you can see and certainly far below the Reuben chips. Going back to the

3:40 bifurcation between general purpose chips and AS6, OpenAI's focus here was to make a chip that is geared towards inference work specifically that's also power efficient. NVIDIA chips are

3:51 supposed to be good at those things as well among many other things given that they're general purpose chips. As the model goes through training and eventually they're used by people for

4:00 inference. While you need an accelerator during these phases, the demand profile for each will look a little bit different at each phase. For example, training as you can see benefits from a

4:10 lot of scaling many GPUs together while having ample compute and memory capacity. But inference looks completely different in both prefill stage and the decode stage. One way to look at it is

4:22 this. The bandwidth column here points directly to the decoding stage that's bandwidthbound and the compute and memory capacity columns points directly at the prefill stage as you can see. So

4:33 being able to add a new row and introducing a chip called Jalapeno to optimize for inference work while also keeping the power consumption low is exactly the approach that OpenAI is

4:43 taking. Now, one thing to keep in mind here is that there is an inherent risk to making custom chips like Jalapeno because it has to make a bolder assumption than general purpose chips on

4:52 what the layers above might need. For example, if researchers in the model layer suddenly came up with a completely new model architecture or maybe how we use agent harness in the application

5:03 layer changes completely, it directly affects the layers below that make these things happen higher up the stack. So while currently the benchmarks certainly reflect a strong number in comparison,

5:13 this chart is really showing us one size of the use case in the application layer. When you look a little bit closer, what you'll find is that the benchmark says the phrase 8K1K. This is

5:23 one of many benchmarks that's typically run in an open- source test called inference X. And out of many other tests that exist, OpenAI only released the 8K1K version, which simulates 8,000

5:35 input tokens processing and 1,000 output token generation. That's not to discount how impressive it is, even by the sheer fact that we even have a chip that performed the way it did on an AK-1

5:46 benchmark. But what we really haven't seen from OpenAI so far is how it performs on tasks like Agent X. Agent X is supposed to measure how the chip actually handles in agentic use cases in

5:59 the application layer where there are many back and forth tool calls that are happening typically with long context usage and long running sessions. This would really test the chip to its limits

6:08 on how it might stand on real life situations. And yet, OpenAI hasn't revealed any of those numbers yet, just the 8K1K for now. My assumption here is that OpenAI might not have really

6:19 figured out how to fully optimize their chip and configuration for certain use cases other than 8K1K. So, while the preliminary numbers really does look good for Jalapeno, it seems to me that

6:31 the chip is perhaps not proven to be as stable with different types of inference work. Now, I hope these things don't come across as if I'm underwhelmed by Jalapeno chip from OpenAI. I'm more than

6:41 shocked by the fact that OpenAI was able to design the chip with Broadcom in just 13 months and work through the already constrained semiconductor supply chain to produce a chip right off the bat that

6:53 competes with Blackwell chips. If you look back to only 7 months ago, OpenAI made headline by securing nearly 40% of the global DRM capacity, which further led to DRM prices going up. And

7:04 thankfully Uber seems to have not wasted their reservation of supply chain and actually using it towards making something that performs really well in the jalapeno chip. Let's look at the

7:13 technical detail a little bit closer. Starting with the scale up and scale out strategy. And if you're not familiar with what that is, scale up is how fast the chips inside the rack can actually

7:23 talk with each other. And scale out is connecting many racks together into a much larger cluster. Now you might be wondering why this is even important at all. models like Deeptick R1 and Kim K

7:34 2.5 can't really fit into a single chip and because of that we have to solve this by scaling up and scaling out at the infrastructure layer by stacking many chips together. Now different

7:45 manufacturers will have different ways to solve the scale up and scale out at the infrastructure layer to get the maximum value out of their own custom chips. Nvidia scales up with NVL72 in a

7:56 single rack by grouping two black wall GPUs with one gray CPU through NVLink C2C. And this rack now containing 72 GPUs are stitched together through NVLink and NV switch, which is how this

8:08 giant rack really moves like one single large machine. This is Nvidia's scale up strategy offering 900 GB per second per GPU. For OpenAI scale up strategy, they're taking a little bit of a

8:19 different approach here where they're putting eight jalapeno chips on a single Vindeloo tray with 16 trays total per rack. That makes it a total of 128 chips rack scale up strategy. And unlike

8:31 having the CPU and GPU in the same module like we saw in Nvidia, OpenAI separates them into Katsu tray where China switch actually ties them together. An opens scaleup strategy in

8:41 this case provides 600 gigabytes per second per GPU which is about 66% slower than Nvidia's NVL72 throughput per GPU in their scale up strategy. Now as far as how the scale out strategy goes,

8:53 Nvidia typically leaves NVL72 rack through infiniband where multiple racks are connected through external switches. Jalapeno is a little bit different by using what they call global scale up

9:05 that connects up to 16 racks at 200 GB per second. This kind of massive scale up resembles to me a lot like Google's TPU where you have a stackable cube in a 3D topology instead. Now, as you can

9:17 see, all these companies propose their own ways to scale. But Nvidia was able to keep their hold on the market due to many factors and one of which is their CUDA kernels. While general purpose

9:27 chips have to support wide variety of use cases, OpenAI can support more specific operations that the models actually might need during inference. And the common narrative is that

9:37 OpenAI's kernel will only work with GPT models. But the fact that the benchmarks here from OpenAI shows them running GPT OSS 12B, Deepseek R1, and Kim K 2.5, which are all different models from

9:49 different labs and varying levels of mixture of experts, attention mechanism, and model architecture suggests that maybe we shouldn't think of Jalapeno as OpenAI's custom chip only for GPT

10:00 models, but could be a much wider use cases like even running Doom on Jalapeno. And I also imagine that LLMs are playing a huge part in helping OpenAI specifically around accelerating

10:11 their support in various kernels. In fact, OpenAI mentions that their latest models are accelerating how they optimize and program the development of Jalapeno. All in all, I'm still trying

10:21 to process all of this. I can't even imagine the amount of effort it took OpenAI to secure DRAM capacity from companies like Samsung and SKH Highix that eventually went to their HBM4 that

10:32 yielded 15.4 4 terabytes per second in bandwidth and also having to work with Celestica to get trays and racks that went up to their scale up strategy and also having to work with security

10:42 optical components and switches that make their scale out happen and having to work with data centers to provide best cooling and networking for procuring deployment and having to work

10:52 with Broadcom to actually design the chips. And while all of that is happening, the first prototype of Jalapeno actually outperforming Nvidia Blackwell chips on certain benchmarks is

11:03 nothing short of crazy. And OpenAI is already working on Gen 2 and Gen 3 of Jalapeno. So it's not just new models that are flooding in the model layer or new agentic applications that are

11:14 flooding in the application layer. We're now seeing high-erforming chips and infrastructures layers moving just as