Inference Engines explained in 10min..

0:00 Inference engine is just one of those topics that cuts through three different layers of the AI stack and a lot of people give up trying to understand how it works because you need a lot of

0:09 lateral knowledge to understand them across all layers and also over the years we've had many different types of model architectures introduced other than LLMs. So to grasp popular inference

0:19 engines that we see today requires us to dig into these layers in the AI stack as well as the history behind AI. So let's get right into inference engines. The concept of inference engines existed

0:31 well before VLM, Lama.cpp, SG Lang, Tensor RTLM and TGI. All these as you can see started to emerge around 2023, right after Chachi PT was first released and Meta's llama model was leaked to the

0:44 public. And these two events caused a pretty big change in what we needed from inference engines. Before 2023, we had a relatively healthy mix of model architectures compared to today. And

0:55 these models needed support like 2D convolution, pooling, deconvolution, and intake calibrated quantization. But auto reggressive LLMs that we use today have a much different type of work like

1:06 attention, batching, KV cache management, and quantization. And they all affect what we really want to see from the inference engines. Let's take a look at PyTorch as an example. We often

1:15 use PyTorch library to build different tensorbased models. And PyTorch could also be used for inference as well. But as you can imagine, the library that was primarily meant to be used as

1:26 generalpurpose work wasn't really optimized to be good at inferencing auto reggressive LMS. What's worse is that after the scaling law was introduced, models over time got bigger and bigger.

1:37 And one of the earliest grassroots movement to address this in 2023 was llama.cpp. And the idea was to specialize for inference workload while also lowering the memory footprint for

1:49 transformer architectures like llama. Now you might look at this and think, wait, couldn't PyTorch be repurposed to do the same thing that llama.cbp is doing? When it comes to quantization,

1:59 methods like GPTQ and AWQ definitely lowered the memory footprint and they could then be used in PyTorch directly. So quantization definitely did coexist with llama.cpp, but you still needed to

2:11 carve out the inference part from PyTorch and optimize them specifically for our workload, which was auto reggressive LLMs. And llama.cpp CPP took a step into this while having the least

2:22 amount of dependencies like PyTorch and early results of this allowed models like Llama's 7B model to run on smartphones. Fast forward to today, this still remains as a big reason why people

2:34 prefer to use llama.cpp because llama.cpp CPP takes a blended approach in how it uses the memory hierarchy in our computers from GPU to CPU and RAM and SSD rather than assuming that our

2:47 model will fit and run in the GPU. And as part of that effort from Georgie Gurgenov went towards quantizing the model as well specifically for that inference engine. Another thing worth

2:57 mentioning about llama.cbp CBP is how it used memory mapping pretty early in its origin, only a few days after it was released as you can see. And memory mapping pretty much targets how this

3:07 entire memory hierarchy gets used to pass the model's weights around from SSD to RAM and the rest. For example, if you're just using GPU to run your model, your weights are typically loaded from

3:19 the SSD through the memory hierarchy and eventually to the GPU, which means now you essentially have a full copy of the model in the GPU, assuming everything will fit there. But llama CPP can't

3:31 really make that kind of assumption. The entire philosophy of llama.cpp CPP is to leverage the existing hardware, which means instead of loading the model eagerly to the GPU, you can map the

3:41 model into virtual memory and let the operating system bring the required pages into physical memory on demand. This way, you're not having another full copy of the model inside of RAM. Now,

3:53 only 3 months after llama.cpp was released, a new inference engine called VLM was introduced along with this research paper, which is certainly worth reading. But when we look at it from the

4:03 AI stack, VLM was trying to optimize the inference engine quite differently than Llama. CPP and there's a lot of things that we need optimization in our lives. And that's why we have to talk about

4:13 Zapier sponsoring this video. As newer and newer things happen in the AI industry, it's important to stay tuned. And having the right automation can be really helpful. Zapier can help you

4:22 build your own custom integration using Zapier CLI, which I can use here to set up my project directly here. And after registering my project, I can just ask clot code to build me the code that will

4:31 check hackernews website for news. And since Zapier allows polling, it can scan the entire website for new information and trigger it automatically. And this code is directly written and I can test

4:40 directly here and publish the app to Zapier. Which means now on the editor, I can just select the app and see that it's pulling the information. And after adding Gmail to send emails like this, I

4:50 can continue building the workflow more by using an agent in between to further customize the result all from CLI. And now I can have a new scanner. All personal to me. And now you can just

4:58 build more complex workflows or go back to your CLI and add more sources like X, Reddit, and more. Link in the description below. Llama. CPP tried to be unassuming about the stack underneath

5:09 and could run the model on the consumer hardware. But VLM on the other hand actually embraced the layers below and tried to squeeze the most throughput out of them. You can think about it this

5:19 way. When we look at typical GPT architectures like this, it has embedding, attention, positional encoding and feed for networks. Llama CPB focused on quantizing portions of

5:29 the model that had large weight matrices and VLM on the other hand focused on the context window, more specifically how KV cache was being managed in the hardware. And in their research paper, we can see

5:42 just how KV cache can easily take up just as much space as the model's weights. Now, if you can imagine multiple users trying to access the model through the inference engine, you

5:51 can imagine just how big this KV cache might grow into. And KV cache is kind of this nasty part of auto reggressive models because they don't really scale very efficiently, especially once you

6:02 have tens and hundreds of users with different problems. It means that we now have varying levels of output tokens generated for each session. So, we had to manage this growing KV cache

6:13 appropriately. But sadly, existing methods back then were wasting up to 60 to 80% of memory through fragmentation and over reservation as it says on their paper. And VLM famously leveraged paged

6:26 attention which adapted how operating system use virtual memory once again but in the context of KV cache to be stored into smaller fixedsiz blocks. And with this, you could have a more flexible

6:39 system that stores the growing KV cache differently and not waste memory trying to lay them out in a contiguous blocks. And part of their efforts also went towards scheduling varying requests in a

6:50 continuous batch so that as the inference engine finished one request, it could keep working through the new batch coming in, which is what we call continuous batching or in-flight

7:00 batching. Okay, let's zoom back out a little bit. LalacP as we saw optimized the inference engine without relying on the underlying infrastructure. VLM tried to maximize the underlying

7:10 infrastructure for maximum throughput and concurrency for more and more users. And one common pattern that you might see here is that if you group enough of these users, you start to notice a

7:21 pattern here where prompts coming in can have very similar prefixes that can be shared. For example, a lot of their users might start their prompt with the phrase, "You are a helpful assistant in

7:32 their prompt followed by their own custom prompts after." So, if you could just stash this commonly used phrase and start building the next pattern that you might notice with enough samples from

7:43 users, you can eventually build this treelike structure called radics attention where you have a dictionary of commonly used prefix and the less commonly used leaves will eventually get

7:53 evicted as you can see. And the whole idea here is to reuse the KV cache for shared prefix instead of recomputing every single time. And that's the idea that SGLANG brought forward. And they

8:04 reported that depending on the task at hand, it could push the inference throughput up to 6.4 times faster. But obviously there's a huge aspect here because it really depends on the

8:15 workload. Now these inference engines that we just talked about all started to emerge around 2023 to 2024. But there's one vendor that continues to follow the everchanging nature of AI. NVIDIA

8:27 introduced Tensor RT back in 2017, Faster Transformer a few years later, and in 2023, Tensor RTLM tracking close to the changing use cases and the supply of models in the AI stack. Nvidia pretty

8:39 much owns the infrastructure and the chip layer. And just like how other inference engines optimized for different use cases they believed was important, Nvidia leveraged their lower

8:50 layers in the AI stack to make an inference engine that could maximize their stack on top of their existing technology. And their earlier results showed strong numbers in performance as

8:59 well as lowering the total cost of ownership which helped them position Nvidia as not just the hardware provider but also the software stack for inference as well. Now looking at the

9:09 progression of inference engines that we just covered in this video with a broad stroke. What I tried to focus on is really what the problem each inference engine tried to optimize for many years

9:20 after and looking back now and many of them in fact converge to provide really similar features that we need from inference engines today but in their own way. So looking at them at a later time

9:30 frame without any context might be a little bit confusing since a lot of them now converge to provide very similar features. Though they certainly optimize for different use cases. And it's

9:40 important to recognize that since 2023 when all of these inference engines all started the demand profile also changed as well as we look at how agentic applications now require a longer

9:51 duration usage as people rely on agents to work on longer duration tasks. We also have a much more complex model architecture now than before. Looking at models from Deepseek, Kimmy, Meta,

10:03 Miniax, and so on. They all have their own unique architecture and kernel optimization, which puts a different pressure on these inference engines. So, as we witness the AI industry unfolding,

10:13 it's important to understand where the bottlenecks are and how they're being optimized architecturally.