NVIDIA's New Local AI Engine Runs Without Python

0:00 I expected Nvidia's new Flare server to sit on the usual Python stack. Its design doc says Rust never calls Python to handle a request. If you've got a new RTX card or you're about to buy one,

0:11 that sounds like an engine built for your desk. Then I read the model card behind its Deep Seek recipe and it wants workstation cards with 96 GB of memory each. Python still does the heavy

0:22 lifting, but only once at build time. After that, a small Rust program replays the frozen GPU work for every request. What pulled me in next was the date. Flare went public the same day Deep Seek

0:34 released Deep Infer, an engine whose readme lists only Nvidia data center GPUs and Wa Ways Ascend chips. Put the two launches side by side and it's tempting to read Flare as Nvidia's

0:44 same-day answer, an Ollama-style local server for the cards Deep Infer skips. I nearly read it that way because a lot of it checks out. Flare is a local server that speaks OpenAI's and Anthropic's

0:56 request formats and on paper the RTX 50 cards qualify. But that reading cracks in three places. One is the Deep Seek hardware you just heard about. The only Windows machine the docs name has an arm

1:07 chip and nothing I read ties Flare to Deep Seek. Someone from Nvidia on its developer forum even calls Flare just a research prototype. So keep a question handy. Which of Flare's ready-made

1:18 recipes fits a card you can actually buy? By the end you'll know how it runs without Python, what that costs you and whether your card's in. Start with which cards it'll even load

1:27 on. Flare runs on the SM 120 family and a close sibling built for the Spark chips. That SM label is Nvidia's compute capability, the version number it gives each chip generation. Nvidia's own table

1:39 puts the GeForce RTX 50 cards and the RTX Pro Blackwell cards in the first family and the DGX Spark and the arm RTX Spark N1X in the sibling. The RTX 4090 sits in an older row of that same table,

1:53 a full generation back. Your 4090 doesn't fail here on memory or speed. It fails on a version number. One caveat on the good news, Flare's docs never name a GeForce RTX 50 card. They say SM120 and

2:07 the RTX 50 match comes from Nvidia's table, not from any test in the sources. Ollama, for contrast, supports Nvidia cards all the way back to compute capability 5, RTX 40 series included.

2:19 Flare's readme is upfront about that trade. It gives up the breadth of servers like VLLM to focus on a few models and local workloads. So, how do you serve a model without Python? Flare

2:31 splits the job between two programs. The first is ignition, a Python process built on PyTorch, and it does every slow, fiddly step ahead of time. The second is Helios, written in Rust, and

2:42 it serves whatever ignition left behind. Ignition finishes and exits before Helios starts. Think of it as a recording session and a playback deck. Ignition records the GPU's work. Helios

2:53 presses play. The recording session runs in order. Ignition downloads the model's weights if they're missing and loads them. It tunes the kernels, the small programs

3:01 the GPU actually executes. It compiles the model's path from your input to the next token. Then it warms everything up and captures CUDA graphs, which are recorded chains of GPU work

3:12 that can be launched again as a single unit. Ignition records one take per batch size, the number of requests handled together on a ladder that starts at a single request and doubles each

3:22 rung. So, there's a take ready whether a single request is waiting or several are. Last, ignition writes everything into a package. The weights, the recorded graphs, and the tuned kernels

3:33 all become blobs named by a hash, web front end included. So, the whole thing can be copied to another machine as it is. Then it exits, and that's the last Python you'll see. There's a catch in

3:44 what this step needs, though, a GPU and the full CUDA toolkit, including NVCC, Nvidia's compiler, and Cupti, its profiling library. Hold on to that. It comes back when we reach Windows.

3:56 Playback is a different program. You type flare run and a separate rust process starts and loads the package. Rust handles everything a request touches after that from the network

4:06 connection to turning your text into tokens and choosing each next one. The design doc is blunt. Rust never calls python to handle a request, run a model step or sample a token. From the

4:16 outside it looks familiar with an open AI style chat endpoint and an anthropic style messages endpoint. So tools you already use can point at your own machine.

4:26 The line that sold me on this design is about steady state. Once Helios is serving, it doesn't allocate GPU memory, compile kernels or capture graphs. Every model step replays a graph ignition

4:36 already recorded. So the slow unpredictable setup all happened before your first request arrived. The picture breaks in a useful place. A recording plays the same song every time and your

4:46 prompts never repeat. What's frozen is the order of the GPUs work and the memory set aside for it. Your words still flow through fresh. So is it a llama style? Partly.

4:57 It's a local server your tools can talk to, but getting a model into flare is a build step, not a quick download. Ignition fetches the weights, then tunes, compiles and records before you

5:06 can serve anything and the package has to match an SM 120 or 121 card. You pay that cost per model upfront, then serve it as often as you like. What limits it? Memory reserved before

5:19 any work starts. Each conversation keeps a cache of what the model has already read, so it doesn't redo that work for every new token. The scheduler takes requests first come,

5:29 first served, but admits one only if it can reserve the cache pages that request will need. Then it alternates. A decode batch writes the next token for every active answer.

5:39 Between those batches it allows at most one bounded chunk of prefill, the step that reads a new prompt. Say your coding agent is in the middle of a long answer and you paste a big file into another

5:48 chat. The file gets read in slices and the agent keeps writing between them. How big is that cash? For the reference Quinn design, the doc puts it at 24 kibibytes for every token

6:00 it keeps. That's with the cash stored in a 16-bit number format. Switch to an 8-bit format and it halves. Say a conversation runs to 100,000 tokens. Multiply it out and the cash comes to a

6:11 bit over 2 GB on top of the model itself. That's exactly the work that readme pitches Flare at. Long conversations, coding agents, and repeated tool use.

6:21 Those are the users who never close the tab. Does all this make Flare faster than Ollama or llama.cpp? The sources don't say. Nothing in them runs Flare and those tools on the same

6:32 card, so speed stays an open question. What the design does promise is fewer surprises mid-answer. Nothing gets compiled and no memory gets grabbed while you're waiting on a reply.

6:42 If a colleague asks how it works, the short version goes like this. A Python build step tunes the GPU code, records the model's work as CUDA graphs at a few batch sizes, and packs it all up. A Rust

6:54 server loads that package, reserves memory before admitting each request, and replays those recordings for every step of every answer. Python never runs while you're chatting.

7:03 That frozen design has a sharp edge and one of the repo's first bug reports found it. A user called Justin T. Nola had Flare on RTX Pro 5000 Blackwell workstation cards running Ubuntu. Their

7:15 Quinn 3.8 package exported cleanly. Then Flare run crashed in its prepare step, where the recorded graphs get rebuilt on the GPU before it served anything. 40 minutes to build. A tenth of a second

7:27 into its prepare step, it fell over. The same day, another user, D. Botwinick, traced it. The recorded steps asked for about 56 kibibytes of the GPU's fast shared memory. Call it a sixth more than

7:39 the limit Flare had set when it rebuilt them. They posted a patch and tested it, but the thread doesn't show it merged, so don't count it as fixed. The lesson for you is that frozen work has to fit

7:49 the limits it meets at replay, even on the machine that built it, and you find out at startup. The happier reports come from Nvidia's developer forum, mostly from DGX Spark owners. A user called

8:00 Cosinus got the 27 billion Quoin 3.8 model running on a single Spark, but first had to swap the Linux distro's rust for rustup. Flare needs the 2024 edition of rust and

8:11 the package compiler was too old. A brand new AI engine and the first wall it hit was the compiler. Another user, alex.martinez.c, ran a Quoin 3.8 flash next model on a

8:23 Spark under Kubernetes and drove coding tasks with it from the Think Qube IDE. People in that thread post single conversation speeds of up to about 50 tokens a second on a single Spark. Those

8:34 are their own figures, not checked by anyone else. Then someone from Nvidia, posting as Yuger, stepped in with a reminder. It's just a research prototype, so adjust your expectations

8:44 accordingly. Which brings me to Windows, the part I most wanted to hold up. Flare builds for Windows 11 on both X64 and ARM. It's Windows runtime talks to the Nvidia driver directly through

8:55 nvcuda.dll. So Python, the CUDA toolkit, and WSL aren't runtime dependencies. If you'd heard it needs just the driver, add Microsoft Visual C++ redistributable to

9:07 that. And the only Windows machine the doc names is an ARM RTX Spark N1X serving 27 billion Quoin 3.8. No X64 gaming card shows up and serving across several machines isn't supported on

9:20 Windows. Then the preparation section changed what no Python meant to me. Packages are usually exported on another machine and copied over. Only the 27 billion Quoin

9:30 3.8 recipe can be prepared on Windows at all with compilation switched off. Even that still needs Python, PyTorch, and the CUDA toolkit. So, the Python moved. It runs wherever the package gets built,

9:41 usually another machine. The server that answers you never touches it. Back to the question from the start, which recipe fits a card you can buy? Two are out straight away. Kimika 3 wants 16

9:53 GPUs. And the DeepSeek V4.1 flash recipe, the one that made this look like a desktop story, is the model card I opened with. On top of those workstation cards, it asked for at least 256

10:05 kibibytes of system memory, a big slice of disk, and Linux. Put the cards together, and by my count, that's 12 RTX 5090s worth of graphics memory. It's a desktop recipe if your desk lives in a

10:17 server room. The single card recipes are where it gets useful. Nematron 3.5 bolt is a mixture of experts model, where each token wakes only a small slice of the network. Nvidia squeezed its expert

10:29 weights down to about two bits each. That's how the whole model fits in 24 gigabytes, and its model card lists the GeForce RTX 5090 as a target. That leaves a 5090 some room for the cash we

10:40 talked about. It's the recipe that clearly fits a gaming card. 27 billion Quoin 3.8 is the other single card recipe, and the Windows one. None of the sources says how much memory its package

10:51 takes, so whether it fits a 16 gigabyte RTX 5070 Ti or 5080 stays open. Which leaves the question the timing raised, is Flare Nvidia's answer to Deep Infer? The evidence doesn't back it. DeepSeek's

11:05 post frames Deep Infer as its production stack, shared to lower compute costs. It serves only DeepSeek V4.1 flash, and DeepSeek measured its decode speed on 64 data center GPUs. The Next Web reports

11:17 it was first built mainly on VLLM, the kind of broad server Flare's readme says it trades away. Nothing in either repo or in that forum thread links the two. Treat the shared release date as a

11:29 coincidence of timing. My call by who you are. If you've got an RTX 5090 and run Linux, flares worth an evening with the Nematron Bolt recipe. None of these sources shows it running on a 5090 yet,

11:41 so you'd be the test. Treat it like the prototype it says it is. If you're on a DGX Spark, the working reports come from people in your seat, so you're the best place tester. On Windows with an X64

11:52 gaming card, I'd wait until someone shows it running. On an RTX 40 card, stay with Ollama, which supports it. And the rule before any download, check that your card is SM 120 or 121, then check

12:05 the package's memory against your cards. A published Windows run on a GeForce card or a same card speed test against Ollama would change my call. I came in expecting a Python stack and a desktop

12:16 deep seek. I found a build machine and a server room. What I'm leaving with is smaller and more useful. An engine that does all its Python at build time and none of it while serving.

12:26 So, if you've built a flare package already, what broke first for you? The compiler, the graph, or something new?