New Local AI Engine Everyone Will Be Using in 2027 ? (FreeToken)

0:00 one graphics card, one desk, and a model with 753 billion parameters running on it. There is no cluster behind it, and no rented node in somebody's data center. One workstation card serving all

0:12 of it. 15 tokens a second in the team's own measurements. Slow enough that you notice, fast enough that you would sit and watch it work. The same software puts a 35 billion model on an 8 GB

0:23 laptop at 39 tokens a second. That paper went up on the archive 5 days ago. The code behind it had gone public 6 days earlier. 11 names are on it. Ion Stoka, Mate, Zaharia, Sanghan, people whose

0:37 earlier work you already use. Stoka co-wrote the paper behind VLLM, the engine underneath a great deal of the inference you pay for. So this is not a weekend project. The crowd who built the

0:48 cloud serving stack just turned around and aimed it at your desk. The engine is called free token. Apache licensed 11 days old in public written by two contributors. What it has to beat is

0:59 Llama.cpp gorgi gurgenoff's engine three and a half years old and what you run today. Free token wins the benchmark and it wins it clearly and the benchmark is not

1:09 the thing it had to beat. That gap is the whole video. Start with the part that sounds impossible because that part is the clever bit. 753 billion parameters do not fit in 96 GB of card

1:21 memory. That is not a near miss. Even quantized down to 4 bits, the weights on disc come to 433 GB. You are off by a factor of 4 and a half before you start. They fit because of how the model is

1:33 built. It is a mixture of experts and the word doing the work in that phrase is mixture. Instead of one enormous feed forward layer, each layer holds hundreds of small ones and a router picks a

1:44 handful per token. Deepseek's V4 flash has 256 of them in each of its 43 layers. I did not take that from a blog post. It is in the model's own configuration file, which anyone can

1:56 open in about 10 seconds. For any single token, six of those 256 fire, six. The other 25 sit there doing nothing. And then the next token arrives and the router picks six again, possibly a

2:09 completely different six. So of 284 billion parameters, roughly 13 billion do the work for any given token. Under 5% of the model is awake at once. That is why a desktop is in this conversation

2:21 at all. But sparse compute is not sparse memory. All 284 billion parameters still have to exist somewhere you can reach in a couple of milliseconds because you cannot know which six the next token

2:32 wants. So they live in your system memory and the card holds whichever ones it can. Every time a token routes to an expert that is not already on the card, that expert has to cross the bus or be

2:43 computed where it sits. That crossing is the entire game. Every design decision in either engine is a different answer to one question. What do you do in the microcond after a token asks for an

2:55 expert you do not have? Llama.cpp has an answer and it has had one for a long time. You can tell it to keep the expert weights on the processor instead of the card and it will run them there rather

3:05 than shipping them back and forth. The flag is minus n minus cpu minus and the documentation for it is one line long which is unusual because that one line is also the entire strategy. Keep the

3:18 mixture of experts weights of the first end layers on the processor. The first end layers that phrasing does all the work here and it is the easiest thing in the sentence to skim past. It is a split

3:29 by layer number and you pick the number before the model has seen a single token. You pick it, the split is fixed and it stays fixed for the entire run, which would be fine if routing were

3:39 fixed. Routing is not fixed. Routing changes token by token and the model decides it at runtime long after you picked your number. A split by layer index cannot see any of that. It is not

3:50 a bad heristic. It is a fast, predictable, sensible heristic. It is simply blind to the one signal that decides everything. Free Tokens paper measures what that blindness costs and

4:01 it measures it the fair way by replaying identical routing traces through all three engines placement policies at the same cache size. Same traces, same memory, different policy at the memory

4:12 of 5090 can spare. Free tokens cache misses 16% of expert reads. The static split misses 62. That is a 46 point gap. Near enough four times the miss rate. And a miss is not a rounding error. A

4:26 miss is the bus every single time. Which raises the question I want you holding for the rest of this. If the fix is that legible, why has llama.cpp not simply done it? A clever mechanism that does

4:37 not show up in the numbers is just a nice paragraph. This one shows up. Then we come back to that question. On a 5090 serving Quen 35B, the paper reports 77 to 83 tokens a second. That is between

4:51 1.8 8 and 2.3 times the strongest engine it tested against depending on the workload. On Deepseek V4 Flash, 22 to 25 tokens a second, 1.5 to 1.9 times. And on the workstation card running GLM,

5:05 5.2, 14.9 tokens a second against llama.cpps 7.3. The one I keep coming back to is the laptop. 8 GB of card memory, a half width bus, 39.3 tokens a second, which

5:19 the paper puts at 92% of what a desktop 4090 manages. That is the machine most people actually own. Now, the caveat, and I would rather say it plainly than bury it in a corner of the frame. Every

5:30 one of those numbers was produced by the people who wrote the engine. That is normal for a systems paper. It is still not independent. To their credit, they did the rigorous version. Every engine

5:40 served bit identical weights which is the step most comparisons skip without saying so and it is the difference between a benchmark and an advertisement. But on the day I wrote

5:49 this, I could not find a single third party benchmark of free token anywhere. And the result I would make a decision on is the tail, not throughput. Not how fast it goes when it goes well, but how

5:59 slow it gets when it goes badly. Free Token's worst single turn stays under 44 seconds. The paper reports llama.cpp CPP crossing 232 seconds somewhere. Alama 179 and K Transformers 946

6:15 almost 16 minutes on one turn. And the papers point about that is sharper than a speed comparison. Open claw ships a 2minute idle watchdog. A turn that takes 232 seconds does not run slowly. It gets

6:29 killed. Tail latency is not a latency statistic there. It is whether the software works. So the case is strong, which is exactly when you should look harder. And there is a crack in how it

6:39 is presented worth seeing because it is the kind of thing that decides whether you trust the rest of the chart. Across its headline figure, the paper draws a dashed line marking what a cloud coding

6:48 agent actually delivers measured from real production traces. 33 tokens a second against bars at 77 and above that reads like a route. So I went and read the trace paper it cites. 33.9 is its

7:01 normalized decode speed. And by that paper's own definition, the figure includes time to first token and all the reasoning in between. It is an endto-end number. The same paper's pure decode

7:12 median for OpenAI's codeex is 61.3. That is the figure that belongs next to a decode throughput bar because that is the same measurement. Put those two side by side and the like forl like ratio is

7:24 about 1.3, not 2.4. Still a win, a much smaller one than the chart implies. Both figures sit in the source correctly cited. So, this is not deception, but a comparison is only as good as its

7:36 denominator. And this one changes the size of the claim by nearly half. None of which would matter much if you could just install it tonight. So, look at what the repository looked like in the

7:45 24 hours after that launch post went up. Eight issues. Please support older hardware 1080 and 2080 series. Support for dual GPU Docker support. GGUF support. The engine install fails on

7:58 Windows. and the one with the most comments on it. Feature request, Mac OS, Apple silicon support. Those eight requests are things llama.cpp already ships 17 hardware backends, Apple's

8:10 metal, AMD, Vulcan, even a Snapdragon phone. Quantization from 1 and a half bits up to 8. Apple silicon described in its own readme as a first class citizen. Free token's own packaging says the same

8:22 thing with no spin at all. Its published classifiers read development status beta environment Nvidia CUDA operating system Pix Linux. That is the supported world written by the people who built it.

8:35 There is no Mac build, not a slow one, not a fallback. The download pages own platform list offers Windows, Abuntu, Arch, and an app image and hands a Mac visitor nothing at all. Two

8:46 contributors, 30 commits of public history, 408 stars on the day I checked against llama.cpp's CPP's 446 contributors and 25,000 downloads of one nightly build in its first day, which

9:00 leaves the argument that will get repeated most, the one in the launch post itself. Run your coding agent against a Frontier model for $0. So, I priced it from a real trace rather than

9:10 a VIP. Researchers released 4,200 real coding agent sessions. The median step reads about 119,000 tokens of context and writes 214. They measured what that step cost them. 11 cents on Anthropics

9:25 Claude or OpenAI's codeex. The same step on DeepS's own published price list for the exact model free token demonstrates costs somewhere between a tenth of a cent and 5 cents depending on caching

9:37 and the hour of the day. The model you would buy a graphics card to run at home is also the cheapest thing on the menu. And the card is not free. One tracked 5090 listing sat above $4,000 all

9:49 through July against a $2,000 list price. $4,000 buys about 500 median agent sessions at Claude prices and somewhere between a,000 and 40,000 at Deep Seeks prices. The invoice was never

10:01 the argument, which means the argument has to be something else. And it is. Nothing you type leaves the room. Nothing rate limits you at the worst possible moment. Your model does not get

10:11 deprecated out from under you 6 months into a project built on it. None of that shows up on an invoice and all of it is real. So, which one runs local AI better? Llama.cpp

10:22 for most people watching this and it is not close because the first thing an engine has to do is run on the hardware you already own. And that is the one benchmark nobody plots. But free token

10:33 wins outright for one specific person. And if that is you, it is not a marginal win. You own a recent Nvidia card. You have plenty of system memory. You run mixture of experts models. and you point

10:44 a coding agent at them all day. For that person, 44 seconds against 232 is not a percentage on a chart. It is the difference between an agent that finishes the task and one that gets

10:55 killed halfway through. And I would install it today at twice the setup pain for that alone. The title says everyone will be using this in 2027. I do not think that is how this ends. So here is

11:06 the version I will be judged on. By the end of 2027, llama.cpp CPP will have merged a routing aware expert cache and a measured split between the bus and the processor. Free Tokens ideas running

11:18 inside the engine you already have. I lose that bet if free tooken ships Apple silicon and passes 10,000 stars first. Their own paper names the precedent. Fiddler proposed treating a missed

11:29 expert as work rather than data. K Transformers made that fast. Neither is a household name and both of those ideas are table stakes today. Which is the part I keep turning over. If the winning

11:39 move in open source is always to fold the good new idea back into the old engine, what is left for anyone who writes a new one?