Your GPU Is Capped for Local AI (NVIDIA Did It on Purpose?)

0:00 If there's an RTX 3090 under your desk, the post going around says Nvidia sold you that card with half its AI speed switched off. A free project on GitHub, it says, switches it back on and takes

0:11 one kind of math from 82 teraflops to 155. What's odd is that the half-speed figure was never a secret. It sits in Nvidia's own spec table for the chip in the white

0:22 paper that describes it. An Nvidia forum moderator pointed people to it back in 2021, years before the viral posts. So, say you're about to load an unofficial kernel module onto a work

0:32 machine or pick a used 3090 over another card because of it. Before you do, it's worth knowing whether the gap is real and whether it was put there on purpose. Those posts are Alexey Fetisov's from

0:43 his own 3090s on October 21st and 22nd, and on the raw tensor math he measured, the jump really is close to 90%. But the same post holds a second number the slogan leaves out, because a real

0:55 matrix multiply through Nvidia's own math library rose only 55%. In one owner's GitHub report, a chatbot running a quantized model, one whose weights are squeezed into fewer bits, wrote its

1:07 answers 0% faster. If the switch doubles the math, why doesn't the chatbot move at all? By the end, you'll know what Nvidia's own numbers already said and how much of the

1:17 on-purpose part anyone can show. You'll also know what the project's benchmarks do and don't prove, and how to tell whether your own work would notice before you put an unofficial

1:26 module on your machine. Start with the strongest form of the accusation, which is that a card you paid for has speed built in that it won't let you use. You'd expect proof of something like

1:35 that to come from a leak or a chip teardown. The trail actually runs through public forums, Nvidia's own among them, which is a little deflating for a conspiracy.

1:44 The complaint is older than the 3090 itself. In December 2018, a user on the fast.ai forums pointed out that GeForce RTX cards other than the Titan ran one kind of mixed-precision math at half

1:57 speed. That was the kind fed with 16-bit numbers that keeps its running totals in 32 bits. In 2021, someone on Nvidia's developer forum asked what the 3090's real peak

2:08 was for its tensor cores, the part of the chip built to multiply grids of numbers. Moderator Robert Crovella answered from table two of Nvidia's white paper for

2:17 the GA102, the chip inside the card. With 16-bit inputs and 32-bit running totals, the card tops out around 70 teraflops, and keeping the totals in 16 bits goes faster. A teraflop is a

2:29 trillion of the single multiplies and adds inside those grids every second, and the moderator's figure sat in a public answer anyone could read. As cover-ups go, this one came with a

2:39 public thread and a table number. Sharper evidence turned up in 2023 when a user on the same forum worked the white paper's numbers per clock. In a tensor format called TF32, the 3090 does

2:51 64 operations per tensor core per clock across 328 tensor cores. Run that at 1,695 MHz, and it comes to 35.6 teraflops, which matches Nvidia's spec. By that

3:04 same formula, the A40, a professional card built on the same GA102 chip, should come out at 37.4. The white paper lists 74.8 instead. Divide one by the other, and you get

3:15 exactly two. So, the professional card's tensor cores each do twice the work per clock. Nvidia's moderator confirmed the numbers, said the tensor units in the two cards don't necessarily work the

3:26 same way, and called the detailed differences unpublished. The closest Nvidia side has come to explaining why. Line the 3090 up against the RTX A6000, a pro card built on the same chip, and

3:38 the gap shows up in 16-bit tensor math. The pro card reads about 155 teraflops against the 3090's 71. Those figures come from a spec aggregator working from Nvidia's data

3:49 sheets. Hold on to one more line from that comparison because it comes back later. Memory on the A6000 moves 768 GB a second, and the 3090's moves 936. On that column, the GeForce card wins.

4:04 So, the half-rate the viral post described is real, and Nvidia's own tables have had it documented for years. A spec sheet can't settle the two questions left over though, which are

4:14 why it's there and whether your work ever runs into it. A download can't add tensor cores to a card, so whatever this project changes has to be about how the existing ones get used. Zaguita, the

4:24 developer who keeps the project, describes it in one line on its GitHub page. In the project's words, it unlocks the issue rate of FP and BF16 instructions, the two 16-bit formats, on

4:36 consumer RTX 30 cards to match their professional equivalent. Issue rate means how many of those instructions the card can start each clock, which lines up with the per clock gap the forum math

4:46 turned up. Inside one of those instructions, a tensor core multiplies two small grids of numbers and adds the products into a running total. The inputs arrive as 16-bit numbers in one

4:57 of those two formats, which are cheap to store and move. That running total, the accumulator, is often kept in 32-bits for precision. 16-bit inputs feeding a 32-bit accumulator make up the path the

5:09 moderator put at about 70 teraflops on the 3090. The 2018 complaint named those same 32-bit totals, and the forum math put TF32, another format that keeps 32-bit totals at half the professional

5:22 card's rate per clock. Put together, the numbers point to a ceiling on tensor math that keeps its running totals in 32-bits, while the path with 16-bit totals already runs faster. That's a

5:33 reading of the numbers though, because neither Nvidia nor the maintainer spells it out. Zaguita's own account stays at that level. A higher limit on how fast those

5:42 instructions can issue, and the more useful question for you is what a higher limit buys. The project's readme publishes one benchmark of its own. An RTX 3090 tie at 450 watts multiplying

5:54 square matrices with 8,192 rows. Each result is the median of five runs with 32-bit totals throughout. BF16 went from 87 and 1/2 teraflops to 157, up

6:07 80%, and TF32 went from 42 and 1/2 to about 82 and 1/2, up 94. You'd expect FP16 to land in the same place since it's the same width on the same card. Yet, the same table has it rising only

6:21 35% from 87 and 1/2 to about 118. And the readme gives no reason for the gap. Rows like that one tend to fall off somewhere between a benchmark table and a thumbnail. So, anyone quoting you one

6:33 number for this switch is flattening a table because the lift depends on which format your model's math uses. On one recipe, tensor math with 32-bit totals, what's being raised is a ceiling. A

6:45 ceiling, though, only tells you how fast the math could go, and you'd reasonably expect a job made mostly of matrix multiplies to speed up about as much as a matrix multiply does. Fatayev's post

6:56 is where that expectation first slips because on the same 3090s at 300 W, 89% on raw tensor math shrank to 55 once the multiply went through Nvidia's own library. Picture a commute where only

7:08 one stretch is motorway, and the switch doubles the speed limit on that stretch alone. Your trip gets faster only by however much of it you spent there. Even a plain library call spends some of its

7:18 time on work besides tensor instructions, though Fatayev's post doesn't split his gap between that and his 300 Where the picture breaks is power

7:27 because on a graphics card, running one stretch harder draws more power. And once the card reaches its limit, everything else slows down along with it. GitHub issue 182 shows both effects

7:38 on one machine. A user called Curias ran an RTX 3070 Ti laptop chip held at 105 W. The plain BF16 matrix multiply rose 50% from about 33 and 1/2 teraflops to just

7:52 over 50. Then came whole jobs where reading a 2,000 token prompt before a language model starts answering, a step called prefill, rose 44%. Training a small GPT

8:03 with 124 million parameters rose 29, and training the image model ResNet 50 rose only 14. Prompt processing in llama.cpp, a program for running language models locally, landed in the middle with a

8:16 BF16 model at about 28%. So, the further a job sits from a bare matrix multiply, the less of the ceiling it collects. Curios' report caught the power side, too. Since under sustained load, the

8:28 chip's clock fell from 1,459 MHz to 1,099 at the same 105 W. Holding a fixed power budget by trading away clock speed is the one benchmark every laptop passes without being asked.

8:41 A Korean account whose first post on the switch drew about 11,000 likes posted a correction the same day saying the switch doesn't overclock the card or raise its power limit. Relaying Zagueta,

8:51 it added that the card lowers its own clocks when it reaches the limit it already had. Before comparing any of this, notice that these are three different tests. The readme used a 3090

9:01 Ti at 450 W. Fattayev used 3090s at 300, and issue 182 used a laptop at 105. They happen to point the same way, but their numbers don't line up against each other.

9:13 Issue 182 alone makes the point because on that one laptop, the same switch read 50% on a bare multiply and 14% in a ResNet training run. So, what mostly predicts your own speed

9:24 up is that one recipe's share of your job. If the share is what counts, you'd expect BF16 image generation to be the best case. A user called Under Null timed it image by image on a 3090 Ti.

9:36 The test went up on the Stable Diffusion subreddit and was written up by ComfyUI Wiki, which is where these numbers come from. Tested in BF16, the models came out

9:45 roughly 56 to 59% faster. One of them, Moss image in BF16, went from just under 72 seconds an image to about 45. Call it 26 and 1/2 seconds back on every image, which adds up over a long batch.

10:00 Eight-bit integer versions of the same models barely moved, one by 3/10 of a percent and the other by under five. That fits the project's own description, since eight-bit integer math isn't on

10:11 the path it lifts. A Across 150 pairs of images, that owner also found identical decoded pixels in their own runs. Language models are where the numbers get strange. On October 24th, WCCF

10:24 Tech's article on the unlock cited the 27 billion Qwen 3.8 jumping from 82 to 381 tokens a second on a 3090. It counted that about 4.6 times faster as a real-world gain. WCCF

10:38 Tech's own article from October 18th, 3 days before Fatyev's post, credits that exact result to inference software, speculative decoding and look-up drafting, with no hardware unlock in it

10:49 at all. Methods like those usually guess several tokens ahead and let the model check them in one pass. Getting 4.6 times out of a switch that lifts a two times cap

10:59 would be quite a trick. And in this case, the switch wasn't even in the room. The chatbot from the start comes out of issue 202.

11:06 On Debian with two 12 GB RTX 3060s, the raw 16-bit tensor math rose 78 to 90%. Then the owner ran three quantized models in llama.cpp, the largest at 40 billion parameters.

11:20 Pre-fill rose somewhere between 4/10 of a percent and 3 and 1/2. Token generation, the part where the model writes its answer one token at a time, didn't change at all. Image and audio

11:31 outputs in the same report came out bit for bit identical. Memory is the reason. Writing each new token on a GPU usually waits on the card feeding numbers out of its own memory to the tensor cores. On a

11:42 3090 for scale, that feed runs at up to 936 GB a second. So, the cores finish their part and wait for the next delivery, and speeding up their part leaves the wait exactly where it was. A

11:54 site laying out that general explanation relays one measurement of the switch on token generation, or decode, 96.7 tokens a second against 97.2, about half a percent apart. In the commute picture,

12:07 decode is the queue at the toll booth, and a higher motorway limit does nothing for a car that isn't moving. Bandwidth also explains the A6000 line from earlier.

12:16 >> [snorts] >> A card with the full tensor rate, but 768 GB a second of memory has less to work with during decode than the 3090's 936.

12:25 On paper then, the pro card's advantage points at the wrong phase for chat. So, the quick test comes down to two things you already know about your own work. The number format the heavy math

12:35 runs in, and which phase you're waiting on. If it's BF16 and you're waiting on prefill, training, or image generation, the ceiling was in your way and owners

12:44 measured roughly 14 to 60%. FP16 and TF32 moved to on a bare multiply in the project's own table by very different amounts, but no owner here timed a real job in them. And if your model is

12:57 quantized to eight bits or lower, or you're watching tokens appear one at a time, the ceiling was barely in your way. There, owners measured anywhere from nothing to about 5%. The title's

13:08 last two words are the part no benchmark can measure because a documented gap says nothing on its own about why it's there or about what trying the switch could cost you. Product segmentation,

13:18 the idea that Nvidia put a small handicap on consumer cards to set its data center and professional cards apart, is the maintainer's explanation from the project's own FAQ. Notebook

13:29 check reported no comment from Nvidia, and Nvidia's forum moderators unpublished is still all there is from that side. So, the gap is documented and the motive

13:38 is the maintainer's reading with nothing from Nvidia to confirm or deny it. A Korean user floated a different idea, offered as a guess, that some chips might be capped because they fell short

13:48 in factory testing. None of the speed numbers in this video come from an independent lab, which is the one limit that sits under all of them. Notebook check's headline says the 3090 nearly

13:57 doubles, while its article leans on community reports and calls the gains very workload dependent, which makes it a fair summary of this whole video on a single page. Issue 184, filed October

14:09 26th, is the failure report worth knowing about for stability. On an RTX 3060 inside a Proxmox virtual machine, the module failed a pre-flight check and the Nvidia driver couldn't use the card

14:20 until a reboot. Once the module was disabled, the card recovered fully and a proposed fix worked for that reporter. Zaguita's FAQ calls damage very unlikely and says normal throttling stays in

14:31 place and owners report identical outputs on their own runs. Beyond what individual owners describe though, no source here measures errors, heat, or wear over the long run. The support side

14:42 is spelled out in Linux's own kernel documentation. Loading a module built outside the kernel's tree or an unsigned one marks the kernel as tainted, which is a fairly strong word for somebody

14:53 else built this. Bug reports from tainted kernels are often ignored, the same page warns, and that's a general Linux note rather than a finding about this project. So, the

15:02 page's own advice for a problem you want fixed is to reproduce it without the extra module loaded, which is one more step on your side. Beyond the project itself, the switch has spread into

15:12 third-party Windows ports with community claims like 92% on a 3060. A Japanese user ran a third-party Windows tool hosted on Hugging Face on a modified 32 GB RTX 4080. Zaguita doesn't maintain

15:26 the Windows side and nobody in these sources has checked their code. Even the project's own supported list now includes the RTX 4090 with no 4090 benchmark beside it. The call then

15:37 depends on what your card spends its day doing. If you run quantized models in llama.cpp or something like it and mostly wait on tokens, the switch has nothing for you. The bigger wins there

15:48 sit in software like the speculative decoding that produced the 4.6 times number. Software has its own fine print though. Per WCCFTech's October 18th article, the lookup mode

16:00 behind that number runs at about half the speed of the software's plain speculative mode on anything but repeated text. If you generate images or train in BF16, the gains owners report

16:10 run from about 14 to 60% though every one of them is owner-measured. One route that skips the module is the path Nvidia's foreign moderator said runs faster FP16 inputs with 16-bit totals.

16:23 That only works if your model can move off BF16 and its accuracy can take the shorter totals. Another is a professional card that carries the full rate on paper. And if

16:32 the machine has to work on Monday, runs as a server or virtual machine, or is where you'd file driver bug reports from, the better call is leaving the module alone. The risks on that side are

16:43 on record in a failure report and Linux's own docs, while the speed up depends on a workload you'd have to check first. Alexey Bataev's post about half the speed switched off got the

16:52 number right and left out two things. The half rate sat in Nvidia's own tables for years and in real jobs, lifting the ceiling bought the owners who measured it anywhere from nothing to about 60%.

17:03 What's still open is why the same chip does half the tensor work per clock in a GeForce card, and only Nvidia can answer that. So, which format does your heaviest job run in, and would that

17:13 ceiling ever have been in its way?