I Opened a New AI Model's Config File. Every Line Was a Paper

0:00 Open a file with me. This is config.json from the newest open model on huggingface downloaded this morning. QWin 3.8 2.4 trillion parameters. It was uploaded 6 days ago. Now scroll down.

0:12 There, num_experts 512 and 10 of them run per token. The rest stay asleep. That single line is a paper from January 2017. It is older than the Transformer paper itself. Keep

0:26 going. rope_theta 10 million. That number traces back to a Chinese blog post from 2021. Then num_key_value_heads four of them against 64 query heads. That gap is a bill and use_cache set to

0:42 true. Three words that carry most of what you pay to serve this thing. Four lines in one file, four separate papers. Not one of them written later than 2023. There is a fifth line lit up here as

0:54 well. No reading list has that one on it yet. Hold on to it. 11 papers built this. So which of the 11 do you have to read and which one can wait a year? I am going to rank them and argue for the

1:05 ranking. Four of the 11 decide what you pay, how long your context can be, and how many users fit on a single graphics card. The other seven are interesting. Those four hold the weight and the gap

1:16 between interesting and holding the weight is what most reading lists get wrong. And every one of them is a fix for the same complaint filed over and over for 9 years. This is too expensive

1:26 to run. So go to the bottom of the file. One afternoon 2017 12 June 2017 5:23 in the evening coordinated universal time. A paper goes up on arXiv called deep reinforcement learning from human

1:39 preferences. Six authors. Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, Dario Amodei. 33 minutes later at 5:57 a different group at Google post attention is all you

1:53 need. The Google paper is the machine. The Christiano paper is the steering wheel. For 5 years, almost the whole industry read one of them. Between them, they are the reason you can talk to a

2:03 computer at all. Here is what Cristiano and his co-authors actually did, and it is smaller than you would expect. They had a simulated robot and no reward function for the behavior they wanted.

2:13 So, they showed a human two short clips of the robot flailing and asked one question, which of these is closer to a backflip? Then, they trained a reward model on the answers. The paper reports

2:24 feedback on, in their words, less than 1% of the agent's interactions with the environment and about an hour of human time to teach a novel behavior. An hour. That is the whole idea. You do not

2:34 describe the goal because you cannot. You judge pairs, and the judging becomes the goal. 5 years later, that trick has a name on every model card, reinforcement learning from human

2:44 feedback. And the paper that made it matter commercially is Instruct GPT, March 2022. Its result is the one worth memorizing. Outputs from a 1.3 billion parameter

2:55 Instruct GPT model were preferred by human labelers over outputs from a 175 billion parameter GPT-3. 100 times fewer parameters, in the paper's own phrasing, winning on preference. That is the first

3:08 thing this list gets right. Alignment arrived dressed as a safety idea and turned out to be a compression technique. Teaching a model what you want is cheaper than making it bigger,

3:17 but preference does not pay for the graphics card. Capacity does. And months before the Transformer paper, one of its authors had already published the fix for that. January 2017, Outrageously

3:28 Large Neural Networks, {comma} the Sparsely Gated Mixture of Experts Layer. That is the actual title. The first author is Noam Shazeer. The problem it opens with is the oldest

3:38 one in the field. A network's ability to absorb knowledge is capped by its parameter count. More knowledge means more parameters, and more parameters means more compute for every single

3:47 example, unless the parameters take turns. Split the layer into a few thousand small feed-forward networks, call them experts, and put a tiny trainable gate in front of them that

3:56 picks a handful per input. The parameter count goes up like a warehouse, the compute per token stays like a corner shop. You are paying rent on the warehouse and electricity on the shop.

4:06 Their claim, verbatim, is greater than a thousand times improvements in model capacity with only minor losses in computational efficiency. The largest thing they built with it

4:16 held 137 billion parameters in 2017. Now go back to the file we opened. 512 experts, 10 fire per token. That is roughly 4% of the model's parameters doing work on any given word. DeepSeek

4:30 V4 flash, uploaded a fortnight ago, runs 256 routed experts plus one shared, six per token. GLM 5.2, 256 and 8. Sparsity stopped being a trend and became the format. But sparsity only cheapens the

4:45 thick part of the network. The attention layer sitting between those experts was still moving every byte of its working set twice, and for five years the field tried to fix that with better

4:54 mathematics. Tri Dao's argument in May 2022 was that the mathematics was not the problem at all. The paper is flash attention, and the word that matters in the title is not fast. It is IO-aware.

5:06 Here's the shape of it. A graphics card has a small pool of extremely fast on-chip memory and a large pool of much slower memory next to it. Standard attention writes a gigantic intermediate

5:16 matrix out to the slow pool and reads it back. Flash attention refuses to write it down. It processes attention in tiles small enough to stay in the fast pool and rebuilds what it needs on the fly.

5:27 Same answer to the bit. Exact, not approximate. The first paper reported three times faster on GPT-2. 15% off the then record BERT large training time and something better than either. Sequence

5:40 lengths that had simply not been reachable before. Then it kept going, and this is the part that decides where it sits on the list. Flash attention two in 2023, Flash attention three in July

5:51 2024, which took hopper cards from 35% utilization to 75, and Flash attention four, March this year, rewritten for Blackwell because Blackwell broke the assumptions. Tensor core throughput

6:03 doubled while shared memory bandwidth and the exponential units did not. Look at the release page, 22nd of July, 29th of July, 5th of August, 12th of August, four releases, seven days apart each

6:16 time, and the repository was pushed again the morning I pulled these numbers. A paper from 2022 is shipping weekly in 2026. You do not cite something like that, you install it. And

6:27 it is the clearest single argument for why the boring papers outrank the exciting ones. But a fast kernel is worthless if the model cannot tell which word came first, which brings up the

6:37 strangest entry on this list because it was not published as a paper. It was published as a blog post. Late March 2021. A researcher named Jianlin Su writes a post on his personal site

6:48 kexue.fm in Chinese about position encoding. The formal paper, RoFormer, lands on arXiv four weeks later. The original transformer added a wave pattern to each token's vector. It

7:00 works. It also means the model has to learn what an absolute position number means, and that learning does not stretch past the lengths it saw in training. Su's

7:08 move is to stop adding and start turning. Every position rotates the token's vector by an angle proportional to where it sits. Rotate the query, rotate the key, and the dot product

7:18 between them ends up depending on the difference between their positions. Relative position for free out of the geometry with nothing added to the vector at all.

7:27 And attention between two tokens decays as they get further apart, which is a property somebody had been bolting on by hand for years. Eleuther AI picked it up from the blog post before most people

7:37 had read the paper and wrote a line I keep coming back to. Very few approaches generalize well across code bases and tasks. Rotary embeddings, they said, performed as well or better in every

7:47 architecture they had tried. They were right, and the receipt is the file we opened. rope_theta 10 million in the QWinConfig, 8 million in GLM, 10,000 in DeepSeek with a scaling block underneath

8:00 it. Soon now works at Moonshot AI, which ships Kimi. Solving position is what let the window grow, and a long window means a long cache. So, the KV cache. This is the one item on the list without a

8:12 single paper behind it. It is a lineage, four papers deep, and worth watching properly because it is where most serving money goes. Start with what the cache is. When a model generates the

8:23 10th word, it needs to attend to the first nine, and recomputing their keys and values every step would be quadratic waste. So, you keep them. That is the whole mechanism. The bill arrives

8:33 because that memory scales with sequence length times batch size times layers times heads, and it lives in the same place your Long context and many users compete for

8:42 the same pool. November 2019, Noam Shazeer again, this time as the only author, publishes fast transformer decoding. His diagnosis is one sentence long, and it names the enemy exactly.

8:55 Incremental inference is slow because of the memory bandwidth cost of repeatedly loading the large keys and values tensors. His fix is blunt. Share one key and value head across every query head,

9:05 multi-query attention. The tensor shrinks by the number of heads, decoding gets much faster, and quality takes a small hit, which raises a question you should answer before the next beat

9:15 because it decides how you architect a product. On a long context request, which is bigger, the weights you are renting or the cash those weights are filling? For most serving setups at real

9:25 context lengths, it is the cash. That is the answer the next four years of research assumed, and it is why the fix went from blunt to careful. 2023, grouped query attention. Instead of one

9:36 shared key-value head or one per query head, use a handful shared in groups. Most of the memory saving, most of the quality, and a recipe to convert an existing checkpoint using 5% of its

9:47 original pre-training compute. That is the line in the Q win file. 64 query heads, four key value heads, 16 to one. A 2023 paper in a model uploaded last week. September 2023, a different

10:01 attack. The V L M team looked at the cache the way an operating system looks at memory and found the waste was fragmentation. Requests reserving contiguous blocks for

10:11 the longest output they might produce, then not producing it. Paged attention gives a cache virtual memory. Non-contiguous pages shared between requests that share a prefix. Reported

10:21 throughput improvement of two to four times at the same latency, and the project built on it now sits near 90,000 stars. Then DeepSeek asked whether you need to store keys and values at all or

10:31 just enough to reconstruct them. Multi-head latent attention compresses both into a low rank latent vector and decompresses on use. Their reported reduction against their own dense 67

10:42 billion parameter baseline is 93.3% of the cache gone. And that is the number that made a million token window something you can put on a price list, which you can watch happen in the

10:53 DeepSeek config. Original position embeddings 65,536, a yarn scaling factor of 16. Multiply them and you get 1,048,576, the exact context length in the line

11:07 below. So, the machine is cheap now. It still could not do arithmetic. January 2022, Jason Wei and colleagues at Google published the paper with the most quoted result in prompt engineering, chain of

11:19 thought prompting. The method is almost insultingly simple. In your few-shot examples, do not just show the answer, show the working. Then the model shows its working, too, and the accuracy

11:29 moves. Their headline, a 540 billion parameter model given eight worked examples reaching state of the art on grade school maths word problems beating a fine-tuned model that had a verifier

11:41 attached. But in September 2024, a group led out of the University of Texas ran the audit and the paper is called to cot or not to cot, a meta-analysis over more than 100 papers plus their own runs

11:53 across 20 data sets in 14 models. The finding is narrow and awkward. Chain of thought helps mainly on maths and symbolic reasoning. On the broad knowledge benchmark, generating the

12:04 answer directly scores almost identically unless the question or the answer contains an equal sign. An equal sign. The most celebrated prompting technique in the field turned out to be

12:14 mostly a way of getting a language model to do symbol manipulation slowly. Their own conclusion asked the field to move beyond prompt-based chain of thought. It did and this is why chain of thought is

12:25 on the list even though you will rarely type one again. Reasoning stopped being something you prompt and became something you train. Every reasoning model since has been taught to produce

12:34 that trace by reinforcement learning against answers a machine can check. The technique got absorbed into the weights. Read the paper anyway, not for the prompt, for the thing it proved, that

12:44 generating intermediate tokens buys computation, which is the assumption every reasoning model is now built on. Therefore, the open question moved, not how the model thinks, but what it does

12:53 between thoughts. October 2022, Shunyu Yao and co-authors published the answer and it is four letters. React, reason and act interleaved. The model emits a thought, then an action, then reads an

13:07 observation from the world, then thinks again. Reasoning keeps the plan on track. Acting keeps the plan attached to reality. On two interactive benchmarks, it beat imitation and reinforcement

13:17 learning baselines by 34 points of absolute success rate on one and 10 on the other, prompted with one or two examples, points, not percent. Strip the framing and you have the shape every

13:28 agent you have used is built on, a while loop. Call the model, run the tool it asked for, append the result, call it again, stop when it says it is finished, and it fails in three ways worth knowing

13:39 before you ship one. It loops retrying the same tool with the same arguments. It blows up context because every thought and observation gets appended, and it cannot re-plan because there was

13:50 no plan to revise. But a loop only acts on what it retrieved, and retrieval has a hole in it that ordinary cannot cover. Ask a corpus what its main themes are, and there is no passage to fetch. The

14:01 answer is not in any chunk, it is in all of them. April 2024, a team at Microsoft Research led by Daren Edge published Graph Rag. Rather than retrieving passages, use the model to build a

14:13 knowledge graph over the corpus first, cluster it into communities, and pre-write a summary for every community. Ask a global question, and it answers from those summaries instead of from

14:22 chunks. Then 7 months later, the same three names, Edge, Ha Tren, Jonathan Larson, published a thing that undercuts their own method, Lazy Graph Rag. Indexing costs, in their words,

14:34 identical to vector search and a thousandth of the cost of full Graph Rag. That is the most useful signal on this entire list, and it has nothing to do with graphs.

14:43 When a technique's own authors race to publish a cheaper version, the original was expensive enough to be blocking adoption. Watch what a lab does next, not what it

14:51 claims. Which leaves the last piece, the loop still has to talk to your tools and to other people's agents. Google announced agent to agent in April 2025 and handed it to the Linux Foundation 2

15:02 months later. At its first anniversary this April, the foundation reported more than 150 organizations and integrations across all three major clouds. But version 1.0 of the specification

15:14 landed in March this year, 11 months after the announcement. Meanwhile, Model Context Protocol shipped a new specification on the 28th of July on software development kits passed a

15:24 billion downloads. Both are real and only one is in your stack this quarter, which is the whole reason ranking a reading list is worth the trouble. That leaves the two papers that gave the

15:33 machine eyes, and they are a lesson in how fast a foundational paper turns into furniture. February 2021, CLIP led by Alec Radford. 400 million image and text pairs off the internet and one training

15:46 objective. Given a batch of images and a batch of captions, predict which goes with which. No labels, no fixed category list. What falls out is a shared space where a picture of a dog and the words a

15:57 photo of a dog land near each other. Which means you can classify an image into categories the model did not train on by writing them down. They match the original ResNet-50 on ImageNet without

16:08 using a single one of its 1.28 million training images. BLIP, a year later from Junnan Li at Salesforce, fixed the data instead of the model. Web captions are noisy, so BLIP trains a captioner to

16:20 write synthetic ones and a filter to throw out the bad ones, then retrains on what survives. Gains of roughly one to three points across retrieval, captioning, and question answering. Both

16:30 are on this list for the idea, not the checkpoint. Contrastive pre-training is how nearly every vision language model still learns to see, but the objective got replaced by a sigmoid version and

16:40 neither original repository has been touched since March. 11 papers, nine years, one file. So, here's the ranking and I'm not going to hedge it. Four of these pay rent. Mixture of

16:51 experts, flash attention, rotary embeddings, and the KV cache lineage. Read those four first in that order. They decide what you pay, how long your context can be, and how many users fit

17:02 on one card. The other seven are worth your time and none of them decide whether your product survives its first invoice. That is the difference and it is not a difference of quality. Chain of

17:12 thought is a better paper than grouped query attention. It is also far less likely to decide whether you ship three receipts because a ranking without receipts is a preference. One, flash

17:23 attention shipped four releases in the last four weeks and was pushed again this morning four years after publication. That is a live dependency, not a citation. Two,

17:33 latent attention cut the cash by 93% against its own dense baseline. That is a line in a serving bill, not a row on a leaderboard, and it is why a million token window has a price at all. Three,

17:46 the newest and loudest protocol on this list reach version one this March with 150 organizations behind it. While the older one shipped a fresh specification last month on a billion downloads, both

17:56 real. One is in your stack today. Who should ignore me? If the model is already a commodity for you and the hard part is what happens between calls, then react and graph rag come first and the

18:07 efficiency four can wait a month. That is a real exception and I would still read all 11. Chain of thought is the highest leverage read for anyone who writes prompts for a living, but four

18:17 weekends is what most people have, so spend them on the papers in your invoice, not the papers on your timeline because the enemy here is not any of these papers. It is the reflex to read

18:26 whatever went up this week. Computer science AI submissions to arXiv ran 138 in January 2017 and 5,490 in June this year, nearly 40 times as many. You cannot read that. Nobody can.

18:40 11 you can read and 11 is enough because the ones that last are the ones that end up in configuration files rather than in threads, which brings us back to the fifth line I asked you to hold on to,

18:51 index_topk 512 sitting in the deep seek file. GLM has it too, right there in the architecture name. That is a learned index that scores every earlier token

19:02 and keeps only the best few hundred to attend to. Sparse attention chosen rather than fixed. The paper describing it came out in December 2025 after every reading list including this one was

19:13 written. So, it goes on the list as number 12 and I will put a date on the bet. By the end of 2027, a learned sparse index like that will be in most new open frontier configs. Two of the

19:24 three files I opened for this video already have one. Here is the part I keep turning over. Chain of thought was a prompting technique for 2 years and then it stopped being a technique and

19:33 became part of the weights. Retrieval looks like it is heading the same way. So, which of the 11 gets swallowed next? And when it does, does it stop being worth reading or does that turn out to

19:43 be the moment it finds