Inside X's (Twitter) Algorithm: Insane Software Engineering Behind It

0:00 There is a file X published this week with one number in it that broke containment minus 234. Four lines above it sits another number 0.5. That one is a like. The big negative one is a

0:12 report. Divide the second into the first and you get 468. Within hours that was the headline. A report cost you 468 likes. It traveled everywhere because it sounds like a receipt. That reading is

0:25 wrong and the fix matters more than the number. What you are looking at is the ratio of two constants in a config file. We know it is wrong because X said so in code. About 27 and a half hours later,

0:37 they pushed a commit to argue back. 29 new lines of comments added to the ranking file aimed at whoever reads it next their wording so that language models or people reading it are more

0:48 likely to understand it correctly. That is where we are. A platform shipping code comments to correct a screenshot going around the internet. Behind the argument is the thing people actually

0:58 wanted, the numbers. All of them in one public repository, 2016 files, the Apache license, 30,000 stars in 2 days, written mostly in Rust, and one file in there lists 665 accounts by name,

1:12 hard-coded that the feed will not recommend. Over the next 20 minutes, what the numbers say, what they do not say, and what is still being held back. Start with the misreading because the

1:23 correction is the fastest way into how any of this works. The weights in that file do not multiply your likes. They multiply a prediction that a model has already made. The shape of it is this. A

1:34 model called Phoenix reads a post and reads your recent behavior then predicts how likely you are to do each thing to it. Like, reply, repost, quote, share it into a direct message. Follow the

1:45 author, mute them, report them. 13 predictions, each a probability. The scorer multiplies each prediction by its weight and adds them up. That sum is the post score and the arithmetic is one

1:57 line of rust sitting underneath everything else on your screen. So minus 234 is not what a report cost. It is what the chance of a report cost. And that chance X says in the new comment is

2:09 more than a thousand times smaller than the chance of a like which turns the number inside out. The weight is not large because reports are devastating. It is large because reports are rare and

2:19 a rare event with a small weight would round away to nothing. Treat it as a unit conversion. Report probabilities arrive in a tiny unit and the weight scales them up so they can be compared

2:29 against likes at all. The ratio is a calibration, not an exchange rate. That distinction matters beyond pedentry and X spells out Y in the same comment block. Readers had concluded that a

2:40 coordinated block and report campaign could bury an account. The answer in the code has two halves. The model predicts how likely you personally are to report the post. So, a brigade mostly moves

2:50 recommendations for people who resemble the brigade. Recommendations are personal, not global. The second half is stranger and more specific. For an action to count in ranking, it has to

3:01 happen on a post that was served in the home timeline. Open a link from a group chat and hit report and the ranking system does not see it. You can argue with all of that and researchers do.

3:11 What you cannot say is that the explanation was withheld because it sits in the file that holds the number in plain English with the wrong reading quoted back in quotation marks. The

3:20 commit itself is worth looking at as an object. 21 files changed. 29 lines added to the scorer. 62 to the params file, 15 to the readme, plus one brand new filter. We will come back to for scale.

3:34 The release the day before was the real event. In January, this repository held 79 files. By the middle of May, 216. On the 13th of August, it jumped to 2015 in one push, roughly nine times the

3:48 files and 13 times the code. And one line in it will date this era better than any launch video. We have added comments to the code they wrote so that language models or people reading it are

3:59 more likely to understand it correctly. Language models first because the way most people are going to read a Rust codebase this size is by pasting it into a chatbot and asking what it means. So

4:09 read the table properly, not as an exchange rate, but as a statement of what X wants its model to chase, written in the only language a ranking system understands. A like is 0.5.

4:20 It is the cheapest positive signal in the entire file, and it is the one most creators optimize for hardest. A repost is one, twice a like, which is lower than most people would guess given that

4:31 the whole culture of the platform treats the repost as the trophy. A reply is five. A quote post is five. Sending a post to someone in a direct message is five. And following the author off the

4:42 back of a post is four. Then copying the link. The plainest, least visible thing you can do with a post is weighted at 20, 40 times a like. Read the positives in order and a pattern falls out of

4:53 them. The cheap actions are the public ones. The expensive actions are the ones that cost you something private. A reply you have to write, a share you have to justify, a link you paste into a group

5:03 chat with your actual friends. The rest of the positive side is small change. Clicking into the post, 0.4. Opening a link, 0.2. Expanding a photo or opening a video, 500s each. Dwell time carries

5:17 4,000 per unit. Small, but it is the only weight that scales with how long you stay. Meanwhile, the plain binary dwell flag is set to zero in the publish defaults, and so is a profile click.

5:28 Bookmarks are the interesting absence. Phoenix does predict a bookmark probability and it sits right there in the list of prediction heads. But the published score carries no bookmark

5:37 weight. So that prediction stays out of the sum entirely. Then the axis flips and the file gets blunt. Not interested - 43.2. Block the authorus 31.2. Mute them - 58.8. Reportus 234. The ordering

5:54 there is worth a second look. Muting somebody is weighted heavier than blocking them, which is not most people's intuition. Blocking feels like the louder act. Muting is the quieter

6:04 one, and in this file, it is the more expensive one. One last negative, the smallest number in the table, not dwelled, minus 200s. The price of a post you scrolled straight past without

6:15 stopping. Put the whole thing on one axis and the argument writes itself. Engagement is cheap. Trust is expensive. The signals that move the feed hardest are the ones a reply guy account cannot

6:26 manufacture. 3 years ago, there was a version of this that did not include any of it. The 2023 repository, shipped the pipeline, and left the constants out, which is why it

6:36 collected 73,000 stars and settled almost nothing. That is the difference this release actually makes. And it puts a question to anyone who posts for a living. Now that the price list is

6:46 public, does knowing it change what you write? Hold that question because there is a better way to answer it than guessing. One of these weights moved in public this summer and the repository

6:55 now ships the whole story of how on the 13th of July a post went out from Nikita Beer's account. We are rolling out a small tweak to boost visibility of your post to your mutuals. It said people who

7:06 you follow back. The reason given was almost domestic. This data was missing from the algorithm and it made your friends appear less in your replies. So, the reply section felt more like a

7:16 battleground with people you do not recognize. That post did 9.7 million views. And 3 weeks later, the code behind it landed with its own explainer document showing what the change looked

7:26 like as a diff. Here is the timeline it gives you. On the 10th of July, an experiment started. A small percentage of users were randomly assigned a boost value of 5, 10, 15, or 20. Everybody

7:38 else sat at zero, which is no boost at all. On the 13th, results came back strong and the value of 20 went out to many users. Then on the 24th, after more results and after complaints that the

7:50 World Cup was passing people by, they moved it down to 15. Note what the boost actually attaches to because this is where most coverage compressed it. It applies to original posts only from an

8:00 author you mutually follow, not to their replies, not to their reposts. And it does not boost the post directly. It raises the weight on the predicted probability that you will reply to it

8:11 from 5 to 20, which puts a mutual's original post at 20 times the weight of a repost. The document opens by saying why it exists. Going forward, future algorithm updates will be published to

8:22 this repository and you should be able to understand what changed by checking the diffs. That is a testable promise and we will test it at the end. None of that happens in a vacuum. A score only

8:32 matters once there is something to score. And before anything is ranked, the feed has to go and find candidate posts. They come from three places. Thunder holds recent posts from the

8:42 accounts you follow in memory as they are published. Phoenix Retrieval embeds you and every post as vectors and returns the nearest ones and SIM clusters groups accounts by who engages

8:52 with what, then pulls candidates out of your cluster, in network from Thunder, out of network from the other two, and both get ranked by the same model. There is no separate algorithm for strangers,

9:02 but there is a discount for them. If you do not follow the author, the final score is multiplied by 0.75. If the post was pulled in because it matches a topic you follow, it is

9:12 multiplied by a half. Turn that around and you get the real number. A stranger's post has to score about a third higher than a friend's post to sit in the same slot. That single constant

9:22 is the entire in network versus out of network argument settled in one line. A second adjustment handles repetition. Every post after an author's first in the same feed is multiplied by a half,

9:33 decaying down to a floor of a quarter. Post five times in an hour, and the fifth one is competing at a quarter of its own score. The third adjustment is the one most coverage missed completely.

9:44 There is a cold start boost written into the timeline for authors who have almost no audience yet. The published conditions, fewer than a,000 impressions on the post, fewer than a,000 followers

9:54 on the account, and the post under 24 hours old. Clear all three and it gets lifted towards slots 15 and 16 of your feed. So there is a hard-coded new account escalator in the timeline and

10:05 its address is roughly the 15th post you scroll past. There is a switch beside it for a bandit sampler to choose which cold start post get promoted currently turned off. Before any of this, 17

10:16 filters run. Duplicates across sources, posts older than 48 hours, your own posts, accounts you block or mute, muted keywords, posts you have already been shown, checked twice against two

10:28 separate records of impressions. That 48 hour cut off is worth sitting with. Whatever else the for you feed is, it has a hard expiry date, and a post that has not found its audience inside 2 days

10:38 is not going to. What survives goes to Phoenix and one design note explains more about your feed than any weight does. During inference, candidate posts cannot attend to each other, only to

10:49 you. There is a second design note worth having. Both retrieval and ranking look up their embeddings through hash functions rather than a fixed vocabulary, which means there is no

10:58 dictionary to maintain, and a post published 1 second ago is representable immediately. Nothing has to warm up. Every post is scored in isolation against your history, which is what

11:08 makes the score consistent and cachable. It also means your timeline is not a contest between posts. It is a thousand separate verdicts about you sorted. After scoring, another service reorders

11:19 the top of the feed to cut similarity between neighbors, trading a little score for variety. Then a blender interle the things the model did not rank at all. Ads, who to follow,

11:29 prompts. So by the time a post reaches your eyes, it has been retrieved, filtered, predicted, weighted, discounted, diversified, reordered, and interled with advertising. The weights

11:40 the whole argument was about are one step out of eight. But ranking only decides the order. Something else decides whether a post is allowed to appear at all. And that part of the

11:49 codebase is what people have been demanding since 2018. It is called visibility filtering. And for every post and every viewer, it returns one of three answers. allow interstitial

12:00 meaning shown behind a warning you can tap through or drop. The rules are listed in evaluation order in a single registry file and 28 of them run for the home timeline. Suspended authors,

12:11 deactivated authors, protected accounts, people you block, people you mute, post with legal takedowns against them. The first rule that answers drop ends the evaluation. Those rules read labels and

12:23 the labels come from a separate machine running offline. One system scores an account on how often people block or report its posts relative to how often they favorite them. Another runs page

12:33 rank over the followraph and turns the result into a score per account. The rules themselves are written in a small language and the repository ships its compiler and its runtime. A rule reads

12:44 on this event if these conditions hold apply this label. 20 of those rule files are published along with 53 helper definitions. Some rules are held back. Then comes the part that is the actual

12:55 answer to the shadowban question. If the post is a recommendation from an account you do not follow, 26 additional rules switch on and those rules can only drop. Same post, two viewers, two outcomes.

13:07 Your followers see it. A stranger does not. 54 rules instead of 28. And the extra 26 exist purely to remove things from the feeds of people who have not opted in by following you. Among them is

13:19 a label called do not amplify. That exact phrase is what the 2023 argument was about. And here it is in a list with a note saying it is being retired in favor of a narrower malicious link

13:30 label. Which brings us to the half of this release that may matter more than the code. X paired it with a tool called under the hood. It reports the visibility limiting labels sitting on

13:40 your own account and your own posts over the past month as a file you can download. 18 post labels, 12 account labels, each with a plain description of what it means and what it does to you,

13:51 not adjectives. Sentences like post hidden from recommendations to non-followers or post discoverability is restricted to the author's profile. That is a platform naming the consequence in

14:02 the same words it uses internally. The catch is the roll out. A pilot group only accounts at least a year old, at least 10 posts in the last month, and aggregate figures rather than a verdict

14:12 on each post. Keith Coleman, who runs product at X, called it the kind of thing that people will be fairly shocked that we are releasing. And X's own suggested next step is to paste a

14:22 downloaded file into a language model and ask it what happened to your account, which brings back that file with 665 accounts hard-coded into it. On the 14th of August, X added a filter for

14:34 the Brazilian general election. Every account registered as a candidate with Brazil's electoral court listed by numeric identifier with the username written next to it in a comment. It

14:44 removes their post from for you along with reposts of them, quotes of them, and replies whose thread contains one, unless you follow the account, in which case nothing changes for you at all.

14:55 Ex's Brazil account posted the reason an hour and 47 minutes before the code landed. A resolution of the electoral court in forced from the 16th of August. The accounts are not suspended and the

15:05 posts stay on their profiles. In their words, this is not a sanction and not a change to X's rules. Whatever you make of the policy, look at what open sourcing did to it. A country specific

15:16 suppression list applied to named politicians is now a file with a diff and a timestamp on it. X makes that argument itself in the readme in one sentence. A benefit of open source is

15:26 that you can see that changes like this exist and exactly how they work. The framing they gave it on the day was the same. The code and the tool are two puzzle pieces that neatly fit together,

15:36 the announcement said so that people can answer for themselves whether a platform is limiting their reach. The account that posted it had existed for 6 days. So the case 4 is strong. Now the case

15:47 against which has been on the record since February and has not been answered. After the first release in January, researchers went through it and were unimpressed. John Thixon, a

15:57 computer science professor at Cornell, said, "These releases give you a pretense that they are being transparent for releasing code." Ruggerero Lazeroni, a doctoral researcher at Graz, put the

16:07 problem in one sentence. "We have the code to run the algorithm," he said, "but we do not have the model that you need to run the algorithm." "That complaint is still live in August.

16:16 Search the whole repository for a trained checkpoint, and there is nothing to find. 2,000 files, no model weights of any kind. The quick start says it outright. This is not a production

16:26 quality model or a production scale setup. Production data, checkpoints, orchestration, and scale are not included. So, you can train a tiny model on synthetic data for six steps and

16:37 watch the machinery turn over. You cannot reproduce what ranked your timeline this morning, and no outside researcher can audit it. Two more things are held back on purpose, and X names

16:46 them both. The prompt files for its content classifiers and some of the labeling rules withheld to reduce the risk of people gaming the system. There is a subtler caveat and it is the one to

16:56 carry out of here. The readme explains that cron jobs write live production values into these files. The repository is a mirror of a running configuration service, not its source of truth, which

17:06 means every number in this video is a snapshot with a shelf life and the accurate way to read the file is this is what production said the last time the job ran. X also states an aim rather

17:16 than a rule that any experiment touching 10% or more of traffic should be visible here. Then there are the promises. In January, the plan was open source within a week with updates every four weeks.

17:28 After the first push on the 20th of January, the repository did not move again for 116 days. In July, the promise got bigger. Once the security review is done, Elon Musk wrote, "The entire

17:39 codebase of X will be open source with no exceptions. What shipped in August is the for you feed and the systems that filter it. That is a great deal of code. It is not the codebase. Here is where it

17:50 lands. X wins this one and it is not close for the 95% of people whose real question is whether a platform is throttling them and why. Meta, Tik Tok, and YouTube publish explainers and

18:02 system cards. No equivalent turned up for any of them. A content you can read, a rule list you can follow in order, a report about your own account you can download. The 5% this fails are the

18:12 researchers and they are right to keep pushing because a system outside parties cannot reproduce is a system you are trusting rather than checking. And the villain in this story is neither X nor

18:22 the press. It is a number that traveled without its footnote. 468 was correct arithmetic on two real constants and it still describes something that does not exist, which is the old warning credited

18:34 to the economist Charles Goodart. Once a measure becomes a target, it stops being a good measure. Publishing the price list is a real advance and it is also the thing most likely to change what the

18:44 prices measure. So here is a bet with a date on it. Six months from now in February, this repository will still ship no production model and the entire codebase with no exceptions will still

18:55 not be opened. If that turns out wrong, it gets its own video. But the question the file leaves you with is not whether X is telling the truth. It is what any of us do with a price list. When the

19:06 cheap signals are the ones most people chase and the expensive ones are the private ones, does publishing the numbers make the feed more honest or just make the gaming better informed?