The AI Problem Nobody Has Been Able to Fix (Context Rot)

Open a brand new chat and ask one sharp question. You get a sharp answer back. That part works beautifully. Now paste in 40 pages of documents first, then ask that exact same question. Something changes. It starts skipping things. It contradicts an instruction you handed it four paragraphs earlier. You handed the model more to work with and it got measurably worse at working. That is the whole problem. Most people blame themselves for that. Bad prompt, wrong wording. Try again in the morning. The prompt is rarely the problem. This effect is documented. It has a name and it dates to 2023. Researchers call it context rot. The uncomfortable part is not that it happens. It is who it happens to. In July 2025, a researcher named Kelly Hong and her team at Chroma ran 18 Frontier models through it. The 2025 lineup, GPT 4.1, Claude 4, Gemini 2.5, Quinn 3, all 18 degraded as the input grew and not at the edge of the window. Degradation started long before the window was anywhere close to full. So what is actually breaking underneath? Why three years of research has managed it rather than cured it? And what the people who build on this do instead? First, the thing people loosely call memory. When you paste a PDF into chat GPT or Claude, it lands in the context window. Think of that window as the model's entire working desk. your instructions, your files, the conversation so far, all visible at once while it answers, which makes one assumption feel obvious. Bigger desk, better answers. Hand it everything and it will connect everything. That assumption is exactly what the Chroma team went after and the curve ran the other way. Across those 18 models, accuracy fell as input grew by 30 to 50% at the far end. The detail that matters is where it starts falling. A model sold with a 200,000 token window was already slipping at around 50,000. Then they made the task easier. Simple retrieval, one fact, nothing clever. It still fell. So they made it trivial. Hand the model a list of repeated words and ask it to copy the list back. Not reason about it. Copy it. At short lengths, clean, longer, and models began dropping words, inventing extras, and putting the odd one out in the wrong position. Copying a list is the least intelligent thing you can ask a language model to do and length broke that too. Two other findings from that report stuck with me. A single plausible wrong answer planted in the document measurably hurt accuracy and four of them compounded it with clawed models tending to abstain while GPT models answered confidently and wrongly. The second one is strange. Models scored better on a shuffled haststack than on the same sentences in their logical order. Coherent structure hurt them. The report does not claim to know why, and I have not found anyone who does. Hong's team is careful about what all of this proves. They observe the effect and state plainly that they do not explain the mechanism, and they warn that real work with synthesis and multi-step reasoning should be worse than their controlled tests. That sits badly next to what you hear at launch. A million tokens of context is standard now. Open AI, Anthropic, Google, and XAI all ship it. And Meta's Llama for Scout advertises 10 million. And the claim gets proven with a test called needle in a haststack. Greg Kumrun popularized it in late 2023. First on GPT4 and then on claude 2.1. Hide one sentence inside a huge document. Ask the model to find it and chart how often it does. It is a real pressure test and it is also mostly a vocabulary match. The hidden sentence usually shares words with the question so a model can pattern match its way there without understanding a line of the document around it. Comr said as much himself in 2024, arguing the field had to move from recall to reasoning, which is the gap that matters because nobody uses AI the first way. You do not paste a contract to find a sentence you already wrote. You paste it to get an answer that is not written anywhere in it. So in February 2025, Ali Modaresi and his co-authors published the version with the shortcut taken out. It is called no lima sack, but the needle is rewritten to share almost no vocabulary with the question. So, the model has to make the connection itself. 13 models, each advertising at least 128,000 tokens at 32,000, a quarter of the way in, 11 of the 13 scored below half of their own short context baseline. In that 2025 test, GPT4 O fell from 99.3% to 69.7 and that was near the top of the group. Same haststack, same length. Take away the matching words and the score collapses. Nvidia's ruler benchmark found the same shape from another angle. Test models at the length printed on the box and almost none hold their own quality bar all the way there. The community benchmark for deep comprehension puts the practical band at roughly 16 to 64,000 tokens. There is a second failure stacked on top of that one and this is the one you can feel directly in your own chats. In July 2023, Nelson Louu and his colleagues at Stanford published a paper called Lost in the Middle. They took a long input, moved the answer around inside it, and watched accuracy move with it. Best when the answer sat at the start, nearly as good at the very end, worst when it sat in the middle. Plot that, and you get a U. In plain terms, the model reads the top of your document, remembers the bottom, and thins out through everything in between, which is almost word for word the complaint people have about feeding it long PDFs. That paper is three years old, and every model in it has been retired since. The shape it found has not been retired. Windows grew from 4,000 tokens to a million and the U did not flatten out. So why can an engineer not simply patch it? A preprint from March 2026 makes a sharp claim about that. It is called lost in the middle at birth and it measured the U-shape in untrained models. Step zero before any training with the position encoding switched off. The curve was already there. The argument is geometric. Causal masking. The rule that each token only sees what came before it. piles influence at the start of the prompt. The residual connection anchors the final token, and between those two, the author derives a dead zone that shrinks factorially with the depth of the network. It is one preprint from one author with no venue yet, so hold it loosely. But if it survives review, the U is not a habit the model picked up from its training data. It is the shape of the building it lives in, and one result closes off the easy fix. A paper at EMNLP last October handed models perfect retrieval. the right passage already located, nothing left to search for, and performance still fell between 14 and 85% as the context around it grew. Length itself is the tax, and the model pays it even when it has already been shown the answer. Better search does not rescue you. It only decides how much you were carrying when the bill arrived, which brings us to the half of the title that actually matters, not why this is hard, why the labs are not racing to solve it. Start with the mechanism which Anthropic put in its own engineering post. A transformer lets every token attend to every other token. So n tokens produce n squared pairwise relationships. Their words as context length increases a model's ability to capture those relationships gets stretched thin. They call it an attention budget and it is finite by construction. You do not get to make it infinite while staying inside the architecture that made these models work in the first place. Then look at what the vendors do rather than what they announce. A pricing page tells you more than a launch post does. Google's own page lists Gemini 3.1 Pro at $2 per million input tokens up to 200,000 tokens and $4 above it. Output goes from 12 to 18. Open AAI draws the same line at 272,000. Its GPT 5.6 flagship is $5 in and 30 out beneath that line, 10 and 45 above it. One extra token roughly doubles what you pay for the input. Read those as admissions. The long window is the headline feature and the premium tier at the same time. And the cliff sits almost exactly where the research says the quality has already gone. You are paying double for the part that works worst. Credit where it is owed though. Anthropic charges flat and its documentation states that a 900,000 token request bills at the same per token rate as a 9,000 token one. That removes the search charge. It does not remove the rot and the incentive underneath does not move either. A context window is a number you can print on a launch slide. Effective context is a number that needs a benchmark the lab does not control and every independent benchmark that exists keeps coming back with a smaller answer. So the people who build on this stopped arguing about window size and started managing the budget instead. Anthropic's own recommendation is three moves and you have almost certainly used the first one without knowing its name. Compaction. When a conversation gets near the limit, summarize it and reopen a fresh window holding only the summary. That is exactly what the compact step in clawed code is doing when a long session starts to drift. Then structured note-taking where the agent writes its state to a file outside the window and reads it back when it needs it. So the window stays small and the memory lives on disk. And sub aents instead of one agent carrying a whole project in its head, specialists each get a clean window and hand back only the result. Rag is the older version of that same instinct. Retrieve the handful of passages that matter, put those in the window, and leave the rest on disk. It gets declared dead every time a bigger window shifts, and it has not died. Its job changed. It is now one of several ways to keep the context small on purpose, and there is evidence the managing works. A benchmark published in February grew the environment around an agent step by step and watched its success rate fall, then showed that proper context management recovered a substantial part of it. A June paper found something sadder inside long searches. Models give up early. The authors call it premature termination. Under a lot of context, the model concludes or guesses well before it has actually used the window it was handed, which sets up the most interesting answer of 2026, and it inverts the problem instead of fighting it. Six weeks into his first year of graduate school at MIT, a 24year-old named Alex Jang wrote a blog post proposing recursive language models. Scientific American profiled him for it in June. His description of the bug is the clearest I have read. A model can do task A. It can also do task B, give it A and B at the same time, and it does both worse than it did either one alone. So, the fix he proposes is to stop giving it both. In a recursive language model, your million token document never enters the model's context at all. It sits in a Python environment as a variable. The model writes code to slice it up and spawns copies of itself to read the slices. Each copy seeing a small clean window. The parent reads the answers, not the document. In the paper's own test on the olong benchmark at 132,000 tokens that set up running the cheaper GPT5 minibe plain GPT5 by more than 30 points, roughly double the correct answers at a comparable cost per query. And Jiang's headline claim is the one to sit with. Recursive models do not degrade at 10 million tokens and beyond because no single call ever sees 10 million tokens. The paper hit Erics on the last day of 2025. The library sits at 5,300 stars and Prime Intellect built its own version and called it the paradigm of 2026. Then March brought the reproduction which complicates the picture. Researchers rebuilt it and pushed the recursion one level deeper and depth 2 made the models overthink. Worse answers on simple retrieval and one task that runs in three and a half seconds at depth one took 344 at depth two. So here is where I land and I am not going to sit on the fence about it. Stop buying context size and start engineering context. For most teams building on these models, a small curated window beats a big one you emptied a folder into. Three receipts. Quality falls before the window fills. 11 of 13 models have by 32,000 tokens. And two of the three largest labs price the far end of their own window as a premium. The villain here is an idea rather than a company, selling the advertised window instead of the effective one. The concession is real, though. If your job is one document under about 30,000 tokens answered once, the big window is the cheap and correct tool and you should not build a retrieval pipeline to read one PDF. The 5% case is the thing that runs for hours and that is where the engineering earns its keep. Which leaves a question I cannot answer. If that U-shape really is there at initialization, then compaction, retrieval, and recursion are all scaffolding around a model that cannot read evenly, and we are getting better at hiding the limit rather than removing it. Does the next architecture take the rod out or do we just keep building smarter walls around it? Recursive language models are the strongest bet on the table right now and they have earned a video of their own. That is the next.