Every Skill You Add Breaks Your Harness (Here's the Fix)

0:00 On the 17th of August, Anthropic shipped a one-line fix to Claude code. Read it slowly. Reduce the context cost of loading the built-in Claude API skill from over 200,000 tokens to about

0:12 25,000. One skill Anthropic's own, eating a context window before you typed anything. A developer had already measured it in July. One migration document inside it was 36,000 tokens. An

0:24 August report found the worst case. The skill could not detect the project language, so it loaded all of them. C#, curl, go, Java, PHP, Python, Ruby, TypeScript, plus 26 shared markdown

0:38 files. 812 kilobytes. The reproduction task needed 33 kilobytes of that. The other 96% was freight, so the diagnosis writes itself. Skills got fat, context filled, agents got worse. Delete tokens,

0:52 fix the agent. That diagnosis is correct. It is also the smaller half of the problem and the half you can see. In May, Hong Wen Song and Song Wei at Databricks measured the other half. It

1:03 does not appear in your token counter. They loaded 202 skills into an agent instead of only the ones it needed. Pass rate fell 21 points. Then they did the thing the field had skipped. They asked

1:14 how much of that drop was actually the tokens. Here's their setup and the control is what makes it worth your time. Start with what they call Oracle skills, only the ones already proven to

1:24 help that exact task. That is the baseline. Then pad the library out to 52 skills, then 102, then 202 without removing anything that works. The task does not change. The helpful skill is

1:37 still sitting right there. The only thing that grows is the crowd it is standing in and the pass rate falls at each step, eight points, then 14, then 21 across two and a half thousand runs.

1:48 Then the split and this is the contribution. There are only two ways extra skills can hurt you. Either the agent picks correctly and executes worse because its

1:56 context got heavier, call that context overhead, or it picks the wrong skill in the first place, call that shadowing. So, the Databricks pair read every run, sorted it by which skills the agent

2:08 actually invoked, and measured the two effects separately. The paper says no prior work had decomposed it. Skills were being evaluated one at a time against a no skill baseline. The two

2:18 numbers behave nothing alike. Context overhead comes in at six points, six points, seven points. It barely moves while the library quadruples, and its confidence interval does not clear zero.

2:29 The authors will not claim it is real. Their word is suggestive. Shadowing goes three, then eight, then 14. It scales with the library. At the largest one, it is 2/3 of the whole

2:40 drop, and across the bigger libraries, it is the only component the statistics can confirm. Their conclusion is one sentence, and I will read it exactly. The skill selection failure, not the

2:50 enlarged context, is the primary bottleneck when expanding the skill libraries. And the behavior underneath that number is stranger than a wrong pick. With only the correct skills

2:59 loaded, the agent invoked the correct ones in 88% of runs. At the largest library, that falls to 52.6. So, where did the other third go? Not mostly into wrong skills. Wrong skill runs went from

3:12 zero to about 9%. The big migration was into runs where the agent invoked nothing at all, 12% up to 38.5. Abandonment, not error. The agent is not making a mistake there. It is looking at

3:25 a shelf of 200 jars, recognizing nothing on it, and doing the job from scratch the hard way. Your skill library did not get overruled. It got skipped, and your logs will not say so. One row of their

3:36 table makes it concrete. The task is called Mario coin counting. The correct answer is a bundle of three skills, FFmpeg, image editing, and an object counter, and all three are loaded and

3:48 available. Also loaded is a skill called video frame extraction, written for a completely different task, whose description happens to match the words of the query better. The agent picked it

3:58 in 26 runs out of 26. So, why does a system that can read your entire codebase fail to recognize a folder somebody handed it? The answer is not incompetence. It is the design, and

4:09 the design is a good one. It is the reason skills are cheap at all. Progressive disclosure. Three levels, and Anthropic's own documentation puts numbers on each. Level one is metadata,

4:20 the name and the description out of the front matter. Loaded at startup, about 100 tokens per skill. Level two is the body. Under 5,000 tokens, and it loads only after the skill has already been

4:32 selected. Level three is bundled reference material, and it costs nothing until something reads it. The documentation summary of all this, you can install many skills without context

4:42 penalty. That sentence is accurate. It is also the whole problem, because it is a claim about tokens, and it gets read as a claim about safety. The description is

4:51 not merely it is the entire ballot your router gets to read, which raises a measurable question. How much of the signal needed to route a task actually lives in a one-line

5:00 description? In March, a team published skill router, a system that routes across roughly 80,000 skills, and they ran an attention analysis on their own re-ranker.

5:10 Across 75 queries, the attention split like this. The name, 7.3%. The description, 1.0%. The body, the part progressive disclosure has deliberately not loaded yet, 91.7%.

5:24 Take the body away, and routing accuracy falls 37 to 44 points. They tried the obvious repair, distilling the body down into better descriptions, and it still came in 7 to 21 points short. The signal

5:36 does not compress into a label. One caveat, because it matters, that is their re-ranker on their benchmark, not Anthropic's model on your machine. What it establishes is where the routing

5:46 signal lives, not how any particular vendor routes. The mechanism generalizes. The exact numbers do not. And out in the wild, that 1% is often not filled in at all. A study of 55,000

5:59 public skills found 26.4% shipping with no routing description whatsoever. A skill with no description cannot be chosen on purpose. It can only be chosen by accident. In July,

6:11 researchers collected 3.8 million skill files off public GitHub, spread across 282,000 repositories. Half of them were verbatim copies of each other. That is the shelf

6:21 your agent is now expected to read, which brings us to the sentence that earns this video its title. And Anthropic wrote it themselves in the Claude Code docs under a heading you

6:30 would not go looking for. Skill descriptions are cut short. The listing has a character budget scaled at 1% of the context window. Names fit. Descriptions do not. When it overflows,

6:42 Claude Code shortens them, and I am quoting, which can strip the keywords Claude needs to match your request. It drops them starting with the skills you invoke least. So installing skill 51 can

6:52 shorten skill 12 until it stops matching. That is the title in shipped code. So what do you actually do about it? Three moves in rising order of effort, and not one of them is delete

7:03 your skills. The first is the one people hate, and it has the best evidence behind it. Cut the shelf. In November 2025, two engineers at GitHub, Anisha Agarwal and Connor Peat, trimmed

7:15 Copilot's default tool set 40 built-in tools down to 13. Success rates went up two to five percentage points on SWE Lancer and SWE Bench verified. Those runs used GPT-5. The other model was

7:28 Sonnet 4.5. Both measured back in November 2025. Response latency fell 400 milliseconds. Their own line for it, "Too many tools doesn't always make it smarter. Sometimes it just makes it

7:40 slower." The tools they cut are not gone. They are routed by embeddings, and Agarwal and Pete measured that, too. 94.5% tool use coverage from embedding routing against 69% from the static list

7:52 it replaced. Move to search instead of load. Five days after that GitHub post, Anthropic shipped tool search on its API. And the way the docs ordered the

8:02 benefits is the giveaway. Bullet one is tokens, over 85% off a 55,000 token setup. Bullet two is this. The model's ability to pick the right tool degrades once you exceed 30 to 50 available

8:15 tools. That is a vendor writing down the shadowing result in its own documentation months before anyone measured it in a paper. Their November 2025 evaluation moved accuracy on tool

8:26 use on Opus 4.5 from 79.5 to 88.1. Same tools, same model. The only change was that the model had to go and find them. It runs today on Opus 5, Fable 5, and Mythos 5. Claude code switches it on for

8:42 you once MCP tool descriptions get large, which tells you Anthropic has already conceded this argument for tools. Skills are the same shape, one layer up. Move three, and hardly any

8:53 harness does it yet. Route on the content, not on the label. Skill router is a small encoder and a small re-ranker, 1.2 billion parameters between them, that actually read the

9:03 body. 74% top one routing across 80,000 skills. The 16 billion parameter pipeline it beats manages 68. 13 times fewer parameters, 5.8 times lower serving

9:16 latency. The winning move was not a bigger router. It was letting the router read the thing it is routing. So, here is where I land without hedging it. Retrieval beats installation, and it is

9:26 not close. For most people running an agent today, the right number of skills in front of it is small and chosen by a search step, not large and chosen by whatever you happen to install. Three

9:37 receipts, 202 skills cost 21 points of pass rate, and 2/3 of that was the agent picking wrong. 91.7% of the routing signal sits in a body the router is not shown. And a shipping harness truncates

9:50 descriptions by design, starting with the ones you use least. Now, the case against me, because it is a real one. The Data Brick study is two models from one provider on a library that tops out

10:00 around 200, and they say so in their own limitations. Their contest overhead interval is wide enough that they refuse to call it zero. At 10 times that library size, tokens

10:11 could come back as the villain, and retrieval is not a cure, either. On the 20th of August, a benchmark landed that measures the failure which survives it, and it is a specific and nasty one.

10:21 Public skill routers find the helpful skill roughly 85 to 89% of the time in their top three. In about a third of those same top three, they also surface a same

10:31 capability sibling, right family, wrong execution contract. The fix moves you from wrong shelf to wrong jar. The villain was not a company, it is a habit. Treating skills like apps, where

10:42 one more install is free, and the pile is an asset. It is a search index that was never built, queried by a model that is not allowed to read it. So, which do you want your routing layer

10:52 to be? A vendor default you cannot inspect, or something you own and can measure?