AI Knows When It’s Wrong Before You Do.

0:00 An AI's written answer isn't the only place to look for clues about whether it's getting the problem right. Researchers report two signals inside the models they studied, how promising

0:09 the reasoning looks, and whether the latest step improved that outlook. That could help an assistant decide when to keep working and when to check its answer. It matters because extra

0:18 reasoning costs time, and a longer explanation can still be wrong. Could something inside the model help choose a better next step? The paper offers an early demonstration. It first appeared

0:28 on February 1st, 2026, with a May 11th revision. This is resurfacing research. The researchers called the two groups value neurons and dopamine neurons. Dopamine is a neuroscience analogy for a

0:40 signal about changes in expected reward. These experiments don't establish feelings or consciousness. Start with a model halfway through a math problem. Inside each layer, it has a hidden

0:51 state, a list of numbers produced as it processes the text. The researchers train a small reader, called a probe, to extract a score from those numbers. Given the question and everything

1:01 written so far, how likely is this model to finish correctly? That's the state's value. A promising start can have high value even though there's still work left to do. Take a simplified example.

1:12 Pause at one paragraph and let the model finish 10 different ways. If four endings are correct, the estimated success rate is four out of 10. These are teaching numbers, not results from

1:22 the paper. Now, keep one additional reasoning step and try 10 completions from there. Suppose eight succeed. That step has changed the outlook from 40% to 80%. The current value is 80%. The

1:34 improvement is 40 percentage points. They answer different questions. A high value doesn't necessarily mean the latest step was useful. The model could repeat something it already established

1:43 and leave the outlook unchanged. Conversely, a helpful correction could improve a weak position without making the whole answer reliable yet. Suppose the next paragraph introduces an error

1:54 and only two of our 10 continuations succeed. Value falls from 80% to 20%, a change of minus 60 percentage points. The second signal is trying to capture that movement, including its direction.

2:06 The paper calls this a temporal difference error. Its formula also includes a discount factor set very close to one in the experiment. For our example, setting it to one makes the

2:16 subtraction easy to see. At the end, the estimate meets the actual outcome. If the model expected success but produces a wrong answer, the final reward is zero. That mismatch provides training

2:27 feedback. Reward has a concrete meaning here. Did the answer pass the task check? So, how do you locate that information? For the value signal, the team runs

2:36 problems through an existing language model and checks the final answers. They record hidden states along the way, then train the probe using those states and rewards.

2:45 The probe connects neighboring estimates through temporal difference learning, with the final answer providing the outcome. The model supplies the internal numbers. The added probe learns to read

2:54 them. The score doesn't arrive as a ready-made dashboard. The researchers rank input coordinates by their connection weights, remove the weakest, and test the probe again. They

3:04 keep narrowing what it can see. In their pruning experiments, prediction often holds up with only about 1% of the coordinates remaining. That's the sparse part of the result. Here, neurons refer

3:14 to coordinates in the hidden states being examined. The experiment reduces what the probe reads, which means it doesn't show that you can delete 99% of the model and keep the same assistant.

3:24 On the original pruning curves, the horizontal axis tracks how much input has been removed. The vertical axis measures how well the score separates eventual correct answers from wrong

3:33 ones. The curve stay fairly level across much of that removal. But does this part of the model matter to its reasoning? To investigate, the team performs a different experiment. They set selected

3:43 activations inside the language model to zero and check its answers. In their 2026 study, they use a 7 billion parameter Qwen model fine-tuned for reasoning. On the MATH 500 dataset, the

3:55 original accuracy is 75.2%. When they zero the selected 1% in layer three, it drops to 13.6%. Zeroing the same proportion at random in that layer leaves accuracy at 73.4%.

4:09 The paper also compares against other selection methods, including neurons picked by a probe trained to predict the next token. The reward-selected group causes much greater damage in this test.

4:19 That strengthens the case that these coordinates matter without proving each has only one job. Disabling part of a network is a fairly blunt instrument for discovering what it normally does. For

4:29 the second signal, the researcher split responses at paragraph boundaries, sampled possible completions, and checked the resulting answers. Comparing success estimates before and after each

4:39 paragraph gives them a training target for whether it helped. A second probe learns to predict that change from the model's activations. The team then prunes its inputs, too. This is a

4:48 learned shortcut to the expensive completion and checking process used to construct its training targets. There's a useful limit in the method. For this analysis, they keep steps whose

4:58 estimated change is larger than 0.3 in magnitude. So, the reported correlation is measured on selected larger changes. It doesn't establish that the probe can reliably judge every tiny edit in a

5:09 chain of reasoning. The paper's example traces show an activation peak near a useful reasoning move and a trough near a mistake. But, to help an assistant, the score has to improve a decision.

5:20 Here's the decision they test. At a paragraph boundary, the model generates four candidate continuations. The probe scores each candidate, and the search keeps the highest-scoring one as the

5:30 next reasoning step. Then, it generates candidates from that updated state and continues until a final answer. Picture four branches growing from the same unfinished solution. One gets selected,

5:40 and the other three stop there. At the next boundary, the retained branch gets another set of possible continuations. This is extra work during answer generation using the internal signal to

5:51 decide where that work goes. The comparison includes random selection, ordinary greedy decoding, and another reward-based scoring method. Random selection asks whether making candidates

6:01 is enough. Choosing with the score should beat picking a branch without it. In this test, the researchers report 72.2% accuracy for both greedy decoding and

6:10 random selection. Their dopamine probe reaches 77.8%. That's a gain of 5.6 percentage points on the validation subset of math 500 averaged over three random seeds. The

6:22 other reward-based method reaches 75%. These are the authors' results, not a benchmark reproduced for this video. They don't tell us how every model or task would behave. There's a bill

6:33 attached to the search because four candidates require generation work that the single candidate baseline doesn't do. A small probe can be cheap to read while the whole procedure still costs

6:42 more. To choose it for an application, you'd want accuracy compared at a match time or token budget. The value signal suggests a different decision. Where should extra effort go in the first

6:52 place? The paper test confidence prediction before the model has written an answer. A useful early estimate could help a system decide which questions deserve

7:00 additional checking, although that complete routing policy isn't demonstrated here. Across the models and benchmarks in its confidence table, the sparse probe has an average AUC of 0.67.

7:12 Asking the models to state their confidence averages 0.52. AUC measures how well a score ranks successful cases above unsuccessful ones. 0.5 corresponds to chance-level ranking, so 0.67 does

7:24 not mean the assistant is correct 67% of the time. It also doesn't mean an individual score is a trustworthy probability. Ranking and calibration are different tests. A

7:34 score can put cases in a useful order while being too optimistic about all of them. For example, take a group of answers assigned 80% confidence. A calibrated estimate would mean roughly

7:45 80% of that group are correct. You have to compare predictions against outcomes to establish that. Naming a number confidence doesn't do the calibration for you. For one small model in the

7:55 paper's table, question length beats the sparse probe on both benchmarks. It's an awkwardly competitive colleague. Simple baselines deserve a place in your evaluation. There's earlier independent

8:05 work on predicting answer accuracy from the questions hidden states, too. Its authors report useful signals on knowledge questions and weaker generalization to mathematical

8:14 reasoning. That supports the broader idea of internal confidence information while showing why transfer to a new task needs its own test. If you wanted to build on this research, you'd need

8:24 access to the hidden states and a way to check answers. You'd train and select a probe on one set of problems, then evaluate it on separate ones. For code, that checking could include actual

8:34 tests. A favorable internal score wouldn't replace running them. The preprocessing needs attention as well. The paper describes normalizing activations using the whole response,

8:44 then averaging them by paragraph. A system that must score an unfinished dream needs an explicit rule for what information is available at that moment. That detail needs resolving before

8:54 treating the method as a live error alarm. The useful result is a research direction with a demonstrated search gain. Small sets of internal coordinates can provide signals about expected

9:04 success and changes in that expectation. For developers, they're candidates for guiding checks and allocating work. The evidence still has to come from the task the assistant is meant to perform. Go

9:15 back to those two readouts beside the unfinished solution. One asks how promising this position is. The other asks whether the last step helped. Keeping those questions separate

9:24 explains both the paper's mechanism and its practical appeal. Use the model's internal signals to choose what to examine next, then check the answer against the world outside the

9:33 model.