The Fatal Flaw in Modern AI (Exposed by a 1998 Game)

0:00 Watch this base. There's an enemy army walking straight into it, burning down buildings one at a time, and the player just sits there, not pulling workers back, not throwing up a single defense,

0:10 nothing. It isn't lag, and it isn't somebody who walked away from the keyboard. It's one of the most capable AI models in the world, and it's losing to a video game from 1998 because it's

0:20 still thinking. The game is StarCraft: Brood War, a real-time strategy game from 1998. On top of it sits a benchmark called Brood War Bench. And the models it takes apart aren't toys. They're the

0:32 same frontier systems people trust to write production code and run long agent tasks all day. Here's how it started, in the words of its creator, Ben Swerdlow, one of the founders of a company called

0:43 Freestyle. He built a version of Brood War you could only play through an agent, just as an experiment to mess around with friends. And those friends, who'd played maybe a couple games of

0:52 StarCraft in their entire lives, did surprisingly well. So, the obvious next question asks itself. What happens if the players aren't casual humans, but the best AI you can pay to run? So, he

1:03 built the real thing. 19 different configurations made from Codex, Claude, and Grok, each one dialed to a different level of reasoning effort. Then he had every configuration play every other one

1:14 in a full round robin, 18 games apiece, all running in parallel on his company's cloud machines. And then he wrote up exactly what happened. The headline finding is brutal in how plainly he says

1:24 it, "Quote, none of the models played beyond a beginner level. Not the winner, not the expensive ones, none of them." He put it even more bluntly than that, "A beginner doing a photon rush, the

1:35 cheap cannon cheese that anyone learns in their first week, would beat every single one of these agents." Now, there is a top of the leaderboard. Codex Astra, on its highest effort setting,

1:45 went 18 and 0, a clean sweep. Claude's fable model landed near the top at 15 and 3. And at the very bottom, a couple of configurations. Claude's smallest model and Grok on low effort when zero

1:57 and 16. They lost basically everything. So, there's a real ranking here. It's just a ranking of who is least bad at a game a 10-year-old can pick up in an afternoon.

2:07 And to understand why it's this hard for them, you have to know that StarCraft is really two games stacked on top of each other. There's the slow game, which is the economy. You mine minerals, you

2:17 build workers, you expand to new bases, you research upgrades, you decide what kind of army to make. That part plays out over minutes and it rewards patient planning.

2:27 Then there's the fast game, which is the fight. Two armies slam together and the winner is usually whoever controls better in the moment. Pulling a wounded unit out, focusing fire, reacting inside

2:37 a fraction of a second. A strong player is running both of those at the same time. These agents struggled with both, but for very different reasons. Take the slow game first because this is the part

2:48 where a reasoning model is supposed to shine. It's basically a planning problem and planning is their whole reputation. But, Swarm learns own summary is that even the top agents couldn't build

2:57 complex armies, couldn't defend simple attacks, and couldn't hold to a real strategy. The benchmark tracks all the tells, the worker count, the army size, the

3:07 upgrades bought, the minerals just sitting unspent in the bank. And across the games, those numbers stay low and flat. The agents weren't out thought at the strategy layer. They mostly never

3:16 built the economy that lets you field a real army in the first place. So, the strategy is weak. But, the more interesting failure and the one the whole benchmark is really about is the

3:25 fast game. Because these are very smart models. So, why does a strategy game from the 90s dismantle them in real time? And the answer is a specific mechanical thing about what a live game

3:36 does to the way they think. Start with what controlling anything in real time actually is. It's a loop. You look at the world, you decide, you act, and then you do it again as fast as you possibly

3:46 can. StarCraft runs on a clock that doesn't pause. Whether or not you've made up your mind, the next second happens anyway. The enemy keeps moving, your buildings keep burning. The harness

3:56 itself gives each agent exactly what a game bot needs, and this part works fine. There's one tool to read the game state, and only the part it's actually allowed to see, because StarCraft hides

4:06 most of the map under fog of war. And there's a second tool to give orders to its units. Read the board, issue commands. No problem there. The problem is the step in the middle. A reasoning

4:17 model makes a decision by writing. It generates its thoughts as text, one token at a time, and then it generates its commands the same slow way. That takes real seconds, and on the higher

4:28 effort settings, it takes a lot of them. And every one of those seconds, the game clock is still running underneath it. So, by the time the order finally goes in, the situation it was reacting to is

4:38 already old news. Swerdlow actually names the deeper version of this. A lot of the models, he says, played the game as if it were turn-based, as if the world would politely freeze and wait for

4:48 their move, the way a chessboard does. It doesn't. And so, in his exact words, they got destroyed while they were thinking. Honestly, that one sentence is the whole finding.

4:58 These are models trained on a world that waits, dropped into a world that flatly refuses to. There's a single game in the logs that captures it perfectly. Grok, on its highest effort setting, in one

5:08 match, spent over 11,000 reasoning tokens. It was thinking extremely hard. And in 43 minutes of game time, it issued six batches of commands. Six. It never built a single fighting unit

5:20 the entire game. All of that deliberation, and it barely touched the board. You can put an actual number on the speed. In actions per minute, the winning agent, Codex on high effort,

5:31 averaged around 13. Grok on high effort averaged under three. And for comparison, when DeepMind built its own StarCraft AI a few years back, they had to cap it at roughly 260

5:42 actions a minute, just to keep it fair against human pros. These agents aren't in the neighborhood of that ceiling. They're barely off the floor. Go back to that base being torn down at the very

5:52 start. That's the first failure and it's the simplest one, response time. Something attacks you and the correct move, pull your workers, put down a defensive building, is completely

6:02 obvious. But obvious still has to travel through a whole paragraph of generated reasoning before it becomes an actual click. And by the time the model has decided to

6:11 defend, there is very often nothing left to defend. The second failure is observation. A human watches the screen continuously. You can feel an attack building before it even lands. The agent

6:22 doesn't watch. It takes a snapshot. It reasons about that snapshot and then it acts. But the snapshot goes stale the instant the game moves on without it. So it isn't only slow to react, it's

6:33 reacting to a version of the game that stopped existing several seconds ago. The third failure is coordination and this one's almost funny. To manage everything at once, the harness splits

6:43 the job across smaller sub agents, one to handle the army and so on. But they don't share a single brain. Swerdlow describes the army agent feeding each new unit straight into a

6:53 fight one at a time with no idea that a bigger force was being gathered somewhere else. So instead of one army hitting together, you get a slow trickle of soldiers walking into their deaths

7:03 one by one. Someone in the discussion afterward put it in a way that stuck with me. StarCraft, they said, is a game about actions per minute. It's about speed.

7:12 And using a slow heavyweight reasoning model to play it is a little like trying to knit with an excavator. All that raw power aimed at exactly the wrong kind of motion for what the job needs. That's

7:23 less a knock on the models than a plain statement about a mismatch. Now, if you know this field at all, you're already objecting because AI has beaten StarCraft convincingly. Back in 2019,

7:35 DeepMind's AlphaStar reached Grandmaster in StarCraft II, the top league on the actual ranked ladder against actual human players. So, the problem clearly is not that AI

7:45 can't do StarCraft. The problem is which kind of AI. And the gap between AlphaStar and these agents is really the entire point. AlphaStar wasn't reasoning in words at all.

7:54 It was trained on thousands and thousands of human replays until it learned to map what it sees directly onto what it does. See the situation, react instantly. No sentences in

8:04 between. It's closer to a reflex built for this one game. These agents are the exact opposite. They're general-purpose reasoners that solve a problem by talking themselves

8:14 through it. And that talking is precisely what makes them so good at writing code. It's also precisely what a real-time game punishes. And to be fair to these models, that contrast is the

8:24 actual lesson worth taking. Hand the same Codex or Claude a real codebase and a nasty bug, and they'll methodically read the files, form a theory, test it, and fix it, often better than most

8:34 engineers would. But look at what that task hands them that StarCraft never will. All the time in the world. The repository doesn't rewrite itself while the model reads. The test suite waits

8:44 patiently. Thinking is pure profit there. It's only when the clock refuses to stop that the exact same behavior flips into a fatal weakness. And you might assume the fix is obvious.

8:55 Just let them think faster, or let them think more. But the data pushes back on that. More reasoning didn't rescue anyone. Grock on its highest effort, the one burning 11,000 tokens a game,

9:06 actually did less than Grock on low effort. And the model that won did it on the quality of its handful of actions, not the sheer quantity of its thoughts. Past a certain point, extra deliberation

9:17 in a real-time game isn't an advantage at all. It's just more time spent not playing. There's a cost angle here, too, and it's a little uncomfortable. That 18 and 0 winner cost about 10 and 1/2

9:27 dollars per game to run. The cheapest agent on the whole board cost around 30 cents a game, and it went 0 and 16. So, you You spend more than 30 times as much per match and still top out at

9:38 beginner who loses to a cannon rush. The money buys you a better beginner. It doesn't buy you a competent player. So, what's the honest takeaway and how far does it actually reach? Keep it narrow

9:49 because narrow is what the evidence supports. This is one person's benchmark, a deliberately minimal setup on these specific models with no chance for them to learn between games. It is

9:59 not proof that AI can't play StarCraft. AlphaStar settled that question years ago. What this shows is sharper and frankly more useful than that. It shows that the one thing these models are most

10:10 celebrated for, careful step-by-step reasoning, carries a hidden cost and that cost is time. On a coding task or a research task or almost any of the benchmarks these models sit on top of,

10:21 the world waits while they think. StarCraft does not wait. And the moment the clock stops waiting, the smartest model in the room can lose to a beginner for the simple reason that it's still

10:31 forming its answer when the game ends. Swerdlow, for what it's worth, doesn't think it stays this way. He points out that his setup gave the agents no memory. No best of five where an agent

10:41 could learn from a loss and start writing its own little automations for the mechanical parts, the way a human slowly builds muscle memory. Hand the fast reflexes off to code, keep the

10:50 model for the strategy, and the picture could look very different. He'd bet on superhuman play inside a year. But that's the thing worth leaving with, whether you build agents or you just use

11:00 them. The benchmarks that made these models famous almost all assume the same thing. That the world will hold still while the model thinks. Enormous parts of the real world don't. Anything with a

11:10 moving clock, live systems, robotics, a trading loop, a game, turns thinking time straight into a liability. And a strategy game from 1998 just gave us the cleanest demonstration of that gap

11:23 anybody's produced yet. The models aren't dumb, they're just slow in a place where slow gets you killed.