Enjoying this issue?
Get tomorrow's AI & engineering digest in your inbox — hand-picked, summarized, and always spam-free.
TLDR
Microsoft's Decision 1 is a decision model that never generates words: it reads the input and candidate options in one pass, then a small head scores each option, and that is why Microsoft claims it is about 35 times faster than GPT-6 Soul. AWS, Cloudflare, and OpenAI all ship the same single-pass idea, and the speed comes from skipping word-by-word generation rather than from a small model, since Decision 1 starts from a 9 billion parameter open language model. The tradeoff is that these models are weaker on complex tasks and always pick an option, so OpenAI's docs recommend adding a fallback such as 'other'.
Key points
Cloudflare publishes a single-pass design that reads the input and all candidate options together once.
Microsoft claims Decision 1 is about 35 times faster than GPT-6 Soul on benchmarks Microsoft chose and ran itself.
AWS's 2 billion parameter Strands decider replaces the word prediction head with a roughly one million parameter scoring head.
AWS reports median latency of about 153 milliseconds on a MacBook M3 for small tasks.
AWS says its model is worse at complex problems and cannot handle coding, chatbots, or summaries.
Tools mentioned
Techniques
- Single-pass decision model that processes the input and candidate options together once
- Classification head over final hidden states that scores each option
- Softmax over option scores to select the answer
- Replacing the word prediction head with a small option-scoring head
- Adding a fallback option class such as 'other'
Stop scrolling. Start reading smarter.
Receive the day's most important AI & engineering updates in one concise email. No spam.
Transcript (captions)
Microsoft says its new Decision 1 model is about 35 times faster than GPT-6 Soul. And the idea behind it is a model that never writes a word. AWS, Cloudflare, and OpenAI ship the same
idea within weeks of Jev, the startup model that started it. So, why would skipping the words buy that much speed? The obvious guess is a tiny model, and that's wrong. Decision 1 starts from a 9
billion parameter can, an ordinary open language model. The saving is the loop. A chat model makes a pass for every word it writes. Cloudflare's published design makes one pass, total. Microsoft's post
doesn't show its internals, so here's the design Cloudflare publishes on its model card. Take a support ticket and four options: billing, bug, refund, other. The model reads a ticket and all
four options together once, the same prefill step a chat model does before it starts writing. Then a small head reads what the model worked out about each option, its final hidden states, and
scores every one. A softmax turns those scores into probabilities. Refund gets the biggest share, and that's the answer. Think of it as a multiple choice answer sheet. Nothing
gets written in the margin, so an answer can't be an option nobody printed. The picture breaks in one place, a student thinks before circling. Here, circling is the whole computation. AWS's 2
billion parameter Strands decider drops the word prediction head and adds a tiny head of about a million parameters that picks among the options. AWS reports a median of about 153 milliseconds on a
MacBook M3 for small tasks. Microsoft's 35 times is median latency against GPT-6 Soul, a general reasoning model on benchmarks Microsoft chose and ran itself. No third party has verified it.
AWS says its model is worse at complex problems and can't handle coding, chatbots, or summaries. Print a bad answer sheet and it'll still circle something every time. That's why
OpenAI's docs say to add a fallback like other. If you're building routing or triage, check the numbers on your own data. The next big AI might not talk at all. That's a prediction.