The harness is the product. GPT-6 Astra just proved it.
GPT-6 Astra launched on 3 September, and for about two hours the story was simple. New flagship model. 99.9% on ARC-AGI-3, the benchmark most people agree is the hardest one going. Greg Brockman called it the start of the AGI era. The number did the rounds and the takes wrote themselves.
Then ARC Prize published its own verification of the same model. Same benchmark, same week. Under their standard conditions, Astra scored 62.7%.
Both numbers are real. Nobody corrected anything, because nothing was wrong. The model didn’t change between the runs. The harness did.
I’ve been chewing on this all week. If you buy or govern AI systems, I think it’s the most useful thing to happen to benchmark reporting in years, and most of the coverage has treated it as a story about the model when it’s really about everything around the model.
What ARC Prize actually ran
ARC Prize tests models two ways and reports both, which is the only reason we know any of this.
The standard harness is their level playing field: a minimal, provider-neutral interface, so every model faces an identical test. Under it, Astra scored 62.7%. That run cost $26,098 at maximum reasoning effort.
The provider adapter is OpenAI’s own setup, the Responses API with the settings they chose. It keeps the model’s reasoning state alive between steps and compacts a long run instead of letting it overflow. Under that, 99.9%, for $18,817 at high effort.
One wrinkle before you quote a 37-point gap at anyone. The effort settings differ between those headline runs. Matched at max effort, the adapter run hit 98.6% for $17,332. So the honest version is slightly narrower and still remarkable. Same weights, same benchmark, same verifier, and 36 points of difference coming purely from the machinery wrapped around the model.
The lower number deserves a second look too. 62.7% isn’t disappointing. It’s the best standard-harness score anyone has recorded. Opus 5 sits at 30.2. GPT-5.6 Sol at 7.8. Astra is a strong model either way, and that matters, because you can’t configure 36 points into a weak model. The gap only shows up because the model underneath is good.
What the adapter changes
OpenAI says what the machinery is, in a footnote to the result. There are two parts.
The first is persistence. In a long agent run the model reasons at every step: partial results, dead ends, working theories. The standard harness treats all of that as transcript. When the context window fills, the earliest tokens get truncated and the run carries on without them. The Responses API harness keeps the reasoning state alive in a form the model can reuse, so step forty can build on what step twelve worked out.
The second is compaction. When the context gets crowded, something has to decide what to keep, what to compress and what to drop. That decision isn’t neutral. Done well, it preserves exactly the state the next step needs and the model behaves as though it never forgot anything. Done badly, or not at all, the same model trips over work it did ten minutes ago.
Two settings, and 36 points live in them.
None of this is a scandal. ARC Prize built the standard harness on purpose, on the argument that a general system should be able to solve hard tasks through a minimal interface. OpenAI disclosed which configuration produced its number. Everyone behaved well and the numbers still landed miles apart. Good faith isn’t the question. Ownership is.
Why this should change how you buy
If your procurement treats the model as the product, you have a structural problem. The number in the deck describes a system. Some of that number is the weights. Some of it is a configuration the vendor controls, operates and can change without asking you.
So who owns the harness? It decides what the agent remembers, which tools it can reach, how it recovers from a failed step and what a run costs. Whoever owns it makes those calls, and if your contract is for model access, that isn’t you.
Where does it run? The best measured configuration of Astra lives inside OpenAI’s platform. A week after launch they made the harness itself available as the Agents API, with US data residency and no zero data retention option. For a lot of work that’s a perfectly reasonable arrangement. For an organisation whose rules say data and processing stay in a particular jurisdiction, it’s a constraint procurement will flag straight away, and in that case the model may be available to you while the 99.9% configuration isn’t.
And what does it log? The reasoning state behind the 98.6% is opaque by design. You can’t inspect it, and neither can an evaluator. If a regulator asks why your system gave a particular answer at a particular moment, that’s the layer that would have to explain itself.
None of this makes Astra a bad product. It makes the benchmark number a property of a system you might not be buying.
The part the launch coverage mostly missed
Nothing in that adapter is magic, and none of it has to happen in someone else’s cloud.
Persistence is a runtime decision. So is compaction, and so is tool scaffolding matched to the model. All of it can be done inside your own boundary, on your hardware or in your tenancy. You give up the vendor’s tuning, and in exchange you get to tune for your own workloads, which the vendor was never going to do for you anyway.
That’s the layer we’re building at Deep Thinking. Kizuna is where agents build software under rules you set, with a record of what they did that your auditors can read. Voxeltron is where the result runs, on your own ground. Spend is capped before it happens rather than reported afterwards, and the model is a slot you can swap, so a setup tuned for Astra this month can run K3 or an MIT-licensed DeepSeek Flash next month without your controls changing.
Those 36 points don’t have to come bundled with someone else’s cloud.
Four questions for the next deck
ARC Prize did everyone a favour by publishing both numbers instead of only the flattering one. The score belongs to the system, and the system is what you’d be running. So the next time a benchmark lands in a slide deck, I’d ask four things before it goes anywhere near a business case.
- Which harness produced this number: a provider-neutral setup, or the vendor’s own?
- Who operates that harness in production, and where does it run?
- What state does it keep between steps, and can my auditors see it?
- What happens to the number, and the cost, when the model underneath is swapped?
A vendor who can answer all four clearly is selling you a system. One who can’t is selling you a screenshot of someone else’s.