We took an AI SRE from 76% to 92% in five days. The only model swap made it cheaper.

Last week our AI site-reliability agent scored 76.1% on AIOpsLab, Microsoft’s benchmark for autonomous incident response. Five days later it scored 91.89%, a point above the best published number we know of, Cerebral Systems’ 91.39%. We did not train a model. We did not buy a bigger one. The only model change in the whole programme moved a third of the workload to a cheaper model, because the cheaper model was measurably better at that task.

This post is the full method. Not the sanitised version: the actual ladder we climbed, the ideas that died on the way, and the two places where we drew a hard line on what counts as honest. If you are evaluating agent vendors, the method matters more than the number, because the method is the part that transfers to your incidents.

The benchmark is a live cluster, not a quiz

AIOpsLab is worth taking seriously because it refuses to be a text benchmark. The harness deploys real microservice applications onto a real Kubernetes cluster, injects a real fault, and drops the agent into a shell. The agent gets the same tools your on-call engineer gets: logs, metrics, traces, kubectl. Then it is graded on four tasks. Did you notice something is wrong? Which service is at fault? What kind of fault is it? And the hard one: fix it, live, and the grader checks the cluster actually recovered.

There is no answer key in the room. A mitigation only scores if the pods come back healthy. That grounding is why we care about the benchmark at all: it is the closest public proxy for the work we build agents to do inside Substrate, where the output is not a paragraph but a changed system with an audit trail.

Cerebral Systems published 91.39% on it, running DeepSeek V4 Flash at 49 seconds per scenario, with per-stage scores of 94.12% detection, 100% localisation, 100% root-cause analysis and 93.3% mitigation (their benchmark page has the table). Our baseline scaffold, on the stronger DeepSeek V4 Pro, sat at 76.1%. Same family of models, fifteen points apart. Whatever separated us from them, it was not parameters.

The trap we refused to walk into

The obvious move when you are fifteen points behind is to guess. Rewrite the prompt, run the whole benchmark, see if the number went up. The full board is 222 scenario runs and the better part of a day. Guess-and-rerun burns a day per guess and teaches you almost nothing, because a two-point swing on one run is indistinguishable from noise.

So we banned it. Every candidate improvement had to climb a ladder instead.

*Interactive: candidate levers flow left to right. Most die at the cheap stages. The board only fires when the banked lower-bound gains cross six points.*

The ladder has four rungs. An idea enters as a hypothesis with a named mechanism and a named class of scenarios it should fix. First it faces an offline kill-check: we replay it against archived transcripts from previous runs, which costs cents and takes minutes. Does the evidence the idea needs actually exist in the data? Most ideas die right here. Our taxonomy rubric died here. A latency heuristic died here. An entire theory about timeout losses died here when we traced all fifteen timeouts and found every one had already been graded before the hang.

Survivors get two or three live probes on scenarios sampled randomly from the target class. Not hand-picked, sampled, because hand-picking your validation scenarios is how you lie to yourself politely. Then we do something slightly unusual: we do not celebrate three-for-three. We compute a Jeffreys lower credible bound on the flip rate, project it across the class, and bank that conservative number. When the banked projections sum past six points, and only then, the full 222-run board fires.

Five days of this produced exactly two board runs. Everything else was cents and minutes. If you take one thing from this post, take the ladder: the expensive measurement is a prize the idea earns, not a debugging tool.

The law we kept re-learning: prompts lose, mechanisms win

Here is the finding I would tattoo on the inside of every agent engineer’s eyelids. We proved it to ourselves five separate times, because apparently we needed five.

If you put a protocol step in the prompt, the model will skip it precisely when it matters.

The clearest case: some faults in this benchmark are invisible in traces and logs. A feature flag gets flipped, a service degrades quietly, and the noisy symptom appears somewhere else entirely. The evidence exists, in a config map an engineer would check. So we added the instruction. Check the flag state before you submit. The model fetched the config map, held the smoking gun in its context window, and then submitted the anchored wrong answer anyway, three times out of three. It preferred its first hypothesis to its own evidence.

The fix was not a better sentence. The fix was taking the decision away from the prompt entirely.

*Interactive: the same rule, once as prose in the prompt and once as a gate in the client. Only one of them survives contact with a confident model.*

Two mechanisms carried most of the fifteen points. The first is the digest: the client parses raw evidence itself, in code, and appends a two-line plain-language conclusion the model cannot misread. Not “here is 40KB of JSON”, but “flag productCatalogFailure is ON and its default is off; the injected fault targets the product-catalog service.” The second is the submit gate: when the model calls submit without having run the required checks, the client intercepts the call, runs the probe itself, materialises the digest, and only then lets the answer through. The model keeps its judgement. It loses the option of skipping the homework.

Once we had the pattern, it compounded. A restart-count digest stopped the model blaming every freshly deployed pod. An exit-code check taught it that a SIGKILL fourteen seconds after boot is an assassination, not a crash. Every one of these started life as a failed prompt instruction and became a working mechanism.

The evidence channel we had to make a call on

One class of scenario deserves its own confession, because it is where benchmark honesty gets tested.

The hotel-reservation network faults were unwinnable through the standard channels. We verified this properly: the target service emits zero trace spans, the logs stay clean, and every cross-service latency gap we measured was under a millisecond. The fault is injected through chaos-mesh, and the injection itself is the only observable evidence, a custom resource sitting in the application namespace, selector labels pointing straight at the afflicted service.

Is reading that cheating? We went back and forth. An on-call SRE running kubectl in that namespace would see the chaos experiment immediately; it is ordinary cluster state, and ignoring it would be malpractice. But it is also the fault injector’s own fingerprint. So we did two things. We drew a quarantine line in code: the agent never reads the benchmark’s answer files or per-scenario mappings, and an import-ban test enforces that. Live cluster state is fair game; the answer key is not. And we report two numbers: every session that used a flag-state or chaos-resource digest is marked in telemetry, so you can see exactly how much of the score leans on that channel. On the final board, localisation went 40 of 46 on marked sessions and 37 of 38 on unmarked ones.

We think that is the right line. We also think a vendor who will not show you their equivalent split is asking you to take rather a lot on faith.

The router: spend where the model is good

By day four we had two full boards measured on two models, which meant we had something better than intuition: a model-by-task scorecard. It did not say what the pricing page implies. The big model was not uniformly better.

*Interactive: tasks route to their measured winner. The dashed path is the one-shot escalation to a heavyweight reasoner, built and proven but off by default.*

Detection, the noticing task, went to V4 Flash: 91.7% against Pro’s 85.4%, at a fraction of the price, with fewer timeouts. Pro overthinks a yes-or-no question; it keeps investigating past the point of diminishing returns. Mitigation is the opposite story. Flash collapses there, 66.7% against Pro’s 95.2%, because live remediation under verification is genuinely hard reasoning. Localisation stays with Pro too. The router binds the model per task at session start, and it carried its solo numbers onto the final board almost exactly.

That is the sentence buyers should sit with. The cheaper model was better at a third of this benchmark. If your agent platform cannot route work by measured capability, you are paying flagship prices for tasks a smaller model does better.

We also built the dramatic version, an escalation path that hands a stuck session to GLM-5.2 with extended thinking. It works mechanically. It flipped nothing in probes, so it ships off by default. Not every clever idea earns its keep, which is rather the point of measuring.

The final board

One run of 222 scenarios, three repetitions of 74, every lever on, router live:

Task	Model	Score	95% CI
Detection	V4 Flash	88/96, 91.7%	84.4 to 95.7
Localisation	V4 Pro	77/84, 91.7%	83.8 to 95.9
Mitigation	V4 Pro	39/42, 92.9%	81.0 to 97.5
Overall	routed	204/222, 91.89%	87.5 to 94.8

The three repetitions landed at 93.2%, 91.9% and 90.5%. Every repetition cleared Cerebral’s 91.39% on its own, which matters more to us than the headline, because a single lucky run proves nothing.

Now the caveats, stated plainly rather than in a footnote. Our confidence interval includes 91.39, so the statistically careful claim is “at or above” the published number, with the point estimate above it. Their number includes root-cause analysis at 100%; ours above excludes it, and the next section explains why rather than hiding it. And their per-stage localisation is 100% to our 91.7%, at 49 seconds a scenario, which we frankly find more interesting than our own result.

The task we could not honestly win

Root-cause analysis on this benchmark is a classification: name the system layer and the fault type from fixed menus. Our score there is 53.8% over three full repetitions, measured separately and published alongside everything else. The repetitions came in at 7, 8 and 6 out of 13, which tells you something real: the model does not flip a coin on easy scenarios, it flip-flops on a handful of genuinely borderline classification calls.

We know precisely where the missing points live, because we decomposed every miss. None were formatting. None were vocabulary. Five scenarios are classification calls where the benchmark’s answer key disagrees with, as far as we can tell, every strong model’s reading of the same evidence. Both mongodb user-record faults, for instance: the model says authentication issue, the oracle says otherwise, and it says so consistently across two model families and a reflection pass. We ran the misses through GLM-5.2 with extended thinking and through Pro with a verify-before-commit loop. Zero of five, both arms.

There is a known cure, and we declined it. Map each scenario to its expected label and whisper it to the agent. That is memorising the answer key, our quarantine line exists to forbid exactly that, and a score bought that way would tell you nothing about the agent your incidents would actually get. So the analysis number stays where the evidence puts it, and folding it in gives a four-task composite of 86.2% against Cerebral’s 91.39. We publish both numbers. The three-task board is where we beat the published bar. The four-task number is what refusing to memorise the answer key costs, and we would make the same trade again.

What five days buys, and what it costs

The whole programme ran on two spare servers and DeepSeek’s API. An earlier full-registry run cost 64 cents in API fees, thanks to aggressive prefix caching; the final board came in around the price of a coffee. The expensive ingredient was discipline, not compute.

It is fair to ask what any of this has to do with Substrate, since we are not in the benchmark business. The answer is that the loop you just read is the product pattern. An agent that must check evidence before acting, through a gate it cannot skip. Every action graded against the real system, not against vibes. Honest telemetry about which evidence channel produced which outcome. A refuted list, so the same bad idea never costs you twice. That is how we run coding and operations agents inside the factory, and the benchmark week was those principles pointed at someone else’s scoreboard.

The scaffold beat the model, again. It usually does. If you want the longer version of why we think the next few years of agent engineering look like instrumentation and governance rather than parameter counts, the related posts below make the argument, and if you want to see the gates and digests running against your own infrastructure, get in touch.

Related reading: DeepSeek V4 just reset the price floor for agentic AI, human approval gates that don’t bottleneck, and cost governance before the invoice.