Four Agents, One Plan

A benchmark that compares models is a claim about the world, and the useful ones are built to embarrass their authors before publication rather than after. This one asked a deliberately narrow question. Take a plan, a decomposition of a research task into concrete work units, written once by a single strong model, and hand that identical plan to four different coding agents as workers. Which agent executes the plan best against a hidden answer key, and when a synthesis model pools the four agents’ output, does the pooling preserve the differences between them or wash them out? The narrowness is the point: this is not a test of whether any model is a good orchestrator, only of who executes a fixed plan and what survives aggregation. Four agents from four vendors ran the same frozen decompositions over an embargoed corpus, and the results are less a leaderboard than a lesson in how much a preregistered test refuses to certify.

One plan, four workers

The plan was fourteen task decompositions authored in advance by Claude Fable 5 and frozen behind a commitment hash before any worker ran. Each decomposition splits a research task into packet-sized work units; a worker reads a packet, searches a fixed document corpus through a metered file-drop channel, and emits structured evidence records, an entity, a criterion, a source document, and a verbatim excerpt, that are later scored against a gold key it never sees. Fable’s role ends at the plan. It is the orchestrator, not one of the workers, and the four workers are the contestants: GPT-5.6 Sol driving the codex CLI at maximum effort, Claude Opus 4.8, Claude Sonnet 5, and Grok 4.5. Two hundred and seventy-six scored runs, one realization each, no retries.

Every worker ran inside its own isolation so that nothing but the model and its harness varied. The two Claude workers ran under a filesystem jail with a tool allowlist; the codex worker under its own kernel workspace-write sandbox; and Grok inside a rootless kernel network jail, its own network namespace with the default route deleted, an nftables egress firewall, and a hostname-allowlist proxy as the only path out, because it was the one worker whose web channel could not otherwise be closed at the harness layer. The gold key, the manifest, and the scoring criteria were absent from every jail. That detail returns later, because the jail that made Grok’s run honest also, in part, made its score look worse than the model is.

The whole apparatus writes to a hash-chained, append-only ledger whose head hash was posted to a timestamped channel before any number was computed, so the outcome is committed before it can be tuned to a result. Every published figure below is recomputed in a witnessed mode that verifies the on-disk ledger against that posted hash and refuses to run if they disagree.

What the scores said

Scored on the primary decomposition across twelve tasks, the mean F1 ranks the four workers cleanly: GPT-5.6 at 0.663, Opus 4.8 at 0.641, Sonnet 5 at 0.529, and Grok 4.5 at 0.295. Read as a ladder, GPT-5.6 and Opus 4.8 sit together at the top, Sonnet 5 a step below, and Grok well beneath the field. The preregistered statistic is not the mean but the mean per-task F1 difference for each ordered pair of workers, and on that statistic the point estimates tell the same ordered story: GPT-5.6 over Grok by 0.368, Opus over Grok by 0.346, Sonnet over Grok by 0.234, GPT-5.6 over Sonnet by 0.134, and the two top pairs, Opus over Sonnet and GPT-5.6 over Opus, separated by a tenth and a fiftieth respectively.

That is the leaderboard, and it is the least trustworthy paragraph in this post.

What survived the honest test

The preregistration fixed the inferential test in advance, and it is not the one that reads the confidence intervals. The frozen test is a Holm-adjusted permutation across the family of all six pairwise contrasts; the bootstrap intervals are, by the same preregistration, a diagnostic of how sensitive a result is to which tasks happen to be in the suite, not a significance test. When I first wrote these results up, I reported four of the six gaps as significant, because their marginal ninety-five-percent intervals excluded zero. That was wrong, and it was wrong in the specific way that testing six things at once is a trap for anyone who tests them one at a time. Under the Holm correction, only one gap survives: GPT-5.6 beats Grok, at an adjusted p of 0.047, a hair under the line. Every other contrast, including the three whose marginal intervals looked clean, is unresolved at twelve tasks.

So the one thing this benchmark actually established is that the strongest worker beats the weakest. The fuller ordering, GPT-5.6 and Opus ahead of Sonnet ahead of Grok, is the honest point estimate and a reasonable thing to believe, but it is not something the data resolved, and the gap between “the intervals exclude zero” and “the family survives correction” is the whole difference between a result and a suggestion. A reviewer caught the overclaim before publication. I will come back to how.

Whether synthesis preserved the differences

The second question was the one the study was really built for. After the workers run, a synthesis model, Opus, at the CLI’s default sampling, pools each worker’s evidence for a cell and produces a single synthesized answer, scored against the same gold key with the same metric. The quantity of interest is attenuation: does the gap between two workers shrink when you look at their synthesized outputs instead of their raw ones? If synthesis rescues a weak worker or collapses a strong one, that shows up as a large attenuation.

It does not happen here. The synthesized per-worker means (0.662, 0.637, 0.484, 0.295) track the raw ones almost exactly, and every one of the six pairwise attenuations is at most 0.045 in magnitude. Synthesis left the ranking where it found it. The single visible movement is that Sonnet 5’s synthesized output scores about 0.045 below its own worker output, nudging every pair that includes it; the other three workers are unmoved to three decimal places, and Grok’s synthesized and raw scores are identical. I want to be careful about what this is: the analysis bootstraps the two gaps that make up each attenuation but never resamples their difference, whose uncertainty depends on a covariance the tool does not compute, so these are descriptive point estimates and not a tested effect. The honest statement is that, at this scale, synthesis neither rescued the weak worker nor collapsed the strong ones, and any effect it has is smaller than the study can see.

What Grok’s number actually measures

The most quoted number in a benchmark like this is the loser’s, and it is the one most likely to be misread. Grok’s 0.295 is not a clean measurement of extraction quality. Of its twelve primary cells, three were zeroed because it hit the six-hundred-second wall clock, every packet of one task, most of another two, while running at maximum effort inside the network jail, which adds real latency to every model call. But four more of its cells completed inside the budget and still scored zero, because the evidence Grok returned credited nothing in the gold key: genuine extraction misses, not timeouts. On the remaining five it scored between 0.47 and a perfect 1.0. So the 0.295 is a blend of a model that can do the task well when it engages, a model that sometimes finishes and produces nothing useful, and a jail-and-budget condition that cost it three cells outright. I ran no unjailed or longer-wall control, which means the portion of the timeout share attributable to the isolation overhead rather than to the model is simply unidentified. The number is honest for the condition it was measured under, and it is not Grok’s ceiling.

The bugs the benchmark caught in itself

Three implementation defects surfaced during the run, and the reason the headline number is trustworthy is that all three were caught before they could reach it.

The first two were the same bug wearing two coats. Output A, the worker score, pools the surviving packet runs of a cell and excludes only the ones that failed to complete, scoring a cell zero only when every packet fails. Output B, the synthesis, was refusing to run at all on any cell that had even one failed packet. On this run that split affected twelve of fifty-six cells, eight of which would have been handed a spurious zero on the synthesis side while their worker side scored normally, biasing the very attenuation the study exists to measure. What made it dangerous is that the asymmetry lived in two coupled places, the synthesis step and the scoring step, and fixing either one alone would have produced the identical wrong answer and looked entirely clean. Both had to be corrected together to match the worker side’s pooling exactly. I caught this in pre-synthesis validation, before spending a single synthesis call.

The third the review caught, not me. A grid in the scoring code filtered the task universe by condition but not by decomposition, so the secondary decomposition, which exists for only two tasks, was being scored as though it spanned all twelve, its pairwise statistics divided across ten zero-filled tasks that had no business in that universe. It moved one secondary contrast from a real 0.768 to a diluted 0.128. The primary decomposition, present for all twelve tasks, was untouched, which is why the headline held; but the secondary was quietly wrong until a reviewer reading the output against its source flagged it.

Two rounds of adversarial review, run by GPT-5.6 in a separate seat from any of the workers, produced the correction of the significance overclaim, the correction of the Grok causal story, an error in how I had counted the biased cells, and the grid defect. The reviewer’s verdict moved from publish-with-fixes to publish-ready only after each was addressed and a fresh pass confirmed no new error had been introduced. A benchmark whose author grades his own work needs an adversary who does not, and the value of the arrangement is measured entirely in the claims it deletes.

The friction is the apparatus

None of the above would mean anything without a record that a skeptic can check, and building that record was most of the work. The plan, the tool manifest, and the answer key were each hashed and committed before the run; the outcome, the ledger’s head hash, was committed after the matrix completed and before any score was computed; and the published numbers are bound to that head, with a negative control confirming that a wrong head is rejected. The gold corpus itself stays embargoed behind its hash, so the benchmark can be replicated without being memorized.

The apparatus was re-frozen five times, and every re-freeze is disclosed in the record rather than smoothed over. Two of them were availability failures that discarded partial runs: one subscription hit a five-hour session limit and killed two dozen worker runs mid-matrix, and a second died when an authentication token expired in flight and the failover logic recognized only the first failure mode and not the second. The fix was a no-trace failover that treats both as an availability event, retries the same run on a second subscription, and leaves nothing scorable behind, because the tamper-proof ledger that makes the outcome trustworthy is exactly what makes a partial run unrepairable in place, so the honest move is to discard and relaunch rather than patch. The other three re-freezes were the two coupled symmetry bugs and the grid defect. The worker matrix that produced the headline is byte-identical across the tool corrections that followed it, and its own committed head hash proves it predates them.

That is the shape of the thing. One resolved gap, a synthesis step that changed nothing, a loser’s number that means less than it looks like, and a correction history kept in the open because a benchmark that hides its retractions is just a leaderboard with better manners. The full method, the per-pair intervals and permutation p-values, the completion profile, and the entire apparatus-correction history are in the companion research report, so every figure here can be checked against its source.