Research
Which Native Coding Agent Executes a Fixed Plan Best, and Does Synthesis Preserve the Difference? A Preregistered Four-Agent Pilot with a Witnessed Provenance Ledger
A common pattern in agentic systems is to have one strong model decompose a task and cheaper or faster agents execute the pieces. This pilot isolates the execution half of that pattern. Fourteen task decompositions were authored once by a single strong model (Claude Fable 5) and frozen behind a commitment hash; four native coding agents (GPT-5.6 Sol via the codex CLI, Claude Opus 4.8, Claude Sonnet 5, and Grok 4.5) then executed the identical frozen plans as workers over an embargoed document corpus, each producing evidence records scored against a hidden gold key with a shared entity-criterion pair metric (Output A). A synthesis model (Claude Opus at default sampling) subsequently pooled each worker's evidence per cell into a single answer scored the same way (Output B), and the attenuation statistic contrasts the two. On the primary decomposition (twelve tasks) the workers rank GPT-5.6 (mean F1 0.663) and Opus 4.8 (0.641) above Sonnet 5 (0.529) above Grok 4.5 (0.295) on point estimate, but under the preregistered Holm-adjusted permutation test over the family of six pairwise contrasts only GPT-5.6 over Grok is resolved at the 0.05 level (adjusted p 0.047); the marginal bootstrap intervals, a preregistered suite-composition diagnostic rather than a significance test, exclude zero for three further pairs that do not survive correction. Synthesis preserves the worker ordering: all six pairwise attenuations are at most 0.045 in magnitude and the analysis computes no test for the difference, so no inferential claim is made. The lowest-scoring worker's value confounds three genuine extraction failures with three wall-clock timeouts under its network-jail isolation, with no unjailed control to separate them. Three implementation defects that would each have biased the result, two coupled Output-A/Output-B pooling asymmetries caught in pre-synthesis validation, and a decomposition-grid filter defect caught in publication review, were corrected before publication and are disclosed in full. The apparatus is a hash-chained, externally-timestamped ledger with pre-run plan commitment and post-run outcome commitment; every published figure is recomputed in a witnessed mode bound to the committed ledger head.
Abstract
A common pattern in agentic systems is to have one strong model decompose a task and cheaper or faster agents execute the pieces. This pilot isolates the execution half of that pattern. Fourteen task decompositions were authored once by a single strong model (Claude Fable 5) and frozen behind a commitment hash; four native coding agents (GPT-5.6 Sol via the codex CLI, Claude Opus 4.8, Claude Sonnet 5, and Grok 4.5) executed the identical frozen plans as workers over an embargoed document corpus, each producing evidence records scored against a hidden gold key with a shared entity-criterion pair metric (Output A). A synthesis model then pooled each worker's evidence per cell into a single answer scored the same way (Output B), and an attenuation statistic contrasts the two. On the primary decomposition (twelve tasks) the workers rank GPT-5.6 (mean F1 0.663) and Opus 4.8 (0.641) above Sonnet 5 (0.529) above Grok 4.5 (0.295) on point estimate, but under the preregistered Holm-adjusted permutation test over the six pairwise contrasts only GPT-5.6 over Grok is resolved (adjusted p 0.047). Synthesis preserves the ordering (all six attenuations at most 0.045, no test computed). The apparatus is a hash-chained, externally-timestamped ledger; every figure is recomputed in a witnessed mode bound to the committed ledger head, and the full apparatus-correction history is disclosed.
1. Introduction
1.1 The Execution Half of Orchestration
The multi-agent pattern in which a strong model plans and other agents execute is now common, but its evaluations tend to entangle two separable questions: whether the planner produced a good plan, and how well a given executor carries it out. This pilot deliberately fixes the plan and varies only the executor. A single strong model authors the decomposition once, that decomposition is frozen, and the measurement is entirely of execution: which native coding agent, handed an identical plan, extracts the required evidence from a fixed corpus most faithfully against a hidden reference, and what happens to the differences between agents when their outputs are pooled by a synthesis step. The estimand is narrow by design, "who executes this fixed plan," not "is the planner a good orchestrator", and the results are correspondingly modest in what they license.
1.2 Two Outputs and One Contrast
The study produces two scored quantities per cell, where a cell is a (task, worker, decomposition) unit. Output A is the worker's own evidence, pooled across the packet runs of the cell and scored against a gold key. Output B is a blinded, evidence-only synthesis of that same pooled evidence, produced by a separate model and scored against the same key with the identical metric. The contrast of interest is the attenuation, defined per ordered worker pair as the difference between the pairwise Output-B gap and the pairwise Output-A gap: it asks whether pooling and synthesizing a worker's evidence widens, preserves, or erases that worker's advantage or deficit relative to another.
1.3 Research Questions
Three questions organize the results. RQ1 asks whether the benchmark discriminates among the four workers on Output A, and under which inferential standard. RQ2 asks whether synthesis (Output B) preserves or attenuates the worker differences. RQ3 asks what the lowest worker's score actually measures, given that it ran under an isolation condition the others did not. A fourth, methodological, thread runs through the paper: what a preregistered familywise test and an adversarial review process refuse to certify that a naive read of the same data would assert.
2. Method
2.1 Tasks and Corpus
The suite is twelve closed-world enumerative tasks, each pairing a document corpus with a gold key of required (entity, criterion) evidence pairs. Two of the twelve additionally carry a second decomposition, yielding fourteen (task, decomposition) plans. The tasks contain no persons and no personal data; the corpus, gold keys, and scoring criteria are embargoed behind a single freeze hash (197 files) so that the benchmark can be replicated without being memorized. All plaintext gold is absent from every worker's execution environment.
2.2 The Frozen Plan
The fourteen decompositions were authored by Claude Fable 5, corpus-blind, and frozen behind a commitment hash before any worker ran (witnessed separately in the Phase A preregistration). Each decomposition splits its task into packet-sized work units; the workers pool across the packets of a cell. Fable is the orchestrator and authored the plan only; it is not one of the executor workers.
2.3 Workers and Isolation
Four native coding agents executed the plans, each in an isolation appropriate to its harness so that only the model and its own tool stack varied:
| Worker | Model | Isolation |
|---|---|---|
| codex | GPT-5.6 Sol (codex CLI, max effort) | kernel workspace-write sandbox |
| opus48 | Claude Opus 4.8 (claude CLI) | bubblewrap filesystem jail + tool allowlist |
| sonnet5 | Claude Sonnet 5 (claude CLI) | bubblewrap filesystem jail + tool allowlist |
| grok | Grok 4.5 (grok CLI, max effort) | filesystem jail inside a rootless kernel network jail |
The Grok worker is the one whose web channel could not be closed at the harness layer, so it ran inside a rootless network namespace with the default route deleted (leaving only an on-link slirp gateway), an in-namespace nftables egress firewall, and a host-side hostname-allowlist CONNECT proxy as the sole egress, restricted to the Grok API endpoints. The corpus is served to every worker through a metered file-drop broker; no worker's jail contains the gold key, the run manifest, or the scoring criteria. The isolation is operator infrastructure around an unchanged model stack, disclosed as such; Section 4.1 addresses the one place it plausibly affects a score.
2.4 Scoring: A Single Shared Pair Metric
Both Output A and Output B are scored by one function on the (entity, criterion_id) pair key space, so that the two share granularity and denominators exactly. A predicted record credits a gold pair (E, C) if and only if its entity normalizes and alias-maps to the canonical entity, its excerpt is verbatim in the cited source document, the entity occurs at a word boundary within that excerpt, and, the binding rule, the cited source document is among the gold pair's evidence documents. The criterion is derived from the cited document's gold binding rather than from any free-text criterion label, so phrasal variation in a criterion string never produces a spurious false positive. True positives are credited gold pairs, false positives are predicted pairs that credit nothing, false negatives are uncredited gold pairs; the primary statistic is the mean per-task F1 difference per ordered worker pair.
A cell pools the evidence of its surviving packet runs. A packet run that did not complete (wall-clock cap, tool-call cap, or a parse failure) is excluded from the pool with its stop reason retained; a cell scores F1 0 only when every one of its packet runs is excluded. A completed cell whose surviving evidence credits no gold pair also scores 0, a genuine extraction miss, distinct from a non-completion.
2.5 Synthesis (Output B)
Output B is a blinded, evidence-only synthesis: for each cell, a synthesis model (Claude Opus, invoked through the claude CLI at the CLI's default sampling) receives the cell's pooled worker evidence and the decomposition's synthesis template, with worker confidence fields stripped, and produces a single answer that is scored against the gold key with the Section 2.4 metric. The CLI cannot pin temperature or top-p, so synthesis is non-deterministic; a preregistered variance subset (three cells per decomposition, two extra calls each) quantifies answer-set instability separately. Exactly one primary synthesis is permitted per cell, the answer is locked on first append and cannot be re-run, which forecloses selecting a favorable synthesis after seeing its score.
2.6 Provenance
Execution writes to a hash-chained, append-only ledger; each entry's hash chains the previous, rooted at the committed plan-manifest hash. The plan manifest, the tool manifest, and the gold-corpus freeze were each hashed and committed to a timestamped channel before the run. After the worker matrix completed, its ledger head hash was committed (the Output-A outcome commitment) before any score was computed; the synthesis entries were then appended and the post-synthesis head committed (the Output-B outcome commitment). Published numbers are produced only in a witnessed mode: the analysis verifies the on-disk ledger's computed head against the committed hash and refuses to run on a mismatch, and a negative control confirms that a wrong head is rejected. The trust model is public self-commitment, the same as the preregistration itself; cryptographically signed appends or an external transparency log would be strictly stronger and are noted as un-built.
3. Results
3.1 Output A: Worker Execution (RQ1)
On the primary decomposition (D1, twelve tasks, four workers), the per-worker mean F1 is GPT-5.6 0.663, Opus 4.8 0.641, Sonnet 5 0.529, Grok 4.5 0.295. The preregistered pairwise statistic and its two resampling views are:
| Pair | ΔA | 95% bootstrap CI | Holm-adjusted p | resolved at α=.05 |
|---|---|---|---|---|
| GPT-5.6 − Grok | +0.368 | [0.170, 0.572] | 0.047 | yes |
| Opus − Grok | +0.346 | [0.101, 0.597] | 0.105 | no |
| Sonnet − Grok | +0.234 | [0.072, 0.399] | 0.078 | no |
| GPT-5.6 − Sonnet | +0.134 | [0.053, 0.216] | 0.078 | no |
| Opus − Sonnet | +0.112 | [−0.069, 0.287] | 0.563 | no |
| GPT-5.6 − Opus | +0.022 | [−0.127, 0.166] | 0.766 | no |
The point estimates rank GPT-5.6 > Opus > Sonnet > Grok. The bootstrap intervals, which the preregistration designates a suite-composition sensitivity diagnostic, not a significance test, exclude zero for four pairs. But the frozen inferential test is the Holm-adjusted permutation across the family of six contrasts, and under it only GPT-5.6 over Grok is resolved (adjusted p 0.047). The three pairs whose marginal intervals exclude zero but whose Holm p exceeds 0.05 are precisely the multiplicity gap that a per-pair interval conceals across six simultaneous tests. The established claim is therefore that the strongest worker beats the weakest; the fuller ordering is a point-estimate suggestion the twelve-task suite did not resolve.
3.2 Output B: Synthesis Attenuation (RQ2)
Synthesized per-worker means (GPT-5.6 0.662, Opus 0.637, Sonnet 0.484, Grok 0.295) track the worker means closely, and all six pairwise attenuations are small:
| Pair | ΔA | ΔB | attenuation |
|---|---|---|---|
| GPT-5.6 − Grok | +0.368 | +0.367 | −0.001 |
| GPT-5.6 − Opus | +0.022 | +0.025 | +0.003 |
| Opus − Grok | +0.346 | +0.342 | −0.004 |
| Opus − Sonnet | +0.112 | +0.152 | +0.041 |
| GPT-5.6 − Sonnet | +0.134 | +0.177 | +0.044 |
| Sonnet − Grok | +0.234 | +0.189 | −0.045 |
The point-estimate ranking is unchanged by synthesis. The analysis bootstraps ΔA and ΔB separately but does not resample their difference, whose uncertainty depends on the paired A/B covariance the tool does not compute; each attenuation is therefore a descriptive point estimate, not a tested effect, and no inferential claim about attenuation is made. Descriptively, the only movement is Sonnet 5's synthesized output scoring about 0.045 below its worker output, which nudges each Sonnet pair; the other three workers are unchanged to three decimals. At this scale synthesis neither rescued the weak worker nor collapsed the strong ones.
3.3 Completion Profile
Of 276 worker runs, 242 completed and 34 were legitimate non-completions: 21 Grok wall-clock caps, 5 Sonnet 5 and 5 Opus parse failures, and 3 codex tool-call caps. There were zero harness errors, and zero availability events (session limits or authentication failures) left in the ledger, by design. The ledger verifies clean, with zero tamper or provenance exclusions.
3.4 Secondary Decomposition (D2)
D2 exists for two tasks only. After the grid-filter correction (Section 5), the binding D2 analysis is computed over its correct two-task universe (per-worker means: Opus 0.861→0.885, Sonnet 0.854→0.895, GPT-5.6 0.768→0.801, Grok 0.000→0.000). At two tasks the Holm permutation resolves nothing (all adjusted p equal to 1.0); D2 is reported for completeness and supports no claim.
4. Limitations
4.1 The Lowest Worker's Score Is Confounded (RQ3)
Grok's 0.295 mixes distinct failure modes. Of its twelve D1 cells, three were zeroed by the 600-second packet wall (all at maximum effort inside the network jail, which adds latency to every model call), four more completed within budget but credited nothing in the gold key (genuine extraction misses), and the remaining five scored between 0.47 and 1.0. No unjailed or longer-wall control was run, so the portion of the timeout share attributable to isolation overhead versus model or provider latency is unidentified. The value is honest for the frozen, disclosed condition and is not the model's ceiling.
4.2 One Retained Realization; Two Discarded Matrices
There is one retained binding run per (task, decomposition, packet, worker). Two earlier partial matrices were discarded wholesale after availability failures and retained only as incident evidence: one aborted at 120 of 276 runs when a subscription session limit killed 24 worker runs, and one stopped at 79 of 276 when an authentication token expired mid-run. The reported intervals resample the retained realized runs at the task level and are not run-to-run variance.
4.3 No Test for the Attenuation Itself
As stated in Section 3.2, the frozen analysis does not compute a confidence interval or test for ΔB−ΔA. The Output-B conclusion is descriptive.
4.4 A/B Divergence Under Tampering
On a tamper or provenance exclusion (a mutated-evidence hash, a run absent from the ledger, an invalidated entry), Output B rejects the whole cell while Output A pools survivors, Output B is intentionally stricter. This can diverge only under tampering, which the witnessed head and hash chain detect; the honest run has zero tamper reasons, so on this data A and B pool identical sets and the attenuation is unconditionally apples-to-apples.
4.5 Trust Model
The witness is a public self-commitment, the same trust model as the preregistration. Signed appends or an external transparency log would be strictly stronger and are un-built.
5. Apparatus Corrections (Full Disclosure)
The apparatus was re-frozen five times; each re-freeze is in the record. The first two were availability failures (Sections 4.2), remediated with a no-trace dual-subscription failover that classifies both a session limit and an authentication-token death as an availability event, retries the same run on the other subscription, and leaves nothing scorable behind. The worker stack is unchanged by this; which subscription served a run is recorded and chosen for availability only.
The third correction was an Output-A/Output-B pooling asymmetry, found in pre-synthesis validation before any synthesis call was spent. Output A pools the surviving packet runs of a cell; the synthesis step and the Output-B scoring path both refused any cell with even one excluded packet, which on this run affected twelve of fifty-six cells and would have handed a spurious zero to the eight partial-completion cells on the Output-B side while their Output-A side scored normally, biasing the attenuation. The defect lived in two coupled places, the synthesis pooling gate and the scoring table, such that fixing either alone would have produced the identical biased result and appeared clean; both were corrected to mirror the worker-side survivor pooling exactly. A cell now produces no synthesis, and both sides score 0, only when all of its packet runs are excluded.
The fourth correction, found in publication review, was a scoring-grid defect: the analysis grid filtered the task universe by condition but not by decomposition, so the two-task secondary was scored as though it spanned all twelve tasks, zero-filling the ten decomposition-absent tasks and dividing its contrasts accordingly (moving one secondary pair from a correct 0.768 to a diluted 0.128). The primary decomposition, present for all twelve tasks, was unaffected, and its published figures are byte-identical before and after the fix.
Only the synthesis and scoring tools changed across the last three corrections; every worker-execution tool is byte-identical across them, and the worker matrix's committed head hash proves the Output-A data predates the corrections. Two rounds of adversarial review by GPT-5.6 (in a seat separate from any worker) produced the correction of an initial significance overclaim (marginal intervals reported as familywise significance), the correction of the Grok causal framing, an error in the count of biased cells, and the grid defect; the verdict advanced to publish-ready only after each was addressed and a fresh pass confirmed no new error. The test suite stands at 170 checks, including regression tests for both the pooling-symmetry and the grid-filter corrections.
6. Provenance
| Artifact | Value |
|---|---|
| Plan manifest | f8966b8e… |
| Tool manifest (v5) | ebc9015c… |
| Gold-corpus freeze (embargoed) | 6e19c24b… |
| Plan decompositions (Phase A) | 3abff9c1… |
| Preregistration (v5) | d1c01a46… |
| Output-A outcome head (worker matrix) | 59f2d4e7… |
| Output-B outcome head (post-synthesis, binding) | a0fe538b… |
Each hash was committed to a timestamped channel; the two outcome heads bind the published Output-A and Output-B numbers respectively, and the analysis refuses to run against a mismatched head. The gold corpus remains embargoed behind its freeze hash for replication without memorization.