Four Models on a Retrieval Contract: Method, Both Realizations, and What the Instrument Cannot Say

Pre-registration

Frozen before any arm ran. The substrate was reused byte for byte from an existing benchmark rather than written for this run: the document corpus, the task packets, the worker prompt template shared by every stack, both tools, the evidence extractor with its single mechanical re-prompt on a parse failure, and the scorer.

The scorer is deterministic and there is no judge model anywhere in the loop. It computes pair-granular precision, recall and F1 over (entity, criterion id) pairs against an embargoed answer key. That choice is what makes two realizations comparable at all: an LLM judge would have added its own variance to the variance being measured.

Task selection was inherited, not re-picked. The three tasks came from an earlier pre-registration that selected them as the packets where one incumbent beat another hardest. That made the selection favourable to one arm then, and it still does. Every conclusion below is qualified by it, and the ranking should be read as a screening result rather than a neutral sample.

The worker contract

A worker receives a scoped packet and two local tools, a corpus search and a document fetch, over a frozen corpus. It must enumerate every in-scope entity and emit one record per entity with a fixed set of fields, including a verbatim excerpt of fifty words or fewer that contains the entity name, as a single strict JSON array with no prose around it.

That contract is what makes the run interesting, because it exercises three things at once: instruction following under a strict schema, agentic tool use through a search and fetch loop, and long-horizon coherence across an exhaustive enumeration. It also produces the two columns the parent post is about, since an excerpt can be checked against the document it claims to come from.

Arms, transports and resolved ids

Model identity was captured per run from each harness's own usage telemetry, not from the flag it was launched with, and each transport was probed before the sweep and re-probed after it. A run whose harness reported a different model than the one requested would have been marked as drift and excluded, retained as evidence. None were.

Arm	Requested	Resolved	Transport	Effort
Opus 5	claude-opus-5	claude-opus-5	vendor CLI, subscription, no proxy	xhigh
GPT-5.6 Sol	gpt-5.6-sol	gpt-5.6-sol	codex exec CLI 0.144.4, subscription	xhigh
Grok 4.6	grok-4.6	grok-4.6-build	xAI coding CLI 0.2.117, OAuth	xhigh
DeepSeek V4-Pro	deepseek-v4-pro	deepseek-v4-pro (build 0813)	Claude Code through a counting proxy to the vendor's first-party API	default, thinking on

Two transport facts worth carrying. Grok 4.6 resolves to a -build variant under the coding CLI: the plain id is what you ask for and the build variant is what serves it, which is exactly the class of substitution that a benchmark trusting its own launch flag would never notice. And the DeepSeek arm ran through a translating proxy that was not strictly necessary, since the vendor serves a native endpoint in the same message format; the proxy was used because it counts tokens and the direct route does not.

Scale: 13 packets × 4 arms × 2 identical realizations = 104 worker runs, zero failures, zero re-prompts, zero excluded cells.

Results, both realizations

Task	Arm	F1 r1	F1 r2	citation validity r1	r2
T05	Grok 4.6	0.848	0.848	0.789	0.833
T05	DeepSeek V4-Pro	0.242	0.667	0.250	0.571
T05	Opus 5	0.909	0.848	1.000	0.947
T05	Sol	0.667	0.812	0.667	0.824
T24	Grok 4.6	0.783	0.783	1.000	1.000
T24	DeepSeek V4-Pro	0.750	0.750	1.000	1.000
T24	Opus 5	0.816	0.833	1.000	1.000
T24	Sol	0.766	0.766	1.000	1.000
T31	Grok 4.6	0.053	0.000	1.000	1.000
T31	DeepSeek V4-Pro	0.632	0.158	1.000	1.000
T31	Opus 5	0.474	0.579	1.000	1.000
T31	Sol	0.474	0.474	1.000	1.000

Pooled across both realizations:

Rank	Arm	mean pair F1	sd	mean entity recall	mean citation validity	run-to-run mean abs delta
1	Opus 5	0.743	0.174	0.826	0.991	0.061
2	GPT-5.6 Sol	0.660	0.152	0.766	0.915	0.049
3	Grok 4.6	0.553	0.409	0.628	0.937	0.018
4	DeepSeek V4-Pro	0.533	0.263	0.751	0.804	0.299

Reproducibility, and the read it inverts

Arm	mean abs delta	max abs delta	per task (T05, T24, T31)
Grok 4.6	0.018	0.053	0.000, 0.000, 0.053
Sol	0.049	0.146	0.146, 0.000, 0.000
Opus 5	0.061	0.105	0.061, 0.017, 0.105
DeepSeek V4-Pro	0.299	0.474	0.424, 0.000, 0.474

The first realization alone supports a conclusion the second one refutes. Grok's standard deviation across tasks is 0.409, the widest in the set, which reads as an unstable model until the same packets run again and it returns almost exactly the same scores. The spread is across tasks, not between runs. The instability belongs to DeepSeek V4-Pro, which nearly tripled its own score on one task and lost three quarters of it on another, from identical inputs.

This is the argument for a second realization stated as cheaply as it can be: without it, the report would have carried a caution about the wrong arm.

The two failures are different failures

A grounding failure. On T05 in the first realization, DeepSeek V4-Pro scored entity recall 1.000 and F1 0.242. It found every correct item, and 75% of its excerpts failed verbatim checking against the documents they were attributed to. Retrieval was perfect and the citation was not. The same cell returned 0.667 with citation validity 0.571 in the second realization, still the worst arm, and the size of that swing is itself the reproducibility finding. The milder version of the same pattern appears on Sol (0.667 to 0.824) and on Grok (0.789 to 0.833). Opus 5 ran 1.000 to 0.947.

A convention failure, and a defect in the key. On T31 Grok returned the correct nineteen items with every excerpt verbatim, and scored 0.053 and then 0.000, because it abbreviated the entity names: the short form where the answer key holds the full sentence. The scorer matches normalized strings against a declared alias map, so an abbreviation is counted as a miss and as a false positive. Both realizations agree, so it is systematic.

That cuts two ways and the investigation refuses to net them out. Against the metric: the task text says in as many words that a short label is acceptable and need not reproduce the whole bullet, so on this task the key punishes an instruction the task gave. Every arm's T31 entity recall equals its F1 exactly, which is the signature of a cell that is scoring canonical strings rather than retrieval. For the metric: an orchestrator whose output is parsed by something downstream has to match the schema its consumer expects, and Grok abbreviated hardest of the four on identical instructions, at 0.053 against 0.474, 0.474 and 0.632.

Tool economy and cost

Across all 26 runs per arm:

Arm	mean wall per packet	mean tool calls per packet	cost
Sol	92.3 s	20.9	subscription
Opus 5	108.2 s	26.9	subscription
Grok 4.6	178.1 s	31.3	subscription
DeepSeek V4-Pro	194.7 s	15.2	$1.65 measured

Grok is the heaviest searcher and the slowest of the subscription arms. DeepSeek is the slowest overall for the opposite reason: it makes the fewest tool calls and spends the time thinking, matching another arm's recall on one task with 32 tool calls against that arm's 83.

The metered arm cost $1.65 for 26 packets, about 6.3 cents per packet, across 232 upstream calls, 7.88M prompt tokens and 338k completion tokens. A naive estimate at the vendor's headline per-token rates would have predicted roughly 2.3 times that, and the gap is prefix caching: the harness re-sends a stable prefix each turn and the cached input tier is roughly two orders of magnitude cheaper than a cache miss. Two consequences. A cost estimate that ignores cache tiers can be wrong by more than a factor of two in the favourable direction, and cheap-tier pricing on a new model is perishable, so any figure of this kind is worth re-reading before it is used.

What this does not establish

• No adoption verdict, for any use. This measures retrieval fidelity under a strict output contract. It does not measure judgment, and it is not evidence for or against putting any of these models in any seat.
• No confidence interval on the ordering. Two realizations and three tasks. No significance test is computed or claimed.
• The sample is a screening instrument. The tasks were inherited from a selection made because one model beat another hardest on them.
• Harness is not held constant. Three arms ran in their native CLI; the fourth has no agentic CLI and ran inside another vendor's harness. An earlier matrix on this estate found harness choice moved turn and tool economy without moving pass rate, which bounds the concern without erasing it.
• Effort tiers are not matched compute. Each stack ran at its own top-ish setting, which is a deployment-shaped comparison rather than a controlled one.
• One task's key is defective, as above, and any reuse of it should widen the alias map or tighten the task text first.

Defects found along the way

Two are worth publishing because they generalize beyond this run.

A spend guard that cannot fire for a new model. The counting proxy prices each call from a table, returns zero for any model with no row, and evaluates its abort ceiling against that accumulated zero. So the runaway-spend guard is inert for exactly the models most likely to be new and unmeasured, which is the same failure shape as any check that reports success it never performed. The real ceiling that night was the account balance.

A launch flag that a CLI silently rejects. A sibling dispatch script still passes an effort level the current CLI refuses by name, which would fail closed at launch with zero tool calls on the next run that used it.