Everyone Found the Answer

A retrieval benchmark usually reports one number per model, and that number cannot see the difference between an answer that is right and an answer that is right while citing a quote the source document does not contain. Scoring four models on a task that required both, with the grading done by a frozen deterministic scorer rather than by a judge model, put that gap on a column of its own.

Thirteen scoped packets over a frozen document corpus, two local tools, and a strict output contract: enumerate every in-scope entity, and for each one emit a verbatim excerpt of fifty words or fewer containing the entity name, as one JSON array and nothing else. Two identical realizations, four arms, 104 worker runs, no failures and no excluded cells. Each arm’s model identity was captured from its harness’s own telemetry rather than from the flag it was launched with, and the transports were probed before the sweep and again after: Claude Opus 5 and GPT-5.6 Sol through their own vendor CLIs on subscriptions, Grok 4.6 through the xAI coding CLI, and DeepSeek V4-Pro through the vendor’s first-party API, since it has no agentic CLI to run in.

The pooled ranking is Opus 5 at 0.743 pair F1, Sol at 0.660, Grok 4.6 at 0.553 and DeepSeek V4-Pro at 0.533, identical in both realizations. That ordering is the least interesting output of the run, and with two realizations and three tasks it carries no confidence interval.

Citation validity is the column that separates them. Measured as the fraction of evidence records whose quoted excerpt appears verbatim in the document it cites, Opus 5 scored 0.991, Grok 0.937, Sol 0.915 and DeepSeek V4-Pro 0.804. The extreme case is one cell: DeepSeek found every correct entity, entity recall 1.000, and three quarters of its excerpts were not in the documents it attributed them to, citation validity 0.250. It knew the answer and misquoted the source, and a summary score cannot express that, because the two components moved in opposite directions inside the same cell.

Reproducibility is the second column, and it inverts the obvious read. Grok 4.6 has by far the widest spread of scores across tasks, which looks like an unstable model until the same tasks run twice: its mean movement between two identical realizations is 0.018, the most reproducible arm in the set. The spread is across tasks, not between runs. DeepSeek V4-Pro moves 0.299 on average and 0.474 at worst, turning 0.242 into 0.667 on one task and 0.632 into 0.158 on another, from identical inputs. A single measurement of that arm means very little.

One cell deserves its own reading, because it is the metric failing rather than the model. Grok scored 0.053 on a task where it had returned all nineteen correct items with every excerpt verbatim; it lost the points for abbreviating the entity names, writing the short form where the answer key held the full sentence. The task text explicitly permitted a short label. So that cell measures naming convention and reports it as retrieval, which is a defect in the key. The same behaviour is still a real signal for anything that parses the output downstream, and Grok abbreviated hardest of the four on identical instructions. Both readings stand; neither cancels the other.

Method, both realizations cell by cell, the transports and resolved model ids, tool economy, cost and the full caveat list are in the companion methodology and raw data. The three tasks were inherited from an earlier pre-registration that chose them because one model beat another hardest on them, so this is a screening instrument rather than a neutral sample, and none of it is an adoption verdict for anything.

What the run costs a reader who takes only the ranking is the two findings that would decide a deployment. Both live in columns a leaderboard does not print, and both were invisible until the same tasks ran a second time.