Investigation
DeepSeek V4 Flash against GPT-5.6 Luna: Methodology, Results and Retraction
Pre-registration, five arms, the full results table, the price sheet, the mislabelled-arm retraction and the build-identity test that established it, plus limitations and spend.
Pre-registration
Frozen before the first call, amended once mid-run on an owner correction that raised the token ceiling for the reasoning heavy arms.
- Substrate: the production move prompt of a browser word game, taken from the shipped code rather than rewritten for the benchmark.
- Grader: the game's own response parser and move validator against its production dictionary, so a move counts as legal exactly when the shipped game would accept it.
- Positions: twelve deterministic seeded board states, identical to those used in an earlier cost report, so numbers from the two runs are directly comparable. The letter rack is the controlled variable.
- Sampling: single trial per cell, temperature 0.4.
- Token room: 16,000 output tokens, raised to 64,000 for the reasoning heavy arms.
- Primary metric: legal move rate. Tie rule: differences of one legal move in twelve are ties and are not ranked.
- Declared in advance: a 10 second latency ceiling for an interactive opponent.
Arms and results
| Arm | Build | Route | Legal | Median latency | Mean output tokens | $/call | $/legal move |
|---|---|---|---|---|---|---|---|
| A2 | GPT-5.6 Luna | OpenRouter to OpenAI (US) | 8/12 | 19.6 s | 1,125 | 0.000765 | 0.001148 |
| A4 | V4 Flash, April base | DeepInfra (US) | 8/12 | 77.9 s | 3,664 | 0.000742 | 0.001113 |
| A1 | V4 Flash, April base | DeepInfra (US) | 6/12 | 95.1 s | 2,996 | 0.000621 | 0.001243 |
| A5 | labelled new, actually April base | OpenRouter to Parasail (US) | 8/12 | 153.8 s | 17,007 | 0.004854 | 0.007281 |
| A3 | V4 Flash 0731 | OpenRouter pinned to DeepSeek | 7/12 | 232.4 s | 28,144 | 0.007896 | 0.013536 |
All arms completed 12 of 12 calls with no unrecovered errors. A1 and A4 ran at 16,000 tokens of room, A3 and A5 at 64,000.
The follow up rerun fourteen hours later, on the same substrate and grader, replaced the A3 row with a cleaner one: 12 of 12 legal, 28,031 mean output tokens, 99.8% of them reasoning, at $0.007865 per legal move, against a Luna control at 7 of 12 and $0.001380. The 5.7× figure in the parent post is that pair. Cost per finished move fell 1.7× between the two DeepSeek runs entirely because failures cleared, not because the model thought less.
Reading the legality column
Applying the frozen tie rule exactly as written: the arms at 8, 8, 8 and 7 of 12 are a four way tie, and the arm at 6 of 12 falls outside the band. That separation cannot be attributed to the model, because the two arms at 6 and 8 are the same build on the same route, distinguished only by a reasoning disable flag that was verified ineffective (their token distributions match).
The pre-registered rule therefore returns an inconclusive verdict. No model outplayed another here, and specifically no claim is made that Luna beat DeepSeek.
That same straddle is evidence the one in twelve tie band was too tight for a single trial instrument at temperature 0.4, since one configuration reproduced itself only to within two of twelve. The tie rule was rewritten mid run on that basis and the rewrite was retracted as post hoc. The observation carries forward as a noise estimate that a future pre-registration should freeze before running, alongside the obvious design fix of multiple trials per cell.
The retraction
The arm presented as the new build served by US providers, A5, pinned the April base slug,
not the July build. Its harness declares deepseek/deepseek-v4-flash with a provider order of
US hosts, and that slug points at the April preview.
The original run inferred the build from a reasoning channel fingerprint: some providers expose a separate reasoning channel and others do not, and the run treated presence of the channel as evidence of the new build. The report stated the weakness itself, that absence is consistent with either an older build or a provider serving reasoning disabled by template. The pair came apart in the direction it did not guard against, since presence is equally consistent with either build. The fingerprint had no discriminating power for the question it was asked.
The test that does discriminate is greedy decoding on a byte identical prompt: same weights and same serving template give a long matching prefix, and different builds diverge almost immediately. One prompt at temperature 0, eight routes, $0.00028 of spend, three clean clusters:
| Cluster | Routes | Reasoning tokens | Register |
|---|---|---|---|
| Terse | 0731 at DeepSeek, 0731 at Cloudflare, base at DeepSeek | 29 to 84 | clipped, telegraphic |
| Verbose | base at Parasail, base at Fireworks | 216 to 231 | full prose |
| Silent | base at DeepInfra | 0 | no reasoning channel at all |
The two verbose routes emitted byte identical reasoning prefixes to each other and nothing resembling either genuine July endpoint, differing in register, in answer formatting, and by three to seven times in token count. The mislabelled arm ran on one of them.
Two consequences worth carrying. A slug does not tell you which build answers, and one vendor's own base slug endpoint landed in the new build cluster, so the provider pin is the only thing that fixes build identity. And the mislabelled arm's measurement was sound: a rerun of that exact route came within 0.8% on mean output tokens. Only the label was wrong, so the row remains usable as a data point about the April build on that provider.
Price sheet, as read live on the run date
| Route | Hosting | Price in/out per million |
|---|---|---|
| DeepInfra, April base build | US | $0.09 / $0.18 |
| Parasail or Fireworks, April base build | US company | $0.14 / $0.28 |
| DeepSeek official, July build | CN | $0.14 / $0.28 |
The vendor's own route is not cheaper than the US routes for the same build. Separately, the routing rule inverts on the larger model in the family, where the third party host is roughly three times the vendor's price, so "keep this family on the cheap host" holds for one size and fails for the other. These are dated figures: they were read from the provider catalog on 2026-07-31 and should be re-read before being used.
Limitations
- Single trial per cell at temperature 0.4, on an instrument with a documented two in twelve replication spread. The aggregate token and cost figures are stable across runs; the legality column should not be over-read.
- One arm was still ceiling limited even at 64,000 tokens of room, with two cells truncating, so its 7 of 12 is a floor rather than a clean measurement.
- A near miss worth recording: the first pass at that arm ran at 16,000 tokens like everything else and scored 4 of 12 with seven cells pinned at the ceiling, which measured the budget rather than the model. The invalid run is retained in the results file, labelled invalid, because deleting it would hide the mistake.
- Mean score is not a ranking. The results file carries a mean quality score per arm and it is inverted by survivorship, since slower arms disproportionately survive the easy open board positions where scores run structurally higher.
- Task family. This is constrained JSON emission for a single game move. It is not agentic coding or tool calling, which is where the July build claims its improvement, and nothing here transfers to that.
- Transport reliability, incidental: 4 of 36 calls to one host failed at the transport layer, one after a 300 second hang, all recovered on retry.
Spend
$0.2318 total across five arms and sixty scored cells, of which about $0.052, or 23%, bought results that were discarded for method reasons: an invalid ceiling limited run and a rate limited first attempt. The discarded portion is stated separately rather than folded into the retained arms, because folding it in would conceal what the two method corrections cost. The follow up rerun cost a further $0.2425 as measured per call.
The provider ledger delta across the rerun session exceeded the sum of per call costs by about $0.23, which is unattributed: candidates include a shared always on service drawing on the same key, eight calls that reported zero cost to the client after failing, and an abandoned probe. Anyone costing this build from these figures should use the ledger delta rather than the per call sum.
Status
Data gathering, not an adoption verdict. The standing ruling on this work is that no adoption call follows from one narrow task family, and three broader evaluations of the same cheap model tier were registered and had not run when this was written.