The Reasoning Floor: Methodology, Cell Data and Control Probe

Design

The question was whether the output token volume of one benchmark arm belonged to the model or to a serving accident, and whether any client parameter could reduce it.

Substrate held byte identical to the original run: the same production move prompt from a browser word game, the same parser and validator, the same twelve deterministic seeded positions, temperature 0.4. The prompt, parser, validator and game engine were confirmed unchanged in version control since two days before the original benchmark, so nothing in the grader moved underneath the comparison.

Every arm is hard pinned with an explicit single provider and fallbacks disabled, and the serving provider is recorded per cell, so the artifact proves the route rather than assuming it. All 48 cells completed with zero transport errors.

Arm	Route	Completed	Legal	Truncated	Mean out tokens	Reasoning share	Median latency	$/legal move
B1	July build at the vendor	12/12	12/12	0	28,031	99.8%	345.7 s	0.007865
A3	same route, original run	12/12	7/12	2	28,144	99.9%	232.4 s	0.013536
B3	April base build at a third party	12/12	9/12	2	17,147 excl. truncated	~76%	281.9 s	0.009452
A5	same route, original run	12/12	8/12	0	17,007	75.5%	153.8 s	0.007281
B4	reference control	12/12	7/12	0	1,190	97.3%	13.1 s	0.001380
A2	control, original run	12/12	8/12	0	1,125	97.1%	19.6 s	0.001148
B2	July build at a second provider	4/12	4/12	0	11,904 †	—	329.0 s	—

† Survivorship. B2's means describe only the four calls that completed and must not be quoted as that endpoint's behaviour.

The control validates the instrument

Before reading anything into the reproduction: the reference arm moved from 8 of 12 to 7 of 12 legal and from 1,125 to 1,190 mean output tokens between the two runs, which is inside the original pre-registered tie band and within 6% on tokens. The harness did not drift, so the comparison is between models rather than between instruments.

Token volume reproduced, failures did not

Cell for cell on the same route, fourteen hours apart:

Position	Original tokens	Rerun tokens	Original	Rerun
p01 open	4,933	1,247	legal	legal
p02 open	879	762	legal	legal
p03 open, vowel heavy	21,675	12,000	illegal	legal
p04 open, junk rack	2,279	1,511	legal	legal
p05 mid	44,947	43,934	legal	legal
p06 mid	31,372	33,614	legal	legal
p07 mid, blank	19,072	37,161	legal	legal
p08 mid	8,691	16,583	legal	legal
p09 deep	63,994	61,118	truncated	legal
p10 deep	63,999	47,289	truncated	legal
p11 deep	38,060	41,696	illegal	legal
p12 deep	37,821	39,452	illegal	legal

Individual cells swing hard in both directions, which is expected of a single trial instrument at temperature 0.4 with a documented two in twelve replication spread. The aggregate is nearly exact: 28,144 to 28,031 mean output tokens, $0.007896 to $0.007865 per call, 99.8% reasoning share in both.

All five failures came back legal, including both cells that had hit the 64,000 token ceiling. A five in twelve legality swing exceeds the documented noise band, but it is still one trial per cell, so the defensible statement is narrower than an improvement factor: at the same spend per call, the route now returns a usable answer where it previously often returned nothing.

The mechanism

The model card for this build states that the release does not include a chat template. Instead the vendor ships an encoder implementation and asks each serving provider to implement the prompt format itself. The reasoning control is not a sampler setting: it is a text prefix.

Requested effort	What is prepended
low, the documented default	nothing
high	"Reasoning Effort: Absolute maximum with no shortcuts permitted. You MUST be very thorough..."
max	"Reasoning Effort: Beyond maximum, exhaustive, relentless, and uncompromising..."

There is no medium. A separate mode closes the thinking block immediately and suppresses the reasoning channel entirely.

With no canonical template, how much reasoning a caller is billed for is a decision each provider makes on their behalf, and providers can disagree by an order of magnitude on byte identical requests. That is not a defect in any one provider. It is the predictable consequence of shipping a reasoning model without a chat template, and it also explains why several third party endpoints for this build sat deranked in the broker's catalog while their operators worked the serving out.

The control probe

Prompt held fixed, reasoning control varied, two mid weight positions, $0.03499 of spend.

Position	Config	Out tokens	Reasoning tokens	Legal?	Finish
p03	baseline	10,823	10,749	legal	stop
p03	effort low, flat parameter	7,305	7,275	legal	stop
p03	effort low, nested parameter	13,827	13,797	legal	stop
p03	reasoning disabled	253	0	illegal	stop
p08	baseline	30,071	30,007	legal	stop
p08	effort low, flat parameter	28,327	28,264	legal	stop
p08	effort low, nested parameter	transport error
p08	reasoning disabled	32,000	0	illegal	length

Two findings, closing the question from opposite directions.

Asking for low effort does nothing, because low is the documented default and prepends no prefix, so requesting it is a no-op. The numbers agree: a 6% move on one position, and on the other the three reasoning enabled configurations land at 7,305, 10,823 and 13,827 with no monotone ordering, a spread of roughly 30% that reads as run to run variation.

Turning reasoning off relocates the thinking rather than removing it. The channel goes cleanly to zero and both cells become illegal: one returns a 253 token stub that fails to parse, and the other runs its full 32,000 token budget into the ceiling with no reasoning channel at all, having inlined its chain of thought into the message body without ever reaching the required JSON.

One transport error occurred on the nested parameter cell of p08 and was not retried, because the flat parameter cell on the same position answers the same question.

What this did not settle

Build versus serving. The second genuine endpoint was included precisely to separate these, and it hard failed 8 of 12 calls, returning an error finish reason with zero prompt tokens and zero cost after ten to twenty minutes each, having generated fifteen to twenty eight thousand reasoning tokens first. So the claim rests on one provider measured twice, not on two providers agreeing. What would settle it: rerun against the other endpoints once they leave deranked status, or self host briefly against the published weights with the reference encoder.
You cannot conclude the failing provider reasons less. Its lower mean covers only the four calls that completed, which is exactly the subset that did not run away.
Why the failures cleared is unexplained. Candidates include ordinary variance, the vendor stabilising a day old endpoint, and this run using lower concurrency. Not separated.
Single trial per cell throughout, at temperature 0.4, on an instrument with a documented two in twelve replication spread.
Task family. Constrained JSON emission for one game move is close to the worst showcase for a build whose headline improvement is agentic work. A large deliberation budget is a liability here and may be an asset there. That was not measured.

Spend

$0.242457 measured from per call provider reports: a build identity probe at $0.000277, a smoke test at $0.004470, four main grid arms totalling $0.202720, and the control probe at $0.034990.

The broker's account ledger moved $0.4738 across the same session, a gap of about $0.231 that could not be attributed. Candidates: an always on speech service drawing on the same key, the eight failed calls that reported zero cost to the client but may have billed upstream, and an abandoned first control probe that was killed after roughly thirty minutes without completing a call. Anyone costing this build from these figures should use the ledger delta rather than the per call sum.