What Replicated, and What Did Not

Teknium of Nous Research announced on 2026-08-02 that Hermes Agent had become dramatically more efficient, especially for smaller and weaker models, attaching an A/B card that reports about 21% fewer agent turns, 29% fewer tool calls and 23% less wall clock on a weak model, with parity on a strong one. A claim of that shape is cheap to check independently and rarely is, so it was checked: thirteen cells, twelve fresh tasks in three families, three repetitions each, 468 runs in 43 minutes for $3.03, with the harness version the only thing varying inside the paired columns.

It reproduces on the weak models, at close to the published magnitudes, on a task set the batch has never seen. Holding model and endpoint fixed, the newer harness against the older one gives 27.6% fewer agent turns, 30.2% fewer tool calls, 44.3% fewer tool errors and 35.0% lower median wall clock on Gemini 3.5 Flash-Lite, and 21.1% fewer turns on GPT-5.6 Luna. Against a published card of 21%, 29% and 23%, that is an independent replication landing inside a few points on a different operator’s instrument.

The half that did not reproduce is the parity claim. On GPT-5.6 Sol the same upgrade still buys 19.0% fewer turns and 9.0% lower median wall clock, which is attenuation rather than absence. The weak and strong split the card describes shows up here as a gradient rather than a cliff, large on the two weak models, moderate on the strong one, and the honest reading is a partial non-replication. The efficiency half lands; the parity half does not.

The counter-example is sharper than either. On DeepSeek V4 Flash the upgrade is near parity on every behavioural axis, at 2.4% fewer turns, and it costs 21% more tokens per solved task. Weak models benefit is too coarse a rule, since one of the two cheap workhorses tested does not benefit at all.

One behaviour should drive any upgrade decision more than the averages do. Three of 36 runs on the newer harness emitted exactly 65,536 output tokens in a single API call, taking between 341 and 643 seconds, against zero such runs in 36 on the older one. Every other call in those same runs ran between 61 and 284 tokens. All three passed, so no correctness metric can see it, and it turns that cell’s mean wall clock into a 127% regression while its median reads as a 16.6% improvement. Those three calls are 51% of the cell’s entire spend.

Because every cell routed through one counting proxy to one upstream, the cross harness comparison is exact rather than approximate, and harness choice moves cost per solved task by about 2.7 times on an identical model with identical serving. On both cheap models the cheapest harness per solved task was Hermes.

No quality claim survives this run. 467 of 468 runs passed, a ceiling effect the design pre-registered as a risk, so the tasks cannot separate these harnesses on correctness and the pass rate column is not a capability ranking. Method, all thirteen cells, the paired version tables, the task families and the reproduction rig are in the companion methodology and raw data.

What the run taught about instruments outweighs what it taught about harnesses. Four measurement artefacts were caught before publication, three of them instrumentation masquerading as harness behaviour: a 58% latency regression that was the proxy retrying a probe it had failed to serve, a multi-minute startup penalty that was one harness paying a database migration on every fresh home directory, and a beautifully clean deterministic failure fingerprint that turned out to be the grader matching on a transcript the instrument itself had written into the graded directory. All three flattered a story the operator already believed, and that is the direction measurement error takes when nobody is looking for it.