Eight of forty, one of forty: rerolling the translations at the extremes

The stress test

A stress test picks the cases that already look extreme and then tries to break them, so whatever it reports describes the cases it picked and says nothing about the population they came from. The experiment underneath this post ran one against a reasoning benchmark. Two hundred and fifty murder mystery narratives from MuSR, each around a thousand words with a binary question at the end, were rewritten by a model into two vocabulary registers, one Germanic and one Latinate, and answered by eight models alongside the untouched originals. Across the full set both registers cost accuracy: the Anglish minimal translations 2.5% below baseline (p=0.019), the Classical minimal 3.7% below (p=0.003). Then the analysis went looking for where that average came from, taking the ten problems translation had hurt most and the ten it had helped most in each of four conditions. Each of those eighty translations was submitted for three further regenerations at temperature 0.7, to see whether the direction held once the wording changed.

Among the forty helped cases, eight produced at least one wording that came out harmful; among the forty hurt cases, one produced a wording that came out helpful. The distance between those two counts is the whole finding. Nothing in it says that a prompt you happen to like carries a twenty percent chance of reversing when you rewrite it, because nothing arbitrary was ever put into the test. These are eighty problem-condition cases the main study had already marked as among the most movable in the corpus, and they were selected on one-sided per-model flip counts before the reroll measured eight-model mean accuracy differences.

What counts as a flip

The counting rule comes from stochastic_analyze.py rather than from the report, because the phrase “at least one harmful variant” hides a threshold. For each problem the script collects every model’s answer under each available wording, then averages correctness across the eight models to get one number per wording. It computes the same average for the untouched baseline. A wording counts as helpful when its mean clears the baseline mean by more than five percentage points, harmful when it falls more than five points below, neutral anywhere between. A case is marked as flipped when any one of its wordings crosses in the wrong direction. The scan runs over every available wording including v1, the original translation generated at temperature 0, so a case can in principle be counted without any reroll having contributed to it.

Metric	Value
Overall consistency	55/80 (69%)
Hurt cases with at least one helpful wording	1/40 (2%)
Helped cases with at least one harmful wording	8/40 (20%)

Sixty nine percent consistency is the third number in that block, and it is measured over a three way split rather than a two way one, since the script records help, hurt and neutral as separate directions. Twenty five of the eighty extremes failed to hold a single direction across their available wordings, while nine of the eighty produced a crossing to the opposite sign. The two counts answer different questions: an extreme that drifts into the neutral band breaks consistency without reversing anything. That drift is its own result. A problem the main study had flagged as among the most movable in the corpus came back, on a fresh wording, with an eight model average sitting within five points of where it started.

The selection is doing most of the work

Both sets were chosen the same way, both got the same treatment, and one of them came apart much more often. That comparison is the part worth keeping. Regression toward the mean is the standing explanation for any extreme that softens on retest, and it pulls a helped case back down toward its baseline and a hurt case back up toward its baseline. Neither drift crosses the line on its own; a crossing needs an overshoot. (The room to overshoot is not equal in the two directions, since helped cases start from a low baseline and hurt cases from a high one, and this design cannot separate that compression from the thing it is trying to measure.)

The automated review narrowed the claim to this reading. Among problems selected for being extreme, apparent help depended on the particular wording more often than apparent harm did, by eight cases to one. That result comes from a sample built to be adversarial, from one translator model with v1 at temperature 0 and v2–v4 at temperature 0.7, scored by averaging eight models whose individual answers are binary. Everything wider than those two sentences was refuted before this post existed.

What a reroll actually changes

Problem 130 in the classical_v2 condition is one of the helped cases, and its four wordings open like this:

v1 Within the formidable topography of an alpine ascension site, Floyd encounters his premature demise via the barrel of a shotgun, leaving Investigator Winston confronting an enigmatic homicide with principal suspects Francis and Camille.

v2 In the formidable topography of an alpine ascent locale, Floyd encounters his premature termination via the muzzle of a shotgun, bequeathing Detective Winston with a frigid enigma of culpability, the principal suspects being Francis and Camille.

v3 In the formidable topography of an alpine ascent locale, Floyd encounters his premature termination via the muzzle of a firearm, presenting Detective Winston with a macabre enigma wherein the principal suspects are Francis and Camille.

v4 In the formidable topography of an alpine ascension locale, Floyd encounters his premature termination via the aperture of a firearm, rendering Detective Winston with a frigid enigma of culpability, the principal suspects being Francis and Camille.

The weapon is a shotgun in the first two wordings and a firearm in the last two. The question at the end of that narrative asks who is the most likely murderer. In a story built so that physical detail carries the inference, a shotgun is a constraint on who could have fired it, and a firearm is that same constraint with its discriminating half removed. Rule three of the instruction the translator was working from says to preserve the meaning precisely and not to change what happens in the story. Two of the three rerolls dropped the weapon specificity anyway. This is one opening paragraph out of roughly a thousand words per narrative, and nothing in the artifacts establishes it as the worst case in the set.

The cells that never ran

The four stochastic data files contain the variant lists for each condition. Anglish and anglish_minimal are complete at twenty problems each with four wordings apiece, and classical_minimal has nineteen complete with one problem holding three. classical_v2 has fourteen complete, one problem holding two, and five problems holding nothing but the original, so seventeen of that condition’s sixty reroll attempts came back with nothing usable. Four of those five stalled problems are helped cases. That puts four of the forty helped cases in the headline ratio into a denominator they were never exposed to, leaving thirty six that were.

The fragility table carries the same gap in a form easier to miss. It lists classical_v2 with fifteen problems rather than twenty, because a standard deviation across wordings needs at least two of them. On the fifteen that survive it reports six fragile (40%) at a mean standard deviation of 0.133, against zero fragile out of twenty and a mean standard deviation of 0.078 for anglish_minimal. The condition whose translations failed most often is the condition whose surviving translations varied most, and one cause covers both readings, since classical_v2 carries the second-shortest of the four instructions and is tied with anglish for the fewest numbered rules. This design cannot tell that apart from the register itself.

Where the movement is

The variance table in the report names five of the eight models. The report lists the Grok 4.1 model at an absolute v1-versus-reroll mean difference of 0.183 in anglish_minimal, Mistral Large at 0.169 in classical_v2, Gemini 2.5 Flash at 0.167 in anglish, DeepSeek at 0.095 in classical_v2, and Llama 3.3 70B at 0.017 in anglish_minimal. The steadiest model in that list is also the weakest of the five the report names, answering 58.4% of the untouched originals correctly on a binary question where a coin gets 50%. Eight points above chance is not much distance to travel in either direction. Its stability is the flatness of something that was barely reading the narrative to begin with. The model at the top of the table, at 73.2% on the same originals, moves about ten times as far in the same anglish_minimal condition, and the three in between cannot be sorted by accuracy from that list at all because their values come from different conditions. Within anglish_minimal, where two of these models can be compared directly, the larger movement belongs to the more accurate of the two. That leaves the obvious next measurement outside this experiment: the same three rerolls over problems chosen for nothing in particular, where regression toward the mean has no extreme to pull on and a flip would have to come from the words themselves.