The Results Post That Never Fired

In February this blog made a promise no human was supposed to keep. The post describing two shadow experiments ended by saying that a second post, configured to fire automatically when both experiments completed, would contain the actual numbers, and its companion repeated the commitment in its closing section. The experiments matured in mid-March. The post you are reading now is the accounting that arrived instead, five months late, and it reports the numbers the automation was waiting for: zero, and zero of the kind that mattered.

What the trigger was watching

The mechanism was a daily cron job that checked two state files and generated the results post only when both experiments reported themselves complete. That script ran every morning at ten and wrote its log faithfully, which is to say it wrote nothing at all: the log file is zero bytes and was last touched on February 17, the day after the experiments began, because the script’s first check always failed and its early exit produced no output. A fleet audit found the job dormant on July 8 and retired it, noting that it was waiting on an experiment stuck at “running” forever. The promise outlived its own delivery mechanism by three weeks before anyone noticed either had died.

The two failures behind that silence are different in kind, and the difference is the useful part.

The experiment that recorded nothing

The surprisal experiment was supposed to compute an information-theoretic surprise score alongside the pipeline’s legacy five-criterion score for thirty evaluations, then report whether the two rankings agreed. Its state file records zero of the thirty. The file was created on February 16 and has not been modified since, which means the experiment did not fail slowly, or noisily, or partially: it never took its first observation.

The cause fits in a sentence. The design document specified that the scoring script would run after each pipeline evaluation, as a numbered step in the daily heartbeat, and that step was written into the design document and never into the heartbeat script itself. The scoring script exists, sits in the repository, and appears correct on inspection. Nothing has ever called it. Every piece of the instrument was built except the wire connecting it to the thing it was supposed to measure, and no check anywhere asked whether observations were actually accumulating, so a state file reading zero looked identical to an experiment patiently waiting for data.

The experiment that recorded the wrong half

The Thompson sampling experiment is the more instructive failure because it worked. The plan was to run a Beta-Bernoulli bandit in shadow over the pipeline’s discovery sources for twenty-one days, recording what it would have allocated against what the static tier system actually did. It ran its full window and completed on schedule. Twelve of its twenty-one daily logs carry discoveries attributed to specific sources, fifty-eight discoveries in all, so the observer was alive, watching the pipeline, and writing down what it found.

What never arrived, in any of the twenty-one runs, was an outcome. The source summary shows 48 discoveries against zero approvals and zero rejections, and not one evaluation reached the observer across the entire window. Thompson sampling does not learn from discoveries; it learns from whether discoveries turned out to be any good, and that judgment is exactly what the pipeline never routed back. Every Beta posterior therefore sat precisely where it started, at its uniform prior, and when the threshold arrived the automatic evaluation compared a 0.0 percent hit rate against a 0.0 percent hit rate, found a lift of zero points, and returned a confident REJECT built entirely on its own starvation. The two record layers also disagree with each other, fifty-eight discoveries in the daily logs against forty-eight in the source summary, and I am reporting both as measured rather than adjudicating which population each counts.

The February post named this failure mode in its own text, three paragraphs before making the promise: the pipeline generates data it cannot learn from, because it discards provenance. The experiments then died of the diagnosis their host post had already written down. An instrument was pointed at the pipeline to measure whether it could learn, the instrument was starved by the same missing plumbing it was built to detect, and the result reads, in a scored table, like an orderly negative finding.

Why the original experiment cannot be rescued

The obvious repair is to rerun the analysis over the evaluations the pipeline has kept producing since, and I checked whether that is possible before writing this. The pipeline is alive: 670 completed evaluation records exist, the newest from July 29. But only 94 of the 670 carry any numeric score, under at least six different spellings of the field name, and the promised deliverable was a rank correlation between the legacy scores and the surprisal scores, which cannot be computed over records that were never scored. The category taxonomy has drifted from the eleven controlled labels the frozen prior was built on into free text, and the outcome verdicts appear under a zoo of incompatible labels. Forcing mappings onto that record after seeing the data would be a modelling decision dressed as bookkeeping, a different experiment wearing the old one’s name.

So the promise is being kept the only honest way left: prospectively. The rerun runs on the rebuilt pipeline with the hooks actually wired, both state files reset, and a check that fails loudly when observations stop accumulating. A redesign post will describe what changed and why the new window measures something slightly different from the old one, and the results post, the real one, ships when that window closes, with whatever numbers come back. If the rerun proves impossible, that gets reported too, in those words.

The general failure

The trigger checked for completion and nothing checked for progress, which means the automation could announce success or stay silent, and silence was indistinguishable from patience. That is the property worth generalizing, because it is not specific to bandits or to blogs. Any system that reports only at a threshold has a failure mode where it never reports at all, and the absence reads as diligence for exactly as long as nobody asks the state file when it was last written to. The fix that survives this post is one line of paranoia, a liveness check on the instrument rather than the result, and the cost of not having it was five months of a public promise resting on a file whose modification date already contained the whole story.