Investigation
Speech to Text Bake-off: Methodology, Reference Design and Full Tables
Corpus selection, the cross-family leave-one-out reference, synthesized noise beds with verified SNR, clean and noisy result tables, vocabulary recall, latency and cost, and what the design cannot support.
Corpus
The audio is real dictation, retained by the transcription server so a bad transcript could be re-run later against a different engine. That retention is the only reason the study could run without recording anything new.
- 354 unique clips, 9.23 hours, spanning two weeks. (420 files on disk; 66 are byte identical retry duplicates, deduplicated by content hash.)
- 26 clips selected, 23.4 minutes, by criteria applied in this order: deduplicate by hash; keep durations between 10 and 150 seconds, since shorter clips are scraps with too few words to score and a longer one would have eaten a fifth of the cloud budget; stratify across the four duration bands the dictation occupies (5 clips at 10 to 20 s, 9 at 20 to 45 s, 8 at 45 to 90 s, 4 above 90 s); prefer clips whose production transcript survives, then the most vocabulary dense; break ties by hash so the selection is deterministic and reproducible.
- Every arm received the identical 16 kHz mono conversion produced by the exact command the production server runs.
The reference problem
There is no ground truth for these clips, and none was invented. Two rules govern the consensus reference that replaced it.
Cross family leave one out. The reference for each arm is the medoid, the hypothesis with minimum total word edit distance to the others, built only from engines of a different family. Two engines sharing a weights lineage agree with each other far more than either agrees with the truth, so letting both vote would have produced a reference meaning "what that lineage thinks" and scored that lineage as most accurate by construction. Every arm is scored against the same three foreign families.
Vocabulary presence needs three votes of four. At that threshold no single engine's vote can create presence, since dropping any one leaves two, so every arm faces an identical denominator and none can inflate its own recall.
What this cannot do: it measures agreement with other engines rather than correctness. An engine that is uniquely right about a word is scored wrong for it, the absolute percentages are not comparable to any published word error rate, and small gaps are noise.
Where the design is strong is the noise half, because two of its three metrics do not require the reference to be true, only to be the same. A paired delta compares an arm to itself across conditions against one fixed clean reference, so the reference is equally right or wrong in both conditions and cancels. Self drift compares an engine's noisy output to its own clean output and touches no consensus at all.
Noise beds
Both beds are synthesized deterministically, with fixed voices, fixed text and a fixed seed, so the exact conditions can be regenerated to score a future engine.
| Condition | What it is | Signal to noise |
|---|---|---|
| babble 20 / 10 / 5 | ten concurrent synthetic voices, different neutral text each, staggered onsets | 20 / 10 / 5 dB |
| stationary 10 | pink noise plus mains hum and a mid shelf, a desk under a fan | 10 dB |
Mixing is calibrated against active speech level with silence excluded, so a clip with long pauses does not quietly receive less noise. Achieved signal to noise was verified within 0.3 dB of target at every level. Three babble depths rather than one give a degradation slope per engine, and the stationary point checks whether a ranking is specific to speech shaped interference.
Limitation: synthesized babble is not a recording of a real room. It is spectrally and temporally speech like, which is what makes it hard, but it has no reverberation and no real crosstalk statistics. The talkers read neutral filler text unrelated to the dictation's subject matter, so no vocabulary term can leak from the noise bed into a transcript and be scored as a hit.
Why these candidates
Candidates were picked on accuracy against a live speech recognition leaderboard rather than inherited from earlier notes.
| Model | Leaderboard WER | Licence | Fits the available GPU | Verdict |
|---|---|---|---|---|
| Voxtral Small 24B | 2.8% | Apache 2.0 | no, roughly 55 GB needed | best open accuracy, excluded on VRAM |
| Voxtral Mini 3B | 3.8% | Apache 2.0 | yes, ~9.3 GB | candidate A |
| Whisper Large v3 | 4.1% | MIT | yes | carried as a reference arm |
| Canary Qwen 2.5B | 4.3% | CC-BY-4.0 | yes | excluded, 40 second training window |
| Parakeet TDT 0.6B v3 | 4.5% | CC-BY-4.0 | yes, ~2 GB | candidate B |
Canary was excluded despite ranking above Parakeet because its training window is 40 seconds while the median clip is 58.5 seconds, so most of the corpus would need hand rolled chunking that the vendor's own card says degrades accuracy. Parakeet is also architecturally unlike the others, a transducer rather than an attention decoder, which makes its errors more independent and its vote in the consensus worth more.
Clean accuracy
Word error against the cross family leave one out consensus, 26 clips, 95% clip level bootstrap intervals. Read as agreement, not truth.
| System | WER | 95% CI |
|---|---|---|
| Voxtral Mini 3B, local | 3.7% | 2.7 to 4.8 |
| cloud audio model, incumbent | 3.9% | 2.5 to 5.9 |
| cloud audio model, mini variant | 4.7% | 3.1 to 7.3 |
| Whisper large v3 turbo, hinted, the installed fallback | 5.4% | 3.5 to 7.8 |
| Whisper large v3, unhinted | 6.2% | 4.1 to 8.6 |
| Parakeet TDT 0.6B v3, local | 7.5% | 4.7 to 11.7 |
Every interval in the top four overlaps every other. The clean audio conclusion is parity rather than a ranking, and the study cannot resolve a difference of about 1.5 word error points.
Noise robustness
Paired on the same 8 clips against one fixed clean reference, where a positive number means the arm got worse under that condition.
| System | clean | babble 20 | babble 10 | babble 5 | fan and hum |
|---|---|---|---|---|---|
| cloud audio model | 1.8% | +0.6 | +4.0 | +20.6 | +0.0 |
| cloud mini variant | 2.1% | +0.2 | +4.1 | +25.9 | −0.3 |
| Voxtral Mini 3B | 2.4% | −0.3 | +2.2 | +12.7 | +0.0 |
| Whisper turbo, hinted | 1.7% | +0.8 | +2.7 | +11.9 | +0.0 |
| Whisper large v3, hinted | 1.3% | +0.6 | +3.8 | +10.9 | +0.2 |
| Parakeet TDT 0.6B v3 | 4.3% | +1.9 | +4.1 | +11.2 | +1.3 |
The reference free confirmation, each engine's noisy transcript against its own clean transcript:
| System | babble 20 | babble 10 | babble 5 | fan and hum |
|---|---|---|---|---|
| cloud audio model | 1.6% | 4.6% | 21.8% | 1.3% |
| cloud mini variant | 1.9% | 6.7% | 28.6% | 1.8% |
| Voxtral Mini 3B | 0.6% | 3.5% | 14.3% | 1.4% |
| Whisper turbo | 2.1% | 3.6% | 13.8% | 0.9% |
| Parakeet TDT 0.6B v3 | 3.1% | 5.8% | 13.9% | 3.3% |
Two independent metrics give the same answer: at 5 dB babble the cloud engine degrades roughly twice as much as any local model. At 20 dB babble and on the fan bed nothing moves more than a point.
Vocabulary
Term recall on the clean set with identical denominators. "Hinted" means the spellings the production server sends the engine; "unhinted" means jargon mined from the speaker's own notes that no engine is told about, which is the uncontaminated signal. Counts only: the term list itself is private material and is not published.
| System | Hinted | Unhinted | Spurious |
|---|---|---|---|
| cloud audio model | 18/18 | 32/33 | 0 |
| Whisper turbo, hinted | 18/18 | 31/33 | 0 |
| Voxtral Mini 3B, chat path | 18/18 | 29/33 | 0 |
| Parakeet TDT 0.6B v3 | 17/18 | 32/33 | 0 |
| cloud mini variant | 16/18 | 31/33 | 0 |
| Whisper large v3, hinted | 15/18 | 29/33 | 0 |
| Voxtral Mini 3B, transcription mode | 12/18 | 32/33 | 0 |
No engine invented a vocabulary term that no other engine heard, across 522 transcriptions on real audio, and no engine echoed the spelling list back as if it had been dictated. That echo failure is real and was reproduced on silence; it did not fire once on speech.
The Voxtral result is a channel problem rather than an ability problem. Its transcription mode is its accuracy optimal path and accepts no text hint, which costs it homophone confusions on exactly the words the hint exists to fix, including writing "Cloud" for "Claude" in three of five occurrences. Its chat path accepts the same hint text and reaches 18 of 18, and the trade is measurable:
| Voxtral Mini 3B | Clean WER | Hinted | Unhinted | babble 5 delta |
|---|---|---|---|---|
| transcription mode, no hint possible | 3.7% (2.7 to 4.8) | 12/18 | 32/33 | +12.7 |
| chat mode, hint accepted | 4.6% (3.4 to 5.8) | 18/18 | 29/33 | +12.4 |
Taking the hint costs about 0.9 word error points and four unhinted terms, since the model spends attention on the instruction instead of the audio. Whisper turbo takes its hint through a dedicated decoder input rather than through the text prompt and pays nothing visible for it.
Speed and cost
| System | Median latency | Speed against realtime | Cost |
|---|---|---|---|
| Parakeet TDT 0.6B v3 | 480 ms | 93× | $0 |
| Whisper large v3 turbo | 812 ms | 35× | $0 |
| Voxtral Mini 3B | 1,826 ms | 26× | $0 |
| cloud mini variant | 2,044 ms | 23× | $0.0004/min |
| cloud audio model | 2,622 ms | 17× | $0.0208/min |
Every local option is faster than the cloud, so the accuracy decision trades nothing away on speed. The cloud price was measured on this study's own billing rather than taken from a price page, at $0.8858 for 42.6 minutes of audio, a figure dated to 2026-07-31 and worth re-reading before it is used.
The mini cloud variant deserves a line: 53 times cheaper for about 0.8 word error points on clean audio, well inside the noise, and the worst arm in the study under heavy babble.
What this does not establish
- Any ranking within about 1.5 word error points. The intervals overlap and 26 clips is a modest sample, capped by the cloud budget.
- Anything about truth, only about agreement. A systematic error shared by all four engine families on this speaker is invisible to this design by construction.
- Anything about real rooms. The noise beds are synthetic and free of reverberation.
- Anything about clips longer than 150 seconds or shorter than 10, or about the cloud model on long form beyond the four clips above 90 seconds that it saw.
- Diarization or timestamps. Not tested, and irrelevant to dictation, though relevant if a model is reused for long form transcription.
The cheapest way to close the largest gap would be twenty minutes of human transcription: correct ten of these clips and the consensus reference becomes a real one, at which point every arm is re-scorable against truth without re-running a single engine, since all 522 transcripts are retained.
Reproduction and containment
The corpus is private dictation, so no clip, audio artifact or transcript left the machine, and none is published here. What is reproducible from the published material is the design: the selection criteria are deterministic, the noise beds are generated from fixed text and a fixed seed, and the scoring rules are stated above in full.
Nothing was installed into a live environment during the study. The two candidate models need dependencies the production environment does not have, so they were installed to a private target directory appended rather than prepended to the module search path, leaving the running services' own numerical libraries in front. One model was resident on the GPU at a time and no production service was evicted.
Sources: a public speech recognition leaderboard retrieved 2026-07-31, the two candidate model cards and their licences, the excluded model's card for its training window, and the deployed server's own configuration for the incumbent arm.