Investigation
Semantic against Keyword: Method, Adjudication and Per-site Verdicts
Task construction from production code paths, the adjudication protocol, per-task results and error asymmetries, cost measured from the provider's own usage surface, the batching correction, and the verdict for each site including the ones that keep their regex.
The sites, and which four were benchmarked
An honest sweep of one working automation estate found eight places where a regular expression or a keyword list classifies natural-language content, later extended by a parallel search to seventeen. The four highest-stakes ones were benchmarked; the rest carry transfer arguments only and are labelled as such.
The four tasks, described by what they decide rather than by which tool decides it:
| Task | Decision | Cost of a miss |
|---|---|---|
| S1 | Does this project-log row claim its work is finished, and is that claim contradicted elsewhere | Live work marked closed; noise escalated to a human |
| S2 | Which of three states is this stalled row in | Misbucketed items waste reading time |
| S3 | Does this free-text answer contain a new instruction | A lost directive |
| S4 | Does this status file claim readiness | A missed claim, or a wasted verification run |
Construction
118 items, sampled deterministically with a fixed seed from live corpora using each tool's own parser: 29 for S1 (every item the tool marked positive, plus a stratified remainder), 29 for S2, 40 for S3 (38 scored after one malformed item was excluded), 20 for S4.
Baselines are the production code paths themselves, imported rather than reimplemented, so the keyword column is what the running system decides rather than a reconstruction of it.
Model arms received the same text through a straightforward prompt: a task definition, the text, and an instruction to reply with one JSON line. No rubric engineering and no examples. Both models ran at their lowest reasoning effort, one call per item, in a read-only sandbox, plus one batched probe.
Adjudication
Every item was hand labelled before any model ran. A stronger model then adjudicated all 53 items where any system disagreed with the hand label, audited ten agreed items as silent-error controls, and independently recomputed one task's arithmetic.
It flipped 8 of the hand labels, several traceable to labelling from a truncated view of the text, excluded one malformed item, and endorsed the rest. All ten silent-error controls came back clean and the arithmetic check was exact. The reported scores use its adjudicated labels, not the originals. Running the adjudication after the hand labelling rather than instead of it is what makes that substitution possible.
Results
| Task | n | keyword | Spark at low | Luna at low |
|---|---|---|---|---|
| S1 closure claim | 29 | 62.1% | 89.7% | 93.1% |
| S2 stalled-row state | 29 | 51.7% | 89.7% | 93.1% |
| S3 answer contains an instruction | 38 | 84.2% | 78.9% | 81.6% |
| S4 status claims readiness | 20 | 75.0% | 90.0% | 90.0% |
Zero parse failures across 236 calls; both models returned clean one-line JSON every time.
Error asymmetries
- S1. Of the twelve rows the keyword tool escalates for a human decision, ten are semantic non-claims: step-level phrases reporting that some component landed or some triage finished, which the word list reads as claims that the whole row is done. The cheap model makes that error once.
- S1, bug classes. Both previously known keyword failure classes reproduced, including rows where a leading verification marker refers to the problem being verified rather than the work. A third was new: a dated closure written in plain text was missed because the pattern required a bold or checkmark prefix.
- S3. The keyword pass catches 17 of 20 instructions with 3 false alarms; the faster model catches 19 of 20 with 7 false alarms, and the other 18 of 20 with 5. The models' only advantage here is recall. If the goal ever becomes catching everything rather than calibrating, the right shape is a recall-side model pass unioned with the lexical one.
- S4. The keyword pass produced four phantom positives from files containing readiness-like substrings, and missed two real claims phrased slightly differently from its pattern. The models halved both error types.
Cost, measured rather than estimated
Every figure comes from the provider CLI's own rollout files, specifically their token-count events and rate-limit payloads, with snapshots preserved beside the working set.
Per-call input runs about 22,000 tokens and is almost entirely fixed harness context, which caches hard: 89% cached on one model, 68% on the other. The marginal item text is one to two thousand tokens.
Two facts decide the deployment shape:
- Unbatched, per-item calls cost roughly 0.13 to 0.14% of a weekly subscription window each on the model that has its own bucket, measured as a clean before-and-after bracket around 118 calls on a bucket that had been untouched that period.
- Batched, the same work is free at this volume. One call carrying all 29 rows of a task used 31,328 input and 1,417 output tokens in 33 seconds, about 24 times fewer tokens than 29 separate calls, and did not move the shared weekly window by a whole percentage point. A full sweep of the live corpus is roughly eight such calls.
The adjudication pass itself, run on a strong model at maximum effort, is a different cost tier entirely, and that is the correct division of labour: the cheap model scans, the expensive model judges, and the expensive model never scans.
The batching correction
The benchmark measured per-item calls. The shipped hybrid batches, and batching costs accuracy: putting both judgments in one call took S1 from 93.1% to 86.2% in production, while S2 held at 93.1%. Against unchanged keyword baselines the gaps are +24 and +41 points rather than +31 and +41.
A second correction runs the other way. The adopted design gives the model only the claims-and-contradiction judgment and keeps the verdict decision table in code, which is a different function from the one benchmarked. Replayed against the same adjudicated ground truth it scores 27 of 29, identical to the model's direct verdict.
Per-site verdicts
| Site | Verdict | Basis |
|---|---|---|
| S1 closure judgment | Adopt, hybrid | 62 to 93 points; ten of twelve human escalations were noise; batched cost is negligible. Keep the deterministic evidence gathering and the verdict table in code; hand the claim judgment to the model. |
| S2 stalled-row state | Adopt, same call | 52 to 93 points, and it rides the same batched call over the same rows. |
| S4 readiness | Keep keywords, tighten | Models win 90 to 75, but this runs inside a shell loop on every pass and its false positives are self-healing, since the check it triggers catches them. The cheaper fix is two pattern additions. |
| S3 instruction detection | Keep keywords | Keywords win outright, and the tool's production role is a frozen calibration comparator where determinism is the point. |
| Rate-limit detection | Keep keywords as the floor | It fires exactly when model providers are failing, so a model call to detect that models are unavailable is circular. A model recheck is useful only when the provider is reachable, which is when it is least needed. |
| Publish-safety scanning | Keyword floor, semantic sweep on top | The failure mode is a leaked credential, and the documented incident was a key whose vendor prefix no rule knew. A semantic pass catches novel shapes; a model must never be the only gate on a publish path. |
| Structural and deterministic-by-contract checks | Keep keywords | No measured deficiency, and reproducibility is load bearing. |
Model choice: the cheaper shared-bucket model at low effort, batched, for the adopted sites, being equal or better on three of four tasks. The faster dedicated-bucket model is the right pick when per-item latency or isolation from the shared window matters, at a measured cost per call.
Live effect after adoption
The hybrid shipped the day after the study. On the adopted surface the human escalation pile fell from 110 items to 47, and the second classifier, which had found zero instances of one category across 263 rows by keyword, found 24. Cost held to the predicted noise across a full 233-row sweep.
Limits
- Single run, one day's corpus, with n of 29, 29, 38 and 20. The thirty and forty point gaps on S1 and S2 are far outside that noise. The S3 keyword edge, 84.2 against 81.6, is inside it, so it reads as no demonstrated semantic advantage rather than as keywords proven better.
- Thirteen further sites were not benchmarked. Their verdicts are transfer arguments from these four measurements and are labelled as such; adoption at any of them earns its own benchmark.
- The benchmark measured a different function from the one that shipped. Both the batching correction and the replay against the same ground truth are reported above for that reason.
- Nothing was changed in production during the study. All eight original sites ran exactly as before; the adoption came afterwards, under a separate decision.