A Cheap Model Against a Regex

Most working automation classifies natural language with a word list, because a word list is deterministic and free, and the question of whether a cheap model would do it better is usually argued rather than measured. Four such classifiers were measured against two cheap models at their lowest reasoning effort, given nothing but a plain description of the task and an instruction to reply with one line of JSON.

The baselines are the production code paths themselves, imported rather than reimplemented, so the keyword column is what the running system actually decides. The items are 118 pieces of real text sampled deterministically from live corpora, hand labelled before any model ran. A stronger model then adjudicated every one of the 53 items where any system disagreed with the hand label, audited ten agreed items for silent errors, and recomputed one task’s arithmetic; it flipped eight hand labels, several of them traceable to labelling from a truncated view, and excluded one malformed item. The scores below use its adjudicated labels rather than mine.

Task	n	keyword	Spark at low	Luna at low
Does this log row claim the work is finished	29	62.1%	89.7%	93.1%
Which state is this stalled row in, of three	29	51.7%	89.7%	93.1%
Does this free-text answer contain a new instruction	38	84.2%	78.9%	81.6%
Does this status file claim readiness	20	75.0%	90.0%	90.0%

Both models returned clean single-line JSON on all 236 calls, with no parse failures.

The error asymmetry matters more than the totals. Of the twelve rows the keyword tool escalated for a human decision, ten were semantic non-claims: phrases reporting that some step had landed, which the word list read as claims that the whole item was done. The cheap model made that error once. After the hybrid shipped, the escalation pile on that surface fell from 110 items to 47, and a second classifier that had found zero instances of one category in 263 rows found 24.

Cost is what makes it deployable rather than interesting. A single batched call carrying all 29 rows of one task used about 24 times fewer tokens than 29 separate calls, and moved the weekly subscription window by less than a whole percentage point. At this volume that is free.

Batching does cost accuracy, and the honest number is the shipped one: putting both judgments in one batched call cost about seven points on the first task, taking 93.1% to 86.2% in production, against a keyword baseline that did not move.

Two of the sites keep their keyword lists, and the reasons are not about accuracy. One detects whether a model provider is rate limiting the caller, where a model call to detect that the models are unavailable is circular, so the word list stays as the floor. The other decides whether staged content is safe to publish, where the failure mode is a leaked credential, and a model must never be the only gate on a publish path. Keywords as the non-negotiable floor wherever a miss means a leak or a false green, semantic as an additional sweep, never a replacement. Method, the site inventory, the adjudication protocol, the cost measurement and the per-site verdicts are in the companion methodology and raw data.

The task the keyword pass won deserves the same discipline as the ones it lost. Its 84.2% against 81.6% is well inside the noise of a 38 item single run, so the correct reading is that no semantic advantage was demonstrated there, rather than that keywords were proven better.