The Lab Notes
More Effort Bought More Searches
Building in conversation with AI. Software, research, games, writing — documenting what happens. About
55 graded research outputs from GPT-5.6 Sol and Claude Opus 5 at five explicit effort settings plus the Opus default, August 2026: higher effort raised search volume, no GPT-5.6 Sol setting above medium improved its mean grade, and Claude Opus 5 averaged lowest at extra high and highest at max.
About this audio
Synthetic narration, adapted from the text — tables and code blocks are described rather than read out.
GPT-5.6 Sol and Claude Opus 5 were each given five web research questions at five explicit reasoning effort settings, to see what a higher setting changes when an agent searches. GPT-5.6 Sol ran through OpenAI’s Codex command line tool with its built in web search, and Claude Opus 5 through Claude Code with web search and page fetching. Each ran once per question on 6 August 2026 at low, medium, high, extra high and max, plus Claude Opus 5’s unspecified default. Higher settings scaled searching while mean grades moved by a few tenths, with GPT-5.6 Sol peaking at medium and Claude Opus 5 lowest at extra high and highest at max.
The questions came from an earlier Exa comparison: four freshness questions that had separated its configurations, about how AI hardware serves large models, a then scheduled API tariff change, the cheapest long context model APIs and the newest flagship model identifiers, and an archival question about wartime economic statistics. Claude Fable 5 graded all 55 outputs from complete reads with the earlier comparison’s five part rubric, checking at least two claims per output against primary sources. The API does not echo Claude Opus 5’s effort level back, so the evidence that it took effect is behavioural: on the archival question it searched 12 times at low and 57 times at max.
| Effort | GPT-5.6 Sol | Claude Opus 5 |
|---|---|---|
| Low | 4.76 | 4.72 |
| Medium | 4.92 | 4.80 |
| High | 4.84 | 4.80 |
| Extra high | 4.84 | 4.56 |
| Max | 4.84 | 5.00 |
| Default (unspecified) | not run | 4.88 |
GPT-5.6 Sol’s search volume climbed with the setting, from 14 searches at medium to 97 at max on the archival question and from 10 to 27 on the hardware question. The max run spent 2,564 seconds, about 43 minutes, on a question the medium run answered to the same grade in 265 seconds, and its 97 searches missed a database the medium run had found. The grading model’s report qualifies this: those searches also produced the broadest map of wartime statistical serials of any GPT-5.6 Sol run, despite the equal grade.
A pricing error repeated across GPT-5.6 Sol runs, and the same report calls its adjudication of that price the emblematic finding of the whole program. At high, extra high and max, GPT-5.6 Sol priced GPT-5.6 Luna at $1.00 per million input tokens and $0.10 cached while linking the OpenAI page that listed $0.20, $0.02 and $1.20. The $1.00 figure was the launch price, and OpenAI’s changelog records an 80% cut to GPT-5.6 Luna starting 30 July, seven days before the run. The Claude Opus 5 runs that fetched the live page reported the current price. Six Exa configurations in the earlier comparison had agreed on about $0.40. The grading model’s report suggests that this figure was most plausibly the long context rate for prompts over 272,000 input tokens repeated without its condition, and concludes that, in this pricing check, agreement among search results was weaker evidence than one fetch of the vendor’s page.
The grading model’s report says that what separated the Claude Opus 5 settings was how their outputs handled what they could not verify. At extra high they made refuted absence claims on two questions and carried stale prices on a third. One of those absence claims was that Google’s Ironwood TPU had no public price entry when the pricing page listed $12.00 per chip hour on demand in Iowa. At max no spot check refuted anything asserted, and its 5.00 was the only perfect configuration mean in either comparison. At low, GPT-5.6 Sol assumed a standard cache discount and priced a DeepSeek-V4-Flash cache hit at $0.014 per million input tokens, where the run’s own check of DeepSeek’s pricing page recorded $0.0028.
The rubric scores accuracy at zero when a checked claim is materially wrong, but the grades gave such outputs partial credit, usually three points, as the Exa comparison had. Scoring every output with a refuted spot check at zero accuracy leaves GPT-5.6 Sol at 4.92 on medium and 4.64 to 4.72 elsewhere, and puts Claude Opus 5 between 4.16 at extra high and 5.00 at max, so the peaks and the trough hold.
Each cell is one run, the grader comes from the same model family as Claude Opus 5, and an instruction file already in Claude Opus 5’s context overlapped the flagship model question, so a tenth or two between settings is not a ranking. The grading model’s report recommends GPT-5.6 Sol at medium and Claude Opus 5 at max where a wrong assertion is expensive, and it advises that when a research agent at a high effort setting claims something does not exist, the claim stays unverified until a second search tool or a direct fetch of the source confirms it. The per question grades, the strict scoring and the price adjudication are in the companion method, per question grades and pricing adjudication.
Comments