Investigation

The Saving Was per Successful Call: Method and Results

How one Claude Code session drove Browser Use's command line tool (release 0.13.8 on its harness 0.1.9, with a local headless Chrome 151) and a Playwright MCP server that answers with accessibility tree snapshots through eight frozen public web tasks on 24 August 2026: the results per task with the calls as tallied, what each payload and latency figure measured, the arm totals as run and why they differ from the figures per result, the session's own call counts against the tally, the run order, the two defects that decided a task written as observations a maintainer could reproduce, the limits of the evidence, and how every figure was recomputed from the raw record.

What was run

On 24 August 2026 one Claude Code session in this workspace drove two browser tools, one after the other, through the same eight tasks on public websites. The task list was frozen before either tool ran, as the list itself states, and the file of expected answers records that each answer but task 6's was established beforehand with curl (from the trial's host). Every task had an exact pass criterion (for the negative control, an exact behaviour), no task required a login, and, as the list states, none submitted private data. The classes and targets below are those of the frozen list.

Task Class Public target
1 form fill and submit httpbin.org/forms/post
2 extraction across one category of items books.toscrape.com, the Travel category
3 lookup in a large table iana.org/domains/root/db
4 search, navigate and verify chain arxiv.org
5 honesty at a login wall (negative control; no login attempted) github.com/settings/profile
6 bot wall a public post on X
7 page rendered by JavaScript (a single page app) quotes.toscrape.com/js/
8 infinite scroll, repeated until no more load quotes.toscrape.com/scroll

The first arm ran the command line tool of Browser Use, which takes a short Python program piped into it and runs that program against Chrome over the DevTools protocol. Its browser was a Chrome 151 dedicated to the arm and launched with --headless=new, and the user agent that browser reported read HeadlessChrome/151. The same tool also ran task 6 against the vendor's cloud browser. The second arm was a Playwright server for the Model Context Protocol (MCP), a locally patched build of an existing server. Its tools open and navigate pages, click, type, press keys and scroll to the bottom of a page, and the page is read through an outline of its accessibility tree or a search of that tree's snapshot by regular expression. The vendor's own autonomous cloud agent took the frozen wording of tasks 2, 6 and 8 as three cloud sessions. By the trial's report, the session drove the Browser Use tool through its shell tool and the snapshot server through its MCP tools, in a single pass, and the record does not name the model behind that session.

Component Tested on 24 August 2026
Browser Use command line tool browser-use 0.13.8
The harness it runs on browser-harness 0.1.9
Local browser of the Browser Use arm Chrome 151 launched with --headless=new, user agent HeadlessChrome/151
Snapshot server a locally patched build of an existing Playwright MCP server, answering with accessibility tree snapshots
Autonomous agent the vendor's cloud agent, on tasks 2, 6 and 8 only

Results per task

The calls below are the driver's own tally from that one pass, which describes one driver and no distribution. The recomputation checked the Browser Use results by program, matching each task's final output file against the frozen pass criteria, while the snapshot server's results are the report's, since that arm left no output files and no program graded its replies.

Task Browser Use, local headless browser Calls, driver's tally Snapshot server, by the report Calls, driver's tally Autonomous agent
1 form fill and submit pass 2 pass 9 not run
2 extraction across a category pass 1 pass 4 session reported success
3 lookup in a large table pass 1 pass 2 not run
4 search, navigate and verify pass 8 pass 11 not run
5 honesty at a login wall pass 1 pass 2 not run
6 bot wall, a public post on X fail, HTTP 403 (cloud browser: page served) none listed; 1 by the total pass 3 session reported success
7 page rendered by JavaScript pass 1 pass 3 not run
8 infinite scroll pass 6 attempts pass about 13 session reported success
Total 7 of 8 about 21 8 of 8 about 47 3 of 3 sessions

On task 6 the local browser landed on Chrome's error page for an HTTP 403 response, which said that access to the site was denied, and the program found no post text. Through the same tool's cloud browser the page was served, with no 403 and no denial, and the program captured the first 600 characters of the page body. The field it used for the post's own text came back empty, so the record shows the page served rather than the text retrieved; the trial's report counts it as a pass. The post's text belongs to an outside individual and is not reproduced here. The total of 7 for the Browser Use arm therefore counts task 6 as a failure of the local mode, and the row carries the result for the cloud mode. The autonomous agent's column gives each session's own success flag for the three tasks it ran.

Payload per result

Each task's final program did its filtering itself and printed its result instead of the page, and the size of that printed output, as saved in the task's output file, is the figure given here for the result.

Task Bytes printed by the final program
1 form fill and submit 858
2 extraction across a category 292
3 lookup in a large table 259
4 search, navigate and verify 340
5 honesty at a login wall 184
6 bot wall (the 403 page, a failure) 194
7 page rendered by JavaScript 218
8 infinite scroll 26

Apart from the form task, whose 858 bytes hold the form as filled, the address it was sent to and the submission the test site echoed back with its request headers, every final output ran from 26 to 340 bytes. Earlier attempts printed more, up to 3,228 bytes on task 4 and 2,553 on task 8. For the snapshot server the trial's report estimated about 600 to 2,600 tokens per task, and about 2,900 at worst on task 4; the raw record holds no token counts, so those figures are the report's. As a check of scale, the session record holds 48 replies from the snapshot server, with a median of 133 characters, a minimum of 59, a maximum of 7,219 and a total of 35,218, counted per call and in characters. With one payload measured in bytes and the other in tokens, the two support at most an order of magnitude between them per successful result.

Latency per operation

Through the Browser Use tool, one timed operation was one program piped into the tool. The program opened a catalogue page at books.toscrape.com in a new tab, waited for it to load, extracted the listed prices with JavaScript (with the titles, in the first set) and printed the result. That time covers the tool's whole path, including the start of its process. Through the snapshot server, one operation was two HTTP requests to the server's own API, a navigation followed by a regular expression search of the snapshot for prices, timed without the MCP wrapper that an agent's calls pass through. The first set ran five times on the Travel category page, the second ran six times while alternating between the Travel and Mystery pages, and a third piped an empty program into the tool five times to time its process start.

Timed set, 24 August 2026 Browser Use runs (ms) Median Snapshot server API runs (ms) Median Ratio of medians
Travel page, 5 runs 447, 446, 449, 446, 447 447 81, 45, 43, 44, 43 44 10.2
Travel and Mystery alternating, 6 runs 450, 1150, 449, 448, 449, 467 449.5 53, 548, 44, 52, 43, 52 52 8.6
Empty program, 5 runs 136, 129, 128, 127, 126 128 not timed

The medians put one operation at about 450 milliseconds through the Browser Use tool, of which an empty invocation of the tool alone took about 130, against a median of 44 to 52 milliseconds through the snapshot server's API. The Browser Use tool was therefore roughly nine to ten times slower per operation. That ratio is an upper bound on the gap an agent would meet, because the snapshot timing leaves out the MCP wrapper hop its real path adds while the Browser Use timing covers the whole path. Each arm's second set held one slow run (1,150 and 548 milliseconds), which the medians absorb. The trial's report weighs the gap against the agent loop, where it assumes each call also costs a model turn, which it takes to be far slower than either tool. On that reading the number of round trips matters more end to end than the speed of each call. No model turn was timed, so that reading rests on the design of the loop alone.

The arms as run

The figures per result describe what a working program printed as its answer. The arm totals below describe everything each arm put into the driver's context as the run actually went, every failed attempt and debugging probe included, together with what the driver wrote into its calls. For the Browser Use arm that writing is each shell command, the program piped in together with the shell lines around it, so the programs count with their wrappers. The span runs from the first program of task 1 to the attempt on task 6, and it leaves out the call that launched the browser and the runs that came after the snapshot arm.

Arm, as run on 24 August 2026 Calls in the session record Reply characters Characters the driver wrote
Browser Use, local headless browser 24 shell calls, carrying 30 programs 23,011 28,654, the commands carrying the programs
Snapshot server 48 tool calls 35,218 1,814, the arguments

Over the arm as run, the Browser Use replies came to 0.65 of the snapshot arm's, so the replies alone stayed smaller even with every retry in the count. With the commands the driver wrote added, the Browser Use arm put 1.40 times as much text into the context as the snapshot arm did. By the report's token estimates, the saving is per successful call, and on this run the failed attempts, the probes and the programs themselves spent it. The two kinds of figure also differ in unit, since the arm totals are lengths in characters taken from the session record and the figures per result are sizes in bytes of the output files.

Call counts in the record

The driver's tally of about 21 calls for the Browser Use arm sums 2, 1, 1, 8, 1, 1, 1 and 6 by task. The 1 for task 6 is inferred from the total, because the report lists no count on that row. The session record shows 30 programs piped into the tool between the first task and task 6, carried in 24 shell calls because some calls piped more than one program. Of the 30, 22 were task attempts and 8 were debugging probes, written to list the harness's helpers, to debug the arXiv search form and to examine the scroll page and the timeouts there. The tally sits close to the 22 task attempts, and with the probes counted the arm made more calls than its tally shows.

For the snapshot server the tally of about 47 sums 9, 4, 2, 11, 2, 3, 3 and about 13 by task, and the session record shows 48 tool calls (one of them opening the page). Taken on the tally, the ratio of calls is 0.45, and with every Browser Use program counted it is 0.625. The Browser Use arm made fewer calls either way, while a ratio below one half holds only on the driver's tally.

Run order

The Browser Use arm ran first, from task 1 through to task 6, and the snapshot arm's first call came after the Browser Use arm's last. By then the driver had met the sites' quirks, among them the arXiv search field kept behind a toggle and the loader on the scroll page, so an order effect that favours the snapshot arm is not ruled out.

The defects that decided a task

Each of the following decided a task on this run. Both are written as observations of the release tested, in a form a maintainer could reproduce, and neither is a verdict on the project.

The first concerns the harness's scroll helper under headless Chrome. On both pages it was tried on the helper raised TimeoutError (the second time after 5.0 seconds), and the same error appears in earlier attempts at the tasks that called it, each made through the mouse wheel event. Task 8 passed only when its program scrolled by script instead.

Environment  browser-use 0.13.8, browser-harness 0.1.9, Chrome 151 launched with --headless=new,
             the tool attached to that local browser over the DevTools protocol
Steps        a program opens https://books.toscrape.com in a new tab, waits for the load and calls
             the harness's scroll() helper once; a second program does the same on
             https://quotes.toscrape.com/scroll and catches the exception
Observed     on both pages the call raised TimeoutError; on the second page it failed after 5.0 s;
             the first page's traceback ends in the harness's IPC request, at the socket receive,
             with "TimeoutError: timed out"; the helper sends Input.dispatchMouseEvent with
             type mouseWheel, and that command did not return
Expected     the page scrolls and the call returns
Workaround   set the window's scroll position with JavaScript and dispatch a synthetic scroll event
             for the page's loader; with it the final program for task 8 counted all 100 quotes

The second concerns the identity of the local headless browser. The browser the tool drove announced itself as headless in its user agent, and on task 6 the site refused it with an HTTP 403, while the same tool's cloud browser was served the page. By the trial's report the snapshot server's browser was served the page as well. The headless user agent is the difference the record shows, since the other browsers' user agents are not recorded, and it was not tested as the cause.

Environment  as above, with the same tool's cloud browser as the comparison
Steps        open a public post on X in the tool's local browser and read the post's text
Observed     Chrome's error page for an HTTP 403 response, saying access to the site was denied,
             and no post text; the local browser's user agent read HeadlessChrome/151
Comparison   through the cloud browser the page was served, with no 403 and no denial, and a
             page body was captured (its first 600 characters); the field for the post's own text came back empty
Open         the cause of the 403 is not isolated

Limits

The Browser Use tool's local browser was only ever Chrome 151 launched with --headless=new, and by the trial's report a headful run might show neither the scroll defect nor the 403. Task 6 was a single trial against a live bot wall, so its result is not durable, and the cause of the 403 remains open. The call counts and the timings come from one driver on one pass, the record does not name the model behind that driver, and the snapshot arm ran second, after the sites' quirks were known.

The snapshot server's pass count and token figures are the report's, because no program graded that arm and the raw record holds no token counts. Its timing leaves out the MCP wrapper, and no model turn was timed on either arm. The autonomous agent ran three of the eight tasks, and its column reports each session's own success flag. On task 6 the cloud browser's record shows the page served, with the field for the post's text empty (the report counts it as a pass), and the payload figures for the two arms are in different units, which limits their comparison to an order of magnitude.

How every figure here is recomputed

Every figure on this page has a line in a recomputation from the trial's raw record, with its rule and its status, and that recomputation is attached to the note's fact-check. A figure that matches the trial's report is given as the report gives it. Two figures depart from it and are given in their raw values: the snapshot server's latency and the scope of the scroll defect. Figures that only the report or the driver states, such as the snapshot server's pass count, its token estimate, the call tallies and the reading of the agent loop, carry that attribution in the sentence that uses them. The arm totals, the call counts in the session record, the ratios of calls, the character check and the run order are the recomputation's own derivations from the record. The raw record is a set of copies of the trial's report, output files, timing runs and session lines, with private material redacted, and no figure on this page rests on anything outside it.

Ask About Projects
Hi! I can answer questions about Ashita's projects, the tech behind them, or how this blog was built. What would you like to know?