The Agent-Computer Interface: Browser, Terminal, API, GUI, and Beyond
Definition: what is an agent-computer interface?
An agent-computer interface — here abbreviated ACI — is the architectural interface through which an AI agent perceives, reasons about, and manipulates a computing environment. It is not just “tool use,” and it is not just “computer use.” It is the whole boundary between a model-driven decision process and the external computational world: observation channels, action channels, state containers, permissions, execution semantics, and feedback loops.
A minimal Agentic Loop looks like this:
| Loop phase | ACI responsibility | Examples |
|---|---|---|
| Observe | Represent environment state to the model or harness | Screenshot, DOM node, accessibility tree, terminal output, file contents, API response, log stream |
| Decide | Select an action under constraints | Click button, run command, call tool, ask user, wait, inspect file |
| Act | Execute the selected action outside the model | Browser click, shell process, MCP request, file patch, HTTP request |
| Verify | Check whether the action changed state as intended | Screenshot after click, test output, API status, file diff, transaction confirmation |
| Govern | Enforce safety, identity, cost, and authority boundaries | Sandbox, OAuth scope, allow/deny rules, watch mode, human confirmation, audit log |
This definition matters because the model alone does not “use a computer.” In Anthropic’s Computer Use documentation, Claude can request actions such as screenshots, clicks, typing, scrolling, dragging, waiting, and keypresses, but the application must execute the tool call; Claude does not execute it directly. Claude Platform OpenAI’s Computer Use API is similarly framed as a loop in which the model inspects screenshots and returns actions that developer code or a harness executes. OpenAI Developers Google’s Gemini Computer Use documentation also describes a loop where the model receives screenshots and action history, returns UI actions and safety decisions, and client code executes the actions. Google AI for Developers
So the ACI is best understood as a system boundary, not a single product feature. It includes the model, but also the harness, the renderer, the executor, the sandbox, the state log, the permission system, and the recovery mechanism. This is the difference between “an LLM that can emit click(x, y)” and a usable Computer Use system.
The agent boundary problem
The central architectural question is: where is the agent?
There are at least four plausible answers.
| Boundary choice | What counts as “the agent”? | Why it is attractive | Why it is incomplete |
|---|---|---|---|
| Model boundary | The language model or multimodal model | Clean conceptual boundary; easy to benchmark | Ignores execution, state, identity, permissions, and recovery |
| Harness boundary | Model plus loop, tools, context management, and policies | Matches how real systems are built | Harnesses differ by vendor and task domain |
| Environment boundary | Model plus harness plus browser, VM, container, repo, worktree, or OS layer | Captures actual capabilities and risks | Blurs product, infrastructure, and model evaluation |
| Organizational boundary | Agent plus auth, audit, delegated authority, approvals, and user/account identity | Required for enterprise deployment | Hard to standardize across services and jurisdictions |
The harness boundary is the most useful engineering boundary. Anthropic’s Claude Code docs describe Claude Code as a harness around Claude: it runs in the terminal, gathers context, takes actions, verifies results, and provides tools and an execution environment. Claude Anthropic’s 2026 Managed Agents engineering post makes the same point more explicitly: the agent system is split into a session, a harness, and a sandbox, with “brain” and “hands” decoupled so that the action environments can fail, be replaced, or be scaled independently. Anthropic
That framing changes how we evaluate agents. A raw model may be excellent at inferring the next action but poor as a deployed agent if the harness has weak permissions, slow feedback, lossy context compression, brittle click execution, or no recovery path. Conversely, a mediocre model with a tight ACI — typed APIs, tests, short feedback loops, scoped credentials, and reliable state — may outperform a stronger model trapped in an ambiguous GUI.
The major modalities
1. Browser-as-interface
The browser is the most visible form of ACI because it gives agents access to the same web interfaces humans use. Browser agents can navigate websites, fill forms, search, purchase, scrape, compare, schedule, and interact with services that do not expose convenient APIs.
OpenAI’s January 2025 Operator release framed this explicitly: Operator used its own browser, looked at webpages through screenshots, and interacted by typing, clicking, and scrolling. It was powered by a Computer-Using Agent model trained to interact with graphical user interfaces rather than requiring custom API integrations. OpenAI Anthropic’s Computer Use release similarly described Claude looking at screens, moving cursors, clicking buttons, and typing text, while warning that the feature was experimental and error-prone. Anthropic Google’s Project Mariner began as a Gemini 2.0 browser prototype that reasoned over browser screens, pixels, text, code, images, forms, and web elements through an experimental Chrome extension. blog.google
The browser ACI is powerful because the web is already the universal interface for many tasks. The same browser session can contain a search engine, email, calendar, spreadsheet, travel site, CRM, shopping cart, or internal dashboard. It also gives agents access to authenticated flows after user takeover or connector authorization. OpenAI’s ChatGPT agent launch made this convergence explicit: the agent combined Operator-style website interaction, deep research, a visual browser, a text browser, a terminal, direct API access, and connectors inside a virtual computer. OpenAI
But browser control is also brittle. OpenAI’s CUA research reported strong headline browser benchmarks but documented wide reliability variation across individual tasks; some simple tasks were solved consistently, while other tasks failed often without prompt hints or interface familiarity. OpenAI Google’s early Mariner post reported an 83.5% WebVoyager score but also said the prototype was not always accurate and could be slow. blog.google OpenAI’s Operator launch likewise called the system a research preview with limitations, and its later ChatGPT agent release still described agent mode as early-stage and capable of mistakes. OpenAI
Browser ACIs also inherit the web’s adversarial surface. Prompt injection is not a side issue; it is a structural risk of letting an agent read untrusted web content and then act in authenticated contexts. OpenAI’s ChatGPT agent post describes malicious instructions hidden in webpages or metadata that could try to exfiltrate connector data or cause harmful actions, and the help center warns that browser and app access create privacy risks that safeguards cannot eliminate. OpenAI
2. Terminal-as-interface
The terminal is the dominant ACI for coding agents because software projects already expose their state and control surfaces through text: files, diffs, compilers, tests, package managers, linters, logs, shells, build tools, and version control. A terminal agent does not need to infer which pixel is a button if it can run grep, inspect a stack trace, edit a file, and execute a test suite.
Claude Code is the clearest Anthropic example. Its docs describe it as an agentic coding tool that reads codebases, edits files, runs commands, and integrates with developer tools across terminal, IDE, desktop, browser, VS Code, JetBrains, and other surfaces. Claude API Docs Anthropic’s “How Claude Code works” page describes the loop as gathering context, taking action, and verifying results, with the terminal as the environment in which the agent can do anything available from the command line. Claude
OpenAI’s Codex CLI follows the same terminal-first pattern. OpenAI’s docs describe Codex CLI as an open-source coding agent that runs in a terminal, reads and changes code in a selected directory, and can execute commands; the terminal UI lets users review actions in real time. OpenAI Developers The same Codex family also expanded into cloud and desktop forms: the May 2025 Codex research preview ran software-engineering tasks in isolated cloud sandboxes, while the February 2026 Codex app framed itself as a command center for multiple agents working in isolated worktrees. OpenAI
OpenCode illustrates the open-source version of this modality. Its docs describe it as an open-source AI coding agent available as a terminal interface, desktop app, or IDE extension; its tool surface includes bash, file reads and writes, grep, glob, LSP, patching, web fetch/search, custom tools, and MCP servers. OpenCode Its permissions system can allow, deny, or require approval for tools, and OpenCode supports per-agent permission overrides and specialized agents such as read-only exploration agents and restricted planning agents. OpenCode
The terminal ACI has better feedback than raw GUI control. A test either passes or fails. A diff can be inspected. A command exits with a code. A compiler points to a file and line number. The interface is still not perfectly reliable — shell commands can be destructive, package managers can mutate environments, secrets can leak, and agents can misread test results — but it offers far more machine-checkable structure than a screenshot of a web page.
Its biggest safety problem is blast radius. Terminal access can mean filesystem mutation, network calls, credential exposure, dependency installation, process spawning, and destructive commands. Anthropic’s Claude Code permissions docs address this with read-only defaults for many operations, approval requirements for Bash and file modification, allow/ask/deny rules, command pattern matching, modes such as plan and auto, and a warning that bypassing permissions should be limited to isolated environments. Claude OpenAI’s Codex CLI docs similarly emphasize sandbox boundaries, platform-native enforcement, and approval flows when commands need to cross boundaries. OpenAI Developers
3. API-as-interface
APIs are the most semantically reliable ACI. A typed function call or protocol request can expose exactly the operation the agent is allowed to perform: create_ticket, search_docs, read_calendar, send_invoice, run_query, list_pull_requests, schedule_meeting. Compared with GUI control, APIs reduce perceptual ambiguity and often reduce latency, token use, and error rate.
OpenAI’s platform docs describe tool use as extending models through built-in tools, function calling, remote MCP servers, file search, code interpreter, web search, and third-party service calls. OpenAI Developers OpenAI’s May 2025 Responses API update added remote MCP server support, image generation, Code Interpreter, file search, background mode, reasoning summaries, and encrypted reasoning items, positioning the Responses API as a primitive for agentic applications. OpenAI
The main standardization effort here is the Model Context Protocol (MCP). Anthropic introduced MCP in November 2024 as an open standard for secure two-way connections between AI-powered tools and external systems, replacing fragmented integrations with a single protocol. Anthropic The MCP specification describes a standardized way for applications to share context, expose tools and capabilities, and build composable workflows using JSON-RPC 2.0 messages among hosts, clients, and servers. Model Context Protocol
By 2025–2026, MCP had become a cross-vendor convergence point rather than just an Anthropic feature. Google’s I/O 2025 material said the Gemini API and SDK were compatible with MCP tools and also discussed Agent2Agent-style interoperability. blog.google OpenAI’s tool docs describe remote MCP servers as a way to connect models to new data sources and capabilities over the internet. OpenAI Developers Anthropic’s 2025–2026 MCP update said MCP had an official community-driven registry, a November 25 specification release with asynchronous operations, statelessness, server identity, and official extensions, and official SDKs across major programming languages. Anthropic
The API ACI’s weakness is not semantics; it is integration burden and context management. Every API has schemas, auth, rate limits, scopes, side effects, and operational edge cases. Too many tools can overwhelm the model context, increase latency, and degrade tool selection. Anthropic’s “Code execution with MCP” post argues that direct tool calls can consume too much context and that agents often scale better by writing code to call tools, filter outputs, and orchestrate control flow programmatically. Anthropic OpenCode’s MCP docs warn similarly that enabling many MCP servers can quickly add to context and exceed limits, especially for token-heavy servers. OpenCode
MCP also does not magically solve security. The protocol’s authorization spec requires OAuth 2.1-oriented mechanisms and protected resource metadata for authorization server discovery, and its transport spec warns implementers to validate origins, bind local servers to localhost where appropriate, and implement proper authentication. Model Context Protocol That is healthy protocol design, but it confirms the point: API ACIs need identity, authorization, origin checks, token storage, audit, and least-privilege scopes as first-class architecture.
4. GUI-as-interface
GUI control is both a modality and a substrate beneath other modalities. Browser agents are GUI agents when they operate through pixels and clicks. Desktop agents are GUI agents when they use screen capture and coordinate actions. Mobile agents are GUI agents when they reason about app screens and touch targets.
There are three major GUI observation/action styles:
| GUI style | Observation channel | Action channel | Strength | Failure mode |
|---|---|---|---|---|
| Pixel/screenshot | Raw screenshot, sometimes with cursor and history | Coordinate click, type, scroll, drag, keypress | Universal; works on human-facing UIs | Expensive, slow, visually ambiguous, resolution-sensitive |
| Accessibility tree / DOM | Structured UI nodes, labels, roles, bounds | Semantic click/fill/press or coordinate-assisted actions | More reliable and compact when available | Platform-specific, incomplete, stale, or hidden behind custom widgets |
| Programmatic browser/desktop harness | Playwright, browser APIs, OS APIs, automation libraries | Code-driven actions and assertions | Fast, testable, repeatable | Requires engineering and may not match human UI state |
Anthropic’s Computer Use model was developed to interpret screenshots and reason about mouse actions; Anthropic notes that Claude may need to count pixels to click correctly. Anthropic OpenAI’s CUA uses raw pixels plus a virtual mouse and keyboard, enabling GUI interaction without OS or web APIs. OpenAI Google’s Gemini 2.5 Computer Use model is optimized for browser control, with screenshots, action history, function-call actions, and follow-up screenshots feeding the loop; Google says it is promising for mobile UI but is not optimized for desktop OS-level control. blog.google
The raw GUI ACI maximizes breadth but minimizes guarantees. It lets the agent act where no API exists. It also makes every visual ambiguity an execution risk: a modal appears, a button shifts, an animation delays, an element is occluded, a site changes copy, a click lands on the wrong row, or a screenshot fails to reveal hidden state. For consequential actions, the safe pattern is therefore not “let the GUI agent do everything.” It is “use GUI control for reach, then constrain, verify, and confirm.”
The lab systems reflect this. OpenAI’s Operator asked users to take over for login, payment details, and CAPTCHAs, supported user takeover, and used safeguards such as task limitations, watch mode, prompt-injection defenses, monitoring, and a detection pipeline. OpenAI Google’s Mariner allowed typing, scrolling, and clicking only in the active browser tab and asked for final confirmation for sensitive actions such as purchases. blog.google Anthropic’s Computer Use docs recommend screenshot verification after steps, keyboard shortcuts for difficult UI interactions, and warnings around login credentials and prompt injection. Claude Platform
5. Code-execution harnesses: beyond direct tools
A major 2025–2026 pattern is that agents do not merely call tools; they write and execute code that calls tools. This creates a hybrid ACI: the model uses the terminal or sandbox to generate a small program, and that program then interacts with APIs, files, browsers, databases, or MCP servers.
OpenAI’s Computer Use docs describe a “code execution harness” option in which the developer gives the model a way to write code using browser APIs, DOM APIs, Playwright, PyAutoGUI, loops, and conditionals, rather than forcing every step through a single screenshot/action turn. OpenAI Developers Anthropic’s “Code execution with MCP” post makes a stronger version of the same argument: agents can reduce context load and improve efficiency by writing code that filters, controls, and orchestrates tool calls instead of passing every tool schema and result through the model context. Anthropic
This is an important architectural shift. A naïve agent loop looks like:
model → tool call → observation → model → tool call → observation
A code-execution harness can instead look like:
model → writes script → script calls many APIs/tools locally → summarized result → model
That changes cost, latency, and reliability. It reduces token pressure, enables loops and conditionals outside the model, and makes some operations reproducible. It also adds sandboxing, resource limits, monitoring, and code-security requirements. Anthropic explicitly notes that code execution introduces sandbox, resource, and monitoring overhead. Anthropic
6. Skills, plugins, hooks, and durable state
As agents move beyond one-off actions, the ACI expands from “what can the model click?” to “what procedures, policies, and memories can the system reuse?”
Anthropic’s Agent Skills are folders of instructions, scripts, and resources that package procedural knowledge for Claude across environments; Anthropic described Skills as a cross-platform standard in late 2025. Anthropic Claude Code plugins can bundle skills, agents, hooks, MCP servers, LSP servers, and monitors, while hooks can run shell commands, HTTP endpoints, or LLM prompts at lifecycle events such as pre-tool-use, permission request, and post-tool-use. Claude
This is an ACI layer because it changes the action grammar. The agent is no longer choosing only among primitive actions such as “click,” “type,” or “run Bash.” It can invoke domain procedures: “run the migration checklist,” “use the release skill,” “call the incident-review hook,” “spin up a read-only explorer,” or “delegate to the build subagent.” That is closer to an operating-system abstraction than a UI automation trick.
Durable state is another sign of maturity. Claude Code docs discuss compaction and context management because context windows fill and performance degrades as they fill. Anthropic Anthropic’s Managed Agents architecture distinguishes the durable session log from Claude’s context window, allowing a harness to recover state, fetch event slices, and transform context before sending it back to the model. Anthropic
Survey of recent primary systems, 2025–2026
| System | Lab/project | Primary ACI stance | Key architectural details | What it shows |
|---|---|---|---|---|
| Anthropic Computer Use | Anthropic | GUI/browser/desktop-style screenshot control | Claude requests screenshots, clicks, typing, scrolling, dragging, keypresses; the developer application executes the tool loop; custom environments need virtualized or containerized execution. Claude Platform | General GUI control is possible, but still experimental and dependent on harness execution. |
| Claude Code | Anthropic | Terminal/repo harness | Reads files, edits code, runs commands, integrates with IDEs and developer tools; loop is gather context, act, verify; permissions, hooks, MCP, skills, subagents, and plugins shape autonomy. Claude API Docs+2Claude+2 | Coding agents are not just models; they are command-line operating environments with policy and verification. |
| Claude Managed Agents | Anthropic | Meta-harness / brain-hands separation | Splits session, harness, and sandbox; decouples Claude and harness from action environments; credentials can be held outside sandboxes; hands can be containers, tools, MCP servers, or other environments. Anthropic | The frontier architecture is explicitly moving toward stable interfaces around replaceable harnesses and sandboxes. |
| Operator / CUA | OpenAI | Browser GUI via screenshots and virtual mouse/keyboard | Operator used its own browser; CUA combined GPT-4o vision and reinforcement-learning reasoning; actions included typing, clicking, scrolling, and browser control without custom APIs. OpenAI | Browser control is the universal web interface, but needs supervision and safety layers. |
| ChatGPT agent | OpenAI | Hybrid virtual computer | Combines visual browser, text browser, terminal, direct API access, connectors, code execution, and user takeover in one agent mode. OpenAI | Mature agents converge on multiple ACIs under a single product surface. |
| Codex CLI / Codex app | OpenAI | Terminal, sandbox, worktree, cloud/desktop agent | CLI reads and edits local code and runs commands; sandbox and approval flows constrain actions; cloud Codex runs tasks in isolated sandboxes; Codex app coordinates multiple agents in worktrees. OpenAI+3OpenAI Developers+3OpenAI Developers+3 | Software agents need isolation, reviewable diffs, tests, and parallel state spaces. |
| Project Mariner | Browser agent | Gemini-based browser prototype reasoned over screen pixels and web elements; allowed click/type/scroll in active browser tab and required final confirmation for sensitive actions. blog.google | Browser agents benefit from combining visual and structural web information. | |
| Gemini 2.5 Computer Use | Browser-control model/API | Receives screenshots and action history, emits UI actions, returns safety decisions, and loops with client-executed actions; optimized for browser control and promising for mobile UI, not desktop OS-level control. blog.google | Computer-use models are becoming API products, not only demos. | |
| MCP | Anthropic-origin, cross-vendor ecosystem | API/protocol ACI | Hosts, clients, and servers communicate via JSON-RPC; MCP standardizes context sharing, tool exposure, and composable workflows; later releases add registry, server identity, async operations, and extensions. Model Context Protocol | Tool interfaces are being standardized, but auth, context load, and security remain hard. |
| OpenCode | Open-source project | Terminal/IDE/desktop coding agent | Provides bash, edit, write, read, grep, glob, LSP, patching, web tools, custom tools, MCP servers, permissions, and per-agent modes. OpenCode+2OpenCode+2 | Open ecosystems are recreating the same harness patterns as lab systems. |
The pattern is unmistakable: no serious agent remains a pure chatbot. Each becomes a governed execution environment.
Tradeoffs across modalities
| Modality | Breadth | Reliability | Latency | Cost | Safety controllability | Best use | Core failure mode |
|---|---|---|---|---|---|---|---|
| Raw GUI / screen capture | Very high | Low–medium | High | High | Medium | Long-tail human interfaces, closed apps, visual tasks | Mis-clicks, hidden state, visual ambiguity |
| Browser GUI | High | Medium | Medium–high | Medium–high | Medium | Web workflows, authenticated sites, shopping, forms, research | Prompt injection, site changes, slow loops |
| Browser DOM / automation harness | Medium–high | Medium–high | Medium | Medium | Medium–high | Repeatable web tasks, scraping, browser tests | DOM mismatch, dynamic sites, brittle selectors |
| Terminal | Medium | High in code domains | Low–medium | Low–medium | Medium | Coding, data work, build/test/debug loops | Destructive commands, secret exposure, environment drift |
| Direct API/function calling | Medium when integrations exist | High | Low | Low | High | Business systems, data retrieval, precise side effects | Integration burden, schema/tool overload, auth complexity |
| MCP/tool protocol | Medium–high | Medium–high | Medium | Medium | Medium–high | Cross-app tool ecosystems | Context bloat, server trust, authorization complexity |
| Code-execution harness | High inside sandbox | Medium–high | Low after setup | Low–medium | Medium | Multi-step data/tool workflows, filtering, orchestration | Unsafe generated code, sandbox/resource overhead |
The table compresses a core principle: breadth and reliability usually trade off. Pixels are broad because every human-facing interface eventually becomes pixels. APIs are reliable because they expose typed operations and structured state. The best agents use both.
A browser agent can buy groceries on a website with no API integration, but it may spend many expensive turns reading screenshots, clicking through modals, and recovering from visual surprises. An API agent can call create_order with a typed payload and receive a confirmation ID, but only if the API exists, the auth is configured, the schema is correct, and the user has granted authority. A terminal agent can fix code efficiently because it has tests, files, and diffs, but that same terminal access can mutate the user’s environment or expose credentials.
The architectural lesson is not that one modality wins. It is that each modality is a different point on the curve:
GUI maximizes reach.
API maximizes semantic clarity.
Terminal maximizes developer leverage and verifiability.
Browser maximizes access to human web workflows.
Code execution maximizes intra-task compression and procedural control.
Policy layers maximize survivability under real-world authority.
The convergence pattern: hybrid ACIs
The recent lab systems all converge toward hybrid interfaces.
OpenAI’s ChatGPT agent is the most explicit example: it combines a visual browser, text browser, terminal, direct API access, connectors, code execution, and a virtual computer that preserves task context across tools. OpenAI Anthropic’s Managed Agents architecture makes the convergence more abstract: the “brain” can connect to many “hands,” and a hand can be a sandbox, a custom tool, an MCP server, or another execution environment. Anthropic Google’s Mariner/Gemini Computer Use trajectory similarly moves from a browser prototype toward API-accessible computer-use models, with agent tools, MCP compatibility, and automation partners mentioned in Google’s 2025 agent announcements. blog.google
Why does convergence happen?
First, no single interface has enough coverage. Browsers handle web tasks but not local repos. Terminals handle code but not arbitrary authenticated web forms. APIs handle structured services but not long-tail websites. Screenshots handle everything visible but poorly. Agents that matter in production need a router across all of them.
Second, verification requires modality switching. A browser action may need API verification. A terminal command may need visual inspection. A spreadsheet edit may require code execution plus GUI export. A web research task may use a text browser for retrieval, a terminal for analysis, and a visual browser for authenticated pages.
Third, safety is easier when the system can choose the least risky interface. Reading a calendar through an API connector may be safer and cheaper than visually logging into a calendar website. Filling a web form may be acceptable through browser control until the final “submit” action, where the system should ask for explicit confirmation. Running a test suite is safer than trusting a model’s belief that a code edit works.
Fourth, cost and latency force specialization. Vision loops are expensive relative to text and structured tool calls. OpenAI’s docs note that a code-execution harness can use programmatic browser APIs, loops, and conditionals for speed and token efficiency. OpenAI Developers Anthropic argues that code execution can reduce token use and latency by letting agents filter and orchestrate large tool outputs programmatically rather than stuffing every tool definition and result into the context window. Anthropic
Hybrid ACI is therefore not a luxury feature. It is the natural endpoint of trying to make agents useful, reliable, and affordable.
Reliability: what makes an interface agent-friendly?
An interface is agent-friendly when the agent can observe state compactly, act precisely, and verify outcomes cheaply.
| Agent-friendly property | Browser GUI | Terminal | API | Raw GUI |
|---|---|---|---|---|
| Compact observation | Medium | High | High | Low |
| Precise action | Medium | High | Very high | Low |
| Cheap verification | Medium | High | High | Low–medium |
| Low integration burden | High | High for dev tasks | Low | Very high |
| Low ambiguity | Medium | High | Very high | Low |
| Works on long-tail systems | High | Medium | Low–medium | Very high |
For AI Engineering, this implies a practical design rule: prefer the most structured interface available, fall back to less structured interfaces only when needed.
A robust agent might use this routing order:
Use an API when a scoped, typed operation exists.
Use terminal/code execution when the task involves files, tests, data, or repeatable computation.
Use browser DOM or programmatic automation when a web workflow is predictable.
Use visual browser/GUI control when the environment is human-facing or otherwise unavailable.
Ask the user or require confirmation when the next action has external, irreversible, private, financial, legal, medical, or reputational consequences.
That ordering is not ideological. It follows from the information structure of the interfaces. A screenshot of a button is a weak affordance. A typed API operation with an idempotency key, permission scope, and response object is a strong affordance.
Safety: the control plane is part of the interface
A deployed ACI must include a Control Plane. Otherwise, the agent has capabilities but no legitimate authority model.
The control plane contains:
| Control-plane component | Function |
|---|---|
| Permissions | Decide which actions are allowed, denied, or require approval |
| Sandboxing | Limit what files, processes, networks, and credentials the agent can reach |
| Identity and auth | Bind actions to users, service accounts, OAuth scopes, or delegated roles |
| Human confirmation | Require approval before consequential or irreversible side effects |
| Monitoring | Detect prompt injection, suspicious behavior, policy violations, or unsafe tool use |
| Audit logging | Record observations, tool calls, diffs, approvals, and external side effects |
| Recovery | Resume after model, harness, sandbox, network, or tool failure |
| Cost control | Limit loops, tokens, tool calls, compute, and wall-clock time |
Every major system in this space exposes some version of that layer. Operator used takeover mode, task limitations, watch mode, prompt-injection detection, and a monitor model. OpenAI ChatGPT agent adds user confirmations for consequential actions, watch mode for certain critical tasks, refusals for high-risk tasks, privacy controls, and secure browser takeover mode. OpenAI Claude Code uses fine-grained allow/ask/deny permissions, command pattern matching, modes, and hooks. Claude+2Claude+2 Codex CLI uses sandbox boundaries and approval flows when actions cross those boundaries. OpenAI Developers
This is why “can the model do X?” is often the wrong question. The better question is: under what interface, authority, verification, and recovery regime can the system do X without unacceptable risk?
Browser vs terminal vs API vs GUI: architectural comparisons
Browser-as-interface
Browser ACIs are best when the task lives on the open or authenticated web and the agent needs to interact with sites designed for humans. They are especially useful for tasks such as comparison shopping, reservation search, form filling, itinerary construction, dashboard reading, public-web research, and workflows across services.
Their main reliability bottleneck is environmental variability. Sites change. Ads appear. Modals interrupt. Infinite scroll behaves inconsistently. Browser state depends on cookies, viewport, device, locale, and login state. The browser is also an adversarial environment: the same page that supplies task-relevant content can contain malicious prompt-injection instructions.
Browser agents therefore need multi-layered verification. After clicking, they should inspect the new page. Before submitting, they should summarize intended side effects. For purchases, account changes, messages, or bookings, they should request confirmation. The primary sources show this norm across labs: Operator hands control back for sensitive tasks, Google Mariner asks final confirmation for sensitive actions, and ChatGPT agent requires confirmations and supervision for certain high-impact tasks. OpenAI+2blog.google+2
Terminal-as-interface
Terminal ACIs are best for software engineering, data analysis, reproducible computation, infrastructure tasks, and workflows where verification can be expressed as commands. They are powerful because they expose both state and action in text.
The terminal is not just a UI. It is a compact Execution Environment. A coding agent can inspect repository history, search for symbols, edit files, run tests, check types, benchmark, install dependencies, and produce diffs. This makes terminal agents more reliable than browser agents in domains where the task has executable tests.
But the terminal is dangerous because it is close to the operating system. Shell access compresses many forms of authority into a small syntax. rm, curl, git push, npm install, ssh, and environment-variable reads are all text commands, but their side effects differ radically. That is why Claude Code, Codex CLI, and OpenCode all make permissions, approval, sandboxing, or per-agent modes central parts of their interface. Claude+2OpenAI Developers+2
API-as-interface
API ACIs are best for reliable access to structured systems. They are the right interface for enterprise workflows when the operation can be represented as a scoped capability with explicit input and output.
The API’s advantage is that it reduces the model’s perceptual burden. Instead of visually identifying a “Send invoice” button, the agent can call send_invoice(customer_id, amount, due_date) and receive a structured result. Instead of reading a dashboard screenshot, it can query metrics. Instead of scraping email visually, it can call a connector.
But APIs are not free. They require integration work, schema design, auth flows, permission scoping, rate limits, and robust error handling. MCP tries to standardize part of that interface, but tool ecosystems still face context bloat and trust problems. OpenCode warns that MCP servers add to context and can exceed limits; Anthropic’s code-execution-with-MCP post argues for code-based tool access to reduce context pressure. OpenCode
GUI-as-interface
GUI ACIs are best as a universal fallback. They let an agent operate where humans operate. They are especially important for legacy applications, SaaS tools without APIs, internal dashboards, mobile apps, and mixed visual workflows.
The problem is that GUIs are optimized for humans, not for agents. Human perception tolerates ambiguity, contextual knowledge, and manual correction. Agents need machine-checkable state. A visual button labeled “Continue” is less informative than a typed operation with declared side effects.
This does not make GUI control unimportant. It makes it a broad but weak substrate. The practical architecture is to combine GUI reach with structured verification and human oversight.
The “agent OS” question
The open question is whether a unified ACI emerges — something like an operating system for agents — or whether interfaces remain fragmented.
The argument for a unified agent OS is strong. Across browser agents, coding agents, and tool protocols, the same abstractions keep recurring:
| Recurring abstraction | OS analogy | Agent-system version |
|---|---|---|
| Process | Running program | Agent session, task, subagent, worker |
| File | Durable state object | Session log, memory file, artifact, diff |
| Syscall | Kernel-mediated operation | Tool call, MCP request, browser action, shell command |
| Permission | User/group/capability | OAuth scope, allow/deny rule, approval prompt |
| Device | Hardware abstraction | Browser, terminal, sandbox, mobile app, API server |
| Scheduler | Process orchestration | Multi-agent task routing, background mode, recurring tasks |
| Signal | Interrupt/control event | User takeover, pause, stop, clarification, monitor intervention |
| Standard library | Common reusable functions | Skills, plugins, connectors, tools, MCP servers |
Anthropic’s Managed Agents post explicitly invokes operating-system history: durable abstractions such as process and file outlast hardware, and Managed Agents attempts a meta-harness with stable interfaces around sessions, harnesses, and sandboxes. Anthropic MCP’s specification likewise positions itself as a standard way to expose tools and context across AI applications, inspired in part by ecosystem-wide standardization patterns such as the Language Server Protocol. Model Context Protocol
But the argument against a single unified agent OS is also strong. Interfaces differ for good reasons. Browser control, terminal access, enterprise APIs, mobile app control, desktop accessibility trees, and cloud sandboxes have different trust models, latency profiles, legal obligations, and platform owners. A payment action, a code change, a medical-record read, a Git push, and a browser click should not be abstracted into the same undifferentiated “action” without preserving authority and risk semantics.
The likely near-term outcome is convergence in architecture, fragmentation in implementation. We should expect common patterns — session logs, sandboxes, MCP-like tool servers, skills, permissions, monitor models, worktrees, and human approvals — but not one universal interface that replaces browsers, terminals, APIs, and GUIs. The “agent OS” may emerge as a control plane and runtime above heterogeneous interfaces, not as a single interface beneath them.
Thin and contested evidence
Several claims in this field are still thin, contested, or vendor-specific.
First, benchmark comparability is weak. Browser and computer-use benchmarks differ in environment, task distribution, allowed tools, number of attempts, prompting, and grading. OpenAI, Google, and Anthropic report useful numbers, but many are self-reported or evaluated under different harnesses. OpenAI’s CUA blog reports OSWorld, WebArena, and WebVoyager scores, while Google’s Gemini Computer Use release claims leading performance and latency on web and mobile-control benchmarks; those are informative but not yet a stable public science of ACI quality. OpenAI
Second, “computer use” sometimes means different things. Anthropic’s Computer Use is a general screenshot/action tool interface. OpenAI’s Operator began browser-first, then ChatGPT agent became a hybrid virtual computer. Google’s Gemini 2.5 Computer Use model is optimized for browsers and promising for mobile UI, but not desktop OS-level control. Claude Platform+2OpenAI+2 Treating these as directly interchangeable hides important differences.
Third, safety mitigations are necessary but not proven complete. The primary sources consistently discuss prompt injection, monitor models, watch mode, confirmations, takeover, refusals, and privacy controls, but they also acknowledge residual risk. OpenAI For agentic systems acting through browsers and authenticated services, there is no evidence that prompt injection is “solved.”
Fourth, standardization is real but incomplete. MCP has broad momentum and formal protocol machinery, including JSON-RPC structure, authorization requirements, transports, registries, SDKs, and vendor adoption. Anthropic+3Model Context Protocol+3Model Context Protocol+3 But tool descriptions, auth scopes, user consent, server trust, context limits, audit semantics, and liability boundaries remain fragmented.
Design principles for ACI architecture
1. Treat the ACI as a first-class architecture layer
Do not bury the interface in a prompt. The ACI should be designed explicitly: what the agent can see, what it can do, what state persists, what authority it has, what gets logged, and when humans intervene.
2. Prefer structured interfaces, but keep GUI fallback
Use APIs and terminal commands when they expose reliable semantics. Use GUI control when the system is only available through human-facing surfaces. A mature agent should route across both.
3. Separate brain, harness, hands, and state
The model should not be conflated with the action environment. Anthropic’s Managed Agents architecture is a useful reference pattern: durable session outside the context window, harness outside the sandbox, credentials outside generated-code environments, and hands exposed as replaceable tools. Anthropic
4. Make permissions compositional
Permissions should be scoped by tool, command pattern, environment, credential, task, and risk level. Claude Code and OpenCode both expose allow/ask/deny-style controls, while Codex uses sandbox and approval boundaries. Claude+2OpenCode+2
5. Design for verification, not just action
Every action channel should have a corresponding verification channel. Browser clicks need screenshots or DOM checks. File edits need diffs and tests. API calls need response IDs and idempotency. Terminal commands need exit codes and logs. GUI actions need state inspection and sometimes human confirmation.
6. Put untrusted content and credentials in different places
Prompt injection becomes more dangerous when the same agent context includes untrusted webpage text, privileged credentials, and action tools. Anthropic’s Managed Agents post describes a structural fix in which tokens are not reachable from the sandbox where generated code runs, with credentials held in a vault or bundled only with specific resources. Anthropic
7. Expect context to be an external object
Long-running agents cannot rely only on the model context window. They need session logs, recoverable event streams, summaries, files, memory, and mechanisms for selective retrieval. Claude Code’s context-management docs and Anthropic’s Managed Agents session-log design both point in this direction. Anthropic
8. Let agents write code when loops are too expensive
For multi-step tool workflows, code execution can collapse many model-tool turns into one script-driven procedure. This can reduce latency and context use, but only when paired with sandboxing, monitoring, and resource limits. OpenAI Developers
Conclusion: the interface is the product
The 2025–2026 evidence suggests that agent progress is not only model progress. It is also interface progress: better browser loops, safer terminals, typed tools, MCP servers, code execution, durable sessions, permissions, sandboxes, monitor models, and hybrid routing.
The strongest agents are becoming less like chatbots and more like governed computing environments. They have browsers for reach, terminals for leverage, APIs for precision, GUIs for the long tail, code execution for compression, and control planes for authority. The open architectural question is whether these pieces become a unified agent runtime — an “agent OS” — or remain a family of partially compatible interfaces. The most likely answer is both: fragmented hands below, increasingly unified control planes above.
Companion entries
Core theory: Agent-Computer Interface, Agentic Loop, Tool Use, Control Plane, Environment State, Human-Computer Interaction, World Models for Agents
Modalities: Browser Agents, Terminal Agents, API Agents, GUI Grounding, Computer Use, Accessibility Tree, Screen Capture Interfaces, Code Execution Harnesses
Protocols and runtimes: Model Context Protocol, Agent2Agent Protocol, Agent Skills, Tool Registries, Remote MCP Servers, Function Calling, Agent OS
Engineering practice: Sandboxing AI Agents, Permission Systems for Agents, Context Engineering, Worktrees for AI Agents, Agent Evaluation Harnesses, Human-in-the-Loop Control, Prompt Injection Defense
Counterarguments and risks: Prompt Injection, Tool Poisoning, AI Agent Misuse, Credential Exposure, Autonomy Limits, Benchmark Fragility, Interface Fragmentation