Long-Horizon Agents: Where Capability and Reliability Diverge

LLM agents have improved sharply on short and medium multi-step tasks, but end-to-end reliability degrades as horizons stretch across more tool calls, more elapsed time, more context substitutions, and more irreversible side effects. The capability curve and the reliability curve are not the same curve — and the gap between them is where most production decisions about agent autonomy actually live. This article maps the divergence: what "long horizon" operationally means, what the published evidence shows, why stepwise success fails to compose, what architectural responses exist, what those responses cost, and what threshold would justify unsupervised deployment for a given workflow class.

Coverage note: verified through May 6, 2026.

1. What "long horizon" means

"Long horizon" is not one axis. The phrase is overloaded across at least five operational dimensions, and conflating them is the most common source of confusion in this literature:

Axis	What it measures	Example
Tool-call count	Number of discrete external actions	A 12-call bug fix versus a 3-call lookup
Wall-clock duration	Elapsed real time the agent operates	A 30-minute coding task versus a three-day research workflow
Context substitutions	How many times the working context must be rebuilt or compacted	A single-shot prompt versus a multi-session investigation
Branching factor	Number of subtask handoffs and merges	A linear plan versus a coordinated multi-agent project
Action risk	Reversibility and blast radius of side effects	An ephemeral sandbox edit versus a production database migration

A 12-tool-call bug fix is "long" along axis one. A three-day investigation is long along axes two and three. A coordinated multi-agent project is long along axis four. An irreversible production migration is long along axis five even when it completes in seconds.

The literature's most prominent metric, METR's task-completion time horizon, sits primarily on axis two but is defined more carefully than the colloquial usage suggests. METR's task-completion time horizon is the human-expert task duration at which a given AI agent reaches a specified success probability — not the wall-clock duration the agent itself can operate unattended (METR — Measuring AI Ability to Complete Long Software Tasks, METR — Time Horizons dashboard). The 50% time horizon is an empirical capability proxy, not a delegation budget.

This distinction matters because the article's central claim — that capability and reliability diverge — depends on holding the axes apart. Capability gains tend to show up first along axes one and two, on tasks with clean grading signals and bounded environments. Reliability is what determines whether those gains survive axes three through five, where context, coordination, and consequence dominate. Treating "long-horizon agent" as a single category collapses the divergence into a vague forecast about autonomy. Treating the axes separately makes the engineering question tractable: which workflow, with which actions, at what required success percentile, under what monitoring and rollback regime, at what cost?

2. The empirical pattern

Three primary-source streams document the capability/reliability gap: METR's time-horizon program, the SWE-bench progression, and the engineering guidance from Anthropic and OpenAI. None of them, taken alone, justifies a smooth autonomy curve. Read together, they describe the gap precisely.

2.1 METR: time horizons are scaling, but 80% lags 50% by roughly 5x

METR's 2025 paper, Measuring AI Ability to Complete Long Software Tasks, introduced the task-completion time horizon as a way to compare AI agents across a battery of self-contained software, ML, and cybersecurity tasks scored against human-expert completion times (Kwa et al. 2025). The headline result: a roughly seven-month doubling in the 50% time horizon between 2019 and 2025, with frontier systems near-perfect on tasks humans complete in under about four minutes and under 10% success on tasks around four hours or more.

The 2026 Time Horizon 1.1 update expanded the task suite, reported faster recent trends on the new suite, and explicitly widened uncertainty at the long end (METR — Time Horizon 1.1). The update notes wide confidence intervals on long tasks, sensitivity to task composition, and limited human baselines for tasks beyond eight hours. These caveats are central, not footnoted: the long-end signal is the one most relevant to deployment claims, and it is also the noisiest part of the measurement.

The most operationally important number on the current METR dashboard is the gap between the 50% horizon and the 80% horizon. Approximate values from the dashboard (verified May 6, 2026) on the TH1.1 suite:

Model	p50 horizon	p80 horizon	p50/p80 ratio
Claude Opus 4.6	~719 min	~70 min	~10x
GPT-5.2	~352 min	~66 min	~5x
GPT-5.4	~342 min	~54 min	~6x

Two observations follow. First, the p50 horizon for frontier systems on this suite is now measured in hours, which is the kind of progress that drives the autonomy narrative. Second, the p80 horizon — closer to but still well below production-grade reliability — is roughly 5–10x shorter. The reliability curve trails the capability curve by something close to an order of magnitude on the same task distribution, and that ratio has not closed.

This is the single most important empirical fact about long-horizon agents in 2026. METR's 50% horizon makes scaling visible; the p50/p80 ratio makes the deployment problem visible.

2.2 SWE-bench: scaffold sensitivity is real and so is benchmark fragility

Where METR measures capability across task durations, SWE-bench measures whether agents can actually resolve real software issues in real repositories. The original SWE-bench paper introduced 2,294 GitHub issue tasks requiring repository understanding, multi-file edits, environment interaction, and gated test passing; the strongest reported model — Claude 2 with retrieval — resolved 1.96% of issues (Jimenez et al. 2024).

The trajectory since is informative in two directions. SWE-agent (Yang et al. 2024) showed that an agent-computer interface designed for the model to navigate, edit, and test code raised resolution rates substantially: 12.47% on the full SWE-bench and 18.00% on SWE-bench Lite with GPT-4 Turbo, but at 8–13× the cost-per-resolved-issue of a non-interactive RAG baseline. The interaction unlocked capability that bare prompting could not express; it also paid for that capability with tokens and time.

OpenAI's SWE-bench Verified (Chowdhury et al. 2024) tightened the benchmark itself: 500 human-verified samples after filtering 68.3% of annotated original samples for underspecification, unfair tests, or other validity issues, with GPT-4o reaching 33.2% on the verified subset under the best tested scaffold. The filtering rate is the part most often skipped in summaries — roughly two-thirds of annotated original samples failed verification, which says something about how much of the original benchmark was measuring task validity rather than agent capability.

The progression is real: 1.96% → 12.47% → 33.2% on increasingly trustworthy versions of the same task family, accompanied by genuine scaffold and model improvements. But two qualifications travel with these numbers. First, scaffold matters substantially — the SWE-agent gain over non-interactive RAG is a clean demonstration that interface design unlocks (or fails to unlock) latent capability. Second, public coding benchmarks are fragile evidence for current frontier reliability because of contamination, scaffold-tuning, and rapid leaderboard saturation. The historical curve is instructive; treating any single SWE-bench score as a stable measurement of present-day production readiness is not.

2.3 Vendor guidance: the convergence is on harnessed delegation

Anthropic and OpenAI engineering guidance, taken at face value, reads less like a marketing argument for autonomous agents and more like an engineering argument for harnessed delegation under explicit budgets, guardrails, and rollback boundaries.

Anthropic's Building effective agents explicitly distinguishes workflows (orchestrated calls with deterministic structure) from agents (models directing their own steps), and recommends starting with the smallest composable system that solves the problem (Anthropic Engineering — Building effective agents). The follow-on harness writeup — Effective harnesses for long-running agents — argues that compaction alone is insufficient for long workflows and recommends progress files, deliberate initializer behavior, frequent commits, and explicit incremental sessions to bound state loss (Anthropic Engineering — Effective harnesses for long-running agents). The eval guidance — Demystifying evals for AI agents — adds that long-horizon evaluation needs trace-level grading, not just outcome grading (Anthropic Engineering — Demystifying evals for AI agents).

OpenAI's practical guide to building AI agents foregrounds guardrails, human intervention thresholds for failure, and explicit handling for high-risk actions (OpenAI — A practical guide to building AI agents). The computer-environment writeup frames long-running work around persistent workspaces, timeouts, retries, and compaction — the language of orchestration rather than agency (OpenAI — Equip Responses API with a computer environment). The 2026 Agents SDK update describes long-horizon harnesses, sandboxes, and tracing as core surfaces, not optional extras (OpenAI — The next evolution of the Agents SDK). The evaluation-best-practices and agent-builder-safety guides add structured outputs, datasets, eval runs, and least-privilege tool design (OpenAI — Evaluation best practices, OpenAI — Agent builder safety).

The convergent message across both vendors: production agents are systems with explicit state, explicit checkpoints, explicit budgets, explicit guardrails, explicit traces, and explicit human escalation rules. The agent loop is necessary but not sufficient. The reliability is in the harness.

3. Why stepwise success does not compose

The capability/reliability divergence has two distinct causes that are easy to merge but worth keeping separate: ordinary reliability arithmetic, and agent-specific state failures.

3.1 The arithmetic floor

If a workflow needs every one of N steps to be correct and independently uncovered, end-to-end success is at most the product of per-step success rates:

Per-step success	10 steps	30 steps	100 steps
95%	60%	21%	0.6%
99%	90%	74%	37%
99.9%	99%	97%	90%

This is a floor, not a ceiling, and it is optimistic in three ways. Real failures are correlated (a context-window event corrupts many subsequent steps simultaneously). Some errors are unrecoverable (an irreversible tool action cannot be retried into success). And recovery itself is an additional reliability surface — a recovery attempt that calls more tools, more model evaluations, and more state mutations may compound rather than restore.

The arithmetic by itself explains a great deal of what is colloquially called "agent failure on long tasks." If each step is merely pretty good, a 30-step workflow is bad. That is not mysterious agency failure; it is multiplication. It is also why per-step accuracy gains in base models translate non-linearly into end-to-end reliability — moving from 95% to 99% is a 5x reduction in per-step error and a roughly 35x improvement in 100-step end-to-end success, which is why frontier capability gains feel disproportionately large at the workflow level.

3.2 Agent-specific failure surfaces

The arithmetic is not the whole story. Agent systems add specific architectural failure surfaces that single-call inference does not have: lossy context compaction, tool side effects on shared state, stale execution state, subtask handoff loss, evaluator blind spots correlated with worker blind spots, and recovery actions that mutate the environment beyond the original failure's blast radius. These are empirically separable from generic compounding error if traces are instrumented to label failures by category at the step where they first become unrecoverable.

The cleanest framing: compounding error is the mathematical backbone, and agent-specific state failures are the engineering surfaces that determine whether per-step error is independent or correlated, recoverable or absorbing, observable or silent. A workflow built on reversible actions, deterministic checks, typed handoffs, and external state is one where the arithmetic is forgiving. A workflow built on free-form natural-language plans, mutable shared state, and recoveries that keep editing the same artifacts is one where the arithmetic is unforgiving and the agent-specific surfaces are load-bearing.

4. A failure-mode taxonomy

The taxonomy below is operational rather than causal. Each row names a failure mode, gives its symptoms during a trace, lists primary mitigations, and — importantly — names the second-order risks the mitigation introduces. Frequency and causal weight in production remain undermeasured outside vendor-internal telemetry; treat the rows as a working trace-labeling vocabulary rather than as established prevalence claims.

Failure mode	Symptoms in trace	Primary mitigation	Second-order risk introduced
Context drift	Agent cites stale goals, ignores newer constraints, reopens settled branches	Externalized task state, summaries with provenance, retrieval tests	Memory layer can preserve a wrong abstraction; stale-context pollution; injection surface
Plan abandonment	Tool calls no longer map to declared next steps	Explicit plan tree with checkpoints and branch status	Bureaucratic commitment to a bad decomposition; failure to replan when premise shifts
Error compounding	Small uncorrected mismatch becomes premise for later work	Step-level verification, reversible actions, golden tests, typed outputs	Verification cost can dominate budget; type discipline can hide ambiguity
Cycle behavior	Repeated searches, edits, or summaries without new evidence	Loop detectors, novelty checks, budget caps	False-positive loop kills on legitimate iteration; budget caps interact with recovery
Recovery failure	Agent notices failure, then retries the same strategy or fabricates completion	Checkpointing, replay buffers, rollback paths, escalation rules	Replay buffers can preserve sensitive traces; rollback assumes reversibility that may not hold
Coordination loss	Subagents duplicate work, disagree on state, or overwrite assumptions	Clear ownership, artifacts as contracts, independent integration review	Handoff overhead grows; integration becomes its own reliability bottleneck
Tool brittleness	Failures correlate with cwd, auth, path, API, or platform differences	Sandbox manifests, dependency pinning, tool affordance tests	Pinned environments diverge from production; sandbox itself becomes a failure surface
Supervisor false assurance	Supervisor approves traces without independent evidence	Diverse verifier models, deterministic checks, human review for irreversible acts	Verifiers can share blind spots with workers; deterministic checks can be evaded by surface compliance
Benchmark overinterpretation	"Agent passes SWE-bench therefore agent is production-ready"	Separate benchmark scores from deployment claims; cite methodology limits	Disciplined benchmarking is slower and harder to publish
Cost and latency blowup	Reliability gains require disproportionate tokens, wall time, or tool calls	Budget accounting, marginal-value evals, cheaper deterministic checks	Aggressive cost ceilings can starve recovery; deterministic substitutes can lose capability
Prompt-injection exposure	Untrusted text influences tool calls or credential-bearing environments	Least-privilege tools, approvals, sandbox isolation, structured extraction	Approval friction reduces benefit of automation; least privilege fragments tool ecosystem

A few remarks on this table.

First, the failure modes are not cleanly separable in practice. A typical bad trace shows context drift, then a plan-abandonment moment as the agent rationalizes a locally plausible next step from drifted state, then error compounding as that step generates artifacts treated as evidence by later steps, then recovery failure as the agent attempts to fix symptoms without diagnosing the underlying state mismatch. The taxonomy is most useful as a labeling vocabulary for trace grading, not as a causal partition.

Second, the second-order risk column is the part most often missing from architecture writeups. Mitigations are not free reliability — they redistribute risk. The articles that argue for "better memory plus plan trees plus replay buffers plus supervisor models" as a reliability stack are describing a control system whose components themselves fail in characteristic ways. A faithful failure-mode catalog includes the mitigations' own failure surfaces.

Third, "benchmark overinterpretation" is included as a failure mode of the observer system, not the agent. It is the most common way that the field misallocates production trust. METR's caveats and SWE-bench Verified's filtering rate are present in the literature precisely so that published numbers can be read responsibly. Treating a 50% time horizon as a delegation threshold, or treating a SWE-bench Verified score as a production reliability claim, is a category error.

5. Architectural responses as a control system

The published architectural responses fall into a small set of mechanism types. Each is necessary for some failure modes and insufficient on its own; collectively they form a control system that is only useful when its components are auditable and task-specific.

Externalized memory. Persistent stores — progress files, structured artifacts, append-only logs — preserve task state across context compactions and session boundaries. Anthropic's harness writeup describes progress files and frequent commits as central to long-running work (Anthropic Engineering — Effective harnesses for long-running agents). The risk surface is staleness: memory without provenance becomes a drift vector, and memory with weak access controls becomes a prompt-injection target.

Plan trees. Explicit decomposition with checkpoints and branch status converts an implicit plan in the model's working context into an externally inspectable artifact. This makes plan abandonment detectable and supports independent integration review across subtasks. The risk is bureaucratic commitment: a plan tree without replanning semantics hardens premature decomposition and slows recovery when the original plan was wrong.

Replay buffers and tracing. Step-by-step traces with deterministic replay support post-hoc failure analysis, regression testing, and recovery from the last known good checkpoint. OpenAI's Agents SDK and evaluation guidance treat tracing as a core surface (OpenAI — The next evolution of the Agents SDK, OpenAI — Evaluation best practices). Anthropic's eval guide makes the same case for trace-level grading (Anthropic Engineering — Demystifying evals for AI agents). The risks: replay preserves whatever was logged, including misleading or sensitive context, and replay-driven recovery can overfit to specific prior failure shapes rather than generalizing.

Supervisor models. A second model, often a smaller one with different prompting or a different specialty, evaluates worker outputs against checkpoints. This is a load-bearing reliability claim only when the supervisor's failure modes are uncorrelated with the worker's. If the supervisor and worker share training data, prompt framing, or task incentives, the supervisor collapses into a correlated rubber stamp. Independent evidence — deterministic checks, gated tests, externally constructed verifiers — is what gives supervision real teeth.

Sandboxing and rollback. Bounded execution environments with explicit reversibility move axis-five risk (action consequence) out of the agent loop and into the infrastructure. The OpenAI Agents SDK update describes sandboxes and timeouts as primary primitives for long-running work (OpenAI — The next evolution of the Agents SDK). The risk: sandbox semantics diverge from production semantics, and "reversible" actions may have side effects (logs, audit trails, downstream notifications) that the sandbox does not capture.

Eval traces and the closing of the loop. All of the above only become a control system when their outputs feed back into the system that decides whether to deploy, retry, escalate, or stop. Anthropic and OpenAI both recommend trace-graded evals over outcome-only evals for long-horizon work (Anthropic Engineering — Demystifying evals for AI agents, OpenAI — Evaluation best practices). The closing of this loop — turning trace data into deployment decisions with explicit thresholds — is what separates an instrumented system from an instrumented demo.

This is the architect's frame, and it is right as far as it goes. But it should not be read as a clean reliability stack. The mechanisms are necessary, not sufficient; they shift the failure surface rather than eliminating it; and they introduce real costs in latency, tokens, complexity, and operational burden. The question for any specific deployment is which mechanisms are load-bearing for this workflow's reliability target, not which mechanisms are nominally present in some harness.

6. Model-shaped or scaffold-shaped?

The most common framing of the open empirical question is binary: are long-horizon failures model-shaped (so they need better base models) or scaffold-shaped (so they need better harnesses)? This binary is false on its own terms, but the underlying question — which intervention changes the marginal failure distribution under controlled tests — is real and undermeasured.

The evidence supports both directions. Larger and newer models shift METR's time horizons upward, with the seven-month doubling trend across 2019–2025 and continued gains on the TH1.1 suite (METR — Time Horizon 1.1). Scaffold design also moves outcomes substantially: SWE-agent's gain over non-interactive RAG (Yang et al. 2024) is a clean demonstration that interface design unlocks latent capability that the same base model cannot express through bare prompting. Anthropic's harness writeup and OpenAI's scaffold comparisons both show that wrappers materially affect end-to-end reliability (Anthropic Engineering — Effective harnesses for long-running agents, OpenAI — Equip Responses API with a computer environment).

Reliability is system-shaped. The relevant inputs are model capability, harness design, task distribution, tool ecosystem, observability, cost budget, evaluator strength, permission scope, and loss function. A better model reduces per-step error. A better harness prevents some errors from compounding. A better workflow makes errors cheap to detect and reverse. None of these alone establishes production reliability, and any of them can dominate the failure distribution for a given workflow.

The cleanest way to make progress on the question is a factorial ablation, not a philosophical position. The design that would actually be decisive:

30–50 private, realistic long tasks requiring 10–50 tool calls, code or document state changes, hidden acceptance tests, and at least one required recovery from an injected fault.
Two or three frontier base models held constant across harnesses.
Five harnesses: bare ReAct-style loop; persistent memory; plan tree plus replay buffer; supervisor-evaluator model; combined scaffold.
One harness held constant across the multiple base models.
Trace instrumentation with failure labels drawn from the taxonomy in section 4.
Metrics: final success, p80/p95 task success, intervention count, cost, wall time, irreversible-error rate, recovery-after-fault rate, supervisor catch rate versus rubber-stamp rate.

This design would directly estimate whether current failures are mostly model-limited, scaffold-limited, or interaction-limited. It would also produce a per-failure-mode breakdown of where each intervention reduces error and where it just shifts errors around. Public versions of this experiment exist in fragments — METR's task suite, SWE-bench Verified's filtering, scaffold-comparison subsets in vendor blog posts — but a coordinated factorial run across realistic, non-contaminated tasks with trace grading is what would settle the question. Until that exists, the model-versus-scaffold binary is best treated as a question waiting for evidence rather than a claim with sides.

7. Critiques: when this whole frame is the wrong frame

The architectural frame in section 5 and the empirical frame in section 2 share an assumption worth surfacing: that the right unit of analysis is the agent and its reliability. There is a serious counter-position that the article should not flatten.

The counter-position runs roughly as follows. The reason long-horizon agent reliability is hard is not primarily that we lack a long-horizon-capable cognitive substrate. It is that we keep asking a single stochastic model to serve simultaneously as planner, executor, state database, workflow engine, QA department, tool selector, recovery operator, and accountable owner. That role allocation creates a single correlated failure domain. The reliable engineering direction in production is often to remove autonomy — to convert open-ended work into typed state machines, narrow tools, deterministic checks, reversible actions, explicit budgets, and human escalation. In that direction, "agentic" is a description of where flexibility is permitted within an otherwise structured workflow, not a description of who is in charge.

This counter-position has empirical support. Anthropic's distinction between workflows and agents, with the recommendation to start from the smallest composable system, is exactly this argument in vendor language (Anthropic Engineering — Building effective agents). The OpenAI guide's emphasis on guardrails, human intervention thresholds, and high-risk action handling is the same argument from the other vendor (OpenAI — A practical guide to building AI agents). The deployments that have most clearly cleared production reliability bars in 2025–2026 — coding assistants, structured document workflows, narrow customer-service flows — are not unsupervised long-horizon agents; they are agents inside well-typed workflows with bounded action surfaces.

The counter-position has limits too. METR's evidence that capability scales is real and matters: the range of tasks an agent can sometimes complete is genuinely growing, and "sometimes" is sufficient for low-risk, reversible, reviewable work where outcome auditing replaces step auditing. SWE-agent-style results show that interface and scaffold design can unlock capability that a non-interactive model cannot express. Pretending agentic work is just "workflows with extra steps" misses the architectural specificity of long traces. The honest synthesis: the workflow-first counter-position is the right default for production today, and the capability scaling is real enough that the boundary between "needs to be a deterministic workflow" and "can be agent-driven" is moving.

The article's net read on this disagreement is that long-horizon agency is progressing, and the workflow-first critique is correct that production reliability is a separate system property that does not arrive automatically with capability. Both are true. The mistake is treating either as the whole story.

8. Deployment thresholds: not a date, a per-workflow gate

The most common version of the deployment question — when do agents become reliable enough for unsupervised production? — is the wrong question because it has no single answer. There is no general property of frontier agents that crosses a threshold one Tuesday in 2027. The right question is per-workflow:

Bounded task class. What is the distribution this agent operates on? Is it tightly scoped (e.g., a specific bug-fix workflow on a known codebase) or open-ended (e.g., "research and recommend")?
Required success percentile. Is the workflow tolerant of 80% success with retry, or does it require 99.9% with no recovery loop?
Reversibility. Are the actions reversible, partially reversible, or absorbing?
Observability. Can failures be detected before they propagate, or only post-hoc?
Independent verification. Is there a verifier whose failure modes are uncorrelated with the worker's?
Escalation rules. What triggers human intervention, and what is the cost of the intervention being late?
Cost envelope. What is the token, latency, and tool-call budget per task, and what is the marginal cost of an incremental reliability gain?
Rollback regime. Is there a known-good checkpoint to roll back to, and is rollback itself reliable?

A workflow that is bounded, observable, reversible, cheaply checkable, and economically tolerant of the supervision overhead is a candidate for unsupervised execution today, where "unsupervised" means no one watches every step rather than no one audits outcomes. A workflow that is open-ended, irreversible, hard to verify independently, and economically intolerant of supervision overhead is not a candidate for unsupervised execution at any METR horizon currently published.

This is why the 50%, 80%, 95%, and 99.9% reliability levels are not points on a single curve. They are different worlds.

50% reliability is a capability claim. It says the agent can sometimes do the task. This is enough for exploration, ideation, brainstorming, and any workflow where the human is sampling outputs and selecting.
80% reliability is a productivity claim. It says human review of every output is no longer the bottleneck — the human samples failures rather than successes. This is the regime where well-designed coding assistants and document-drafting agents currently operate.
95% reliability is a delegation claim. It says outcome auditing replaces step auditing for most tasks, and the human intervenes on flagged exceptions. This is roughly where unsupervised customer-service flows and well-typed back-office automation operate, with substantial structural support.
99.9% reliability is an infrastructure claim. It says the workflow is acceptable as a system component that other systems depend on. This is where most safety-critical, financial, medical, or regulated deployments need to be, and the current evidence does not support frontier agents operating at this percentile on long-horizon work without strong deterministic substrate underneath.

The METR p80 horizons for frontier systems are roughly an hour. That is impressive as capability and weak as deployment evidence. The gap between p80 and the 99.9% reliability needed for infrastructure-grade deployment is not a small interpolation; it spans roughly two orders of magnitude in tail-risk performance and is not closing at the same rate as p50 horizons.

9. What would change this assessment

The article's confidence interval is asymmetric. The claim that capability and reliability are diverging is well-supported by current evidence. The claim that the divergence will close on a predictable timeline is not. Specifically, the assessment would update toward more optimistic deployment posture if:

METR-style evaluations begin showing 80%, 95%, and 99% time horizons scaling at rates comparable to the 50% horizon, across independently maintained task suites with messier task distributions and lower contamination risk. The current pattern is that 50% horizons scale fastest, and tail percentiles trail; a regime shift in tail scaling would be the strongest single signal.
Controlled factorial ablations — of the kind sketched in section 6 — show that modern harnesses eliminate most 10+ tool-call failures at high reliability across realistic, non-contaminated tasks, with stable cost and low recovery-failure rates under injected faults.
External audits of frontier agents on multi-day workflows with realistic permissions, adversarial state changes, fixed budgets, and trace-graded evaluation show low correlated failure rates and high tail-percentile reliability.

The assessment would update toward more pessimistic deployment posture if:

Ablation evidence shows that the architectural mitigations mostly shift errors around rather than reducing them, while increasing correlated failures, cost, and attack surface.
Production telemetry from scaled deployments shows that observed reliability trails benchmark reliability by a larger margin than current evidence suggests, particularly on irreversible-action workflows.
METR-style suites converge on a ceiling at the long end, with frontier models plateauing on multi-hour tasks even as compute and parameter counts scale.

What would not change the assessment, in either direction, is more headline numbers from contaminated leaderboards, more vendor demonstrations of impressive single-workflow runs, or more aggregate benchmark improvements without trace-level grading. The reliability question lives in the tail, in the recovery, in the irreversibility, and in the verification — not in the median or the mean.

10. Conclusion

Long-horizon agency is progressing. Capability is moving along the axes that benchmarks measure most cleanly: tool-call count, wall-clock duration on bounded tasks, multi-step coding under filtered grading regimes. Reliability is not moving at the same rate, and the gap between capability and reliability is the part of the picture that most determines what can actually be deployed.

The honest summary, given the evidence verified through May 6, 2026: METR's 50% time horizons demonstrate real and continuing capability scaling; the gap between p50 and p80 horizons demonstrates that this scaling has not yet crossed the deployment threshold for unsupervised production work; SWE-bench's progression demonstrates that scaffold and benchmark design substantially affect what we measure; and the convergent vendor guidance from Anthropic and OpenAI describes harnessed delegation, not autonomous agents, as the production-relevant pattern.

Long-horizon agents will become deployable workflow by workflow, not all at once. Reliability will arrive first where tasks are observable, reversible, cheaply checkable, and economically worth the supervision overhead. It will arrive later, or never, where tasks are open-ended, irreversible, hard to verify independently, and economically intolerant of the supervision cost. The question is not when this happens. It is which workflow you mean.

Companion entries

Core theory:

Capability vs Reliability in AI Systems
Reliability Arithmetic and Compounding Error
Time-Horizon Metrics for Agent Capability
The p50/p80 Gap in Agent Evaluation
Workflow vs Agent: The Anthropic Distinction

Practice:

Agent Harness Design Patterns
Externalized Memory and Progress Files
Plan Trees and Checkpoint Semantics
Replay Buffers and Trace-Level Evaluation
Supervisor Models and Verifier Independence
Sandboxing and Reversible Actions
Context Window Economics
Multi-Model Agent Orchestration Patterns
Tool Affordance Tests and Sandbox Manifests

Empirical methods:

Factorial Ablation Design for Agent Evaluation
Trace-Graded Failure Mode Labeling
METR Time Horizons: Methodology and Caveats
SWE-bench: From Original to Verified
Benchmark Contamination and Frontier Measurement

Counterarguments and limits:

The Workflow-First Critique of Agentic Systems
Deterministic State Machines as Reliability Substrate
Autonomy as Marketing vs Autonomy as Engineering
When Mitigations Become Risk Surfaces
Production Reliability vs Benchmark Reliability

Deployment and governance:

Per-Workflow Deployment Thresholds
Reversibility and Blast Radius in Agent Actions
Escalation Rules and Human-in-the-Loop Design
Cost Budgets and Marginal-Reliability Tradeoffs
Prompt Injection and Least-Privilege Tool Design