AlphaGo Zero

AlphaGo Zero showed that, for a clean adversarial domain with exact rules and an automatic outcome signal, a system could start from random play, learn only through Self-Play Reinforcement Learning, and surpass versions of AlphaGo that had been bootstrapped from human games. Its lasting significance is not merely that it became a stronger Go engine, but that it crystallized a scalable recipe: combine Monte Carlo Tree Search with a neural policy/value learner, use search to improve play, and train the network to absorb the improved search policy.

Coverage note: verified through May 18, 2026.

AlphaGo Zero: Mastery From Self-Play Alone

AlphaGo Zero is the system introduced by Silver et al. in the 2017 Nature paper “Mastering the game of Go without human knowledge.” The official Nature abstract frames the result directly: AlphaGo Zero used reinforcement learning without human data, guidance, or domain knowledge beyond game rules, and achieved superhuman performance while winning 100–0 against the previously published champion-defeating AlphaGo. Nature

At a high level, AlphaGo Zero replaced the older AlphaGo pipeline with a tighter loop:

1. Start with a randomly initialized neural network.

2. Use that network to guide MCTS during self-play.

3. Treat the MCTS move distribution as an improved policy target.

4. Treat the final game result as the value target.

5. Train the same network to predict both.

6. Repeat.

This made AlphaGo Zero an early canonical example of Search-Guided Learning: search improves the policy locally, learning distills search into the network globally, and the better network then makes future search stronger.

Historical setting: from AlphaGo to AlphaGo Zero

The original 2016 AlphaGo system was already a breakthrough. It used separate policy and value networks, trained the policy initially by supervised learning from human expert games, refined it with reinforcement learning from self-play, and combined these networks with MCTS and rollouts. In that form, AlphaGo defeated the European Go champion Fan Hui 5–0 and was presented as the first computer program to defeat a human professional player in full-board Go. Google Research

AlphaGo Zero changed the emphasis from “learn from humans, then improve” to “learn from the game itself.” The 2017 paper explicitly lists four central differences from AlphaGo Fan and AlphaGo Lee: AlphaGo Zero was trained solely by self-play from random play, used only black and white stones as input features, used one network rather than separate policy and value networks, and used a simpler tree search without Monte Carlo rollouts. UCL Discovery

Dimension	Earlier AlphaGo versions	AlphaGo Zero
Initial training data	Human amateur/professional games plus self-play	No human game data; starts from random play
Network structure	Separate policy and value networks	One network with policy and value outputs
Search	MCTS plus neural networks plus rollouts	MCTS guided by one neural network; no rollouts
Input representation	Board plus hand-engineered features in earlier versions	Raw board-history feature planes
Methodological claim	Superhuman Go via hybrid human-data/self-play pipeline	Superhuman Go from self-play alone, given rules

The important caveat is that “without human knowledge” did not mean “without all priors.” The system still used Go rules, legal move generation, board symmetries in the Go-specific setup, a convolutional residual architecture suited to grid-like boards, MCTS, and a carefully engineered distributed training system. The claim was narrower and stronger: AlphaGo Zero did not learn from human move data or expert demonstrations, and it did not rely on human-designed Go features in the way earlier versions did. UCL Discovery

The algorithmic core

AlphaGo Zero’s network is written as fθ(s)=(p,v)f_\theta(s) = (p, v)fθ​(s)=(p,v). Given a board state sss, it outputs a policy vector ppp, assigning probabilities to legal moves, and a scalar value vvv, estimating the current player’s chance of winning. This single network combines the roles that earlier AlphaGo versions split between a policy network and a value network. UCL Discovery

The training loop is an approximate Policy Iteration procedure. At each position during self-play, the current network guides MCTS. The resulting search probabilities π\piπ are usually stronger than the raw network policy ppp, so the paper describes MCTS as a policy-improvement operator. The completed game supplies an outcome zzz, which becomes a value target. The network is then trained to make its policy match π\piπ and its value match zzz. UCL Discovery

The loss function combines value error, policy cross-entropy, and L2 regularization:

l=(z−v)2−π⊤log⁡p+c∥θ∥2l = (z - v)^2 - \pi^\top \log p + c|\theta|^2l=(z−v)2−π⊤logp+c∥θ∥2 This objective is deceptively compact. It says: imitate the stronger policy found by search, predict the final winner, and regularize the parameters. In effect, each game creates supervised targets, but the labels are generated by the agent’s own improved search and by the game’s terminal result, not by human examples. UCL Discovery

The MCTS procedure stores, for each edge (s,a)(s,a)(s,a), visit counts, priors, and value estimates. Search selects moves using a combination of action value Q(s,a)Q(s,a)Q(s,a) and an exploration bonus derived from the prior P(s,a)P(s,a)P(s,a) and visit count N(s,a)N(s,a)N(s,a). At a newly reached leaf, the neural network evaluates the position once, producing both priors and value; those values are backed up through the tree. UCL Discovery

This is the conceptual center of AlphaGo Zero: the network does not replace search, and search does not replace learning. Search produces better local decisions than the network alone; learning compresses those search-improved decisions into a reusable global function approximator.

Architectural simplifications

AlphaGo Zero simplified AlphaGo in ways that made the system stronger, not merely cleaner.

The network input is a 19×19×1719 \times 19 \times 1719×19×17 stack of binary feature planes: eight history planes for the current player’s stones, eight for the opponent’s stones, and one plane indicating the color to play. The history planes are needed because Go’s repetition rules mean the current board alone is not fully observable, and the color plane is needed because komi is not observable from the stones alone. UCL Discovery

The network uses a residual tower. The 3-day run used 20 residual blocks; the larger 40-day run used 40 residual blocks. Each residual block applies convolution, batch normalization, rectification, another convolution, batch normalization, a skip connection, and another rectifier. The tower then branches into a policy head and a value head. UCL Discovery

Simplification	Technical change	Why it mattered
One network	Policy and value share a residual representation	Reduces duplicated computation and lets both objectives regularize the same representation
No rollouts	Leaf evaluation comes from the neural value output	Removes fast handcrafted rollout policy and makes search rely on learned evaluation
Rawer input	Board-history planes instead of human Go features	Weakens the role of human feature engineering
Residual blocks	Deep residual convolutional tower	Improves accuracy, value prediction, and playing strength
Search inside training loop	MCTS supplies policy targets for learning	Turns search into an iterative policy-improvement mechanism

The paper’s architecture comparison found that residual networks improved performance by over 600 Elo, and combining policy and value into one network added roughly another 600 Elo despite slightly reducing move-prediction accuracy. The authors attributed part of this to efficiency and part to the dual objective regularizing a common representation. UCL Discovery

The deeper lesson is that simplification was not a synonym for naivety. AlphaGo Zero removed human game data and handcrafted Go features, but kept strong structural priors: convolution over a spatial board, residual learning, MCTS, exact legal moves, exact scoring, and massive self-generated experience.

Empirical results

The headline 3-day result is often summarized as “superhuman Go from scratch.” In the first main experiment, AlphaGo Zero trained for roughly three days from random behavior, generated 4.9 million self-play games, used 1,600 MCTS simulations per move, updated parameters from 700,000 mini-batches of 2,048 positions, and used a 20-block residual network. UCL Discovery

After 36 hours, the system outperformed AlphaGo Lee, the version that defeated Lee Sedol. After 72 hours, the authors evaluated AlphaGo Zero against the exact AlphaGo Lee version under the same two-hour match conditions used in Seoul; AlphaGo Zero used one machine with 4 TPUs, while AlphaGo Lee used a distributed setup with 48 TPUs. AlphaGo Zero won 100–0. UCL Discovery

Result	Details	Interpretation
Superhuman Go in ~3 days	Training began from random behavior and used only self-play	Human game data was not necessary for superhuman play in this setting
100–0 over AlphaGo Lee	72-hour AlphaGo Zero defeated the Lee Sedol-match version	The self-play-only system surpassed the famous hybrid system
Better than supervised human-data baseline	The self-learned player beat the human-trained player within the first 24 hours	Predicting human moves was not the same as optimal play
Stronger 40-day system	Larger 40-block network trained for ~40 days	Self-play scaling continued to improve strength
89–11 over AlphaGo Master	40-day AlphaGo Zero beat AlphaGo Master in a 100-game match	The self-play-only version exceeded a human-data-initialized stronger predecessor

The comparison to supervised learning is especially important. A supervised network trained on KGS human games achieved higher move-prediction accuracy, but the self-play-trained player was stronger and defeated the human-trained player within the first 24 hours of training. The paper explicitly suggests that AlphaGo Zero may have been learning strategies qualitatively different from human play. UCL Discovery

The larger 40-day run trained a 40-block residual network from random behavior, generated 29 million self-play games, and reached an Elo rating of 5,185 in the paper’s internal tournament, compared with 4,858 for AlphaGo Master, 3,739 for AlphaGo Lee, and 3,144 for AlphaGo Fan. In a 100-game match against AlphaGo Master with two-hour time controls, the 40-day AlphaGo Zero won 89–11. UCL Discovery

The system also rediscovered known Go concepts and developed nonstandard strategies. The paper reports that AlphaGo Zero progressed from random and capture-focused play toward sophisticated concepts such as fuseki, tesuji, life-and-death, ko, yose, sente, shape, influence, and territory, while also preferring some new joseki variants that were not part of standard human tradition. UCL Discovery

The methodological contribution

AlphaGo Zero’s methodological contribution was not “reinforcement learning works,” which was already known, nor “self-play works,” which had precedents such as TD-Gammon. The contribution was the scale and cleanliness of the demonstration: a single system, starting from random play, using self-play and search, exceeded expert-human-data pipelines in a domain long treated as a grand challenge for AI. Nature

The core methodological shift was from imitation-first to improvement-first. Earlier AlphaGo used human games to initialize a policy, then improved through self-play. AlphaGo Zero used MCTS-guided self-play to generate its own curriculum and its own targets from the start. Human games were no longer the anchor; the game rules and outcome function were.

This matters because supervised learning from expert behavior has a built-in ceiling when the goal is to exceed the experts. It can be useful for initialization, but it also biases the model toward the distribution of human moves. AlphaGo Zero showed that, in at least one clean domain, the better teacher was not the human corpus but the iterated combination of search, self-play, and exact scoring.

A precise reading is needed. AlphaGo Zero did not prove that human data is always harmful, or that imitation learning is obsolete. It proved that, when a domain has a perfect simulator, exact rules, an unambiguous outcome, and enough compute, self-play can generate training data whose quality rises beyond the human distribution.

Relationship to AlphaZero

AlphaZero generalized the AlphaGo Zero recipe beyond Go to chess and shogi. DeepMind described AlphaZero as a single system that taught itself from scratch to master chess, shogi, and Go, beating a world-champion program in each case. Google DeepMind

The AlphaZero paper retained the broad AlphaGo Zero loop—neural policy/value network, self-play, and MCTS—but adapted it for games with draws and for rule sets without Go’s exact symmetries. AlphaZero optimized expected outcome rather than only win probability, did not assume rotation/reflection symmetry, and maintained a continually updated single network instead of the previous best-player replacement scheme used in AlphaGo Zero. Google Cloud Storage

AlphaZero also served as evidence that the approach was not merely a Go hack. The paper reports that AlphaZero first outperformed Stockfish after 4 hours of chess training, Elmo after 2 hours of shogi training, and AlphaGo Lee after 30 hours of Go training. In final evaluations, it defeated Stockfish with 155 wins and 6 losses out of 1,000 games, defeated Elmo with a 91.2% overall win rate in shogi, and defeated the previous AlphaGo Zero in Go by winning 61% of games. Google Cloud Storage

This generalization strengthens the Bitter Lesson reading: the same general pattern—search plus learning plus self-play—scaled across games that had historically depended on specialized engines and human-crafted evaluation machinery. But AlphaZero was still within a favorable family of domains: board games with exact rules, cheap simulation, clear legal actions, and objective scoring.

Relationship to broader self-play RL

AlphaGo Zero belongs to a long lineage of self-play systems. The 2017 paper itself points to earlier self-play reinforcement learning in Go, chess, checkers, backgammon, Othello, Scrabble, and poker, while noting that many prior systems still relied on handcrafted features, supervised initialization, pre-existing programs, or other domain-specific structure. UCL Discovery

TD-Gammon is a key predecessor. It trained a neural evaluation function for backgammon by playing against itself and learning from outcomes, and it became a classic example of reinforcement learning producing strong play through self-generated experience. dl.acm.org

After AlphaGo Zero and AlphaZero, self-play scaled into richer multi-agent environments. OpenAI Five used large-scale self-play reinforcement learning in Dota 2, a domain with long horizons, imperfect information, and a complex continuous state-action space, and defeated the world champion team OG. arXiv DeepMind’s AlphaStar used multi-agent reinforcement learning, supervised learning, imitation learning, and league-based self-play to reach Grandmaster level in StarCraft II across all three races. Nature

Those later systems also show the limits of the clean “self-play alone” story. AlphaStar did not simply start from random play in a perfect-information board game; it used human data and a league of agents to handle strategy diversity, partial observability, and nontransitive exploitability. OpenAI Five demonstrated the power of self-play at scale, but in a constrained version of Dota 2 and with enormous engineering around distributed training.

MuZero pushed in a different direction. AlphaZero assumes known game rules and uses those rules for search. MuZero combined tree search with a learned model and matched AlphaZero-like performance in Go, chess, and shogi while also achieving state-of-the-art results on Atari, where dynamics are more complex and not supplied as rules. Nature But MuZero still depends on rewards and interaction data; it weakens the need for known dynamics, not the need for an objective signal.

The bitter-lesson reading

Richard Sutton’s “bitter lesson” argues that, over the long run, general methods that leverage computation—especially search and learning—tend to beat methods that rely on encoding human domain knowledge. Sutton explicitly uses computer Go as an example: early efforts tried to exploit human understanding to avoid search, but much greater success came when search and learning were applied effectively at scale. UT Austin Computer Science

AlphaGo Zero is one of the cleanest case studies for that thesis. It removed human game records, handcrafted Go features, rollout policies based on human-like heuristics, and separate hand-staged training pipelines. What remained was a scalable loop: generate experience, search harder, train the model, repeat.

The “bitter” part is that the system did not need to respect human Go pedagogy. It did not learn joseki because a teacher explained joseki. It learned sequences because they improved expected game outcome under self-play. It rediscovered some human concepts and abandoned or revised others. That is exactly the uncomfortable lesson: human understanding may be useful locally, but it can become a constraint when the system can search and learn at sufficient scale.

Still, AlphaGo Zero should not be read as “no priors are best.” It used powerful priors, just not human Go move data. The right bitter-lesson interpretation is narrower: prefer priors that scale with computation and discovery. MCTS, residual networks, exact simulators, legal-action masks, and outcome-based training targets are not empty assumptions; they are meta-methods that let computation search through strategies humans did not enumerate.

Why self-play worked so well in Go

Go is unusually favorable for self-play reinforcement learning.

First, it is a two-player zero-sum game with perfect information. The AlphaGo Zero paper states that its approach is most directly applicable to zero-sum games of perfect information. UCL Discovery

Second, the rules provide an exact simulator. A self-play game can be generated cheaply and scored automatically. There is no need for a human to label whether a move was good; the terminal outcome gives a hard training signal.

Third, the opponent improves automatically. In self-play, the agent’s current weaknesses generate the curriculum. When the agent becomes stronger, it faces a stronger opponent: itself. This can create an implicit Automated Curriculum without a handcrafted lesson plan.

Fourth, the reward is sparse but clean. Winning or losing at the end of a game is delayed, but it is not ambiguous. The value head learns to predict that outcome, while MCTS helps assign practical decision pressure to intermediate moves.

Fifth, search and learning have complementary failure modes. The neural network can generalize across board positions but may make local mistakes. MCTS can correct some local mistakes by lookahead, but it needs a good prior and value estimate to search efficiently. AlphaGo Zero’s loop turns each into training fuel for the other.

The open question: domains without clean reward signals

The largest open question is whether the self-play-only approach generalizes to domains without clean reward signals, exact simulators, and objective win/loss criteria. The answer is not “yes” or “no”; it is conditional.

In Go, the reward is not a proxy. The game outcome is the thing being optimized. In many real-world domains, the reward is a proxy for what humans actually care about. That changes the problem. Reward misspecification, reward hacking, side effects, scalable supervision, and distribution shift are not incidental engineering inconveniences; they are central obstacles when the target objective is hard to formalize. Amodei et al.’s “Concrete Problems in AI Safety” categorizes several accident risks by whether they arise from the wrong objective, an objective that is expensive to evaluate, or undesirable behavior during learning. arXiv

Self-play also needs an opponent or environment that reveals mistakes. In Go, every bad move eventually meets a better counterstrategy. In software engineering, scientific discovery, management, ethics, or philosophy, the “opponent” is not always formalizable, and the score may not be known for months or years. A self-improving system can only bootstrap as far as its evaluator is both cheap and aligned with the true task.

This is why AlphaGo Zero is more directly relevant to Simulator-Centric AI than to arbitrary autonomous learning. If a domain can be turned into a game-like environment with faithful dynamics and truthful scoring—protein folding benchmarks, theorem proving with proof checkers, compiler optimization with executable tests, chip design with simulators—then AlphaGo Zero’s pattern becomes highly suggestive. If the domain lacks a reliable evaluator, self-play can amplify the wrong signal just as efficiently as the right one.

MuZero partially relaxes the need for known dynamics by learning a model useful for planning, but it still learns predictions of reward, value, and policy. That is progress on the “unknown rules” problem, not a complete solution to the “unknown objective” problem. Nature

Engineering lessons for self-improving systems

AlphaGo Zero offers several design lessons for Self-Improving Systems.

The first is that the evaluator is central. AlphaGo Zero’s evaluator was the Go rules and terminal score. The entire bootstrapping process depends on that evaluator being cheap, unambiguous, and resistant to gaming outside the intended game dynamics.

The second is that search can serve as a temporary expert. MCTS is too expensive to run exhaustively over all possible positions, but it can produce better local decisions than the network alone. Training then distills those local improvements into the network. This pattern generalizes beyond Go: use expensive inference to create higher-quality targets, then train a cheaper model to approximate them.

The third is that self-play can create a moving frontier. A static dataset fixes the learner to yesterday’s distribution. Self-play continually shifts the data distribution toward positions the current agent finds difficult or strategically relevant.

The fourth is that “from scratch” is mostly about data, not about system design. AlphaGo Zero began from random network weights, but the training loop, architecture, optimizer, simulator, search algorithm, and objective were all human-designed. For engineering practice, this is a feature, not a flaw: the system’s success came from putting human effort into scalable machinery rather than into enumerating domain tactics.

The fifth is that interpretability lags capability. AlphaGo Zero learned concepts Go players could name, but it also learned preferences that were alien or surprising. In systems trained by search and self-play, human-understandable abstractions may appear as after-the-fact interpretations rather than as the substrate of learning.

Common misreadings

Misreading 1: AlphaGo Zero proves that human data is useless.

No. It proves that in Go, with exact rules and enough compute, human data was not necessary for superhuman performance and may have imposed a ceiling relative to self-play. In domains where exploration is dangerous, simulation is poor, or rewards are unavailable, demonstrations may still be essential.

Misreading 2: AlphaGo Zero was model-free.

No. It did not learn a model of Go dynamics, but it used the known rules of Go inside MCTS. It was not model-free in the practical planning sense; search depended on a simulator. AlphaZero similarly used game rules, while MuZero later addressed the separate problem of planning with a learned model. Nature

Misreading 3: AlphaGo Zero removed search.

No. Search was more central than ever. The neural network guided MCTS, and MCTS produced the improved policy targets used for training. The system is best understood as search and learning coupled in a feedback loop.

Misreading 4: AlphaGo Zero was cheap.

No. The 3-day system used millions of games and specialized hardware, and the larger 40-day system generated 29 million self-play games. Its importance lies in the scalability and autonomy of the data-generation loop, not in low resource requirements. UCL Discovery

Misreading 5: AlphaGo Zero settles general intelligence.

No. It is a landmark result in a highly structured domain. Its broader importance comes from the pattern it exhibits—self-play, search, learned value functions, scalable data generation—not from directly solving open-ended reasoning, robotics, or social objectives.

Primary source map

Source	Role in this article
Silver et al., “Mastering the game of Go without human knowledge,” Nature 2017	Primary source for AlphaGo Zero’s algorithm, architecture, and empirical claims Nature
Silver et al., “Mastering the game of Go with deep neural networks and tree search,” Nature 2016	Baseline for earlier AlphaGo’s supervised-learning plus reinforcement-learning pipeline Google Research
Silver et al., “A general reinforcement learning algorithm that masters chess, shogi and Go through self-play,” Science / AlphaZero paper	Primary source for AlphaZero’s generalization beyond Go Google Cloud Storage
Sutton, “The Bitter Lesson”	Interpretive frame for search, learning, computation, and human-prior skepticism UT Austin Computer Science
Schrittwieser et al., “Mastering Atari, Go, chess and shogi by planning with a learned model,” Nature 2020	Follow-on source for MuZero and the move from known rules to learned dynamics Nature
AlphaStar and OpenAI Five papers/blogs	Evidence for later self-play systems in richer multi-agent environments Nature

Companion entries

Core theory: Self-Play Reinforcement Learning, Monte Carlo Tree Search, Approximate Policy Iteration, Policy Improvement, Value Functions, Policy Distillation, Exploration-Exploitation Tradeoff

Systems lineage: AlphaGo, AlphaGo Lee, AlphaGo Master, AlphaZero, MuZero, TD-Gammon, OpenAI Five, AlphaStar

Architecture and training: Residual Networks, Policy-Value Networks, Search-Guided Learning, Neural MCTS, Automated Curriculum, Simulator-Centric AI, Experience Generation

Bitter-lesson interpretation: The Bitter Lesson, Human Priors in AI, Tabula Rasa Learning, Scaling Laws and Search, Human Data as a Ceiling, Compute-Leveraging Methods

Open problems: Reward Specification, Reward Hacking, Sparse Rewards, Scalable Oversight, Self-Play Beyond Games, Evaluator Bottleneck, Objective Misspecification