AlphaZero
AlphaZero showed that a single self-play reinforcement learning recipe could master three historically important perfect-information games—chess, shogi, and Go—without human game data. Its durable contribution was not merely that it beat Stockfish, Elmo, and AlphaGo-family baselines, but that it made Search and Learning look like one feedback system: a neural network guides tree search, tree search improves play, and improved play becomes the next training distribution.
Coverage note: verified through May 11, 2026.
Publication note
The canonical AlphaZero record has two closely related entries. The December 2017 preprint, “Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm,” introduced AlphaZero as a tabula-rasa algorithm that reached superhuman play in chess, shogi, and Go within 24 hours, using no domain knowledge beyond the game rules. The peer-reviewed chess/shogi/Go AlphaZero publication appeared in Science in December 2018, not Nature; the directly relevant Nature papers in the same lineage are AlphaGo, AlphaGo Zero, MuZero, and AlphaProof. Nature+3arXiv+3Google DeepMind+3
Why AlphaZero mattered
AlphaZero was a forcing move in AI research. Chess engines had long been archetypes of domain specialization: hand-engineered evaluation functions, alpha-beta search, opening books, endgame tablebases, and decades of expert tuning. AlphaZero replaced that stack with a general Self-Play Reinforcement Learning loop, a deep policy-value network, and Monte Carlo Tree Search; it learned from games it generated against itself rather than from human games. In DeepMind’s 2017 formulation, AlphaZero started from random parameters and learned solely from self-play, while its only externally supplied knowledge was the legal game dynamics needed to generate moves and score terminal positions. ar5iv
The sharper claim is methodological: AlphaZero made “general game-playing” empirically credible in domains where hand-designed engines had dominated. It was not a universal intelligence system; it operated in finite, deterministic, perfect-information games with exact rules and terminal rewards. But inside that class, the same learning/search machinery adapted across games with different board structures, action encodings, draw rules, and tactical cultures. DeepMind’s Science-era evaluation reported that the same single system taught itself chess, shogi, and Go, beating world-champion or prior self-taught systems in each case. Google DeepMind
System at a glance
| Layer | AlphaZero design | Why it mattered |
|---|---|---|
| Environment | Legal move generator and terminal scoring for chess, shogi, or Go | Rules are supplied, but human game data, openings, heuristics, and handcrafted evaluations are not. |
| Neural network | Deep residual network with policy and value outputs | The policy suggests promising actions; the value estimates expected outcome from a position. |
| Search | MCTS guided by network priors and values | Search improves local decisions beyond the raw network policy. |
| Training data | Self-play games generated by the latest agent | The agent creates its own curriculum as its strength rises. |
| Targets | MCTS visit distribution and final game outcome | Search becomes the policy teacher; game results become the value teacher. |
| Update loop | Continual reinforcement learning from self-play | The learned network amortizes and generalizes search improvements. |
In the 2017 paper’s formulation, the network takes a board position as input and returns two objects: a vector of move probabilities and a scalar expected outcome. MCTS uses those predictions to explore a tree of possible continuations; at the root, the visit counts define an improved move distribution. Self-play games are then stored as examples: board state, search-improved policy, and final outcome. The network is trained to predict the game outcome and to match the MCTS-improved move distribution, combining value loss, policy cross-entropy, and regularization. ar5iv
A compact way to describe the mechanism is Approximate Policy Iteration with a learned planner in the loop. The current network proposes a policy and value estimate. MCTS performs local policy improvement by looking ahead. The improved behavior generates stronger self-play games. Gradient descent distills those search results back into the network. The next search starts from a better prior.
The learning loop
AlphaZero’s core algorithm is simple enough to summarize without hiding the important feedback structure:
Initialize a neural network randomly.
For each self-play position, run MCTS guided by the current network.
Select moves according to the MCTS visit distribution.
At game end, assign the terminal result: win, draw, or loss.
Train the network to predict both the search policy and the game outcome.
Use the updated network to generate the next wave of self-play.
The decisive part is that the system does not merely imitate its current policy. It trains on search-improved play. MCTS is a temporary expert instantiated at every move; the neural network is the long-lived memory that compresses the expert’s output into a function approximator. This is the core of AlphaZero’s Search-Policy Distillation pattern.
The loop also avoids one major limitation of earlier supervised systems: the ceiling imposed by human data. AlphaGo’s original Nature paper used value networks and policy networks trained by a combination of supervised learning from human expert games and reinforcement learning from self-play. AlphaGo Zero removed human games for Go, showing that a system could become its own teacher in that domain; AlphaZero generalized the same idea beyond Go to chess and shogi. Nature+2Nature+2
Architecture and methodological details
AlphaZero inherited the AlphaGo Zero architecture family: a single deep neural network with policy and value heads, coupled to MCTS. But the generalization from Go to chess and shogi forced several important changes.
First, AlphaZero predicted expected outcome, not merely probability of winning. This mattered because chess and shogi can include draws, unlike the binary win/loss framing used in Go-centric AlphaGo Zero. Second, AlphaZero removed Go-specific board symmetries from the training and search pipeline; Go’s rotations and reflections are valid data augmentations, but chess and shogi are asymmetric because pieces have direction, promotion rules, castling, drops, and other position-dependent mechanics. Third, AlphaZero used a single continually updated network rather than AlphaGo Zero’s iteration scheme that selected a new “best player” after evaluation matches. Fourth, DeepMind reported reusing the same algorithm settings, network architecture, and hyperparameters across the three games, with separate instances trained per game. ar5iv
The “no domain knowledge” phrase needs precision. AlphaZero did not receive human game records, opening books, endgame tablebases, handcrafted material values, piece-square tables, joseki, shogi castles, or Go patterns. It did receive the rules required to generate legal moves and evaluate terminal positions; it also received board/action encodings based on the basic rules. That is not a trivial input, but it is qualitatively different from encoding centuries of human strategic knowledge into evaluation features. The paper explicitly contrasts AlphaZero with Stockfish-style engines that use handcrafted features, quiescence search, alpha-beta pruning, transposition tables, opening books, and endgame tablebases. ar5iv
Empirical results that defined the era
AlphaZero’s empirical story has two phases: the fast 2017 preprint results and the fuller 2018 Science evaluation. The 2017 preprint reported that AlphaZero surpassed Stockfish in chess after about 4 hours of training, surpassed Elmo in shogi after less than 2 hours, and surpassed AlphaGo Lee in Go after 8 hours, using Elo estimates from evaluation games. It also reported tournament matches at one minute per move in which AlphaZero lost zero games to Stockfish, lost eight to Elmo, and beat the previously published three-day AlphaGo Zero. ar5iv
The 2018 Science-era evaluation was more robust and is usually the cleaner result to cite. DeepMind reported that AlphaZero took approximately 9 hours of training for chess, 12 hours for shogi, and 13 days for Go; in chess, it first outperformed Stockfish after 4 hours, in shogi it first outperformed Elmo after 2 hours, and in Go it first outperformed the Lee Sedol version of AlphaGo after 30 hours. In the full evaluation, each program ran on hardware suited to its design, matches used three hours per game plus 15 seconds per move, and AlphaZero defeated Stockfish with 155 wins and 6 losses out of 1,000 games, defeated Elmo with a 91.2% win rate, and defeated AlphaGo Zero with a 61% win rate. Google DeepMind
| Domain | Baseline | Reported AlphaZero result | Key interpretive point |
|---|---|---|---|
| Chess | Stockfish, TCEC champion lineage | 155 wins, 6 losses, 839 draws in 1,000 games in the Science-era evaluation | Neural-guided MCTS competed with and beat a highly optimized alpha-beta engine under reported match conditions. |
| Shogi | Elmo, CSA world champion version | 91.2% win rate | The method handled drops and a larger, tactically explosive action space. |
| Go | AlphaGo Zero | 61% win rate | AlphaZero generalized back to Go while using the broader AlphaZero recipe. |
The search comparison was as culturally important as the match score. The 2017 paper reported that AlphaZero searched roughly 80,000 positions per second in chess and 40,000 in shogi, compared with roughly 70 million for Stockfish and 35 million for Elmo. AlphaZero therefore did not win by brute-force node count; it won by making each evaluated node more semantically informed through the network’s policy and value predictions. ar5iv
This is one reason AlphaZero felt different from prior chess milestones. Deep Blue’s victory over Garry Kasparov had already shown that machines could beat world champions, but its technical aesthetic was specialized search plus expert engineering. AlphaZero suggested that the same game strength might emerge from a more general learning loop, at least when the world is a board game with exact rules.
AlphaGo, AlphaGo Zero, AlphaZero, MuZero: the family lineage
AlphaZero is best understood as a member of a sequence rather than as an isolated system.
| System | Main publication | Human data? | Model of environment | Core contribution |
|---|---|---|---|---|
| AlphaGo | Nature 2016 | Yes, then self-play | Rules/simulator supplied | Combined policy networks, value networks, and MCTS to beat professional Go players. |
| AlphaGo Zero | Nature 2017 | No human game data | Rules/simulator supplied | Learned Go tabula rasa from self-play; single policy-value network became its own teacher. |
| AlphaZero | arXiv 2017; Science 2018 | No human game data | Rules/simulator supplied | Generalized the AlphaGo Zero recipe to chess, shogi, and Go. |
| MuZero | Nature 2020 | No external data for reported results | Learned model | Learned a planning model predicting reward, policy, and value, matching AlphaZero in board games without being given game dynamics and reaching state-of-the-art Atari results. |
| AlphaProof | Nature 2025/2026 | Uses pretraining/SFT plus RL | Formal Lean environment | Adapted AlphaZero-like RL and tree search to formal proof search, with Lean verification and test-time RL. |
The original AlphaGo used deep networks plus search, but it still learned initially from human expert games. Its Nature paper reported policy and value networks trained from human expert play and self-play, combined with Monte Carlo search; AlphaGo achieved a 99.8% win rate against other Go programs and defeated European champion Fan Hui 5–0. Nature
AlphaGo Zero made the decisive philosophical shift: no human games. Its Nature paper described an algorithm based solely on reinforcement learning, without human data, guidance, or domain knowledge beyond game rules; the network learned to predict its own MCTS move selections and game winners, improving the search and thereby improving future self-play. Starting tabula rasa, AlphaGo Zero beat the previously published champion-defeating AlphaGo 100–0. Nature
AlphaZero then abstracted the recipe away from Go. Chess and shogi were not naturally as convolution-friendly as Go: their rules are asymmetric, include long-range piece movement, and permit draws or drops. That made AlphaZero a stronger generality claim than AlphaGo Zero. It demonstrated that the self-play-plus-search loop was not just a Go exploit. ar5iv
MuZero removed an even deeper assumption: the rules. MuZero learned a model that predicts the quantities useful for planning—reward, policy, and value—rather than reconstructing the full environment. Nature’s MuZero paper states that it achieved superhuman performance across Go, chess, shogi, and Atari without knowledge of underlying dynamics, and matched AlphaZero’s superhuman performance in Go, chess, and shogi without being supplied the rules of the games. Nature
AlphaProof later brought the lineage into formal reasoning. The Nature article describes AlphaProof as an AlphaZero-inspired agent that learns to find formal proofs through RL by training on millions of auto-formalized problems in Lean; its main RL loop uses self-generated experience from proof and disproof attempts, guided by tree search and verified by the Lean environment. In the 2024 IMO evaluation, AlphaProof solved three of five non-geometry problems, and together with AlphaGeometry 2 reached silver-medal-level performance under multi-day computation. Nature+2Nature+2
The methodological contribution: a general self-play recipe
AlphaZero’s contribution can be phrased as a recipe:
Use search to improve action selection; use learning to amortize search; use self-play to generate an adaptive curriculum; use exact rules and outcomes to keep the learning signal grounded.
This recipe has four important properties.
First, the data distribution is endogenous. The agent trains on positions produced by its current and recent policies. As the policy improves, the curriculum shifts automatically toward stronger play. There is no static dataset of “expert” games whose weaknesses are inherited forever.
Second, the target policy is stronger than the behavior policy’s raw network output. MCTS transforms a noisy prior into a more accurate local policy by spending computation on a specific position. The neural network then learns to imitate that improved distribution. This is why AlphaZero is not simply reinforcement learning from terminal rewards; it is reinforcement learning with search-based policy improvement.
Third, value learning makes search selective. A traditional rollout-heavy MCTS estimates outcomes by simulating many continuations. AlphaZero’s network provides a learned value estimate at leaf nodes, enabling fewer but more meaningful expansions. The paper’s node-per-second comparison with Stockfish and Elmo dramatizes this point: AlphaZero searched orders of magnitude fewer positions, but each node carried richer learned evaluation. ar5iv
Fourth, the environment supplies a verifier. A game engine can say whether a move is legal, what state follows, and who won. This is often underemphasized. AlphaZero did not need human labels because the environment itself produced reliable labels. In chess, shogi, and Go, correctness is cheap compared with open-ended domains.
AlphaZero and the bitter lesson
AlphaZero is one of the cleanest case studies for The Bitter Lesson. Rich Sutton’s essay argues that the major long-run wins in AI come from general methods that exploit computation—especially search and learning—rather than from building human domain knowledge into agents. incompleteideas.net
AlphaZero vindicated that thesis, but with a nuance: it did not show that learning alone beats search. It showed that learning and search can be mutually amplifying. The neural network made search more intelligent; search made the neural network’s targets stronger. AlphaZero was therefore a bitter lesson for handcrafted evaluation, but not for planning. It replaced human-coded chess knowledge with learned evaluation, yet it preserved the idea that intelligence often requires looking ahead.
This matters for interpreting the broader AI trajectory. A superficial lesson would be “deep learning beat search.” The better lesson is “learned representations made search scalable in domains where raw search was too broad, and search made learning more data-efficient and more strategically grounded.” In AlphaZero, neither engine is optional.
The result also complicates the “human-like” framing. DeepMind’s blog noted that AlphaZero searched only tens of thousands of positions per second in chess compared with tens of millions for Stockfish, and commentators often described its play as intuitive or creative. But the system’s apparent intuition was not mystical. It was the behavioral residue of enormous self-play, function approximation, and repeated local planning. Google DeepMind
Contemporary echoes in LLM reasoning
The AlphaZero pattern has reappeared, in altered form, in LLM Reasoning research. The analogy is not exact, but it is strong enough to be useful.
OpenAI’s o1 writeup describes a large-scale reinforcement learning algorithm that teaches a language model to reason using chain-of-thought, with performance improving both with more train-time reinforcement learning and with more test-time thinking. The same page says o1 learns to refine strategies, correct mistakes, break tricky steps into simpler ones, and try different approaches when the current one fails. OpenAI
DeepSeek-R1’s paper makes the connection more explicit from the open literature side: it reports that reasoning abilities can be incentivized through pure RL without human-labeled reasoning trajectories, with emergent behaviors such as self-reflection, verification, and dynamic strategy adaptation. The paper’s abstract frames this as reducing dependence on human-annotated demonstrations for reasoning. arXiv
Earlier, STaR—Self-Taught Reasoner—showed a simpler bootstrapping loop: generate rationales, keep those that lead to correct answers or regenerate with the correct answer, fine-tune on successful rationales, and repeat. That is not AlphaZero-style MCTS, but it shares the core idea that a model can improve by training on traces produced by its own attempts, filtered by correctness. arXiv
A still closer analogy appears in reinforced self-play reasoning systems such as Absolute Zero Reasoner, which proposes tasks, solves them, and uses a code executor to validate tasks and verify answers, aiming for reasoning improvement without external data. The OpenReview version describes it as self-play reasoning RL with no data, where a single model self-evolves its curriculum and reasoning ability through verifiable feedback. arXiv
| AlphaZero game setting | LLM or formal-reasoning echo | Similarity | Critical difference |
|---|---|---|---|
| Board state | Prompt, proof state, code state, problem state | The agent acts from a structured context. | Natural language states are often ambiguous and underspecified. |
| Legal move | Token, tactic, tool call, code edit | The agent chooses an action from a large action space. | Token spaces are vastly larger and less semantically constrained than game moves. |
| MCTS line of play | Reasoning trace, proof search, tool-augmented search | Test-time computation improves answers. | Search transitions may be unreliable unless grounded in tools or formal systems. |
| Terminal win/loss/draw | Verifier result, unit test, proof checker, math answer | Outcome reward can train behavior. | Many real tasks lack cheap, exact reward. |
| Self-play curriculum | Synthetic tasks, generated proofs, generated code problems | The system can generate its own training distribution. | Self-generated tasks can collapse, become biased, or optimize the wrong proxy. |
| Network learns from search | Model learns from successful traces | Computation is distilled into parameters. | Reasoning traces may be unfaithful, hidden, or optimized for reward rather than truth. |
AlphaProof is the strongest current bridge between AlphaZero and reasoning because it restores the missing ingredient: a formal environment. Lean provides machine-checkable proof states and correctness verification. AlphaProof’s training loop uses tree search over tactics, Lean-verified outcomes, and self-generated proof/disproof attempts to improve a proof network. This is much closer to AlphaZero than ordinary chatbot RL because the verifier is exact and the state transitions are grounded. Nature
Why transfer beyond game-tree search remains open
The central open question is whether the AlphaZero recipe transfers to open-ended reasoning domains that do not have compact game trees, exact simulators, and cheap terminal rewards.
AlphaZero worked because several conditions aligned:
| Requirement | Present in chess/shogi/Go? | Why it matters |
|---|---|---|
| Exact legal transition function | Yes | Search can expand valid futures. |
| Cheap terminal reward | Yes | Self-play labels are reliable. |
| Closed world | Yes | The game contains all relevant state. |
| Competitive curriculum | Yes | Self-play naturally generates harder opponents. |
| Manageable action semantics | Mostly | Moves are discrete and rule-bounded. |
| Verifiable improvement | Yes | Elo and match play expose progress. |
Open-ended reasoning breaks these assumptions. There may be no exact simulator for “write a good research plan,” “debug this distributed system,” or “derive a useful philosophical distinction.” The reward may be delayed, subjective, or socially mediated. The state may be partially observable. The action space may include arbitrary text, tool use, experiments, and interactions with humans. A self-generated curriculum may reward fluency, puzzle-like artifacts, or exploitative shortcuts rather than genuine problem solving.
Formal math, code, and constrained tool environments are the most plausible transfer zones because they partially recreate AlphaZero’s verification structure. A proof assistant can reject invalid proof steps. A code executor can run tests. A theorem prover can search over tactics. A simulator can evaluate robot policies. But these are still bounded environments, not open-ended general reasoning. AlphaProof’s own evaluation highlights the remaining friction: IMO 2024 problems were manually formalized into Lean for the main competition protocol, geometry required AlphaGeometry 2 because of limitations in Lean formalization coverage, and successful solutions could require multi-day computation. Nature
This suggests a sober conclusion: AlphaZero transfers best where reasoning can be made game-like without destroying the task. The recipe is powerful when a domain can supply exact state transitions, legal actions, and reliable reward. It becomes speculative when those anchors disappear.
What AlphaZero did not prove
AlphaZero did not prove that human knowledge is useless. It proved that, in certain closed domains, human strategic data is not necessary if the system has enough compute, an exact environment, and a scalable search-learning loop. Human knowledge still appears in the choice of problem, architecture, board encoding, rules interface, optimization method, and evaluation protocol.
AlphaZero also did not prove that MCTS is always superior to alpha-beta search in chess-like games. It showed that neural-guided MCTS plus self-play learning could beat highly optimized engines under reported conditions. Since then, modern chess engines have incorporated neural evaluation in different ways, including NNUE-style evaluation inside alpha-beta search; the broader lesson is not “MCTS wins,” but “learned evaluation plus scalable search wins.”
Finally, AlphaZero did not solve general game playing in the classical sense of receiving an arbitrary formal game description and immediately mastering it. DeepMind trained separate instances for each game. The generality was algorithmic, not a single universal trained policy that instantly transferred across games. ar5iv
Primary reference spine
| Descriptive reference label | Role in the lineage |
|---|---|
| Silver et al., “Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm” | 2017 AlphaZero preprint introducing the general self-play algorithm for chess, shogi, and Go. arXiv |
| Silver et al., “A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play” | 2018 peer-reviewed Science AlphaZero publication and full evaluation. Google DeepMind |
| Silver et al., “Mastering the game of Go with deep neural networks and tree search” | 2016 Nature AlphaGo paper combining human-data training, self-play, deep networks, and MCTS. Nature |
| Silver et al., “Mastering the game of Go without human knowledge” | 2017 Nature AlphaGo Zero paper establishing tabula-rasa self-play for Go. Nature |
| Schrittwieser et al., “Mastering Atari, Go, chess and shogi by planning with a learned model” | 2020 Nature MuZero paper replacing known rules with a learned planning model. Nature |
| Hubert et al., “Olympiad-level formal mathematical reasoning with reinforcement learning” | 2025/2026 Nature AlphaProof paper adapting AlphaZero-like RL to formal proof search. Nature |
| Sutton, “The Bitter Lesson” | Conceptual frame for interpreting AlphaZero as search and learning displacing handcrafted domain knowledge. incompleteideas.net |
Core takeaways
AlphaZero’s main claim was not “AI can play chess.” That had been true since Deep Blue and became increasingly true through decades of engine progress. AlphaZero’s claim was that a comparatively general self-play system could rediscover and exceed domain-specific expertise across multiple hard games.
The architecture is best read as a closed learning loop: MCTS improves the policy locally; the neural network learns the improved policy globally; self-play supplies the next training distribution. Search and learning are not rivals in this design. They are reciprocal mechanisms.
The result is one of the clearest empirical supports for the Bitter Lesson, but the lesson is narrower than slogans imply. AlphaZero succeeded in environments where legal action, transition, and reward were exact. The open research frontier is whether similarly grounded self-improvement loops can be built for domains where “winning” is not already formalized.
Companion entries
Core theory: Self-Play Reinforcement Learning, Monte Carlo Tree Search, Approximate Policy Iteration, Policy-Value Networks, Search and Learning, The Bitter Lesson
AlphaZero lineage: AlphaGo, AlphaGo Zero, AlphaZero, MuZero, AlphaProof, AlphaGeometry, AlphaFold and Scientific Discovery
Game AI and planning: Deep Blue, Stockfish, Elmo Shogi Engine, Computer Go, Alpha-Beta Search, Neural-Guided Search
Reasoning systems: LLM Reasoning, Chain-of-Thought Reinforcement Learning, Reinforcement Learning with Verifiable Rewards, Self-Taught Reasoner, Formal Verification, Lean Theorem Proving
Open questions: Open-Ended Self-Improvement, Verifier-Based AI Training, Synthetic Data Curricula, Test-Time Compute, Search in Language Models, Limits of Self-Play