Multi-Head Latent Attention
Abstract. Multi-Head Latent Attention (MLA) is DeepSeek’s attention mechanism for reducing inference-time KV-cache memory by storing a compressed latent representation of keys and values rather than full per-head K/V tensors. Its central claim is stronger than ordinary cache optimization: DeepSeek reports that MLA preserves, and in its V2 ablations often improves, model quality while cutting cache size dramatically relative to standard Multi-Head Attention and while avoiding the quality regressions DeepSeek observed for Multi-Query Attention and Grouped-Query Attention. Architecturally, MLA is important because it turns long-context inference from a brute-force memory-bandwidth problem into a model-design problem, but its adoption outside DeepSeek remains unsettled.
Coverage note: verified through May 18, 2026.
Core idea
In autoregressive decoder inference, each generated token attends to all previous tokens. To avoid recomputing previous hidden states, serving systems cache each layer’s previous keys and values: the KV Cache. For standard Multi-Head Attention (MHA), that cache scales roughly as:
O(L⋅T⋅2nhdh)O(L \cdot T \cdot 2 n_h d_h)O(L⋅T⋅2nhdh) where LLL is number of layers, TTT is sequence length, nhn_hnh is attention-head count, and dhd_hdh is per-head dimension. This is linear in context length but large in practice, especially when the model is deployed with large batches, long contexts, and high concurrency. Shazeer’s Multi-Query Attention paper framed incremental Transformer decoding as being bottlenecked by repeatedly loading the large K/V tensors, not just by arithmetic; DeepSeek-V2 makes the same practical point, saying MHA’s heavy cache limits maximum batch size and sequence length. ([Fast Transformer Decoding / MQA]) arXiv ([DeepSeek-V2 Technical Report]) ar5iv
MLA attacks that bottleneck by changing what gets cached. Instead of caching all full-rank per-head keys and values, each token’s hidden state is projected into a lower-dimensional latent ctKVc_t^{KV}ctKV, from which per-head key/value components can be reconstructed or algebraically absorbed into adjacent projections. Because Rotary Position Embeddings (RoPE) are position-sensitive and interfere with that absorb trick, DeepSeek separates positional key/query dimensions from the compressed no-position latent part. ([DeepSeek-V3 Technical Report]) arXiv
The result is not merely “fewer KV heads,” as in Grouped-Query Attention. MLA keeps a richer head-specific attention structure while storing a much smaller latent state plus a small positional key. That makes it part of the broader KV-Cache Compression family, but it is more invasive than MQA/GQA because it changes the learned attention parameterization itself.
Background: MHA, MQA, and GQA
The original Transformer uses scaled dot-product attention and multi-head attention: multiple query/key/value projections run in parallel, their outputs are concatenated, and a final output projection mixes them. This design gives each head independent K/V projections, but at inference time the cache must preserve those per-head keys and values for every previous token. ([Attention Is All You Need]) arXiv
Multi-Query Attention (MQA) compresses the cache by sharing one key head and one value head across all query heads. It greatly reduces K/V memory bandwidth in incremental decoding, but it removes much of the per-head K/V diversity. Shazeer reported much faster decoding with only minor quality degradation in the original MQA setting, but DeepSeek’s V2 experiments found that, on their 7B dense baselines, MQA lagged MHA on harder benchmarks. ([Fast Transformer Decoding / MQA]) arXiv ([DeepSeek-V2 Ablations]) ar5iv
Grouped-Query Attention (GQA) sits between MHA and MQA. Query heads are divided into groups; each group shares a key/value head. Ainslie et al. introduced GQA as a generalization of MQA and showed that uptraining MHA checkpoints into GQA could recover quality close to MHA while approaching MQA-like speed. ([GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints]) arXiv
MLA differs from both. MQA and GQA reduce cache by reducing or sharing K/V heads. MLA reduces cache by storing a learned latent representation and using up-projection matrices to recover expressive key/value behavior. In DeepSeek’s framing, this avoids the direct “share K/V heads, lose expressivity” tradeoff.
MLA architecture
DeepSeek-V3’s report gives the clearest compact formulation of MLA. Let ht∈Rdh_t \in \mathbb{R}^dht∈Rd be the attention input for token ttt. MLA first creates a compressed latent vector for keys and values:
ctKV=WDKVhtc_t^{KV} = W^{DKV} h_tctKV=WDKVht where ctKV∈Rdcc_t^{KV} \in \mathbb{R}^{d_c}ctKV∈Rdc and dc≪nhdhd_c \ll n_h d_hdc≪nhdh. It then uses up-projections to form key and value components:
ktC=WUKctKV,vtC=WUVctKVk_t^C = W^{UK} c_t^{KV}, \quad v_t^C = W^{UV} c_t^{KV}ktC=WUKctKV,vtC=WUVctKV The superscript CCC marks the compressed/no-position part. For positional information, MLA separately computes a RoPE-carrying key:
ktR=RoPE(WKRht)k_t^R = \text{RoPE}(W^{KR} h_t)ktR=RoPE(WKRht) and concatenates that shared positional key with each head’s compressed key component:
kt,i=[kt,iC;ktR]k_{t,i} = [k_{t,i}^{C}; k_t^R]kt,i=[kt,iC;ktR] Only ctKVc_t^{KV}ctKV and ktRk_t^RktR need to be cached during generation. DeepSeek explicitly states that this is the cache-reduction mechanism and that it maintains performance comparable to standard MHA. ([DeepSeek-V3 Technical Report]) arXiv
Queries are also low-rank-compressed, but for a different reason. The query latent ctQ=WDQhtc_t^Q = W^{DQ} h_tctQ=WDQht does not reduce the inference KV cache because queries are for the current token rather than the history. It can, however, reduce activation memory during training. DeepSeek then forms compressed query components and RoPE query components:
qtC=WUQctQ,qtR=RoPE(WQRctQ),qt,i=[qt,iC;qt,iR]q_t^C = W^{UQ} c_t^Q,\quad q_t^R = \text{RoPE}(W^{QR}c_t^Q),\quad q_{t,i}=[q_{t,i}^{C};q_{t,i}^{R}]qtC=WUQctQ,qtR=RoPE(WQRctQ),qt,i=[qt,iC;qt,iR] The attention output is then computed with the concatenated query/key and the compressed value component:
ot,i=∑j=1tSoftmaxj(qt,iTkj,idh+dhR)vj,iCo_{t,i} = \sum_{j=1}^{t} \text{Softmax}j \left( \frac{q{t,i}^{T} k_{j,i}}{\sqrt{d_h+d_h^R}} \right) v_{j,i}^{C}ot,i=j=1∑tSoftmaxjdh+dhRqt,iTkj,ivj,iC ut=WO[ot,1;ot,2;…;ot,nh]u_t = W^O [o_{t,1};o_{t,2};\ldots;o_{t,n_h}]ut=WO[ot,1;ot,2;…;ot,nh] ([DeepSeek-V3 Technical Report]) arXiv
Why RoPE must be decoupled
RoPE is useful because it injects positional information into query/key geometry through rotations and gives attention access to relative-position structure. RoFormer introduced this mechanism as a way to encode absolute position while inducing relative-position dependency in self-attention. ([RoFormer / RoPE]) arXiv
The difficulty is that RoPE is position-sensitive. If the full key projection is compressed and then passed through RoPE, the up-projection cannot be cleanly absorbed into a fixed matrix during inference, because a token-position-dependent rotation lies between the matrices. DeepSeek-V2 says this would force recomputation of keys for prefix tokens and therefore destroy the inference-efficiency objective. Its solution is “decoupled RoPE”: keep a small RoPE-bearing key/query path while allowing the larger no-position K/V path to remain low-rank-compressible. ([DeepSeek-V2 Technical Report]) ar5iv
This is the subtle part of MLA. The compression is not just “project K/V down.” It is “project the no-position K/V information down, keep a small positional side channel, and algebraically arrange inference so the latent cache stays latent.”
Cache accounting
A useful simplified comparison is:
| Attention mechanism | What is cached per layer per token | Cache intuition | Main tradeoff |
|---|---|---|---|
| MHA | K,VK,VK,V for every head | Largest cache; maximal per-head K/V diversity | Strong quality, high memory bandwidth |
| MQA | One shared K,VK,VK,V pair | Smallest simple shared-head cache | Can lose head-specific expressivity |
| GQA | K,VK,VK,V for each group | Middle ground between MHA and MQA | Better quality/speed balance than MQA |
| MLA | cKVc^{KV}cKV plus small RoPE key kRk^RkR | Stores compressed latent state, not full K/V | Strong compression, more complex kernels and architecture |
DeepSeek-V2’s own cache formula compares MHA, GQA, MQA, and MLA by number of cached scalar elements per token. For DeepSeek-V2, the report says MLA’s cache is equivalent to GQA with only 2.25 groups, while claiming stronger-than-MHA capability in the same table. ([DeepSeek-V2 Technical Report]) ar5iv
For DeepSeek-V3, the published hyperparameters are nh=128n_h=128nh=128, dh=128d_h=128dh=128, KV compression dimension dc=512d_c=512dc=512, query compression dimension dc′=1536d'_c=1536dc′=1536, and decoupled RoPE per-head dimension dhR=64d_h^R=64dhR=64, across 61 Transformer layers. That means the MLA cache path stores a small latent and positional key rather than 128128128 full key/value heads per layer. ([DeepSeek-V3 Hyperparameters]) arXiv
Empirical evidence from DeepSeek-V2
DeepSeek-V2 is the primary source for MLA’s empirical claim. The report introduces MLA as a replacement for conventional MHA, MQA, and GQA in a 236B-total-parameter MoE model with 21B activated parameters and 128K context support. In the abstract and introduction, DeepSeek reports that, compared with DeepSeek 67B, DeepSeek-V2 achieved stronger performance while saving 42.5% training cost, reducing KV cache by 93.3%, and increasing maximum generation throughput by 5.76×. ([DeepSeek-V2 Technical Report]) ar5iv
DeepSeek’s Appendix D first establishes why the team was not satisfied with MQA/GQA alone. On 7B dense models trained on 1.33T tokens, all else aligned except attention mechanism, MHA outperformed MQA and GQA on four hard benchmarks:
| 7B dense attention ablation | BBH EM | MMLU Acc. | C-Eval Acc. | CMMLU Acc. |
|---|---|---|---|---|
| MQA | 33.2 | 37.9 | 30.0 | 34.6 |
| GQA, 8 groups | 35.6 | 41.2 | 37.7 | 38.4 |
| MHA | 37.0 | 45.2 | 42.9 | 43.5 |
Condensed from DeepSeek-V2 Appendix D.1. The paper’s conclusion is blunt: MHA had “significant advantages” over MQA and GQA on these benchmarks. ([DeepSeek-V2 Ablations]) ar5iv
The stronger claim comes from the MLA-vs-MHA ablation. DeepSeek trained small and large MoE models that differed in attention mechanism and compared both capability and cache size:
| MoE attention ablation | KV cache per token | BBH EM | MMLU Acc. | C-Eval Acc. | CMMLU Acc. |
|---|---|---|---|---|---|
| Small MoE w/ MHA | 110.6K | 37.9 | 48.7 | 51.6 | 52.3 |
| Small MoE w/ MLA | 15.6K | 39.0 | 50.0 | 50.9 | 53.4 |
| Large MoE w/ MHA | 860.2K | 46.6 | 57.5 | 57.9 | 60.7 |
| Large MoE w/ MLA | 34.6K | 50.7 | 59.0 | 59.2 | 62.5 |
Condensed from DeepSeek-V2 Appendix D.2. The small MLA model is slightly lower on C-Eval but higher on BBH, MMLU, and CMMLU; the large MLA model is higher on all four listed benchmarks while using about 4% of the large MHA cache. ([DeepSeek-V2 Ablations]) ar5iv
This is why the right interpretation is not merely “MLA compresses KV cache.” The DeepSeek claim is: for their architecture and training setup, MLA compresses KV cache while maintaining or improving benchmark quality. That is still an empirical claim from the model developer, not a theorem that MLA is always lossless.
Evidence from DeepSeek-V3
DeepSeek-V3 is the scale-up validation, not a clean MLA ablation. The V3 report says the model remains within the Transformer framework and adopts MLA for efficient inference and DeepSeekMoE for economical training, with those choices “thoroughly validated” in DeepSeek-V2. V3 has 671B total parameters, 37B activated parameters per token, and was pretrained on 14.8T tokens. ([DeepSeek-V3 Technical Report]) arXiv
The V3 report also gives the systems-level efficiency claim: full training, including pretraining, context extension, and post-training, required 2.788M H800 GPU hours. But this number should not be attributed to MLA alone. V3’s efficiency story is a stack: MLA for inference and attention-memory efficiency; DeepSeekMoE for sparse activation; FP8 mixed-precision training; auxiliary-loss-free load balancing; DualPipe and communication overlap; and Multi-Token Prediction for extra decoding speed. ([DeepSeek-V3 Technical Report]) arXiv
DeepSeek’s public V3 repository summarizes the same stack: MLA and DeepSeekMoE for efficient inference and cost-effective training, FP8 mixed precision, communication co-design, and 2.664M H800 GPU hours for pretraining, with later stages adding only about 0.1M GPU hours. It also lists multiple serving frameworks—SGLang, LMDeploy, TensorRT-LLM, vLLM, LightLLM, AMD GPU support, and Huawei Ascend support—showing that real deployment requires model-specific inference integration rather than generic “run any Transformer” assumptions. ([DeepSeek-V3 GitHub]) GitHub
The careful conclusion: V3 shows MLA can survive at frontier-scale open-weight model size inside a strong system. It does not isolate MLA as the reason V3 performs well.
The “absorb” operation
A recurring implementation term around MLA is the Absorb Operation. In plain terms, some MLA up-projection matrices do not need to be materialized at every decode step. Matrix associativity lets the serving implementation merge fixed parameter matrices offline or into adjacent projections. For example, the no-position key score can be arranged so the current query-side computation interacts directly with the cached latent cjKVc_j^{KV}cjKV, rather than first reconstructing a full key for every cached token. DeepSeek-V2 notes that this avoids recomputing full keys and values during inference. ([DeepSeek-V2 Technical Report]) ar5iv
The absorb trick is why decoupled RoPE is not incidental. The no-position part can be absorbed because its matrices are fixed. The RoPE part cannot be fully absorbed because the rotation depends on position. Later work such as MHA2MLA makes this explicit: MLA stores latent cKVc^{KV}cKV plus the RoPE component, and RoPE separation is necessary because the position-dependent rotation prevents a single fixed matrix merge. ([MHA2MLA / ACL 2025]) ACL Anthology
Relationship to the KV-cache-compression family
MLA belongs beside MQA and GQA, but it should not be collapsed into them.
| Method | Compression axis | Does it alter attention expressivity? | Typical integration difficulty |
|---|---|---|---|
| MQA | Share K/V across all query heads | Yes; one K/V head for many Q heads | Relatively simple |
| GQA | Share K/V across groups of query heads | Yes; fewer K/V heads than Q heads | Moderate; widely deployed |
| MLA | Store low-rank latent K/V plus positional side channel | Less direct pruning; expressivity moved into latent/up-projection structure | High; requires architecture and kernel support |
| KV quantization | Store cache in fewer bits | Usually preserves architecture, changes numeric precision | Serving-system dependent |
| Token/cache pruning | Drop or summarize cached tokens | Can lose long-range dependencies | Task- and policy-dependent |
MHA2MLA’s related-work section divides economical KV-cache methods into architecture changes, quantization, and dynamic pruning; it identifies MLA as an architecture method and notes that it had been successfully validated in DeepSeek’s LLMs. It also reports that converting pretrained models to MLA is nontrivial because MHA/GQA and MLA have inherent architectural disparities. ([MHA2MLA / ACL 2025]) ACL Anthology
This distinction matters for engineering strategy. Quantization can often be applied after training. GQA can sometimes be introduced by uptraining from MHA checkpoints. MLA is more like a model-family design decision: easiest when chosen before pretraining, harder when retrofitted.
Why MLA mattered in DeepSeek’s efficiency story
DeepSeek’s public narrative around V2/V3 is not only “we trained a strong model cheaply.” It is “we co-designed architecture, training, and inference around cost.” MLA is one of the recognizable architectural ingredients because it directly lowers the memory pressure that grows with context length and batch size. DeepSeek-V2 explicitly presents MLA and DeepSeekMoE as the two major Transformer-block changes: MLA for inference efficiency, DeepSeekMoE for economical training through sparsity. ([DeepSeek-V2 Technical Report]) ar5iv
The architectural significance is that MLA makes cache size a learned representation-design problem. MQA/GQA choose a point on a sharing spectrum; MLA learns a bottleneck representation and then uses projections to recover head-specific behavior. That is closer in spirit to low-rank model design than to simple head sharing.
The practical significance is that cache compression can improve serving economics even if raw parameter count remains enormous. In long-context serving, the KV cache can dominate memory; once context length grows, a model’s active parameter count is not the only limiting factor. MLA therefore complements MoE: MoE reduces activated compute per token, while MLA reduces per-token historical state.
But MLA should not be credited for all of DeepSeek’s training efficiency. DeepSeek-V3’s report attributes cost-effectiveness to FP8 training and engineering optimizations as well as architecture, and its conclusion separately identifies MLA, DeepSeekMoE, auxiliary-loss-free balancing, and Multi-Token Prediction. ([DeepSeek-V3 Technical Report]) arXiv
Implementation realities
MLA is specialized. DeepSeek-V2 says its MLA was optimized using an improved version of FlashAttention-2, and the broader literature on FlashAttention-2 is about exploiting GPU memory hierarchy, work partitioning, and better parallelism for attention kernels. ([DeepSeek-V2 Technical Report]) ar5iv ([FlashAttention-2]) arXiv
Modern serving engines now expose MLA-specific implementation paths. vLLM’s MLA documentation distinguishes data-movement-friendly and compute-friendly MLA computation, preferring different modes for decode versus prefill. That is exactly the kind of systems detail hidden by a high-level architecture diagram: MLA changes the serving kernel’s shape, not just the model config. ([vLLM MLA docs]) docs.vllm.ai
DeepSeek also released FlashMLA, a library of optimized MLA kernels used for DeepSeek models. Its README lists dense and sparse MLA decoding/prefill kernels, hardware requirements such as SM90/SM100 and CUDA 12.8+, and performance numbers for H800/B200-class GPUs. This is useful evidence that MLA’s best efficiency depends on specialized GPU kernels and hardware-aware implementation. ([DeepSeek FlashMLA]) GitHub
DeepSeek-V3 itself acknowledges deployment limitations. The report says the recommended deployment unit is relatively large, which can burden small teams, and that there remains room to improve generation speed even after V3’s deployment strategy exceeded V2’s end-to-end generation speed by more than 2×. ([DeepSeek-V3 Technical Report]) arXiv
Limitations and unresolved issues
1. MLA is not a universal no-loss guarantee. DeepSeek’s V2 ablations are strong evidence for its own model family, but not proof that every architecture, scale, dataset, or training recipe will benefit. The safe claim is “DeepSeek reports maintained or improved quality in V2 ablations and successful scaling in V3,” not “MLA is mathematically lossless.”
2. It complicates model implementation. Decoupled RoPE, latent K/V storage, absorb logic, and prefill/decode-specific kernels are more complex than GQA. GQA is attractive partly because it is simple: fewer K/V heads, same broad attention pattern.
3. It changes the training recipe. Choosing MLA before pretraining is straightforward for a lab designing a new model. Retrofitting an existing MHA/GQA checkpoint is harder. MHA2MLA proposes partial-RoPE and joint SVD approximations and reports recovery with 0.6%–1% of training data, but that paper exists precisely because the migration is not trivial. ([MHA2MLA / ACL 2025]) ACL Anthology
4. It may shift bottlenecks rather than remove them. MLA trades memory and communication for extra projection structure and specialized compute. TransMLA’s abstract explicitly frames MLA as using up-projection to trade additional computation for lower communication overhead. That trade is favorable on memory-bandwidth-bound hardware, but not automatically optimal for every accelerator, batch regime, or context length. ([TransMLA]) arXiv
5. Ecosystem inertia favors GQA. Many widely used model families and inference stacks have invested heavily in GQA. Qwen3’s report says its dense models use GQA, RoPE, SwiGLU, and RMSNorm; Google’s Gemma 3 materials say Gemma 2 and Gemma 3 use GQA; Meta’s Llama 3 launch said GQA was adopted across both 8B and 70B sizes. ([Qwen3 Technical Report]) arXiv ([Google Gemma 3]) developers.googleblog.com ([Meta Llama 3]) ai.meta.com
Will MLA become standard?
There are two plausible futures.
In the first, MLA becomes the next standard attention-efficiency primitive for large long-context models. The argument is straightforward: cache memory is a real deployment bottleneck; MLA appears to dominate GQA in the DeepSeek ablations; and post-training migration work is emerging. MHA2MLA reports a Llama2-7B KV-cache reduction of 92.19% with only a 1% LongBench drop, using a data-efficient conversion method. TransMLA argues that GQA can be represented by MLA with the same KV-cache overhead, while the reverse does not hold, and proposes converting GQA models such as LLaMA, Qwen, Gemma, Mistral/Mixtral into DeepSeek-compatible MLA forms. ([MHA2MLA / ACL 2025]) ACL Anthology ([TransMLA / OpenReview]) OpenReview
In the second, MLA remains a DeepSeek-specific or DeepSeek-adjacent design. The counterargument is also strong: GQA is simple, well-understood, broadly deployed, and supported across modern inference stacks. Many model developers may prefer GQA plus cache quantization, sliding/local attention, or hybrid sparse attention over adopting a more complex MLA parameterization. Public evidence through the coverage date shows MLA-centered work growing, but not yet displacing GQA as the default in major non-DeepSeek open model reports.
The most likely near-term outcome is selective adoption. MLA is attractive where long-context inference, batch concurrency, and memory bandwidth dominate costs. GQA remains attractive where implementation simplicity, portability, and pretrained-checkpoint continuity matter more. The unresolved frontier question is whether future foundation-model labs decide that MLA’s extra architectural complexity is worth standardizing before pretraining, or whether they treat it as one specialized point in a broader menu of Inference-Time Memory Optimization techniques.
Selected primary sources
DeepSeek-V2 Technical Report. Primary source for MLA’s introduction, low-rank joint K/V compression, decoupled RoPE, cache comparison, and V2 ablations. ar5iv
DeepSeek-V3 Technical Report. Primary source for V3’s scaled MLA formulation, hyperparameters, efficiency claims, and deployment limitations. arXiv+2arXiv+2
Attention Is All You Need. Baseline source for the Transformer and standard multi-head attention. arXiv
Fast Transformer Decoding: One Write-Head Is All You Need. Primary source for MQA and the memory-bandwidth framing of incremental decoding. arXiv
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. Primary source for GQA as the MQA/MHA middle ground and uptraining approach. arXiv
RoFormer: Enhanced Transformer with Rotary Position Embedding. Primary source for RoPE. arXiv
MHA2MLA and TransMLA. Post-DeepSeek research on migrating existing MHA/GQA models toward MLA-like architectures. ACL Anthology
Companion entries
Core theory: Transformer Architecture, Multi-Head Attention, Scaled Dot-Product Attention, Rotary Position Embeddings, Low-Rank Factorization, Latent Representations
Inference systems: KV Cache, KV-Cache Compression, Long-Context Inference, Memory-Bandwidth Bottlenecks, FlashAttention, PagedAttention, Inference-Time Memory Optimization
Attention variants: Multi-Query Attention, Grouped-Query Attention, Multi-Head Latent Attention, Sliding Window Attention, Sparse Attention, Linear Attention, State Space Models
DeepSeek architecture: DeepSeek-V2, DeepSeek-V3, DeepSeekMoE, Auxiliary-Loss-Free Load Balancing, Multi-Token Prediction, FP8 Training, Absorb Operation
Practice: Serving Large Language Models, Prefill and Decode, Speculative Decoding, Tensor Parallelism, Expert Parallelism, GPU Kernel Co-Design
Counterarguments: GQA as the Default Attention Compromise, Architecture-Specific Optimization Risk, Benchmark Attribution Problems, Frontier Model Secrecy, Simplicity Bias in Model Architecture