Wilson Confidence Intervals
Wilson confidence intervals are a practical default for estimating a single binomial proportion because they retain the computational simplicity of normal approximations while avoiding the worst failures of the textbook Wald interval. Their main value is not that they are “exact,” but that they invert the score test rather than plugging the observed proportion into its own standard error, giving much better behavior for small samples and proportions near 0 or 1. In modern LLM evaluation, Wilson intervals are useful for independent binary quantities—win rates, pass/fail rates, violation rates—but they are not a substitute for cluster-aware uncertainty when examples, authors, scenarios, or judges are reused.
Coverage note: verified through May 19, 2026.
Problem statement: estimating one binomial proportion
The core setting is simple. We observe X=xX=xX=x successes in nnn Bernoulli trials and want an interval estimate for an unknown success probability ppp:
X∼Binomial(n,p),p^=xn.X \sim \mathrm{Binomial}(n,p), \qquad \hat p = \frac{x}{n}.X∼Binomial(n,p),p^=nx. This covers coin flips, defect rates, model accuracy on a fixed benchmark slice, refusal-rate estimates, toxicity-flag rates, and pairwise win rates after the evaluator has reduced each comparison to a binary success/failure event. The simplicity is deceptive. Because XXX is discrete and ppp is bounded in [0,1][0,1][0,1], standard continuous normal approximations can produce intervals that are too narrow, outside the valid range, or even degenerate at exactly the cases where uncertainty should be highest.
The Wilson interval, also called the Wilson score interval, is named for Edwin B. Wilson’s 1927 note “Probable Inference, the Law of Succession, and Statistical Inference.” Wilson’s key move was to use the unknown ppp in the binomial standard error while solving for the range of ppp values compatible with the observation, rather than substituting p^\hat pp^ into the standard error and pretending that solves the inverse problem. The original note explicitly shows the quadratic equation and the resulting interval center shifted toward 1/21/21/2. jhanley.biostat.mcgill.ca
In the Confidence Intervals taxonomy, Wilson is an approximate frequentist interval obtained by inverting a Score Test. It sits between two extremes: the Wald Test interval, which is very simple but unreliable, and the Clopper-Pearson Interval, which has guaranteed minimum coverage under the exact binomial model but is often conservative and wider than practitioners want.
The four common binomial CI methods
Let
z=Φ−1(1−α/2)z = \Phi^{-1}(1-\alpha/2)z=Φ−1(1−α/2) be the standard normal critical value for a two-sided 100(1−α)%100(1-\alpha)%100(1−α)% confidence interval. For a 95% interval, z≈1.96z \approx 1.96z≈1.96.
| Method | Formula / construction | Main advantage | Main failure mode | Practical status |
|---|---|---|---|---|
| Wald / asymptotic normal | p^±zp^(1−p^)/n\hat p \pm z\sqrt{\hat p(1-\hat p)/n}p^±zp^(1−p^)/n | Fast, familiar, easy to teach | Can leave [0,1][0,1][0,1], collapses to zero width at x=0x=0x=0 or x=nx=nx=n, poor and erratic coverage | Avoid except for rough large-nnn, central-ppp diagnostics |
| Wilson score | Invert the score test: p^−pp(1−p)/n\frac{\hat p-p}{\sqrt{p(1-p)/n}}p(1−p)/np^−p | Boundary-respecting, closed form, good coverage/width tradeoff | Still approximate; assumes independent Bernoulli trials; not guaranteed exact coverage | Strong default for one independent proportion, especially small nnn |
| Clopper-Pearson exact | Invert exact binomial tail tests; beta-quantile endpoints | Guarantees at least nominal coverage under the binomial model | Conservative because of discreteness; intervals often wider than needed | Use when minimum coverage is mandatory |
| Agresti-Coull | Adjusted Wald: n~=n+z2\tilde n=n+z^2n~=n+z2, p~=(x+z2/2)/(n+z2)\tilde p=(x+z^2/2)/(n+z^2)p |
Very simple; for 95%, approximately “add two successes and two failures” | Slightly more conservative than Wilson; can still need truncation for display | Good teaching/simple-reporting alternative |
The Wald interval is the common textbook interval. Agresti and Coull describe it as the interval obtained by inverting the Wald test, and they contrast it with the score interval, which uses the null standard error rather than the estimated standard error. They also state the Clopper-Pearson construction as an inversion of equal-tailed binomial tests and note its guaranteed coverage property. Department of Mathematics & Statistics
Clopper and Pearson’s original 1934 paper, “The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial,” appeared in Biometrika 26(4), pages 404–413. The modern “exact” interval associated with their names is exact in the sense of meeting or exceeding nominal coverage under the binomial model, not exact in the sense of being shortest or automatically best for practice. OUP Academic
Agresti and Coull’s 1998 paper made the provocative but defensible point that “approximate” intervals can be better than “exact” intervals for practical interval estimation, because exact binomial inversion tends to over-cover while Wald intervals under-cover. They argue that Wilson’s score interval gives coverage close to nominal even for very small samples and that the adjusted Wald interval behaves similarly for 95% intervals after adding two successes and two failures. Department of Mathematics & Statistics
Wilson score interval: formula
The Wilson interval for xxx successes in nnn trials is:
Wilson1−α(x,n)=p^+z22n1+z2n ± z1+z2np^(1−p^)n+z24n2.\mathrm{Wilson}_{1-\alpha}(x,n)
\frac{\hat p + \frac{z^2}{2n}}{1+\frac{z^2}{n}} ;\pm; \frac{z}{1+\frac{z^2}{n}} \sqrt{ \frac{\hat p(1-\hat p)}{n} + \frac{z^2}{4n^2} }.Wilson1−α(x,n)=1+nz2p^+2nz2±1+nz2znp^(1−p^)+4n2z2. Equivalently, its center is
cW=p^+z2/(2n)1+z2/n=x+z2/2n+z2.c_W = \frac{\hat p + z^2/(2n)}{1+z^2/n}
\frac{x+z^2/2}{n+z^2}.cW=1+z2/np^+z2/(2n)=n+z2x+z2/2. That center is a weighted average of the observed proportion p^\hat pp^ and 1/21/21/2:
cW=(nn+z2)p^+(z2n+z2)12.c_W = \left(\frac{n}{n+z^2}\right)\hat p + \left(\frac{z^2}{n+z^2}\right)\frac12.cW=(n+z2n)p^+(n+z2z2)21. This shrinkage toward 1/21/21/2 is not Bayesian magic, although it resembles pseudocount reasoning. It arises algebraically from score-test inversion. When nnn is small or the confidence level is high, the z2z^2z2 term is large relative to nnn, so the center moves farther from the boundary. As n→∞n\to\inftyn→∞, the adjustment vanishes and the Wilson interval approaches the familiar normal approximation.
A useful boundary case is x=0x=0x=0. The Wald interval gives [0,0][0,0][0,0], which is clearly not a plausible 95% uncertainty statement after only a finite number of failures. Wilson gives
[0,z2n+z2].\left[0,\frac{z^2}{n+z^2}\right].[0,n+z2z2]. For n=20n=20n=20 and 95% confidence, this is approximately [0,0.161][0,0.161][0,0.161], while Clopper-Pearson gives approximately [0,0.168][0,0.168][0,0.168]. The two are close in this case, but their philosophies differ: Wilson is approximate and score-based; Clopper-Pearson is exact-tail inversion and conservative. Wilson’s original note explicitly called the zero-width all-failure inference “out of the question” in substance, even if not in those exact modern terms. jhanley.biostat.mcgill.ca
Derivation: score test inversion
Wilson begins from the binomial normal approximation
p^≈N(p,p(1−p)n).\hat p \approx N\left(p, \frac{p(1-p)}{n}\right).p^≈N(p,np(1−p)). For a candidate null value p0p_0p0, the score-test statistic is
Z(p0)=p^−p0p0(1−p0)/n.Z(p_0)=\frac{\hat p-p_0}{\sqrt{p_0(1-p_0)/n}}.Z(p0)=p0(1−p0)/np^−p0. A two-sided score test fails to reject H0:p=p0H_0:p=p_0H0:p=p0 when
∣p^−p0p0(1−p0)/n∣≤z.\left|\frac{\hat p-p_0}{\sqrt{p_0(1-p_0)/n}}\right| \le z.p0(1−p0)/np^−p0≤z. A confidence interval is the set of p0p_0p0 values that would not be rejected:
n(p^−p)2≤z2p(1−p).n(\hat p-p)^2 \le z^2 p(1-p).n(p^−p)2≤z2p(1−p). Expanding gives a quadratic inequality in ppp:
(n+z2)p2−(2np^+z2)p+np^2≤0.(n+z^2)p^2-(2n\hat p+z^2)p+n\hat p^2 \le 0.(n+z2)p2−(2np^+z2)p+np^2≤0. The endpoints are the two roots:
p=2np^+z2±zz2+4np^(1−p^)2(n+z2).p = \frac{2n\hat p+z^2 \pm z\sqrt{z^2+4n\hat p(1-\hat p)}}{2(n+z^2)}.p=2(n+z2)2np^+z2±zz2+4np^(1−p^). Algebraic rearrangement yields the standard Wilson formula above. The important conceptual point is that the standard error is evaluated at the null candidate ppp, not at p^\hat pp^. Agresti and Coull describe this as inverting the approximately normal test that uses the null standard error, and identify it as the score confidence interval first discussed by Wilson. Department of Mathematics & Statistics
This also explains why Wilson respects the boundary. The quadratic knows that p(1−p)p(1-p)p(1−p) collapses near 0 or 1, but it does not let an observed p^=0\hat p=0p^=0 imply zero uncertainty. The observed all-failure case is treated as evidence about ppp, not as proof that p=0p=0p=0.
Wald derivation: why the textbook interval fails
The Wald interval begins from the same asymptotic normal idea but substitutes p^\hat pp^ into the standard error:
p^±zp^(1−p^)n.\hat p \pm z\sqrt{\frac{\hat p(1-\hat p)}{n}}.p^±znp^(1−p^). This is computationally convenient, but it is a poor inversion of the binomial problem. At x=0x=0x=0, it gives zero standard error. At x=nx=nx=n, it also gives zero standard error. When xxx is small but nonzero, it can produce negative lower bounds; when xxx is near nnn, it can produce upper bounds above 1.
The deeper problem is coverage. Brown, Cai, and DasGupta’s 2001 review is the canonical modern citation here. Their abstract says the Wald interval’s coverage behavior is erratic and that its “chaotic coverage properties” are more persistent than commonly appreciated; they recommend Wilson or equal-tailed Jeffreys intervals for small nnn, and Agresti-Coull for larger nnn. Buffalo ACSU
Their examples are severe. For a nominal 95% Wald interval with n=100n=100n=100, actual coverage is about 0.952 at p=0.106p=0.106p=0.106 but only 0.911 at p=0.107p=0.107p=0.107. At p=0.5p=0.5p=0.5, the nominal 95% Wald interval has coverage about 0.953 at n=17n=17n=17 but only 0.919 at n=40n=40n=40. For p=0.005p=0.005p=0.005, coverage can rise near nominal and then drop abruptly when nnn increases by one, e.g., from near 0.945 at n=591n=591n=591 to about 0.792 at n=592n=592n=592. Buffalo ACSU
The cause is not merely “small samples.” The binomial distribution is discrete, so coverage oscillates as a function of both ppp and nnn. Brown, Cai, and DasGupta attribute these oscillations to the lattice structure of the binomial distribution and emphasize that some oscillation is unavoidable for nonrandomized procedures. The issue is that Wald combines discreteness, boundary problems, and skewness in a particularly bad way. Buffalo ACSU
Clopper-Pearson: exact enumeration
The Clopper-Pearson interval inverts exact binomial tail probabilities. For 0<x<n0<x<n0<x<n, the usual equal-tailed endpoints can be written using beta quantiles:
L=Bα/2−1(x, n−x+1),L = B^{-1}{\alpha/2}(x,; n-x+1),L=Bα/2−1(x,n−x+1), U=B1−α/2−1(x+1, n−x),U = B^{-1}{1-\alpha/2}(x+1,; n-x),U=B1−α/2−1(x+1,n−x), with boundary conventions L=0L=0L=0 when x=0x=0x=0 and U=1U=1U=1 when x=nx=nx=n.
Equivalently, the lower endpoint solves an exact upper-tail equation and the upper endpoint solves an exact lower-tail equation:
∑k=xn(nk)Lk(1−L)n−k=α/2,\sum_{k=x}^{n} {n \choose k} L^k(1-L)^{n-k} = \alpha/2,k=x∑n(kn)Lk(1−L)n−k=α/2, ∑k=0x(nk)Uk(1−U)n−k=α/2.\sum_{k=0}^{x} {n \choose k} U^k(1-U)^{n-k} = \alpha/2.k=0∑x(kn)Uk(1−U)n−k=α/2. This is Exact Tests logic: enumerate the binomial distribution under each candidate ppp, then keep the ppp values whose observed result would not fall into the rejection region.
The advantage is a hard coverage guarantee under the independent-binomial model. The disadvantage is conservatism. Because XXX is discrete, exact equal-tailed inversion cannot usually hit exactly 1−α1-\alpha1−α; it often exceeds it. Agresti and Coull argue that this makes Clopper-Pearson inappropriate as an unquestioned “gold standard” for ordinary statistical practice, because actual coverage can be much larger than nominal and the resulting intervals wider than necessary. Department of Mathematics & Statistics
Use Clopper-Pearson when the analysis context values minimum coverage above width: regulatory testing, formal safety thresholds, or settings where undercoverage is unacceptable by policy. Do not use it merely because the word “exact” sounds more rigorous.
Likelihood inversion: related but not Wilson
Another route is to invert the binomial likelihood ratio test. The binomial log-likelihood is
ℓ(p)=xlogp+(n−x)log(1−p),\ell(p)=x\log p+(n-x)\log(1-p),ℓ(p)=xlogp+(n−x)log(1−p), with maximum at p^=x/n\hat p=x/np^=x/n. A likelihood-ratio interval keeps ppp values satisfying
2{ℓ(p^)−ℓ(p)}≤χ1,1−α2.2{\ell(\hat p)-\ell(p)} \le \chi^2_{1,1-\alpha}.2{ℓ(p^)−ℓ(p)}≤χ1,1−α2. Equivalently,
2[xlogp^p+(n−x)log1−p^1−p]≤χ1,1−α2,2\left[ x\log\frac{\hat p}{p} + (n-x)\log\frac{1-\hat p}{1-p} \right] \le \chi^2_{1,1-\alpha},2[xlogpp^+(n−x)log1−p1−p^]≤χ1,1−α2, with the usual limiting conventions when x=0x=0x=0 or x=nx=nx=n.
Likelihood inversion is conceptually clean and often performs well, but it typically requires numerical root-finding rather than a simple closed-form expression. Agresti and Coull note that likelihood-ratio inversion is similar to the score interval in its comparison with exact intervals but more complex to construct. Department of Mathematics & Statistics
The distinction matters for wiki use: Wilson is not a likelihood interval. It is a score-test inversion. Wald, Wilson, and likelihood-ratio intervals correspond to three classic asymptotic test families—Wald, score, and likelihood-ratio—applied to the same binomial model.
Agresti-Coull: the adjusted Wald compromise
The Agresti-Coull interval takes the Wilson center and then uses a Wald-style interval around the adjusted estimate:
n~=n+z2,\tilde n = n+z^2,n~=n+z2,
p~=x+z2/2n+z2,\tilde p = \frac{x+z^2/2}{n+z^2},p=n+z2x+z2/2,
p±zp~(1−p~)n~.\tilde p \pm z\sqrt{\frac{\tilde p(1-\tilde p)}{\tilde n}}.p±znp(1−p).
For 95% confidence, z2≈3.84z^2\approx 3.84z2≈3.84, which is close to 4. Thus
p~≈x+2n+4.\tilde p \approx \frac{x+2}{n+4}.p~≈n+4x+2. This is the famous “add two successes and two failures” rule. It is not identical to Wilson, but it inherits Wilson’s shrinkage intuition and is easier to explain to nontechnical audiences. Brown, Cai, and DasGupta describe Agresti-Coull as having good minimum coverage but being more conservative than Wilson, especially for small nnn; they also state that the Agresti-Coull interval contains the Wilson interval as a proper subinterval in their formulation. Buffalo ACSU
For a 95% interval with x=2,n=10x=2,n=10x=2,n=10:
| Method | Approximate interval |
|---|---|
| Wald | [−0.048, 0.448][-0.048,;0.448][−0.048,0.448] |
| Wilson | [0.057, 0.510][0.057,;0.510][0.057,0.510] |
| Clopper-Pearson | [0.025, 0.556][0.025,;0.556][0.025,0.556] |
| Agresti-Coull | [0.046, 0.521][0.046,;0.521][0.046,0.521] |
The Wald lower bound is impossible. Wilson and Agresti-Coull are both boundary-respecting after construction or conventional truncation, and Clopper-Pearson is wider because it protects minimum coverage.
Empirical case for Wilson as the default
The empirical case for Wilson is not that it dominates every criterion. It does not. Brown, Cai, and DasGupta also recommend the equal-tailed Jeffreys prior interval for small nnn, and there are mid-PPP, likelihood-ratio, and continuity-corrected variants with their own advantages. The case for Wilson is that it occupies a particularly attractive engineering point: closed form, no numerical search, boundary-respecting, good coverage behavior, and reasonable expected width.
Brown, Cai, and DasGupta’s review is especially damaging to the Wald default. They conclude that the standard Wald interval is persistently chaotic and unacceptably poor, and they recommend Wilson or Jeffreys for small nnn, with Agresti-Coull as a simple option for larger nnn. Buffalo ACSU
Agresti and Coull independently make the same broad practical point: Wald intervals tend to under-cover, exact intervals tend to over-cover, and Wilson’s score interval tends to stay close to nominal coverage even for very small samples. Their paper also reframes Agresti-Coull as a teaching-friendly approximation to score-interval behavior. Department of Mathematics & Statistics
Newcombe’s 1998 single-proportion comparison evaluated seven methods over 96,000 parameter-space points and recommended intervals based on tail areas and simpler score methods. That result is consistent with the Wilson-default view: avoid naive Wald; use score or carefully chosen exact/tail-area methods depending on the objective. PubMed
The honest conclusion is:
| Decision criterion | Best practical choice |
|---|---|
| One independent proportion, ordinary engineering report | Wilson |
| One independent proportion, audience needs a very simple formula | Agresti-Coull |
| Minimum coverage guarantee is required | Clopper-Pearson |
| Rough exploratory calculation with large nnn, p^\hat pp^ far from 0/1 | Wald is tolerable but still not preferred |
| Small nnn, high-stakes inference | Wilson, Jeffreys, or exact/tail-area methods; report method explicitly |
| Difference of two proportions | Newcombe hybrid score or other two-sample method, not single-proportion Wilson |
Why Wilson matters for LLM evaluation
Many LLM Evaluation metrics are binomial proportions in disguise.
A pairwise win-rate experiment asks: out of nnn comparisons, how often did model or condition A beat B? If ties are excluded and each remaining comparison is treated as decisive, the win count is binomial under an independence model. If ties are counted as half-wins, the statistic is no longer a Bernoulli count and a simple Wilson interval is not strictly appropriate without redefining the target.
A red-flag-rate experiment asks: out of nnn generations, how many triggered a safety violation, hallucination flag, refusal error, privacy leak, or other binary event? That is a binomial proportion if each generation is an independent Bernoulli trial with a common event probability.
A benchmark-slice accuracy report asks: out of nnn examples in a narrow category, how many did the model answer correctly? Again, that is a binomial proportion under the independent-item model. This is especially important for small slices: “model got 9/10” is not a stable estimate of 90% ability; Wilson will make the uncertainty visible.
The same is true for Small-N Benchmark Slices in eval dashboards. If a model passes 3 of 4 adversarial prompts in a scenario family, a Wald interval can be absurdly confident or badly placed. A Wilson interval keeps the estimate grounded: the point estimate may be 0.75, but the interval will communicate that four trials are not enough for a precise claim.
This is why Wilson belongs in LLM-eval infrastructure as a default leaf statistic. It prevents the most common error in empirical AI reporting: treating a small observed percentage as if it were a stable property of the model.
PsycheEval and analyzer practice
The public PsycheEval v0.2 report is a useful example because it shows both sides of the issue. The report’s first analyzer pass used pairwise win-rate tables across same-author judges and explicitly says those rates were cluster-bootstrapped by persona × scenario × author. It also reports a length-matched subset with n=78n=78n=78, a controlled lo_win of 0.449, and a Wilson CI of [0.343,0.559][0.343,0.559][0.343,0.559]. Later AB/BA-corrected headline tables use 2,000-resample cluster bootstrap CIs with the same persona × scenario × author clustering unit. Ashita Orbis
That supports the narrower, source-grounded statement: Wilson appears in PsycheEval’s visible analyzer output for independent-binomial-style subsets, while cluster bootstrap is used for grouped headline pairwise rates. A stronger code-level claim that every PsycheEval analyzer default is Wilson would require public analyzer documentation or source code beyond the post. The methodological pattern, however, is exactly the one this article recommends: Wilson for leaf binomial slices; cluster-aware intervals for grouped eval claims.
PsycheEval v0.2 also demonstrates why pairwise LLM evaluation should not stop at binomial win-rate intervals. Its original pairwise table was undermined by position bias; the AB/BA counterbalanced rejudge changed which headline claims survived. The report cites LLM-as-judge position-bias literature and shows that the apparent C5_CONTRACT advantages over C3 and C4 collapsed after counterbalancing. Ashita Orbis
That is not a Wilson failure. It is a design-dependence failure. Wilson answers: “Given this binary independent-trial model, how uncertain is the observed proportion?” It does not answer: “Was the judge prompt biased, were the trials exchangeable, or did the same scenario family induce correlation?”
Wilson’s independence assumption
Wilson assumes the data can be summarized as
X=∑i=1nYi,Yi∼iidBernoulli(p).X=\sum_{i=1}^{n}Y_i,\qquad Y_i\overset{\mathrm{iid}}{\sim}\mathrm{Bernoulli}(p).X=i=1∑nYi,Yi∼iidBernoulli(p). That assumption is often false in real evals. Consider an LLM pairwise study with repeated structure:
the same author/model contributes many outputs;
the same scenario family appears repeatedly;
the same judge model scores many pairs;
the same persona or prompt template recurs;
AB/BA swaps create paired observations;
multiple outputs are generated from the same base question.
In such settings, the row count nnn overstates the amount of independent information. A Wilson interval computed over rows will often be too narrow because within-cluster correlation reduces effective sample size.
For example, 1,000 pairwise judgments across 20 scenarios are not equivalent to 1,000 independent scenarios. If a model performs especially well on a few scenario families and poorly on others, resampling rows erases the structure that matters most. The uncertainty should reflect scenario-to-scenario variation, not only row-level Bernoulli noise.
Cluster bootstrap relationship
The cluster bootstrap is a design-aware alternative. Instead of resampling individual rows, resample clusters: authors, scenarios, persona × scenario × author units, users, benchmark tasks, or whatever unit best represents independent variation in the data-generating process. The CRAN ClusterBootstrap documentation describes the package as performing bootstrapping on hierarchically structured data with clustered or nested resampling at arbitrary hierarchy levels, and allowing arbitrary statistics to be computed on each resampled dataset. cran.r-project.org
For an LLM evaluation report, a basic cluster-bootstrap workflow is:
Define the estimand: win rate, violation rate, accuracy, controlled AB/BA win rate, difference in win rates, etc.
Define the independent resampling unit: often scenario, user, document, author, or persona × scenario × author.
Resample clusters with replacement.
Recompute the statistic on each bootstrap sample.
Use percentile, BCa, or another bootstrap interval procedure.
Report the cluster unit and number of bootstrap replicates.
The key is that the statistic can still be a proportion. The difference is the uncertainty model. Wilson treats nnn rows as independent Bernoulli trials. Cluster bootstrap treats the clustered design as the object whose sampling variability must be approximated.
This creates a clean hierarchy for Evaluation Uncertainty:
| Level | Typical statistic | Appropriate interval |
|---|---|---|
| Independent binary leaf metric | “7/20 examples passed” | Wilson |
| Repeated examples within scenario family | “Mean pass rate over scenario clusters” | Cluster bootstrap or hierarchical model |
| Pairwise judge with AB/BA swaps | “Controlled win rate” | Paired or clustered bootstrap |
| Multi-author, multi-scenario eval | “Overall model advantage” | Cluster bootstrap / hierarchical CI |
| Formal binomial acceptance test | “At most kkk failures allowed” | Exact binomial / Clopper-Pearson |
Wilson can still appear inside a clustered report as a diagnostic. For example, each scenario family may show Wilson intervals for its local pass rate, while the headline cross-family claim uses a cluster bootstrap.
Hierarchical CIs and random effects
Cluster bootstrap is not the only fix. A hierarchical model can represent repeated structure explicitly:
Yij∼Bernoulli(pij),Y_{ij} \sim \mathrm{Bernoulli}(p_{ij}),Yij∼Bernoulli(pij), logit(pij)=β0+uscenario(i)+vauthor(i)+wjudge(i)+⋯ .\mathrm{logit}(p_{ij}) = \beta_0 + u_{\text{scenario}(i)} + v_{\text{author}(i)} + w_{\text{judge}(i)} + \cdots.logit(pij)=β0+uscenario(i)+vauthor(i)+wjudge(i)+⋯. A hierarchical interval estimates uncertainty over both row-level Bernoulli variation and higher-level random effects. This can be more efficient and interpretable than bootstrap when the model is well specified, but it adds modeling assumptions that may be hard to defend in a fast-moving evaluation setting.
Cluster bootstrap is attractive because it is closer to the design: resample the units that could plausibly have varied. It is less parametric than a random-effects logistic model, but it can be unstable with very few clusters. A bootstrap over six scenario families is not magic; it cannot create design information that does not exist.
The practical rule is:
Wilson is appropriate when the independent Bernoulli abstraction is credible.
Cluster bootstrap is appropriate when the statistic is simple but observations share design structure.
Hierarchical modeling is appropriate when the dependence structure itself is part of the scientific claim.
Exact intervals are appropriate when the binomial model is credible and coverage guarantees matter more than width.
Difference of proportions: use Newcombe, not Wilson
Wilson is for one proportion. It does not directly solve the problem of estimating a difference:
Δ=p1−p2.\Delta = p_1-p_2.Δ=p1−p2. A common but flawed move is to compute two separate Wilson intervals and subtract endpoints naively. That is not the standard method.
For two independent proportions, Newcombe’s 1998 “hybrid score” interval is the usual Wilson-derived method. Let p^1=x1/n1\hat p_1=x_1/n_1p^1=x1/n1, p^2=x2/n2\hat p_2=x_2/n_2p^2=x2/n2, and let Wilson intervals for the two proportions be
[L1,U1],[L2,U2].[L_1,U_1],\qquad [L_2,U_2].[L1,U1],[L2,U2]. Then Newcombe’s interval for Δ=p^1−p^2\Delta=\hat p_1-\hat p_2Δ=p^1−p^2 is
[Δ^−(p^1−L1)2+(U2−p^2)2,Δ^+(U1−p^1)2+(p^2−L2)2].\left[ \hat\Delta - \sqrt{(\hat p_1-L_1)^2+(U_2-\hat p_2)^2}, \quad \hat\Delta + \sqrt{(U_1-\hat p_1)^2+(\hat p_2-L_2)^2} \right].[Δ^−(p^1−L1)2+(U2−p^2)2,Δ^+(U1−p^1)2+(p^2−L2)2]. Newcombe’s independent-proportions comparison found that a method combining Wilson score intervals for the two proportions performed well and was readily implementable regardless of sample size. PubMed
For paired proportions, use paired methods. Newcombe’s paired-data paper explicitly says that independent-proportion score methods can be adapted by incorporating a correction for non-independence, and it discusses methods that remain within [−1,+1][-1,+1][−1,+1]. EITI
This matters for LLM eval. If two systems are evaluated on the same prompts, the outcomes are paired. If two prompts are judged in AB/BA order, the outcomes are paired. If two safety policies are tested on the same scenario set, the outcomes are paired. A difference-of-proportions CI should respect that structure.
Reporting recommendations for AI evaluation
A minimal Wilson-based report for one independent proportion should include:
| Field | Example |
|---|---|
| Numerator | x=17x=17x=17 wins |
| Denominator | n=29n=29n=29 decisive comparisons |
| Point estimate | p^=0.586\hat p=0.586p^=0.586 |
| Interval | 95% Wilson CI [0.405,0.744][0.405,0.744][0.405,0.744] |
| Tie handling | Ties excluded / ties counted separately / ties modeled as third outcome |
| Independence claim | One comparison per scenario, no repeated author or judge cluster |
| Method note | Wilson score interval, two-sided |
For LLM-eval dashboards, a good convention is:
| Eval object | Default display |
|---|---|
| Binary metric on independent examples | Wilson CI |
| Small benchmark slice | Wilson CI plus raw x/nx/nx/n |
| Pairwise win rate with repeated scenarios/authors | Cluster bootstrap headline; Wilson per slice |
| AB/BA pairwise judge result | Controlled win rate with paired/cluster bootstrap |
| Red-flag rate with repeated users/documents | Cluster bootstrap or hierarchical model |
| Difference between two models on same items | Paired bootstrap or paired-proportion method |
| Difference between two independent proportions | Newcombe hybrid score |
The display should not hide raw counts. A proportion without nnn is almost useless; a confidence interval without a method is ambiguous; a bootstrap interval without a cluster unit is under-specified.
Limitations of Wilson
Wilson’s limitations are sharp:
| Limitation | Consequence |
|---|---|
| Single-proportion only | Do not use it directly for differences, ratios, odds ratios, or multi-class outcomes |
| Independent Bernoulli model | Repeated scenarios, authors, users, or judges make naive Wilson too narrow |
| Approximate coverage | It does not guarantee exact minimum coverage like Clopper-Pearson |
| Binary outcome required | Ties, ordinal ratings, severity scores, and continuous judge scores need other methods |
| Fixed denominator assumed | Adaptive sampling or post-hoc slice selection complicates interpretation |
| No correction for multiple looks | Many benchmark slices create selection and multiplicity problems |
| No protection against biased labels | A precise CI around biased judge outputs is still biased |
The last point is crucial. Wilson can make uncertainty honest after the binary labels are accepted. It cannot make the labels valid. In LLM-as-judge settings, position bias, verbosity bias, self-preference, rubric drift, and prompt sensitivity can dominate sampling error. Zheng et al. identify position, verbosity, and self-enhancement biases in LLM-as-judge evaluation, and Shi et al. systematically study position bias across judges, tasks, and model outputs. arXiv
Open question: Wilson default or bootstrap default for LLM eval?
The narrow statistical answer is easy: if the estimand is one independent binomial proportion, Wilson should usually replace Wald as the default. The broader LLM-evaluation answer is contested because many real evals are not independent binomial experiments.
The case for Wilson-first reporting is strong:
| Wilson-first advantage | Why it matters |
|---|---|
| Closed form | Easy to implement consistently |
| Fast | Works in dashboards, notebooks, CI pipelines |
| Boundary-aware | Handles low red-flag rates and high pass rates |
| Small-nnn honest | Makes thin slices visibly uncertain |
| Explainable | Easier to audit than complex resampling |
| Literature-backed | Supported by Brown-Cai-DasGupta, Agresti-Coull, Newcombe |
The case for bootstrap-first reporting is also strong:
| Bootstrap-first advantage | Why it matters |
|---|---|
| Handles clustered designs | Most LLM eval datasets reuse scenarios, users, authors, or judges |
| Handles complex statistics | AB/BA controlled rates, macro-averages, weighted rates, filtered subsets |
| Avoids false row-level nnn | Prevents overconfident intervals from repeated correlated records |
| Matches evaluation design | Resamples the unit that could plausibly vary |
| Works for paired comparisons | Better for same-prompt A/B model comparisons |
The most defensible policy is not “Wilson everywhere” or “bootstrap everywhere.” It is:
Use Wilson as the default interval for leaf-level independent binary proportions; use cluster bootstrap or hierarchical CIs for headline LLM-eval claims whenever the design has repeated scenarios, authors, documents, users, judges, or AB/BA pairs.
This is a conservative engineering compromise. It preserves Wilson’s advantages where its assumptions hold and prevents it from being abused as a row-count uncertainty wrapper around dependent eval data.
Methodological source map
| Source | Role in this entry |
|---|---|
| Wilson, “Probable Inference, the Law of Succession, and Statistical Inference” (1927) | Original score-style interval derivation and boundary critique jhanley.biostat.mcgill.ca |
| Clopper & Pearson, “The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial” (1934) | Exact binomial interval lineage OUP Academic |
| Agresti & Coull, “Approximate is Better than ‘Exact’ for Interval Estimation of Binomial Proportions” (1998) | Adjusted Wald interval, score interval advocacy, critique of exact conservatism Department of Mathematics & Statistics |
| Newcombe, “Two-sided confidence intervals for the single proportion” (1998) | Comparative methodological evidence for score/tail-area methods PubMed |
| Newcombe, “Interval estimation for the difference between independent proportions” (1998) | Hybrid score interval for p1−p2p_1-p_2p1−p2 PubMed |
| Brown, Cai & DasGupta, “Interval Estimation for a Binomial Proportion” (2001) | Major review showing Wald instability and recommending Wilson/Jeffreys/Agresti-Coull alternatives Buffalo ACSU |
| PsycheEval v0.2 public report | Example of Wilson for a binomial subset and cluster bootstrap for grouped LLM-eval win rates Ashita Orbis |
Companion entries
Core theory: Binomial Distribution, Bernoulli Trials, Confidence Intervals, Coverage Probability, Score Tests, Wald Tests, Likelihood-Ratio Intervals, Exact Tests
Binomial interval methods: Wald Interval, Wilson Score Interval, Clopper-Pearson Interval, Agresti-Coull Interval, Jeffreys Interval for Binomial Proportions, Mid-P Interval, Newcombe Hybrid Score Interval
LLM evaluation practice: LLM Evaluation, LLM-as-a-Judge, Pairwise Win Rate Evaluation, AB/BA Counterbalanced Judging, Red-Flag Rate Evaluation, Small-N Benchmark Slices, PsycheEval
Dependence and resampling: Cluster Bootstrap for Evaluation, Hierarchical Evaluation Models, Paired Proportion Tests, Scenario-Level Resampling, Judge Bias in LLM Evaluation
Counterarguments and open problems: Coverage vs Interval Width, Exactness vs Practicality, Bootstrap Default in LLM Evaluation, Multiplicity in Benchmark Slices, Evaluation Uncertainty