Law of large numbers and central limit theorem: why averages behave so predictably

The sample mean, the law of large numbers and the central limit theorem with animations — why the normal distribution is everywhere, where the ±3% polling error comes from and why an estimate twice as precise costs four times as much data.

13 minutes Matěj Žalmánek 2026-10-15

A single roll of a die is pure chance. But the average of ten thousand rolls is almost certainly 3.5 — and with a precision you can calculate in advance. This part is about two theorems that turn the chaos of individual trials into predictability: the law of large numbers tells you where the average is heading, and the central limit theorem tells you how it fluctuates around that target. Opinion polls, quality control, A/B tests and the training of neural networks all rest on them.

It builds on the parts Expected value and variance and Overview of distributions.

1. Sample mean

2. Law of large numbers

The proof fits on one line: by Chebyshev's inequality, P(|X̄_n − μ| ≥ ε) ≤ σ²/nε², and that goes to zero as n grows. For a die, a tolerance of 0.1 and 10,000 rolls, the bound is 2.917/10,000 · 0.1² = 2.917/100 ≈ 0.029 (colours as in the formula below).

The bound can also be used the other way round — how many rolls are enough for the average to miss by less than 0.1 with probability at least 95%? It suffices that 2.917/n · 0.01 ≤ 0.05, i.e. n ≥ 2.917/0.0005 ≈ 5833.3 — 5834 rolls will do. But Chebyshev is cautious: it holds for every distribution, and so the number actually needed is usually much smaller. Exactly how much smaller is what the central limit theorem tells you.

After ten rolls the averages of the four runs lie between 3.4 and 4.1; after 10,000 rolls, between 3.496 and 3.502. The funnel μ ± 2σ/√n narrows exactly as the theorem on the variance of the mean predicts — and the lines stray outside it only rarely. Why 2σ in particular, and why “only rarely”, is what the next theorem explains.

The law of large numbers does not say that chance “evens out”. When a coin lands heads five times in a row, tails is not “due” — the coin has no memory. The average approaches 0.5 not because the surplus of heads gets cancelled out, but because the next thousand tosses dilute it.

3. Convergence in distribution

4. Central limit theorem

The distribution of a single die is flat — no bell at all. The average of two dice is a triangle, the average of five already resembles a bell, and the average of thirty is almost impossible to tell apart from a normal distribution. The bars are the exact distribution, computed by enumerating every possibility; the dashed line is the normal approximation:

It works even for a strongly skewed distribution — such as waiting for an e-mail, which follows an exponential distribution. A single wait is very asymmetric (skewness 2), the average of five waits less so, and the average of thirty is almost symmetric (skewness 0.37):

How good is the approximation? Take a concrete question: what is the probability that the average of 30 die rolls is greater than 4? The exact calculation over all possibilities gives 0.0488. The central limit theorem says: 4 is 4 − 3.5/1.708/√30 ≈ 1.60 standard deviations above the mean (colours as in the breakdown of the theorem above), so P ≈ 1 − Φ(1.60) ≈ 0.0544. Without a single sum over 6³⁰ possibilities, and with an error below six thousandths.

5. What it is good for: the error of an estimate

The central limit theorem says that an estimate based on the average lies within 1.96 · σ/√n of the true value in 95% of cases. This gives the most widely used formula in statistics:

• An opinion poll with 1,000 respondents and support of around 50% has an error of 1.96 · √0.25/1,000 ≈ ±3.1 percentage points (colours as in the formula above). That is why polls ask about a thousand people — to halve the error, they would need four thousand.
• Model accuracy measured as 90% on 1,000 test examples is in reality 90% ± 1.9 percentage points. A model scoring 91% on a test set of the same size is therefore not necessarily better.

Where statistics gets σ from when it does not know it, and what exactly that interval means, is the topic of the next part, on statistics and estimation.

6. When the theorems do not hold

• Infinite variance or no expected value. The Cauchy distribution has such heavy tails that it has no expected value at all — and the average of n Cauchy variables has exactly the same distribution as a single one. More data does not help in the slightest. City sizes, fortunes and stock market crashes tend to be similarly wild. (The Cauchy distribution has the density f(x) = 1/π(1 + x²), x ∈ ℝ; the integral ∫_−∞^∞ |x| f(x) dx diverges, because |x|/π(1 + x²) falls only like 1/π|x| — and so E[X] does not exist.)
• Dependent data. A thousand answers from people living on one street, or a thousand frames of the same video, are not a thousand independent observations, and σ/√n badly underestimates the error.
• Little data and a skewed distribution. The theorem is a limit theorem — for the exponential wait, the distribution of the average is still noticeably asymmetric after five observations.

7. Limit theorems in artificial intelligence

• Stochastic gradient descent. A gradient computed from a minibatch of B examples is a sample mean. On average it points the same way as the gradient over the full data (law of large numbers), but it fluctuates with a standard deviation ∝ 1/√B. A batch four times larger means only half the noise — which is why it does not pay to keep growing batches indefinitely.
• Monte Carlo methods. When something cannot be computed exactly — the expected reward in reinforcement learning, an integral in high dimensions — you average a large number of random samples. The law of large numbers guarantees the right result; the central limit theorem tells you how precise it is.
• Model evaluation and A/B tests rest on the error 1.96 · σ/√n: without it, you cannot tell whether a new model is really better or just got lucky.
• Network initialization. The input of a neuron is a sum of many weights times activations — by the central limit theorem, it is approximately normally distributed. This is the basis of the Xavier and He initialization schemes, which choose the variance of the weights so that the signal neither blows up nor dies out as it passes through the network.

A batch of 32 examples has noise σ_∇/√32 ≈ 0.177 σ_∇, a batch of 128 examples σ_∇/√128 ≈ 0.088 σ_∇ — four times the computation for half the noise (colours as in the formula above).

monte_carlo(h, vzorek, n):           # estimate of E[h(X)]
    for i = 1 … n:
        x ← vzorek()                      # independent sample of X
        y[i] ← h(x)
    odhad ← mean(y)                       # law of large numbers: → E[h(X)]
    s ← sample standard deviation(y)      # estimate of σ
    return odhad ± 1.96 · s / √n          # central limit theorem: 95% interval

8. Summary

• The sample mean is a random variable with expected value μ and standard deviation σ/√n.
• Law of large numbers: as n grows, the average almost certainly approaches μ.
• Central limit theorem: the average of many independent variables is approximately normally distributed — no matter how the individual variables are distributed.
• The error of an estimate based on the average is roughly ±1.96 · σ/√n; an estimate twice as precise costs four times as much data.
• Without finite variance or without independence, the theorems do not hold.

Do you see it differently? Write to support@coden-app.com.