Statistics and parameter estimation: maximum likelihood, MAP and confidence intervals

Samples, estimators, unbiasedness and why the sample variance divides by n − 1, maximum likelihood estimation, Bayesian MAP estimation and confidence intervals — and why training a neural network is parameter estimation.

14 minutes Matěj Žalmánek 2026-10-15

Series AI basics Part 19 of 52

Full series syllabus 52 of 83 topics published

What you will take away

What a parameter estimator is, when it is unbiased and consistent, and why the sample variance divides by n − 1.
How maximum likelihood and Bayesian MAP estimation work — and why they are training and regularization.
What a 95% confidence interval really means and the common mistake people make when reading one.

Useful to know first: Law of large numbers and central limit theorem: why averages behave so predictably

Probability theory goes from the model to the data: if you know a coin is fair, you can work out what will come up. Statistics goes the other way — from the data to the model: 6 tails out of 20, so what kind of coin is it? And that is exactly what machine learning does. A model has parameters, the data are what came up, and training is parameter estimation. This part shows what makes a good estimator, where maximum likelihood estimation — the basis of almost every loss function — comes from, and how to say honestly how sure we can be of an estimate.

It builds on the part Law of large numbers and central limit theorem.

1. Sample, statistic, estimator

Definition — random sample

Let (Ω, 𝓕, P) be a probability space, Θ ⊆ ℝ^d the set of possible values of the parameter (the parameter space), and for every θ ∈ Θ let f(·; θ) be the density, or probability mass function, of some distribution on ℝ. Let n ∈ ℕ, n ≥ 1, and θ ∈ Θ. A random sample of size n from the distribution f(·; θ) is an n-tuple of random variables X₁, …, X_n : Ω → ℝ such that

X₁, …, X_n are independent: for all x₁, …, x_n ∈ ℝ, P(X₁ ≤ x₁, …, X_n ≤ x_n) = ∏_i=1ⁿ P(X_i ≤ x_i);
each of the variables X_i, i = 1, …, n, has the distribution with density (probability mass function) f(·; θ).

The value θ is fixed but unknown. The numbers x_i = X_i(ω), i = 1, …, n, measured for a particular outcome ω ∈ Ω, are called the realization of the sample (the measured data).

In plain words

We treat the data we have as if they were drawn at random from some distribution. We choose the type of distribution ourselves (say Bernoulli — spam or not), but we do not know its parameter (what share of the mail is spam), and we want to find it out from the data.

Definition — statistic and estimator

Let X₁, …, X_n be a random sample from the distribution f(·; θ) with an unknown parameter θ ∈ Θ ⊆ ℝ^d. A statistic is a random variable (for k > 1 a random vector)

T = t(X₁, …, X_n),

where t : ℝⁿ → ℝ^k is a function whose formula does not depend on θ (and is measurable, so that T is a random variable). An estimator of the parameter θ is a statistic θ̂ = t(X₁, …, X_n) with values in ℝ^d that we use as an approximate value of θ. The number t(x₁, …, x_n) computed from a realization is called the estimate.

A statistic is thus anything that can be computed from the data alone, without knowing θ.

In plain words

A statistic is any calculation from the data: the mean, the maximum, the median, the share of spam. An estimator is a statistic with a job — to hit the unknown parameter. Because the data are random, the estimate is random too: with different data it would come out differently. That is why we ask how estimators behave on average and as data accumulate.

Definition — sample mean

Let X₁, …, X_n (n ≥ 1) be a random sample. The sample mean is the statistic

X̄ = 1/n ∑_i=1ⁿ X_i.

For a realization x₁, …, x_n we write x̄ = 1/n ∑_i=1ⁿ x_i.

If the distribution has a finite expected value μ = E[X₁] and variance σ² = var X₁, then E[X̄] = μ and var X̄ = σ²/n (the part on limit theorems): the variance of a sum of independent variables is the sum of the variances, i.e. nσ², and multiplying by the constant 1/n makes it n² times smaller.

Example — the mean of five e-mail waits

Measured waits (minutes): 1.2, 3.5, 0.4, 2.1, 2.8. Their sum is 10, so x̄ = 10/5 = 2 minutes. We keep working with these data below.

2. A good estimator: unbiasedness and consistency

Definition — unbiased estimator

Let X₁, …, X_n be a random sample from the distribution f(·; θ), θ ∈ Θ ⊆ ℝ, and let θ̂ be an estimator of θ. The symbols E_θ, var_θ and P_θ denote the expected value, variance and probability computed under the assumption that the true value of the parameter is θ. Suppose E_θ[θ̂] exists for every θ ∈ Θ. The bias of the estimator θ̂ is the function b : Θ → ℝ,

b(θ) = E_θ[θ̂] − θ.

The estimator θ̂ is unbiased if and only if b(θ) = 0 for all θ, i.e.

∀θ ∈ Θ : E_θ[θ̂] = θ.

An unbiased estimator thus aims exactly at θ on average over all possible data, whatever the true θ is. Where no confusion can arise, we drop the index θ from now on.

In plain words

An unbiased estimator misses in both directions in individual trials, but on average it aims exactly right. The sample mean is an unbiased estimator of the expected value — we saw that in the previous part.

Definition — consistent estimator

Let X₁, X₂, … be independent variables with the distribution f(·; θ), θ ∈ Θ ⊆ ℝ, and for every n ≥ 1 let θ̂_n be an estimator of θ from the sample X₁, …, X_n. The sequence of estimators (θ̂_n)_n≥1 is consistent if and only if

∀θ ∈ Θ ∀ε > 0 : lim_n→∞ P_θ(|θ̂_n − θ| ≥ ε) = 0,

i.e. θ̂_n converges to θ in probability. The probability that the estimate differs from the true value by at least ε thus falls to zero as n grows, however small ε is.

In plain words

With enough data the estimate gets as close as you like. The sample mean is consistent — that is precisely the law of large numbers.

Unbiasedness cannot be taken for granted. The obvious way to estimate the variance is the mean squared deviation from the mean, 1/n ∑ (X_i − X̄)². But this estimate systematically underestimates the variance: we measure the deviations from X̄, which has adapted itself to the data, instead of from the true μ.

Theorem — sample variance

Let X₁, …, X_n (n ≥ 2) be a random sample from a distribution with expected value μ and finite variance σ², and let X̄ be its sample mean. Then

E[1/n ∑_i=1ⁿ (X_i − X̄)²] = n − 1/n σ².

Therefore the sample variance

s² = 1/n − 1 ∑_i=1ⁿ (X_i − X̄)²

satisfies E[s²] = σ², i.e. it is an unbiased estimator of σ².

Proof. Write X_i − X̄ = (X_i − μ) − (X̄ − μ). Since ∑ (X_i − μ) = n(X̄ − μ), expanding gives

∑ (X_i − X̄)² = ∑ (X_i − μ)² − 2(X̄ − μ) ∑ (X_i − μ) + n(X̄ − μ)² = ∑ (X_i − μ)² − n(X̄ − μ)².

The expected value of the first sum is nσ², because E[(X_i − μ)²] = var X_i = σ². Since E[X̄] = μ, we have E[(X̄ − μ)²] = var X̄ = σ²/n, so the expected value of the second term is n · σ²/n = σ². Altogether E[∑ (X_i − X̄)²] = (n − 1)σ²; dividing by n gives the first equality, dividing by n − 1 the equality E[s²] = σ². ∎

In plain words

You should divide by n − 1, not by n (Bessel's correction). One “degree of freedom” is used up on estimating the mean. With small samples the difference shows — with five values, dividing by five is off by a fifth — but with a million values it does not matter.

s² = 1/n − 1 · ∑_i=1ⁿ (X_i − X̄)²

1/n − 1: turns the total into an average — but divides by a number one smaller, because deviations from X̄ come out systematically smaller than from μ; this is exactly what offsets that bias
∑_i=1ⁿ: every observation contributes its deviation; variance is a property of the data as a whole, not of a single value
X_i − X̄: how far the observation is from the centre; we do not know μ, so we measure from its estimate X̄
²: the square — so that deviations up and down do not cancel out (their sum is always 0) and large deviations weigh more

The mean squared deviation from the mean, corrected for the fact that we estimated the mean from the same data.

Example — the sample variance of five waits

The mean is 2. Deviations x_i − x̄: −0.8, 1.5, −1.6, 0.1, 0.8 (sum 0). Their squares added up: 0.64 + 2.25 + 2.56 + 0.01 + 0.64 = 6.10 (colours as in the formula above).

s² = 1/4 · 6.10 = 1.525 min², whereas dividing by n gives 6.10/5 = 1.22 min². The standard deviation is s = √1.525 ≈ 1.23 minutes.

Let us check it: 2,000 times we rolled five dice and each time computed both estimates of the variance. The true variance of a die is 35/12 ≈ 2.92. The average of the estimates that divide by n ended up at 2.38 (it aims at 4/5 · 2.92 ≈ 2.33), the one dividing by n − 1 at 2.97:

Average of variance estimates over 2,000 samples of five rolls dividing by n: 2.38 dividing by n − 1: 2.97

The bias of the estimator that divides by n is thus E[θ̂] − θ = (4/5 − 1) · 35/12 = −7/12 ≈ −0.58 pips². But unbiasedness is not everything: an estimator can aim exactly right on average and still fluctuate a lot. One measure combines both.

Theorem — decomposition of the mean squared error

Let θ ∈ ℝ be the true value of the parameter and θ̂ its estimator with finite variance var θ̂. The mean squared error of the estimator θ̂ is the number

MSE(θ̂) = E[(θ̂ − θ)²].

Then

MSE(θ̂) = (E[θ̂] − θ)² + var θ̂,

i.e. the mean squared error is the sum of the squared bias and the variance of the estimator.

Proof. Write m = E[θ̂]. Then θ̂ − θ = (θ̂ − m) + (m − θ), where m − θ is a constant, and (θ̂ − θ)² = (θ̂ − m)² + 2(m − θ)(θ̂ − m) + (m − θ)². By the linearity of expectation the cross term is 2(m − θ) · E[θ̂ − m] = 0, because E[θ̂ − m] = m − m = 0. What remains is E[(θ̂ − m)²] + (m − θ)² = var θ̂ + (E[θ̂] − θ)². ∎

E[(θ̂ − θ)²] = (E[θ̂] − θ)² + var θ̂

E[(θ̂ − θ)²]: how far the estimate is from the truth on average over all possible data — what we ultimately care about
(E[θ̂] − θ)²: the bias squared: a systematic error that no amount of repetition removes — we are aiming off target
var θ̂: the variance of the estimator: how much the estimate changes with the randomness in the data — we aim right, but our hand shakes

Total error = systematic error + fluctuation. Reducing one at the cost of the other can pay off — that is the bias–variance trade-off.

3. Maximum likelihood estimation

Definition — likelihood function

Let Θ ⊆ ℝ^d be a parameter space, for every θ ∈ Θ let f(·; θ) be a density, or probability mass function, and let x₁, …, x_n ∈ ℝ (n ≥ 1) be measured data — a realization of a random sample from the distribution f(·; θ). The likelihood function is the function L : Θ → [0, ∞),

L(θ) = ∏_i=1ⁿ f(x_i; θ).

For θ where L(θ) > 0 we define the log-likelihood

ℓ(θ) = ln L(θ) = ∑_i=1ⁿ ln f(x_i; θ).

The data are fixed and the variable is θ. For a discrete distribution, L(θ) is the probability that the sample takes exactly the values x₁, …, x_n when the parameter equals θ; the product is there thanks to independence.

In plain words

The data are fixed; we vary the parameter and ask: “if the parameter were θ, how probable would exactly these data be?” For independent observations the probabilities multiply. Careful: L(θ) is not the probability of the parameter — it need not sum to 1 over θ.

Definition — maximum likelihood estimate

Let L : Θ → [0, ∞) be the likelihood function of the data x₁, …, x_n. The maximum likelihood estimate (MLE) is a value θ̂ ∈ Θ for which

∀θ ∈ Θ : L(θ̂) ≥ L(θ), i.e. θ̂ = argmax_{θ ∈ Θ} L(θ).

The maximum need not exist and need not be unique; the definition assumes that it exists. Because ln is increasing on (0, ∞), for L(θ̂), L(θ) > 0 we have the equivalence L(θ̂) ≥ L(θ) ⇔ ℓ(θ̂) ≥ ℓ(θ), so

θ̂ = argmax_{θ ∈ Θ} ℓ(θ) = argmax_{θ ∈ Θ} ∑_i=1ⁿ ln f(x_i; θ).

Viewed as a function of the data, θ̂ is an estimator in the sense of the definition above: the parameter under which the measured data are most likely.

In plain words

Pick the parameter that makes the observed data most probable. The logarithm turns the product into a sum — which is easier to work with and keeps tiny numbers from underflowing in the computer.

Out of 20 e-mails, 6 were spam. For the share of spam p, L(p) = p⁶(1 − p)¹⁴. The derivative of its logarithm, 6/p − 14/1 − p, is zero at p̂ = 6/20 = 0.3 — exactly the share of spam in the data (colours as in the formula below). In general, k successes out of n give p̂ = k/n. With 60 spams out of 200 the estimate is the same, but the peak is much narrower: the data tell us more about the parameter.

ℓ(p) = k · ln p + (n − k) · ln(1 − p)

ℓ(p): the log-likelihood, ln L(p); it has its maximum at the same point as L, but it is a sum instead of a product, so it is easy to differentiate
k: the number of spams — each spam contributed a factor p to the product, and the logarithm turns k equal factors into a k-fold multiple
ln p: the “reward” for one spam: it grows with p, so spams pull the estimate up
(n − k): the number of other e-mails — the same reason as for k, just for the other outcome
ln(1 − p): the “reward” for one non-spam: it falls with p, so the other e-mails pull the estimate down; the maximum is where the two pulls balance

The log-likelihood of a Bernoulli sample x₁, …, x_n ∈ {0, 1} with k = ∑ x_i successes out of n and parameter p ∈ (0, 1).

Example — deriving p̂ = k/n

Let 0 < k < n. The derivative on (0, 1): ℓ′(p) = k/p − n − k/1 − p. We set it equal to zero and multiply by p(1 − p) > 0: k(1 − p) = (n − k)p, so k = np and p̂ = k/n.

It is a maximum: ℓ″(p) = −k/p² − n − k/(1 − p)² < 0 on the whole of (0, 1), the function is concave.

For 6 out of 20: ℓ′(0.3) = 6/0.3 − 14/0.7 = 20 − 20 = 0. ✓

Likelihood of the share of spam p (divided by its maximum) 6 of 20 60 of 200

Example — how narrow the peak is

Let us compare p = 0.4 with the best p̂ = 0.3. For 6 out of 20 we subtract ℓ(0.4) − ℓ(0.3) term by term (colours as in the formula for ℓ(p) above): ln L(0.4)/L(0.3) = 6 · ln 4/3 + 14 · ln 6/7 = 1.726 − 2.158 = −0.432, so the likelihood ratio is e^−0.432 ≈ 0.65. The value 0.4 is still well compatible with the data.

For 60 out of 200 both exponents are ten times larger, so the logarithm of the ratio is ten times larger too: −4.32, and the ratio is 0.013. Ten times more data, and the value 0.4 is suddenly about 49 times less likely than before.

The same thing is said by the standard error of the estimate — the standard deviation of the estimator as a random variable. For p̂ = k/n, var p̂ = p(1 − p)/n, so √0.3 · 0.7/20 ≈ 0.102 and √0.3 · 0.7/200 ≈ 0.032: ten times more data narrows the uncertainty √10 ≈ 3.16 times.

We solved the equation ℓ′(p) = 0 by hand. For a neural network with millions of parameters that is impossible, and the maximum is found step by step — by climbing in the direction of the derivative:

mle_gradientem(k, n, p, η, tolerance):
    repeat:
        sklon ← k/p − (n − k)/(1 − p)      # ℓ′(p): the way the log-likelihood grows
        if |sklon| < tolerance:            # the peak: no further climbing possible
            return p
        p ← p + η · sklon                   # a step uphill, proportional to the steepness

For 6 out of 20 starting at p = 0.5 with η = 0.005: the slope is −16, so p = 0.5 + 0.005 · (−16) = 0.42; then the slope is −9.85 and p = 0.371; then 0.340, 0.322… After 16 steps the slope is below 0.001 and p = 0.30001. The steps shorten by themselves, because near the peak the log-likelihood is flat. Training networks is the same procedure, only going “down” the negative log-likelihood (the part Gradient descent).

For waiting for an e-mail with an exponential distribution, maximum likelihood estimation gives λ̂ = 1/X̄. From five measured waits of 1.2, 3.5, 0.4, 2.1 and 2.8 minutes, the mean is 2 minutes and λ̂ = 0.5 per minute.

Example — deriving λ̂ = 1/X̄

The exponential distribution with parameter λ > 0 has the density f(x; λ) = λe^−λx for x ≥ 0. For data x₁, …, x_n ≥ 0 the log-likelihood is

ℓ(λ) = ∑ ln(λe^−λx_i) = n ln λ − λ ∑ x_i.

ℓ′(λ) = n/λ − ∑ x_i = 0 gives λ̂ = n/∑ x_i = 1/x̄; ℓ″(λ) = −n/λ² < 0, so it is a maximum. With our data: λ̂ = 5/10 = 0.5 per minute — on average one e-mail every 2 minutes.

4. Bayesian MAP estimation

Definition — maximum a posteriori estimate

Let Θ ⊆ ℝ^d be a parameter space, L : Θ → [0, ∞) the likelihood function of the data x₁, …, x_n and π : Θ → [0, ∞) the prior density of the parameter, i.e. ∫_Θ π(θ) dθ = 1 — we treat the parameter as a random variable, and π expresses what we assume about it before observing the data. The maximum a posteriori estimate (MAP) is a value θ̂_MAP ∈ Θ for which

∀θ ∈ Θ : L(θ̂_MAP) · π(θ̂_MAP) ≥ L(θ) · π(θ), i.e. θ̂_MAP = argmax_{θ ∈ Θ} L(θ) · π(θ).

By Bayes' theorem, the posterior density of the parameter after observing the data is (if the denominator is finite and positive)

π(θ | x₁, …, x_n) = L(θ) · π(θ)/∫_Θ L(θ′) π(θ′) dθ′;

the denominator does not depend on θ, so MAP is the point at which the posterior density attains its maximum (the mode of the posterior distribution).

In plain words

Bayes' theorem from the part on conditional probability, applied to parameters: posterior ∝ likelihood × prior. MAP picks the most probable parameter after taking into account what we believed beforehand. If we give extremes a small weight up front (a prior density proportional to p(1 − p)), the share of spam comes out as 7/22 ≈ 0.318 — the estimate is pulled slightly towards the middle. As data accumulate, the influence of the prior fades away.

θ̂_MAP = argmax_θ [ ln L(θ) + ln π(θ) ]

ln L(θ): what the data say — the log-likelihood from maximum likelihood estimation; it grows with the number of observations
ln π(θ): what we believed beforehand; it does not depend on the data, which is why the first term drowns it out as data accumulate

The logarithm of the product L(θ) · π(θ): MAP is MLE with an added term for the prior belief.

Example — MAP for the share of spam

The prior density π(p) = 6p(1 − p) on [0, 1] (the Beta(2, 2) distribution; the six makes the integral 1 and does not affect where the maximum is). Then

L(p) · π(p) = 6 · p⁶(1 − p)¹⁴ · p(1 − p) = 6p⁷(1 − p)¹⁵.

The derivative of the logarithm term by term (colours as in the formula above): 6/p − 14/1 − p + 1/p − 1/1 − p = 7/p − 15/1 − p = 0 gives p̂ = 7/22 ≈ 0.318. In general k + 1/n + 2: as if one extra spam and one extra non-spam had been added to the data.

For 60 out of 200 it comes out as 61/202 ≈ 0.302 — only 0.002 away from the MLE 0.3 instead of 0.018.

5. Confidence interval

Definition — confidence interval

Let X₁, …, X_n be a random sample from a distribution with parameter θ ∈ Θ ⊆ ℝ, α ∈ (0, 1), and let T₁, T₂ be statistics with T₁ ≤ T₂. The random interval (T₁, T₂) is a confidence interval for θ with confidence level 1 − α if and only if

∀θ ∈ Θ : P_θ(T₁ < θ < T₂) = 1 − α.

If the equality holds only in the limit, lim_n→∞ P_θ(T₁ < θ < T₂) = 1 − α (the bounds then depend on n), it is an asymptotic (approximate) confidence interval. What is random are the bounds T₁, T₂; the parameter θ is a fixed number.

In plain words

Instead of a single number you give a range, together with a guarantee: the procedure you use to compute the range hits the true parameter in 95% of cases. What is random are the endpoints of the interval, not the parameter.

For the expected value and a large enough n, the central limit theorem gives an approximate (asymptotic) 95% interval X̄ ± 1.96 · s/√n. Forty times we rolled 40 dice and computed an interval from each sample. The vertical line is the true expected value, 3.5:

40 confidence intervals from 40 samples of 40 rollscontains μ: 38/40

The true value is contained in 38 of the 40 intervals — 95%. Over 20,000 repetitions it was 94.1%, slightly below the promised 95%, because we estimate σ from the data and n = 40 is not yet infinity.

X̄ ± 1.96 · s/√n

X̄: the centre of the interval — the best point estimate of μ; the uncertainty spreads symmetrically around it
1.96: how many standard errors to add on each side to hit 95%: for the standard normal distribution P(−1.96 < Z < 1.96) ≈ 0.95 (Φ(1.96) ≈ 0.975, a value from statistical tables); for 99% it would be 2.576
s: the sample standard deviation — how much the individual observations fluctuate; we do not know the true σ, so we plug in an estimate (which is why the coverage came out slightly below 95%)
√n: the average fluctuates √n times less than a single observation (var X̄ = σ²/n); four times more data halves the interval

s/√n is the standard error of the mean; by the central limit theorem X̄ is approximately normal, so a quantile of the normal distribution is all we need.

Example — the first interval in the picture

In the first sample of 40 rolls, the mean came out as x̄ = 3.525 and s ≈ 1.768. The standard error is 1.768/√40 = 1.768/6.325 ≈ 0.280, times 1.96 that is 0.548 (colours as in the formula above).

The interval: (3.525 − 0.548, 3.525 + 0.548) = (2.977, 4.073). It contains 3.5 — this sample hit the target.

The coverage of 94.1% is the result of this experiment:

pokryti(pocet, n):
    trefy ← 0
    repeat pocet times:
        x ← n rolls of a die                    # a new random sample
        polovina ← 1.96 · s(x) / √n
        if x̄ − polovina ≤ 3.5 ≤ x̄ + polovina:
            trefy ← trefy + 1                   # the interval contains the true μ
    return trefy / pocet                        # share of intervals that hit the target

A common mistake: “μ lies in the interval (3.2, 3.9) with probability 95%”. For one particular interval, μ either is inside or it is not. The 95% is a property of the procedure — how many intervals out of many repetitions hit the target. Statements about the probability of a parameter only come with the Bayesian approach.

6. Estimation in artificial intelligence

Training is maximum likelihood estimation. Cross-entropy in classification is the negative log-likelihood divided by the number of examples, and mean squared error is the same thing for data with normal noise. Minimizing the loss = maximizing the likelihood.
Regularization is MAP. The weight penalty λ‖w‖² (L2, weight decay) corresponds to a normal prior on the weights, L1 to a Laplace prior. The prior belief that “weights should be small” holds back overfitting.
Bias and variance of a model. A model is an estimator; complex models have low bias but high variance (overfitting), simple ones the other way round. Regularization trades a little bias for lower variance.
Test set and confidence interval. Accuracy on test data is an estimate of the true accuracy; without a confidence interval you cannot tell whether the difference between two models is real. Batch normalization estimates the mean and variance from each batch, and language models are trained as MLE: they maximize the likelihood of the next word.

Theorem — mean squared error is MLE for normal noise

Let n ≥ 1, let σ > 0 be fixed, y₁, …, y_n ∈ ℝ measured values and ŷ_i(w) ∈ ℝ, i = 1, …, n, the predictions of a model with parameters w (σ does not depend on w). Assume that y_i = ŷ_i(w) + ε_i, where ε₁, …, ε_n are independent and ε_i ~ N(0, σ²) (normal distribution). Then for any two choices of parameters w, w′

L(w) ≥ L(w′) ⇔ ∑_i=1ⁿ (y_i − ŷ_i(w))² ≤ ∑_i=1ⁿ (y_i − ŷ_i(w′))²,

i.e. maximizing the likelihood is the same as minimizing the sum of squared errors.

Proof. Under the parameters w, y_i has the density 1/√2πσ² · exp(−(y_i − ŷ_i)²/2σ²), so

−ℓ(w) = 1/2σ² ∑ (y_i − ŷ_i)² + n/2 ln(2πσ²).

The second term does not depend on w and the factor 1/2σ² is a positive constant, so −ℓ(w) ≤ −ℓ(w′) if and only if the sum of squares for w is at most that for w′. Since ln is increasing, −ℓ(w) ≤ −ℓ(w′) is equivalent to L(w) ≥ L(w′). ∎

Regularization can be written out in the same way. Give each weight w_j, j = 1, …, d, independently the prior distribution N(0, τ²); then −ln π(w) = ‖w‖²/2τ² + constant, and minimizing the negative logarithm of L · π means minimizing

−ln L(w) + λ ‖w‖², λ = 1/2τ²

−ln L(w): the usual loss (cross-entropy, squared error) — it pushes the weights towards the model fitting the training data
λ: the strength of regularization; a small τ (a strong belief that the weights are close to zero) means a large λ — this is how a prior belief becomes a single number that we tune
‖w‖²: the sum of squared weights — a penalty for large weights; it comes from the exponent of the normal density, which is why it is a square (the Laplace distribution would give the absolute value, L1)

L2 regularization (weight decay) is MAP with a normal prior on the weights.

7. Summary

Statistics goes from the data to the model: an estimator is a calculation from the data meant to hit an unknown parameter.
A good estimator is unbiased (right on average) and consistent (more precise with more data); the sample variance divides by n − 1.
Maximum likelihood estimation picks the parameter that makes the data most probable; for a proportion it is k/n.
MAP adds a prior belief; in machine learning, that is regularization.
A 95% confidence interval comes from a procedure that hits the true parameter in 95% of cases.

Do you see it differently? Write to support@coden-app.com.

Statistics and parameter estimation: maximum likelihood, MAP and confidence intervals

1. Sample, statistic, estimator

2. A good estimator: unbiasedness and consistency

3. Maximum likelihood estimation

4. Bayesian MAP estimation

5. Confidence interval

6. Estimation in artificial intelligence

7. Summary

An Agent on your computer, driven from your phone