Overview of probability distributions: Bernoulli, binomial, Poisson, uniform, normal and exponential

Let (Ω, 𝓕, P) be a probability space, X : Ω → ℝ a random variable and p ∈ [0, 1]. For x ∈ ℝ we write P(X = x) as shorthand for P({ω ∈ Ω : X(ω) = x}). The variable X has the Bernoulli distribution with parameter p, written X ~ Bernoulli(p), if and only if

P(X = 1) = p   ∧   P(X = 0) = 1 − p.

Since p + (1 − p) = 1, X takes, with probability 1, only the values 0 and 1: the value 1 (“success”) with probability p, the value 0 (“failure”) with probability 1 − p.

For p = 0.5 the result would be 0.25 — a variable with values 0 and 1 cannot do better, because p(1 − p) is a parabola with its vertex at 0.5.

Let (Ω, 𝓕, P) be a probability space, X : Ω → ℝ a random variable, n ∈ ℕ, n ≥ 1, and p ∈ [0, 1]. For k ∈ {0, 1, …, n} denote by

(n choose k) = n!/k! · (n − k)!

the binomial coefficient (the number of k-element subsets of an n-element set; m! = 1 · 2 · … · m and 0! = 1). The variable X has the binomial distribution with parameters n and p, written X ~ Bin(n, p), if and only if

∀k ∈ {0, 1, …, n} :   P(X = k) = (n choose k) · p^k · (1 − p)^n−k.

By the binomial theorem these probabilities add up to (p + 1 − p)ⁿ = 1, so X takes, with probability 1, only the values 0, 1, …, n. Bin(1, p) is the same as Bernoulli(p).

The colours match the terms of the formula above. For a coin p = 1 − p = 0.5, so the second and third terms give the same 1/1024 for every k and everything is decided by the binomial coefficient.

Let (Ω, 𝓕, P) be a probability space, X : Ω → ℝ a random variable and λ ∈ ℝ, λ > 0. Denote ℕ₀ = {0, 1, 2, …}. The variable X has the Poisson distribution with parameter λ, written X ~ Pois(λ), if and only if

∀k ∈ ℕ₀ :   P(X = k) = e^−λ · λ^k/k!.

Since ∑_k=0^∞ λ^k/k! = e^λ, these probabilities add up to 1 and X takes, with probability 1, only values from ℕ₀.

The colours match the terms of the formula above. Each next probability is the previous one times λ/k. As long as k < λ it grows, then it falls — so for a whole-number λ the two most likely values are λ − 1 and λ.

Let λ ∈ ℝ, λ > 0, and k ∈ ℕ₀ be fixed. For every n ∈ ℕ, n > λ, let X_n ~ Bin(n, λ/n) (the condition n > λ guarantees λ/n ∈ [0, 1]). Then

lim_n→∞ P(X_n = k) = e^−λ · λ^k/k!.

Proof. For n ≥ k we split the binomial probability with p = λ/n into four factors:

Let P, Q : ℤ → [0, 1] be two probability distributions on the set of integers ℤ, i.e. ∑_{k ∈ ℤ} P(k) = ∑_{k ∈ ℤ} Q(k) = 1, and for A ⊆ ℤ write P(A) = ∑_{k ∈ A} P(k) and Q(A) = ∑_{k ∈ A} Q(k). The total variation distance (total deviation) between P and Q is the number

d(P, Q) = 1/2 · ∑_{k ∈ ℤ} |P(k) − Q(k)| ∈ [0, 1].

We have d(P, Q) = max_{A ⊆ ℤ} |P(A) − Q(A)|, and the maximum is attained for A = {k ∈ ℤ : P(k) > Q(k)}. The total deviation is thus the largest difference the two distributions can give in the probability of the same event. The half is there because whatever one distribution has in excess at some k, it lacks elsewhere — without it we would count every difference twice.

Let (Ω, 𝓕, P) be a probability space, X : Ω → ℝ a random variable and a, b ∈ ℝ, a < b. The variable X has the uniform distribution on the interval [a, b], written X ~ U(a, b), if and only if it has the density f : ℝ → [0, ∞),

f(x) = 1/b − a for x ∈ [a, b],   f(x) = 0 for x ∉ [a, b].

That is, for every x ∈ ℝ, P(X ≤ x) = ∫_−∞^x f(t) dt; the probability of every subinterval [c, d] ⊆ [a, b] is then d − c/b − a and depends only on its length.

The probability of a stretch is its length times the height 1/b − a: the probability that you wait longer than 7 minutes is 10 − 7/10 = 0.3.

Let (Ω, 𝓕, P) be a probability space, X : Ω → ℝ a random variable, μ ∈ ℝ and σ ∈ ℝ, σ > 0. For y ∈ ℝ we write exp(y) = e^y. The variable X has the normal (Gaussian) distribution with parameters μ and σ², written X ~ N(μ, σ²), if and only if it has the density f : ℝ → (0, ∞),

f(x) = 1/σ√2π · exp(−(x − μ)²/2σ²),   x ∈ ℝ.

The parameter μ is then the expected value and σ² the variance of X (σ is its standard deviation).

The colours match the terms of the formula above.

Two σ further out the density is e⁻² ≈ 7.4 times lower. Density is not probability: roughly 0.057 · 1 cm ≈ 5.7% of men are between 175 and 176 cm tall.

The normal distribution N(0, 1), i.e. with parameters μ = 0 and σ = 1, is called the standard normal distribution. Its density is denoted φ : ℝ → (0, ∞),

φ(z) = 1/√2π · exp(−z²/2),   z ∈ ℝ.

Standardization. Let μ ∈ ℝ, σ > 0 and X ~ N(μ, σ²). Then the random variable

Z = X − μ/σ   satisfies   Z ~ N(0, 1).

Let Z ~ N(0, 1) and let φ be the density of the standard normal distribution. The distribution function of the standard normal distribution is the function Φ : ℝ → (0, 1),

Φ(z) = P(Z ≤ z) = ∫_−∞^z φ(t) dt = ∫_−∞^z 1/√2π · exp(−t²/2) dt,   z ∈ ℝ.

Let μ ∈ ℝ, σ > 0 and X ~ N(μ, σ²). From standardization and the symmetry φ(−t) = φ(t), for every x ∈ ℝ and every k > 0

P(X ≤ x) = Φ(x − μ/σ),   P(|X − μ| < kσ) = Φ(k) − Φ(−k) = 2Φ(k) − 1.

Input: a function f : [a, b] → ℝ (computable at every point), bounds a, b ∈ ℝ, a < b, and an even number n ∈ ℕ, n ≥ 2 (the number of pieces).

Output: a number S_n ∈ ℝ, an approximate value of the integral ∫_a^b f(x) dx.

Steps:

1. Piece width: h = b − a/n.
2. Division points: x_i = a + ih for i = 0, 1, …, n (so x₀ = a, x_n = b).
3. On each pair of neighbouring pieces [x_2j, x_2j+2], j = 0, …, n/2 − 1, we replace f by the parabola through the three points (x_2j, f(x_2j)), (x_2j+1, f(x_2j+1)), (x_2j+2, f(x_2j+2)) and add up the areas under the parabolas; the result is S_n = h/3 · [f(x₀) + 4f(x₁) + 2f(x₂) + 4f(x₃) + … + 4f(x_n−1) + f(x_n)], i.e. inner points with an odd index have weight 4, with an even index weight 2, and the end points weight 1.
4. Return S_n.

Termination: the algorithm always finishes, after evaluating f at n + 1 points. The accuracy is chosen in advance through the number of pieces n; for f with a continuous fourth derivative the error shrinks in proportion to h⁴.

simpson(f, a, b, n):                     # n even
    h ← (b − a) / n
    s ← f(a) + f(b)
    for i = 1 … n − 1:
        s ← s + (4 if i is odd, otherwise 2) · f(a + i·h)
    return s · h / 3

φ is the density of N(0, 1). Even with two pieces the error is only about 1 percentage point, and it keeps shrinking with the fourth power of the piece width h — twice as many pieces make it roughly sixteen times smaller.

Let (Ω, 𝓕, P) be a probability space, X : Ω → ℝ a random variable and λ ∈ ℝ, λ > 0. The variable X has the exponential distribution with parameter (rate) λ, written X ~ Exp(λ), if and only if it has the density f : ℝ → [0, ∞),

f(x) = λ e^−λx for x ≥ 0,   f(x) = 0 for x < 0.

Equivalently, X has the distribution function F(x) = 1 − e^−λx for x ≥ 0 and F(x) = 0 for x < 0.

Let λ > 0, X ~ Exp(λ) and s, t ∈ ℝ, s, t ≥ 0. Then P(X > s) > 0, so the conditional probability below is defined, and

P(X > s + t | X > s) = P(X > t).

Proof. P(X > x) = ∫_x^∞ λe^−λu du = e^−λx. The event “X > s + t” is part of the event “X > s”, so by the definition of conditional probability