Random variables and the cumulative distribution function: one function that tells you everything about a distribution

Let Ω be a non-empty set (the sample space of a random experiment, see the part What probability is) and 2^Ω the set of all its subsets. A system 𝓕 ⊆ 2^Ω is a σ-algebra if and only if

1. Ω ∈ 𝓕;
2. if A ∈ 𝓕, then A^c = Ω ∖ A ∈ 𝓕;
3. if A₁, A₂, … ∈ 𝓕, then ⋃_i=1^∞ A_i ∈ 𝓕.

Let 𝓕 ⊆ 2^Ω be a σ-algebra; its elements are called events. A function P : 𝓕 → ℝ is a probability if and only if it satisfies the Kolmogorov axioms:

1. P(A) ≥ 0 for every A ∈ 𝓕;
2. P(Ω) = 1;
3. for every sequence A₁, A₂, … ∈ 𝓕 with A_i ∩ A_j = ∅ for all i ≠ j, we have P(⋃_i=1^∞ A_i) = ∑_i=1^∞ P(A_i).

A probability space is a triple

(Ω, 𝓕, P),

where Ω is a non-empty set, 𝓕 ⊆ 2^Ω is a σ-algebra and P : 𝓕 → ℝ is a probability. The axioms imply 0 ≤ P(A) ≤ 1 for every A ∈ 𝓕, so P maps into [0, 1].

Let (Ω, 𝓕, P) be a probability space. A random variable is a function X : Ω → ℝ that satisfies

∀x ∈ ℝ :   {ω ∈ Ω : X(ω) ≤ x} ∈ 𝓕.

So a function X is a random variable if and only if for every real x the set of outcomes for which X comes out at most x is an event. Instead of {ω ∈ Ω : X(ω) ≤ x} we write {X ≤ x} for short, and instead of P({X ≤ x}) just P(X ≤ x); likewise {X = x}, {X ∈ B} and so on.

The notation {S ≤ 3} is short for {ω ∈ Ω : S(ω) ≤ 3} — the set of outcomes that the function S sends to a number at most 3 (it is called the preimage of the interval (−∞, 3]).

For two dice every set of outcomes is an event, so the condition in the definition holds automatically.

Let (Ω, 𝓕, P) be a probability space and X : Ω → ℝ a random variable. Denote by 𝓑 the Borel σ-algebra on ℝ — the smallest σ-algebra of subsets of ℝ that contains all intervals (−∞, x], x ∈ ℝ (it then also contains all other intervals, single-point sets and their countable unions). The definition of a random variable implies {ω ∈ Ω : X(ω) ∈ B} ∈ 𝓕 for every B ∈ 𝓑. The distribution of the random variable X is the function P_X : 𝓑 → [0, 1],

P_X(B) = P(X ∈ B) = P({ω ∈ Ω : X(ω) ∈ B})   for B ∈ 𝓑.

P_X is a probability on (ℝ, 𝓑), i.e. (ℝ, 𝓑, P_X) is a probability space. The distribution thus assigns to every set of numbers B the probability that the value of X lands in B.

Let (Ω, 𝓕, P) be a probability space and X a discrete random variable (defined in the section Discrete and continuous random variables), i.e. there is an at most countable set H ⊆ ℝ with P(X ∈ H) = 1. The probability mass function of X is the function p_X : ℝ → [0, 1],

p_X(x) = P(X = x) = P({ω ∈ Ω : X(ω) = x}).

It determines the whole distribution: for every B ∈ 𝓑 we have P(X ∈ B) = ∑_{x ∈ B ∩ H} p_X(x) — we sum over the possible values that lie in B (the events {X = x} are mutually exclusive and there are at most countably many of them, so this follows from additivity).

The sum s = a + b comes from the pairs (a, s − a) with both numbers between 1 and 6. For s ≤ 7 there are s − 1 of them, for s ≥ 7 there are 13 − s; together

The number of sixes N: no six means 5 options on each die, 5 · 5 = 25; exactly one six 2 · 5 = 10 (the six on the first or on the second die, the other one showing one of the 5 other numbers); two sixes 1. Check: 25 + 10 + 1 = 36.

Let (Ω, 𝓕, P) be a probability space and X : Ω → ℝ a random variable. The cumulative distribution function (CDF) of X is the function F_X : ℝ → [0, 1],

F_X(x) = P(X ≤ x) = P({ω ∈ Ω : X(ω) ≤ x})   for x ∈ ℝ.

The probability on the right-hand side exists, because by the definition of a random variable {X ≤ x} ∈ 𝓕. The value F_X(x) is thus the probability that X comes out at most x. Where it is clear which variable we mean, we write just F.

Let X and Y be random variables (they may even be defined on different probability spaces) with distributions P_X, P_Y and distribution functions F_X, F_Y. Then

(∀B ∈ 𝓑 : P_X(B) = P_Y(B))   ⇔   (∀x ∈ ℝ : F_X(x) = F_Y(x)).

So two random variables have the same distribution if and only if they have the same distribution function.

Let X be a random variable and F = F_X : ℝ → [0, 1] its distribution function. Then

1. F is non-decreasing: ∀x, y ∈ ℝ : x ≤ y ⇒ F(x) ≤ F(y);
2. lim_x→−∞ F(x) = 0 and lim_x→+∞ F(x) = 1;
3. F is right-continuous: ∀x ∈ ℝ : lim_y→x⁺ F(y) = F(x).

Conversely: if G : ℝ → ℝ is any function with properties 1–3, then there is a probability space and a random variable X on it such that F_X = G.

Let X be a random variable with distribution function F and a, b ∈ ℝ, a < b. Then

P(a < X ≤ b) = F(b) − F(a),

where {a < X ≤ b} = {ω ∈ Ω : a < X(ω) ≤ b}. The proof is below the breakdown of the formula.

Let X be a random variable with distribution function F and x ∈ ℝ. Denote by F(x⁻) = lim_y→x⁻ F(y) the limit from the left (it exists, because F is non-decreasing and bounded). Then F(x⁻) = P(X < x) and

P(X = x) = F(x) − F(x⁻).

Proof. For y < x we have F(x) − F(y) = P(y < X ≤ x); as y increases towards x, the intervals shrink to the single point {x} and their probabilities decrease to P(X = x) (continuity of probability, which follows from countable additivity). ∎

Let (Ω, 𝓕, P) be a probability space and X : Ω → ℝ a random variable. X is discrete if and only if there is an at most countable (finite or countable) set H = {x₁, x₂, …} ⊆ ℝ of pairwise distinct numbers such that

P(X ∈ H) = ∑_i P(X = x_i) = 1.

So a discrete variable takes, with probability 1, only values from some finite or countable list.

Let X be a random variable with distribution function F_X : ℝ → [0, 1]. X is continuous if and only if F_X is continuous at every point, i.e.

∀x ∈ ℝ :   lim_y→x F_X(y) = F_X(x).

By the theorem on the height of the jump, P(X = x) = F_X(x) − F_X(x⁻) = 0 for every x ∈ ℝ. (Some textbooks call a variable “continuous” only if it has a density — the topic of the next part; such a variable is continuous in this sense as well.)

Let a, b ∈ ℝ, a < b, and let X be a random variable with distribution function F. X has the uniform distribution on the interval [a, b] if and only if for every x ∈ ℝ

F(x) = 0   for x < a,    F(x) = x − a/b − a   for a ≤ x ≤ b,    F(x) = 1   for x > b.

The probability that X lands in a subinterval of [a, b] is thus its length divided by the length of [a, b].

The event {X = 3} lies in every interval (3 − ε, 3], however small ε > 0 is. The probability of a subset cannot be larger:

The only number that is ≥ 0 and at the same time smaller than every positive ε/10 is zero.

Let n ∈ ℕ, n ≥ 1, and let x₁, …, x_n ∈ ℝ be measured values (they may repeat). For a finite set M, |M| denotes the number of its elements. The empirical distribution function of these values is the function F̂_n : ℝ → [0, 1],

F̂_n(x) = |{i ∈ {1, …, n} : x_i ≤ x}|/n   for x ∈ ℝ.

F̂_n(x) is thus the share of measured values that are at most x.

The simulation gave (sorted, rounded to hundredths of a minute) 0.03; 2.81; 4.26; 5.27; 6.13; 6.27; 7.21; 9.68; 9.81; 9.95.

Between 7.21 and 9.68 no measurement fell, so the staircase stays at 0.7 there while the true F keeps rising — which is why the deviation is largest exactly there.

Let (Ω, 𝓕, P) be a probability space, F a distribution function and X₁, X₂, … : Ω → ℝ random variables that

1. all have the distribution function F: F_Xi = F for every i ≥ 1;
2. are independent: for every n ≥ 1 and all x₁, …, x_n ∈ ℝ we have P(X₁ ≤ x₁, …, X_n ≤ x_n) = F(x₁) · … · F(x_n) (compare independent events in the part Conditional probability and Bayes’ theorem).

For n ≥ 1 and ω ∈ Ω let F̂_n be the empirical distribution function of the values X₁(ω), …, X_n(ω) and

D_n = sup_{x ∈ ℝ} |F̂_n(x) − F(x)|.

Then P(lim_n→∞ D_n = 0) = 1, i.e. with probability 1, D_n → 0 as n → ∞.

How fast is told by the Dvoretzky–Kiefer–Wolfowitz inequality (1956) with the constant 2, proved by P. Massart (1990): under the same assumptions, for every n ≥ 1 and every ε > 0

P(D_n > ε) ≤ 2 exp(−2nε²).

Let F : ℝ → [0, 1] be a distribution function (of some random variable). The quantile function (the generalised inverse of F) is the function F⁻¹ : (0, 1) → ℝ,

F⁻¹(u) = min {x ∈ ℝ : F(x) ≥ u}   for u ∈ (0, 1).

The minimum exists: the set {x ∈ ℝ : F(x) ≥ u} is non-empty (because F(x) → 1 > u as x → +∞), bounded below (because F(x) → 0 < u as x → −∞), and thanks to the monotonicity and right-continuity of F it is an interval of the form [c, ∞), which contains its infimum c.

For u ∈ (0, 1), the value F⁻¹(u) is the u-quantile of the distribution with distribution function F; F⁻¹(0.5) is the median. A quantile is thus the smallest number for which the probability of “at most that much” is at least u.

Let F : ℝ → [0, 1] be a distribution function, F⁻¹ : (0, 1) → ℝ its quantile function and U : Ω → (0, 1) a random variable with the uniform distribution on [0, 1] (the values 0 and 1 have probability 0, so we may leave them out). Then X = F⁻¹ ∘ U, i.e. X(ω) = F⁻¹(U(ω)), is a random variable and

F_X(x) = F(x)   for every x ∈ ℝ.

Proof. The definition of the minimum and the monotonicity of F give F⁻¹(u) ≤ x ⇔ u ≤ F(x). Hence P(X ≤ x) = P(U ≤ F(x)) = F(x), because for a uniform U, P(U ≤ t) = t. ∎