What probability is: from rolling dice to the Kolmogorov axioms

Random experiment, sample space, events and operations on them, classical and frequency probability and the axioms everything else rests on — illustrated with two dice.

12 minutes Matěj Žalmánek 2026-10-15

Series AI basics Part 12 of 52

Full series syllabus 52 of 83 topics published

What you will take away

What a random experiment, a sample space and an event are — and why events are sets.
How classical probability is calculated and why relative frequency settles down as the number of trials grows.
The three Kolmogorov axioms and the rules that follow from them: the complementary event and the union.

Useful to know first: Sets and relations: the building blocks of definitions in AI

A language model does not pick the next word, it picks probabilities for all words. A classifier does not say “it is a cat”, but “with probability 0.93 it is a cat”. All of machine learning is built on calculating with uncertainty — and probability is the language in which we talk about uncertainty precisely.

At school, probability is usually defined as “the number of favourable cases divided by the number of all cases”. That is a good start, but it is not enough. In this part we will go from a random experiment through events all the way to the precise definition that everything else rests on. All the examples use two dice. It helps to know the part on sets — events are sets, after all.

1. Random experiment and sample space

Definition — random experiment

A random experiment is a process that satisfies three conditions:

its outcome is not uniquely determined in advance — we cannot calculate it from the conditions of the experiment;
the set of all outcomes that can occur is known in advance;
it can be repeated (at least in our minds) any number of times under the same conditions, and the individual repetitions do not influence each other.

It is a basic notion that mathematics does not break down further: in the theory it is represented by its sample space (next definition), and everything else is defined in terms of that.

In plain words

Rolling a die, tossing a coin, drawing a card, but also “which e-mail arrives next” or “which word a person types”. We do not know what will come up, but we know what can.

Definition — sample space

Let a random experiment be given. Its sample space is a non-empty set Ω whose elements correspond to the possible outcomes of the experiment in such a way that

every possible outcome of the experiment corresponds to some element ω ∈ Ω (the list is complete),
every outcome corresponds to exactly one element of Ω — two different elements cannot occur at the same time (the outcomes are mutually exclusive).

The elements ω ∈ Ω are called elementary outcomes.

In plain words

The sample space is a complete list of everything that can happen. For one die it is Ω = {1, 2, 3, 4, 5, 6}. For two dice that we tell apart (first and second), it is all pairs (first, second), that is Ω = {1, …, 6} × {1, …, 6} = {(a, b) : a, b ∈ {1, …, 6}} — 6 · 6 = 36 outcomes.

2. Events

Definition — event

Let Ω be the sample space of a random experiment. An event is any set A such that A ⊆ Ω.

Let the experiment end with the outcome ω ∈ Ω. We say that the event A occurred if and only if ω ∈ A, and that it did not occur if and only if ω ∉ A.

In plain words

An event is a question that can be answered yes or no after the experiment: “is the sum 7?” As a set, we write it by listing the outcomes for which the answer is yes: {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)}.

Definition — certain and impossible event

Let Ω be a sample space. The certain event is the event Ω (the whole sample space). The impossible event is the event ∅ (the empty set).

For every outcome ω ∈ Ω we have ω ∈ Ω and ω ∉ ∅ — so the certain event occurs with every outcome of the experiment and the impossible event with none.

In plain words

“The sum is between 2 and 12” is a certain event — it cannot turn out otherwise. “The sum is 13” is an impossible event.

The grid shows all 36 outcomes of two dice; the row is the first die, the column the second, and the number in the cell is the sum. Two events are coloured: A = “the sum is 7” = {(a, b) ∈ Ω : a + b = 7} and B = “at least one die shows a six” = {(a, b) ∈ Ω : a = 6 ∨ b = 6}:

36 outcomes of rolling two dice only A (sum 7): 6 only B (at least one six): 11 both A and B: 2

Event A has 6 outcomes, event B has 11. Two outcomes, (1, 6) and (6, 1), belong to both — they are purple in the grid.

3. Operations on events

Since events are sets, they are combined with the same operations as sets. But each operation also has a natural meaning in words. In all the definitions of this section Ω is a sample space, ∨ means “or” (at least one of the statements holds) and ∧ “and at the same time”:

Definition — union of events

Let Ω be a sample space and A, B ⊆ Ω events. The union of the events A and B is the event

A ∪ B = {ω ∈ Ω : ω ∈ A ∨ ω ∈ B}.

So the event A ∪ B occurs if and only if A occurs, or B occurs, or both occur.

In plain words

“The sum is 7, or at least one six came up” — in the grid, all the coloured cells together; there are 15 of them.

Definition — intersection of events

Let Ω be a sample space and A, B ⊆ Ω events. The intersection of the events A and B is the event

A ∩ B = {ω ∈ Ω : ω ∈ A ∧ ω ∈ B}.

So the event A ∩ B occurs if and only if A and B occur at the same time.

In plain words

“The sum is 7 and a six came up at the same time” — that is only possible as (1, 6) or (6, 1), the two purple cells in the grid.

Definition — complementary event

Let Ω be a sample space and A ⊆ Ω an event. The complementary event (complement) of the event A is the event

A^c = Ω ∖ A = {ω ∈ Ω : ω ∉ A}.

So the event A^c occurs if and only if A does not occur.

In plain words

The complement of “at least one six” is “no six” — in the grid, all the cells outside the last row and column, 5 · 5 = 25.

Definition — mutually exclusive events

Let Ω be a sample space and A, B ⊆ Ω events. The events A and B are mutually exclusive (disjoint) if and only if

A ∩ B = ∅,

i.e. there is no outcome ω ∈ Ω with which both would occur. Events A₁, A₂, … ⊆ Ω (finitely or countably many) are pairwise mutually exclusive if and only if A_i ∩ A_j = ∅ for all i ≠ j.

In plain words

“The sum is 7” and “a double came up” (both dice the same) cannot occur together — the sum of two equal numbers is always even.

4. Classical probability

Definition — classical probability (Laplace)

Let Ω be a finite non-empty set of elementary outcomes that are equally likely (none has a reason to occur more often than another). For a finite set M let |M| denote the number of its elements. The classical probability of an event A ⊆ Ω is the number

P(A) = |A|/|Ω|.

In plain words

When every outcome has the same chance, you just count how many outcomes are “in favour” and divide by the number of all of them. The sum 7 has 6 favourable outcomes out of 36, so P(A) = 6/36 = 1/6 ≈ 0.167 (the colours match the terms of the formula below).

P(A) = |A|/|Ω|

P(A): the probability of the event — a number between 0 and 1 measuring how large a part of the possibilities the event takes up
|A|: the number of favourable outcomes (elements of the set A) — we count outcomes, not “kinds” of outcomes; that is why with dice we need to tell the first from the second
|Ω|: the number of all outcomes — dividing turns a count into a proportion, so the certain event gets 1 and the result does not depend on how large the space is

It holds only when all outcomes in Ω are equally likely — otherwise each would wrongly be counted with the same weight.

The catch is in the words “equally likely”. With two dice, the sums are not equally likely — the sum 2 comes up only as (1, 1), the sum 7 in six ways. That is why we count pairs, not sums. And for plenty of real events we have no equally likely outcomes at all: what is the probability that it rains tomorrow, or that an e-mail is spam? There the classical definition does not help.

Example — the probabilities of all sums of two dice

The sum s comes up with those pairs (a, b) for which b = s − a and both numbers are from 1 to 6, that is max(1, s − 6) ≤ a ≤ min(6, s − 1). There are 6 − |s − 7| such a — the distribution is symmetric about 7, so the sums s and 14 − s have equally many outcomes:

sum `s`	2, 12	3, 11	4, 10	5, 9	6, 8	7
\|`A`_s\|	1	2	3	4	5	6
`P`(`A`_s)	0.028	0.056	0.083	0.111	0.139	0.167

Check: 1 + 2 + 3 + 4 + 5 + 6 + 5 + 4 + 3 + 2 + 1 = 36 — every outcome has exactly one sum, so the events “sum s” split Ω into mutually exclusive parts. If we took the sums as equally likely, each would get 1/11 ≈ 0.091 — the sum 2 would then come up 36/11 ≈ 3.3 times more often than it really does, and the sum 7 almost half as often (0.091 instead of 0.167).

5. Probability as long-run frequency

Definition — relative frequency

Let Ω be the sample space of a random experiment, A ⊆ Ω an event and n ∈ ℕ, n ≥ 1. Let us repeat the experiment n times under the same conditions, and let ω₁, …, ω_n ∈ Ω be the outcomes of the individual repetitions. Let

n_A = |{i ∈ {1, …, n} : ω_i ∈ A}|

denote the number of repetitions in which the event A occurred (clearly 0 ≤ n_A ≤ n). The relative frequency of the event A in these n repetitions is the number

f_n(A) = n_A/n ∈ [0, 1].

In plain words

Relative frequency is what actually came out: you tossed a coin ten times and got heads seven times — the relative frequency is 7/10 = 0.7.

With few trials the relative frequency jumps back and forth. But as the trials add up, it settles down. Here are four independent simulations of coin tosses:

The relative frequency of heads settles around 0.5 as the number of tosses grows

The first simulation had 70% heads after 10 tosses, 40% after 100 tosses, 49.4% after 1000 tosses and 49.92% after 10,000 tosses. Probability can be understood as the number the relative frequency approaches — exactly what the law of large numbers will later formulate. This is how probabilities are estimated from real data: how many e-mails with a given word were spam, how often the word “good” was followed by “morning”.

The simulation in the figure is a simple algorithm — repeat the experiment and keep computing the relative frequency:

def relativni_cetnost(pokus, jev, n):
    n_A = 0
    for _ in range(n):
        vysledek = pokus()          # e.g. heads or tails at random
        if jev(vysledek):           # did the event A occur?
            n_A += 1
    return n_A / n                  # estimate of P(A)

Example — how much fluctuation is normal

Without proof (we derive it in the parts Expected value, variance and covariance and Law of large numbers and central limit theorem): after n independent trials, the relative frequency of an event with probability p typically differs from p by roughly

σ_n = √p(1 − p)/n

σ_n: the standard deviation of the relative frequency — the “typical” distance from p; in about two thirds of repetitions the deviation is smaller
p(1 − p): the variance of one trial — largest for p = ½ (a coin), zero for a certain or impossible event, where there is nothing to fluctuate
n: the number of trials — the random deviations of the individual trials partly cancel out in the average, which is why the fluctuation decreases; under a square root, so a ten times more precise estimate costs a hundred times more trials

a coin, p = 1/2: σ_n = √0.5 · 0.5/n = 0.5/√n

n = 10: 0.158, n = 100: 0.05, n = 1000: 0.016, n = 10,000: 0.005

the first simulation differed from 0.5 by 0.2; 0.1; 0.006; 0.0008 (the same n)

The deviations of the simulation are of the same order as σ_n; after 100 tosses, at 0.1, it is twice as much — unusual, but in no way exceptional.

6. Kolmogorov axioms

The classical definition needs equally likely outcomes, the frequency one infinitely many trials. In 1933, Andrey Kolmogorov defined probability differently: he does not say how to calculate it, only which rules it must satisfy. Each of the previous views is then just one of the ways to fulfil those rules.

Definition — probability (Kolmogorov axioms)

Let Ω be a sample space and 𝓕 the system of events to which we want to assign a probability (𝓕 is a set of subsets of Ω with the properties of a σ-algebra, see the definition of a probability space below). Probability is a function P : 𝓕 → ℝ that satisfies these three axioms:

non-negativity: for every A ∈ 𝓕, P(A) ≥ 0;
normalization: P(Ω) = 1;
(countable) additivity: for every sequence of events A₁, A₂, … ∈ 𝓕 that are pairwise mutually exclusive (A_i ∩ A_j = ∅ for i ≠ j), P(⋃_i=1^∞ A_i) = ∑_i=1^∞ P(A_i).

For finitely many events A₁, …, A_m, additivity follows from axiom 3 by choosing A_m+1 = A_m+2 = … = ∅ (this requires P(∅) = 0, derived below).

In plain words

Three sensible rules. Probability is never negative. Something has to happen, so the whole space has probability 1. And when two things exclude each other, the chance that one or the other happens is the sum of their chances. Everything else follows from that.

Definition — probability space

Let Ω be a non-empty set. A system 𝓕 of subsets of Ω (i.e. 𝓕 ⊆ 2^Ω, where 2^Ω is the set of all subsets of Ω) is a σ-algebra if and only if

Ω ∈ 𝓕;
if A ∈ 𝓕, then A^c = Ω ∖ A ∈ 𝓕;
if A₁, A₂, … ∈ 𝓕, then ⋃_i=1^∞ A_i ∈ 𝓕.

A probability space is a triple (Ω, 𝓕, P), where Ω is a sample space, 𝓕 ⊆ 2^Ω is a σ-algebra (its elements are events) and P : 𝓕 → ℝ is a probability, i.e. it satisfies the Kolmogorov axioms. From the axioms and the theorems below it follows that P takes only values in [0, 1].

If Ω is finite, one usually takes 𝓕 = 2^Ω (every subset is an event) — for two dice there are 2³⁶ ≈ 6.9 · 10¹⁰ events. For uncountable Ω (say, “a random number from the interval [0, 1]”) this is not possible: not all subsets of the interval can be assigned a probability consistent with the axioms, and 𝓕 must therefore be a smaller σ-algebra.

Definition — probability distribution on a finite space

Let Ω be a finite non-empty set. A probability distribution on Ω is a function p : Ω → ℝ that satisfies

p(ω) ≥ 0 for every ω ∈ Ω,
∑_{ω ∈ Ω} p(ω) = 1.

The probability of an event A ⊆ Ω is then defined as the sum over the outcomes that belong to it (for A = ∅ the sum is empty and equals 0):

P(A) = ∑_{ω ∈ A} p(ω)

P(A): the probability of an event — a set of outcomes
∑_{ω ∈ A}: the sum only over the outcomes with which the event occurs — an event is the union of its one-element events {ω}, these are mutually exclusive, and additivity therefore allows us to add them
p(ω): the probability of one outcome — it need not be the same for all; this is how the definition gets rid of the requirement of “equally likely” outcomes

P defined in this way satisfies all three axioms: a sum of non-negative numbers is non-negative, the sum over the whole Ω is 1, and the sum over a union of disjoint sets is the sum of the partial sums.

Theorem — classical probability is a special case

Let Ω be a finite non-empty set and p(ω) = 1/|Ω| for all ω ∈ Ω. Then p is a probability distribution, and for every event A ⊆ Ω, P(A) = |A|/|Ω|.

Proof. p(ω) = 1/|Ω| > 0 and ∑_{ω ∈ Ω} 1/|Ω| = |Ω| · 1/|Ω| = 1, so p is a distribution. Further, P(A) = ∑_{ω ∈ A} 1/|Ω| = |A| · 1/|Ω| = |A|/|Ω| — exactly Laplace’s formula. So classical probability satisfies the axioms. ∎

A loaded die on which a six comes up with probability 0.25 and each of the other numbers with 0.15 is also a distribution (5 · 0.15 + 0.25 = 1) — just no longer a classical one: P(even number) = 0.15 + 0.15 + 0.25 = 0.55, not 3/6.

In plain words

A probability distribution is an “outcome → chance” table in which the numbers are non-negative and add up to 1. The chance of an event is the sum of the rows with which it occurs. A language model returns exactly such a table for the next word — it just has tens of thousands of rows.

7. Rules that follow from the axioms

Theorem — probability of the complementary event

Let (Ω, 𝓕, P) be a probability space and A ∈ 𝓕. Then P(A^c) = 1 − P(A). In particular, P(∅) = 0 and 0 ≤ P(A) ≤ 1.

Proof. A and A^c are mutually exclusive and A ∪ A^c = Ω. By additivity and normalization, P(A) + P(A^c) = P(Ω) = 1. For A = Ω we have A^c = ∅, so P(∅) = 1 − 1 = 0. And since P(A^c) ≥ 0 (non-negativity), P(A) = 1 − P(A^c) ≤ 1. ∎

In plain words

It is often easier to calculate the opposite. “At least one six” has the opposite “no six”, which has 25 outcomes out of 36. So P(at least one six) = 1 − 25/36 = 11/36 ≈ 0.306.

Theorem — probability of a union

Let (Ω, 𝓕, P) be a probability space and A, B ∈ 𝓕 any events. Then P(A ∪ B) = P(A) + P(B) − P(A ∩ B).

Proof. We split into mutually exclusive parts: A ∪ B = A ∪ (B ∖ A) and B = (B ∖ A) ∪ (A ∩ B). By additivity, P(A ∪ B) = P(A) + P(B ∖ A) and P(B ∖ A) = P(B) − P(A ∩ B). Substituting gives the formula. ∎

In plain words

When you add up events that overlap, you count the shared part twice — and you have to subtract it once. For a sum of 7 or a six: 6/36 + 11/36 − 2/36 = 15/36 (colours as in the formula below). That is exactly how many coloured cells there are in the grid.

P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

P(A ∪ B): the chance that at least one of the events occurs — in the example 15/36 (a sum of 7 or a six)
P(A): all outcomes in A, including the shared ones — 6/36
P(B): all outcomes in B, again including the shared ones — 11/36; the shared outcomes are now counted a second time
P(A ∩ B): the correction for double counting — we subtract the shared outcomes (1, 6) and (6, 1) once, 2/36. For mutually exclusive events it is zero and plain additivity remains

6/36 + 11/36 − 2/36 = 15/36 ≈ 0.417.

8. Probability in artificial intelligence

Classifiers return class probabilities; softmax from the part on classification ensures they are non-negative and sum to 1 — exactly according to the axioms.
Language models are probability distributions over the next word. When writing text, they sample from it.
Estimating from data is relative frequency: the model learns the probabilities that “came out” in the training data.
Uncertainty — a good model knows when it is not sure. Probabilities 0.51 and 0.99 give the same class, but a completely different level of confidence.

How softmax turns arbitrary scores z₁, …, z_K (they may even be negative) into a probability distribution over K classes:

p_i = e^z_i/∑_k=1^K e^z_k

p_i: the probability of class i — one row of the distribution table
e^z_i: the exponential of the score — it is always positive, which satisfies non-negativity; and it preserves the order, so a higher score gives a higher probability
∑ e^z_k: the sum over all classes — dividing by it “rescales” the numbers so that they add up to 1; this satisfies normalization

Additivity needs no separate treatment: the probability of a group of classes is computed by adding, as with every distribution on a finite space.

scores z = (2, 1, 0.1): e² ≈ 7.389, e¹ ≈ 2.718, e^0.1 ≈ 1.105, sum ≈ 11.213

p ≈ (7.389/11.213, 2.718/11.213, 1.105/11.213) ≈ (0.659, 0.242, 0.099), 0.659 + 0.242 + 0.099 = 1

The next part will show how to update probability when we learn something new — conditional probability and Bayes’ theorem.

9. Summary

A random experiment has a sample space Ω; an event is a subset of Ω.
Events combine like sets: union is “or”, intersection “and at the same time”, the complementary event “not”.
Classical probability is |A|/|Ω|, favourable divided by possible — only for equally likely outcomes.
Relative frequency settles down as the number of trials grows; probability is the number it approaches.
Kolmogorov axioms: non-negativity, P(Ω) = 1 and addition for mutually exclusive events. The complement rule and the union formula follow from them.

Do you see it differently? Write to support@coden-app.com.

What probability is: from rolling dice to the Kolmogorov axioms

1. Random experiment and sample space

2. Events

3. Operations on events

4. Classical probability

5. Probability as long-run frequency

6. Kolmogorov axioms

7. Rules that follow from the axioms

8. Probability in artificial intelligence

9. Summary

An Agent on your computer, driven from your phone