Conditional probability and Bayes’ theorem: how to update your estimate when you learn something new

Conditional probability, the product rule, independence, the law of total probability and Bayes’ theorem — and why a positive test for a rare disease means only 16.7%.

13 minutes Matěj Žalmánek 2026-10-15

Series AI basics Part 13 of 52

Full series syllabus 52 of 83 topics published

What you will take away

What conditional probability is and why P(A | B) is not the same as P(B | A).
What independence means and how the product rule and the law of total probability work.
How Bayes’ theorem turns the condition around — and why with rare events you must not forget the base rate.

Useful to know first: What probability is: from rolling dice to the Kolmogorov axioms

Probability is not a fixed number carved in stone. When you learn something new, it changes. The probability that it is raining is different once you know the sky is overcast. The probability that an e-mail is spam is different once it says “you have won”. This part is about how to recalculate a probability precisely when new information arrives — and about the theorem that spam filters, medical diagnostics and a large part of machine learning are built on.

It follows on from the part What probability is and once again uses two dice.

1. Conditional probability

Definition — conditional probability

Let (Ω, 𝓕, P) be a probability space (see the part What probability is) and A, B ∈ 𝓕 events with P(B) > 0. The conditional probability of the event A given B is the number

P(A | B) = P(A ∩ B)/P(B).

For P(B) = 0 this definition does not determine P(A | B). In words: P(A | B) is the share of the probability of the event B that falls on outcomes lying in A as well.

In plain words

A condition shrinks the world. Once you know that B has happened, you can cross out everything else and work only inside B: what share of B also belongs to A?

P(A | B) = P(A ∩ B)/P(B)

P(A | B): the probability of A after we know that B has happened — what we want to compute
P(A ∩ B): only the part of A that lies inside B — outcomes of A outside B can no longer happen, so we cross them out
P(B): the new whole — dividing by it rescales the remaining world so that it again has probability 1 (P(B | B) = 1); that is why P(B) must be > 0

The share of the event A in the narrowed world B.

A friend rolls two dice and only tells you that at least one six came up. What is the probability that the sum is 7? Of the 36 outcomes, 11 contain a six — and only two of them, (1, 6) and (6, 1), add up to 7:

The condition “at least one six” narrows 36 outcomes down to 11 outcomes with a six: 11 of them with sum 7: 2

P(sum 7 | at least one six) = 2/11 ≈ 0.182 — a little more than without the condition, where it is 1/6 ≈ 0.167. And watch the order: P(at least one six | sum 7) = 2/6 = 1/3. P(A | B) and P(B | A) are in general completely different numbers.

Example — computing straight from the definition

If all outcomes are equally likely (classical probability), the number 36 cancels in the fraction and a ratio of counts remains: P(A | B) = |A ∩ B|/|B|, where |·| is the number of outcomes.

A = sum 7, B = at least one six, A ∩ B = {(1, 6), (6, 1)}

P(B) = 11/36, P(A ∩ B) = 2/36, P(A) = 6/36

P(A | B) = 2/36/11/36 = 2/11 ≈ 0.182, P(B | A) = 2/36/6/36 = 1/3

(colours as in the formula above; in P(B | A) the event A plays the role of the condition, so the denominator is P(A))

The numerator is the same in both cases; only what we divide by differs — that is, which world we have selected with the condition.

Theorem — conditional probability is a probability

Let (Ω, 𝓕, P) be a probability space and B ∈ 𝓕 an event with P(B) > 0. Then the function

P_B : 𝓕 → ℝ, P_B(A) = P(A | B),

is a probability on (Ω, 𝓕), i.e. it satisfies all three Kolmogorov axioms.

Proof. Non-negativity: P(A | B) is the ratio of a non-negative and a positive number. Normalization: P(Ω | B) = P(Ω ∩ B)/P(B) = P(B)/P(B) = 1. Additivity: if A₁, A₂, … ∈ 𝓕 are pairwise mutually exclusive, so are the A_i ∩ B, and since (⋃_i A_i) ∩ B = ⋃_i (A_i ∩ B), we have P((⋃_i A_i) ∩ B) = ∑_i P(A_i ∩ B); dividing by P(B) gives P(⋃_i A_i | B) = ∑_i P(A_i | B). ∎

In plain words

Inside the condition, all the rules you already know apply — for example P(A^c | B) = 1 − P(A | B): a sum other than 7, given that a six came up, has probability 1 − 2/11 = 9/11.

2. The product rule

Theorem — product rule

Let (Ω, 𝓕, P) be a probability space and A, B ∈ 𝓕 events. Then

if P(B) > 0, then P(A ∩ B) = P(A | B) · P(B);
if P(A) > 0, then P(A ∩ B) = P(B | A) · P(A).

Proof. We multiply the equality from the definition P(A | B) = P(A ∩ B)/P(B) by the number P(B); the second item in the same way with the roles of A and B swapped. ∎

In plain words

For both to happen, B has to happen first, and then, knowing about B, A has to happen as well. The chances multiply. “Sum 7 and also a six”: 11/36 · 2/11 = 2/36 — exactly the two squares in the picture.

The product rule is not a new assumption — it is the definition of conditional probability multiplied by the denominator. Repeat it and you get a rule for any number of events:

Theorem — chain rule

Let (Ω, 𝓕, P) be a probability space, n ≥ 2 and A₁, …, A_n ∈ 𝓕 events with P(A₁ ∩ … ∩ A_n−1) > 0. (Then P(A₁ ∩ … ∩ A_i) is also positive for every i < n, because these are supersets, so all the conditional probabilities below are defined.) Then

P(A₁ ∩ … ∩ A_n) = P(A₁) · P(A₂ | A₁) · P(A₃ | A₁ ∩ A₂) · … · P(A_n | A₁ ∩ … ∩ A_n−1).

Proof. By induction on n: P(A₁ ∩ … ∩ A_n) = P(A_n | A₁ ∩ … ∩ A_n−1) · P(A₁ ∩ … ∩ A_n−1) is the product rule for A = A_n and B = A₁ ∩ … ∩ A_n−1; we break down the second factor in the same way. ∎

P(A₁ ∩ … ∩ A_n) = ∏_i=1ⁿ P(A_i | A₁ ∩ … ∩ A_i−1)

P(A₁ ∩ … ∩ A_n): the probability that everything happens at once — it often cannot be computed directly
∏: a product — we go through the events one after another and the chances multiply, because each further step has to come out in addition to the previous ones
P(A_i | …): the chance of the i-th step — the only thing we need to be able to estimate at each step
A₁ ∩ … ∩ A_i−1: everything that has already happened — the condition carries the history, so the rule holds for dependent events too (for i = 1 the condition is empty)

It breaks a complicated probability down into a chain of simple steps — this is how a language model puts together the probability of a whole sentence from words.

Example — three aces in a row

From a shuffled deck of 52 cards (4 aces) we draw three cards without replacement. A_i = “the i-th card is an ace”.

P(A₁) = 4/52, P(A₂ | A₁) = 3/51, P(A₃ | A₁ ∩ A₂) = 2/50

P(three aces) = 4/52 · 3/51 · 2/50 = 24/132,600 = 1/5,525 ≈ 0.000181 (colours as in the formula above)

Each condition captures what has changed: after the first ace there are 3 aces left in 51 cards.

3. Independence

Definition — independent events

Let (Ω, 𝓕, P) be a probability space and A, B ∈ 𝓕 events. The events A and B are independent if and only if

P(A ∩ B) = P(A) · P(B).

If P(B) > 0, A and B are independent if and only if P(A | B) = P(A) (dividing both sides by P(B)) — the information that the event B has happened does not change the probability of the event A.

In plain words

Independent events tell each other nothing: knowing that B has happened tells you nothing more about A. The rolls of the first and second die are independent — two sixes have probability 1/6 · 1/6 = 1/36.

Independence can be surprising. “The sum is 7” and “the first die shows a six” are independent: P of both is 1/36, and 1/6 · 1/6 = 1/36 — whatever the first die shows, there is exactly one number on the second that makes the sum 7. On the other hand, “the sum is 12” and “the first die is a six” are not independent: without the condition a sum of 12 has probability only 1/36, but with a six on the first die it is already 1/6.

And do not confuse independent events with mutually exclusive ones. Mutually exclusive events are in fact strongly dependent — when one happens, you know for certain the other did not.

Definition — independence of several events

Let (Ω, 𝓕, P) be a probability space, n ≥ 2 and A₁, …, A_n ∈ 𝓕 events. The events A₁, …, A_n are (mutually) independent if and only if for every set of indices I ⊆ {1, …, n} with |I| ≥ 2

P(⋂_{i ∈ I} A_i) = ∏_{i ∈ I} P(A_i).

So the product rule must hold for every pair, every triple, …, and for all n events at once.

In plain words

Checking pairs is not enough. A = “the first die is odd”, B = “the second die is odd”, C = “the sum is odd”. Each has probability 1/2, and each pair occurs with probability 9/36 = 1/4 = 1/2 · 1/2 — they are pairwise independent. But all three cannot occur (two odd numbers give an even sum): P(A ∩ B ∩ C) = 0 ≠ 1/8. Whoever knows A and B also knows C.

Definition — conditional independence

Let (Ω, 𝓕, P) be a probability space and A, B, C ∈ 𝓕 events with P(C) > 0. The events A and B are conditionally independent given C if and only if

P(A ∩ B | C) = P(A | C) · P(B | C),

i.e. if and only if A and B are independent with respect to the probability P_C = P( · | C). Conditional independence does not follow from independence, nor the other way round.

In plain words

Independence inside the narrowed world C. Two positive tests of the same person are not independent — when the first came out positive, it is more likely that the person is ill, and so also that the second comes out positive. But once we know whether the person is ill, they may well be independent: the error of one test says nothing about the error of the other. The calculation for repeated tests and the naive Bayes classifier below both rest on this assumption.

4. The law of total probability

Theorem — total probability

Let (Ω, 𝓕, P) be a probability space, k ≥ 1, and let the events H₁, …, H_k ∈ 𝓕 form a partition of the space Ω, i.e.

H_i ∩ H_j = ∅ for all i ≠ j (pairwise mutually exclusive),
H₁ ∪ … ∪ H_k = Ω (together they cover Ω),
P(H_i) > 0 for all i = 1, …, k.

Then for every event A ∈ 𝓕

P(A) = ∑_i=1^k P(A | H_i) · P(H_i).

In plain words

When you do not know which of the possibilities holds, go through all of them: for each one, work out the chance of A and weight it by how common that possibility is. A positive test result comes either from someone who is ill or from someone who is healthy — and the two paths add up.

P(A) = ∑_i=1^k P(A | H_i) · P(H_i)

P(A): the total probability of the event, whichever path it happened by
∑_i=1^k: the sum over all possibilities — we may add because the possibilities exclude each other (additivity), and nothing is missing because together they cover Ω
P(A | H_i): the chance of A if the possibility H_i were the one that holds — this tends to be easy to estimate
P(H_i): the weight of the possibility — a common possibility contributes more; without it we would average as if they were all equally common

A weighted average of the conditional probabilities, with the probabilities of the possibilities as weights.

Proof and example — total probability

Proof. The event A splits into mutually exclusive pieces A = (A ∩ H₁) ∪ … ∪ (A ∩ H_k). By additivity P(A) = ∑ P(A ∩ H_i), and by the product rule each term is P(A | H_i) · P(H_i). ∎

Condition 3 is needed only so that the P(A | H_i) are defined. If some H_i had P(H_i) = 0, then P(A ∩ H_i) = 0 and we simply leave its term out of the sum.

A partition by what the first die shows: H_i = “the first die shows i”, i = 1, …, 6.

P(sum 7) = ∑_i=1⁶ P(second = 7 − i | H_i) · 1/6 = 6 · (1/6 · 1/6) = 1/6

P(sum 12) = 0 · 1/6 + … + 0 · 1/6 + 1/6 · 1/6 = 1/36 (only the path H₆ contributes; colours as in the formula above)

5. Bayes’ theorem

Theorem — Bayes’ theorem

Let (Ω, 𝓕, P) be a probability space and A, H ∈ 𝓕 events with P(A) > 0 and P(H) > 0. Then

P(H | A) = P(A | H) · P(H)/P(A),

where the denominator P(A) is computed with the law of total probability.

In plain words

Bayes’ theorem turns the condition around. We often know how a cause shows itself — how often someone who is ill tests positive. But we want the opposite: given a positive test, how likely is the disease? The theorem follows directly from the product rule; it just writes it out both ways.

Proof — Bayes’ theorem

The product rule written twice: P(H | A) · P(A) = P(H ∩ A) = P(A | H) · P(H). We divide by P(A) > 0. ∎

If moreover H₁, …, H_k ∈ 𝓕 form a partition of Ω with P(H_i) > 0 for all i (as in the law of total probability), we substitute the total probability into the denominator and for every j = 1, …, k we get

P(H_j | A) = P(A | H_j) · P(H_j)/∑_i=1^k P(A | H_i) · P(H_i).

The parts of the theorem have their own names:

Definition — prior probability

Let (Ω, 𝓕, P) be a probability space, H ∈ 𝓕 an event that we regard as a hypothesis and A ∈ 𝓕 an event that we regard as the observed data. The prior probability of the hypothesis H is the number P(H) — the probability of the hypothesis regardless of the data A, that is, before we see them.

In plain words

What you believe beforehand. The disease affects 1% of people, so without a test the chance that you have it is 1%.

Definition — likelihood

Let H, A ∈ 𝓕 be a hypothesis and data as above, and P(H) > 0. The likelihood of the hypothesis H given the data A is the number P(A | H) — the probability of the data A assuming that the hypothesis H holds.

The likelihood is understood as a function of the hypothesis for fixed data, H ↦ P(A | H); as such it is not a probability distribution over the hypotheses — for a partition H₁, …, H_k the sum ∑_i P(A | H_i) in general does not equal 1.

In plain words

How well the hypothesis explains what you see. Someone who is ill tests positive in 99% of cases.

Definition — posterior probability

Let H, A ∈ 𝓕 be a hypothesis and data as above, and P(A) > 0. The posterior probability of the hypothesis H is the number P(H | A) — the probability of the hypothesis after we have seen the data A.

In plain words

What you believe afterwards. Bayes’ theorem tells you how to turn “beforehand” and “how well it fits” into “afterwards”.

P(H | A) = P(A | H) · P(H)/P(A)

P(H | A): the posterior probability — how much we believe the hypothesis after looking at the data; what we want
P(A | H): the likelihood — a hypothesis that predicts the data well gains; this is the condition “in the opposite direction” that we usually know
P(H): the prior probability — a rare hypothesis needs stronger evidence; leaving this term out is what causes base rate neglect
P(A): the total probability of the data (the evidence) — the same for all hypotheses; it divides only so that the posterior probabilities of all hypotheses add up to 1

The estimate afterwards = how well the hypothesis explains the data × the estimate beforehand, normalized.

6. A positive test: why only 16.7%

A disease affects 1% of people. A test detects it in 99% of people who have it, but in 5% of healthy people it raises a false alarm. Your test is positive. How likely is it that you have the disease? Most people — including many doctors — guess around 95%. Bayes’ theorem says:

P(positive) = 0.99 · 0.01 + 0.05 · 0.99 = 0.0594

P(disease | positive) = 0.99 · 0.01/0.0594 ≈ 0.167

(colours as in the breakdown of Bayes’ theorem above: likelihood, prior probability, total probability of the data, posterior probability)

Only 16.7%. It is easiest to see with natural frequencies — instead of probabilities, imagine 10,000 people:

10,000 people, one test

There are 100 ill people and the test catches 99 of them. But there are 9,900 healthy people, and even just 5% of them makes 495 false alarms. That is 594 positives in total, and only 99 of them are ill — one in six. This error in intuition has a name: base rate neglect. When something is rare, even an accurate test produces more false alarms than real findings.

Definition — sensitivity, specificity and predictive value

Let (Ω, 𝓕, P) be a probability space (a randomly chosen tested person), D ∈ 𝓕 the event “the person is ill”, + ∈ 𝓕 the event “the test came out positive” and − = +^c the event “the test came out negative”; let 0 < P(D) < 1 and P(+) > 0. We define:

sensitivity = P(+ | D) — the share of ill people that the test catches (here 0.99);
specificity = P(− | D^c) — the share of healthy people that the test correctly lets through (here 0.95); the false positive rate is 1 − specificity = 0.05;
positive predictive value = P(D | +) — what interests a patient with a positive result.

In plain words

Sensitivity and specificity describe the test, and the manufacturer measures them on people whose illness status they know. The predictive value also depends on how common the disease is — that is why the manufacturer cannot print it on the box. From the tree: 99 truly ill out of 99 + 495 = 594 positives, 99/594 = 1/6 ≈ 0.167 — the same number Bayes’ theorem gave.

7. Learning from data: the posterior becomes the prior

What if the test is repeated and comes out positive again? Now we no longer start at 1% — the result of the first test is the new starting estimate. The posterior probability becomes the prior:

Probability of the disease after consecutive positive tests

After the second positive test it is already 79.8%, after the third 98.7%. Each piece of evidence shifts the estimate, and the next one builds on the previous. This is exactly how Bayesian models learn — they start with some assumption and refine it with every bit of data.

The calculation really is just Bayes’ theorem repeated — but with one hidden assumption: the tests are conditionally independent both given “ill” and given “healthy” (the error of one says nothing about the other). Then the likelihood of each further test is still 0.99 or 0.05, respectively. For a real test repeated with the same procedure this need not hold.

def aktualizuj(prior, citlivost, falesne_poz, vysledky):
    p = prior
    for pozitivni in vysledky:
        if pozitivni:
            ano, ne = citlivost, falesne_poz              # P(+ | ill), P(+ | healthy)
        else:
            ano, ne = 1 - citlivost, 1 - falesne_poz      # P(− | ill), P(− | healthy)
        p = ano * p / (ano * p + ne * (1 - p))            # Bayes’ theorem
    return p                                              # posterior = the new prior

It is even simpler to compute in odds — the ratio of “for” to “against”:

Theorem — Bayes’ theorem in odds form

Definition. For p ∈ [0, 1), the odds corresponding to the probability p are the number

O = p/1 − p ∈ [0, ∞); back p = O/1 + O.

For an event H with P(H) < 1 we write O(H) = P(H)/P(H^c), and for P(A) > 0, P(H | A) < 1 similarly O(H | A) = P(H | A)/P(H^c | A).

Theorem. Let (Ω, 𝓕, P) be a probability space and A, H ∈ 𝓕 events for which 0 < P(H) < 1, P(A) > 0 and P(A | H^c) > 0. Then

P(H | A)/P(H^c | A) = P(A | H)/P(A | H^c) · P(H)/P(H^c), i.e. O(H | A) = P(A | H)/P(A | H^c) · O(H).

Proof. Bayes’ theorem for H and for H^c (both have positive probability) gives P(H | A) = P(A | H) P(H)/P(A) and P(H^c | A) = P(A | H^c) P(H^c)/P(A) > 0. We divide the first equality by the second and the denominator P(A) cancels. ∎

O(H | A) = P(A | H)/P(A | H^c) · O(H)

O(H | A): the odds of the hypothesis after the observation — the probability is then computed from them as O/1 + O
P(A | H)/P(A | H^c): the likelihood ratio — the strength of the evidence: how many times more probable the observation is when the hypothesis holds than when it does not; it does not depend on the prior probability, so it is a property of the test
O(H): the odds beforehand — this is where the base rate comes in; the denominator P(A) dropped out of the ratio, so we do not need to compute it

Each further (conditionally independent) piece of evidence just multiplies the odds by its likelihood ratio.

Example — three positive tests in odds

likelihood ratio of a positive test: 0.99/0.05 = 19.8; odds beforehand: 0.01/0.99 = 1 : 99

1st test: 1/99 · 19.8 = 0.2 → p = 0.2/1.2 ≈ 0.167

2nd test: 0.2 · 19.8 = 3.96 → p = 3.96/4.96 ≈ 0.798

3rd test: 3.96 · 19.8 ≈ 78.41 → p = 78.41/79.41 ≈ 0.987

(colours as in the formula above; the odds after one test are the odds beforehand for the next test)

The same numbers as in the chart (16.7%, 79.8%, 98.7%), just with one multiplication per test.

8. Bayes in artificial intelligence

Spam filters — if 40% of mail is spam, and the word “prize” appears in 30% of spam but in only 1% of other e-mails, then an e-mail containing “prize” is spam with probability 0.12/0.126 ≈ 0.952. A naive Bayes classifier does this for every word.
Language models compute the conditional probability of the next word given all the previous ones: P(x_t | x₁, …, x_t−1). A whole text is a product of such conditionals — the product rule.
Bayesian networks record which quantities depend on each other as a directed graph and compute with conditional probabilities along its edges.
Model uncertainty — Bayesian methods do not give just one answer but a distribution: how sure the model is, and where more data would help it.

The naive Bayes classifier

Definition — naive Bayes classifier

Let K be a finite non-empty set of classes (e.g. {spam, other}), n ≥ 1, and let an observation be an n-tuple of features (w₁, …, w_n) (e.g. the words of an e-mail). Suppose that for each class c ∈ K we know its prior probability P(c) > 0 and for each feature the conditional probability P(w_i | c). The naive Bayes classifier

assumes that the features are conditionally independent given each class: P(w₁, …, w_n | c) = ∏_i=1ⁿ P(w_i | c) for all c ∈ K;
assigns the observation to the class ĉ = arg max_{c ∈ K} P(c) · ∏_i=1ⁿ P(w_i | c).

If P(w₁, …, w_n) > 0, Bayes’ theorem together with item 1 gives

P(c | w₁, …, w_n) = P(c) · ∏_i=1ⁿ P(w_i | c)/P(w₁, …, w_n) ∝ P(c) · ∏_i=1ⁿ P(w_i | c),

where ∝ means “equals up to a positive constant independent of c” (the denominator). So the classifier chooses the class with the largest posterior probability.

P(c | w₁, …, w_n) ∝ P(c) · ∏_i P(w_i | c)

P(c | w₁, …): how probable the class is when we see these words — this decides where the e-mail goes
P(c): the prior probability of the class — how much of all mail is spam; without it the filter would misjudge how common each class is
∏_i P(w_i | c): the likelihood broken down into words — the “naive” assumption of conditional independence replaces one unobtainable probability of the whole combination of words with a product of numbers that can be counted from training e-mails word by word

Bayes’ theorem + the chain rule + conditional independence; we drop the denominator because it has no effect on the order of the classes.

A product of hundreds of small numbers would be rounded to zero in a computer, so we compute with a sum of logarithms instead — the logarithm is increasing, so it does not change the order of the classes:

def klasifikuj(slova, prior, verohodnost):
    # prior[c] = P(c), verohodnost[c][w] = P(w | c), estimated from the training data
    skore = {}
    for c in prior:
        skore[c] = log(prior[c])
        for w in slova:
            skore[c] += log(verohodnost[c][w])   # product → sum of logarithms
    return max(skore, key=skore.get)              # the class with the largest score

Example — an e-mail with the words “prize” and “free”

Spam is 40% of mail, “prize” appears in 30% of spam and 1% of other e-mails; “free” (illustrative numbers) in 20% of spam and 2% of other e-mails.

spam: 0.4 · 0.3 · 0.2 = 0.024, other: 0.6 · 0.01 · 0.02 = 0.00012 (colours as in the formula above)

P(spam | prize, free) = 0.024/0.024 + 0.00012 ≈ 0.995

in logarithms: ln 0.4 + ln 0.3 + ln 0.2 ≈ −3.730, ln 0.6 + ln 0.01 + ln 0.02 ≈ −9.028

The second word raised the estimate from 0.952 to 0.995. The score difference 5.298 is the logarithm of the posterior odds: e^5.298 ≈ 200 = 0.024/0.00012.

9. Summary

Conditional probability P(A | B) works only inside B; P(A | B) ≠ P(B | A).
Product rule: P(A ∩ B) = P(A | B) · P(B). Independent events tell each other nothing.
The law of total probability adds up all the paths by which an event can happen.
Bayes’ theorem turns the condition around: from the prior probability and the likelihood it produces the posterior.
For rare events, even an accurate test gives many false alarms — do not forget the base rate.

Do you see it differently? Write to support@coden-app.com.

Conditional probability and Bayes’ theorem: how to update your estimate when you learn something new

1. Conditional probability

2. The product rule

3. Independence

4. The law of total probability

5. Bayes’ theorem

6. A positive test: why only 16.7%

7. Learning from data: the posterior becomes the prior

8. Bayes in artificial intelligence

The naive Bayes classifier

9. Summary

An Agent on your computer, driven from your phone