Probability density and probability mass functions: why a continuous variable has zero probability at a point

The probability mass function, the probability density and the area under the curve — why density is not probability, why it can be greater than 1 and how a histogram estimates it from data.

13 minutes Matěj Žalmánek 2026-10-15

Series AI basics Part 15 of 52

1. Map of artificial intelligence: what is what and how it all fits together
2. A brief history of artificial intelligence: from Turing to language models
3. Data, model, training, inference: the basic terms on a single example
4. How to read mathematical notation: symbols, indices, sums and quantifiers
5. Sets and relations: the building blocks of definitions in AI
6. Functions and mappings: why every AI model is a function
7. Logic and proofs: the rules AI reasons by
8. Graphs and trees: the structure that both search and neural networks stand on
9. Linear algebra for machine learning: vectors, matrices and why AI mostly multiplies
10. Derivatives, the gradient and the chain rule: how a model finds out which way to learn
11. Gradient descent: how a model learns
12. What probability is: from rolling dice to the Kolmogorov axioms
13. Conditional probability and Bayes’ theorem: how to update your estimate when you learn something new
14. Random variables and the cumulative distribution function: one function that tells you everything about a distribution
15. Probability density and probability mass functions: why a continuous variable has zero probability at a point
16. Expected value, variance and covariance: where a distribution lies, how wide it is and what goes with what
17. Overview of probability distributions: Bernoulli, binomial, Poisson, uniform, normal and exponential
18. Law of large numbers and central limit theorem: why averages behave so predictably
19. Statistics and parameter estimation: maximum likelihood, MAP and confidence intervals
20. Information theory: entropy, cross-entropy and KL divergence — where loss functions come from
21. State-space search: BFS, DFS, UCS and A* with animations
22. Games and adversaries: minimax, alpha-beta pruning and Monte Carlo tree search
23. Constraint satisfaction: backtracking, constraint propagation and AC-3 on sudoku and a map of Czech regions
24. Logic and knowledge systems: forward and backward chaining, resolution and expert systems
25. Planning: STRIPS, the blocks world, heuristics from relaxation and planning in AI agents
26. Local search and simulated annealing: a travelling salesman through the Czech regional capitals
27. Genetic algorithms: population, crossover, mutation and selection on a backpack for a hike
28. Evolution strategies and genetic programming: from the one-fifth rule to Kepler’s law
29. Ant colony optimization: how a pheromone trail finds a short path
30. Particle swarm optimisation: inertia, personal experience and the swarm’s experience
31. Neuroevolution: evolving the weights and shape of neural networks, and the NEAT algorithm
32. Fuzzy sets and membership functions: how to write down “somewhat warm”
33. Fuzzy logic and probability: why “somewhat warm” does not mean “probably warm”
34. Fuzzy logic: t-norms, implications and IF–THEN rules
35. Fuzzy control: Mamdani and Sugeno from a fan to a pole on a cart
36. Fuzzy clustering: the fuzzy c-means algorithm and points between clusters
37. Neuro-fuzzy systems (ANFIS): fuzzy rules that learn from data
38. Markov chains: the transition matrix, the stationary distribution and PageRank
39. Hidden Markov models: the Viterbi algorithm and the dishonest casino
40. Bayesian networks: why the grass is wet and what it means for the sprinkler
41. The Kalman filter: how to estimate where a car is and how fast it goes from imprecise measurements
42. Types of machine learning: supervised, unsupervised, semi-supervised and more
43. Prompts, context and RAG: how to give a model the right information
44. Variational autoencoders (VAE): new images from a continuous latent space
45. Generative adversarial networks (GAN): the forger versus the detective
46. Diffusion models: how pure noise becomes an image
47. Markov decision processes: states, actions, rewards and the Bellman equations
48. Q-learning and DQN: learning the value of actions, from a table to a neural network
49. Policy gradient and PPO: learning the policy directly
50. How an AI agent works: a language model in a loop with tools
51. Programming with AI agents: the task, tests and review
52. Evaluating and securing AI systems: how to measure and how to keep watch

Full series syllabus 52 of 83 topics published

We ended the part on the random variable with a paradox: the probability that you wait for the bus for exactly 3 minutes is zero — and yet you will wait for some exact amount of time. The cumulative distribution function can handle that, but it is not very intuitive. This part shows two functions that are much easier to draw and to read: the probability mass function for discrete variables and the density for continuous ones.

1. Probability mass function

Definition — probability mass function

Let (Ω, 𝓕, P) be a probability space and X : Ω → ℝ a discrete random variable (see the part Random variables and the cumulative distribution function), i.e. there is an at most countable set of pairwise distinct values H = {x₁, x₂, …} ⊆ ℝ with P(X ∈ H) = 1. The probability mass function of X is the function p_X : ℝ → [0, 1],

p_X(x) = P(X = x) = P({ω ∈ Ω : X(ω) = x})   for x ∈ ℝ.

The value p_X(x) is thus the probability that X comes out exactly x; for x ∉ H it is zero.

Theorem — properties of the probability mass function

Let X be a discrete random variable, H = {x₁, x₂, …} ⊆ ℝ an at most countable set of pairwise distinct values with P(X ∈ H) = 1, p : ℝ → [0, 1] the probability mass function and F : ℝ → [0, 1] the distribution function of X. Then

1. p(x) ≥ 0 for every x ∈ ℝ;
2. ∑_i p(x_i) = 1;
3. F(x) = ∑_{i : xi ≤ x} p(x_i) for every x ∈ ℝ.

Items 2 and 3 follow from additivity: the events {X = x_i} are mutually exclusive and their union is {X ∈ H}, or {X ∈ H, X ≤ x} respectively, which differs from {X ≤ x} only by an event of probability 0.

Example — the probability mass function of the sum of two dice in one formula

All 36 pairs (a, b) are equally likely. The sum k comes from the pairs (a, k − a) with 1 ≤ a ≤ 6 and at the same time 1 ≤ k − a ≤ 6, i.e. max(1, k − 6) ≤ a ≤ min(6, k − 1). The number of such a is

min(6, k − 1) − max(1, k − 6) + 1 = 6 − |k − 7|,   k = 2, 3, …, 12,

p(k) = 6 − |k − 7|/36:   p(2) = 6 − 5/36 = 1/36,   p(7) = 6 − 0/36 = 6/36,   p(10) = 6 − 3/36 = 3/36.

Checking the second property: 1 + 2 + 3 + 4 + 5 + 6 + 5 + 4 + 3 + 2 + 1 = 36, so ∑ p(k) = 36/36 = 1.

The bars grow by 1/36 up to seven and then fall in the same way — which is why the graph is a triangle.

2. Why a continuous variable has no probability at a point

Take a continuous variable: the time X in minutes until your next e-mail arrives, when on average one comes every 2 minutes. Waiting like this has an exponential distribution with distribution function F(x) = 1 − e^−0.5x for x ≥ 0 (why exactly this one is shown in the part on distributions). A probability mass function is no help here — it would be zero at every point. So instead of a point, let us take the interval [2, 2 + h] and make it narrower and narrower:

Definition — exponential distribution

Let λ ∈ ℝ, λ > 0, and let X be a random variable with distribution function F : ℝ → [0, 1]. X has the exponential distribution with parameter (rate) λ if and only if for every x ∈ ℝ

F(x) = 1 − e^−λx   for x ≥ 0   and   F(x) = 0   for x < 0.

The number e = lim_n→∞ (1 + 1/n)ⁿ ≈ 2.718 is the base of the natural logarithm. The variable thus takes only non-negative values, and the probability that it exceeds x ≥ 0 is 1 − F(x) = e^−λx.

width h	P(2 < X ≤ 2 + h)	P / h
2	0.2325	0.1163
1	0.1447	0.1447
0.5	0.0814	0.1627
0.1	0.0179	0.1794
0.01	0.0018	0.1835
→ 0	→ 0	→ f(2) ≈ 0.1839

The probability of the interval goes to zero — 0.2325, 0.1447, …, 0.0018 — which is why a single point gets zero. But the probability per unit of length, that is P/h, settles down: 0.1163, 0.1447, 0.1627, 0.1794, 0.1835 … and approaches 0.1839. This number does say something about the point 2: how much probability falls on one minute around the second minute. It is the density.

f(x) = lim_{h → 0⁺} P(x < X ≤ x + h)/h

f(x)

the density at the point x — a number that is meant to say something about a single point, even though its probability is zero

lim_{h → 0⁺}

narrowing the interval — as long as the interval has a non-zero width, it describes a whole stretch, not the point x; in the limit only the point remains

P(x < X ≤ x + h)

the probability of a narrow interval just after the point — it goes to zero itself, so we cannot take it directly

h

the width of the interval — dividing by it turns a probability into a probability per unit of length, and that does not approach zero

Example — the first row of the table and the limit

P(2 < X ≤ 3) = F(3) − F(2) = (1 − e^−1.5) − (1 − e⁻¹) = e⁻¹ − e^−1.5 ≈ 0.3679 − 0.2231 = 0.1447

h = 1,   so P/h = 0.1447/1 = 0.1447

In the limit h → 0⁺ this is the derivative of F at the point 2: (1 − e^−0.5x)′ = 0.5 · e^−0.5x,   f(2) = 0.5 · e⁻¹ ≈ 0.1839

(colours as in the formula above)

Theorem — a continuous variable has probability zero at every point

Let X be a random variable whose distribution function F : ℝ → [0, 1] is continuous at every point of ℝ. Then

∀x ∈ ℝ :   P(X = x) = 0.

Proof. For every h > 0 the event {X = x} is part of the event {x − h < X ≤ x}, so 0 ≤ P(X = x) ≤ F(x) − F(x − h). By the continuity of F, the right-hand side goes to zero as h → 0, and therefore P(X = x) = 0. ∎

3. Area under the curve: the definite integral

Definition — definite integral

Let a, b ∈ ℝ, a < b, and let f : [a, b] → ℝ be a function. For n ∈ ℕ, n ≥ 1, split [a, b] into n pieces [t_i−1, t_i], i = 1, …, n, where

Δx = b − a/n,   t_i = a + i · Δx   (i = 0, 1, …, n),

and in each piece choose any point x_i ∈ [t_i−1, t_i]. The function f is (Riemann) integrable on [a, b] if and only if there is a number I ∈ ℝ such that for every choice of the points x_i (separately for each n)

lim_n→∞ ∑_i=1ⁿ f(x_i) · Δx = I.

The number I is then the definite integral of f from a to b, written ∫_a^b f(x) dx. The integral is thus the limit of the sums of rectangle areas, if it exists and does not depend on where in the pieces we measure the height. (The usual Riemann definition also allows pieces of unequal width whose largest width goes to zero; it gives the same notion.) For x ∈ ℝ and a function integrable on every [a, x], the improper integral is defined as ∫_−∞^x f(t) dt = lim_a→−∞ ∫_a^x f(t) dt, if the limit exists; likewise ∫_a^∞ and ∫_−∞^∞.

∫_a^b f(x) dx = lim_n→∞ ∑_i=1ⁿ f(x_i) · Δx

∫_a^b f(x) dx

the area under the graph between a and b — what we are after; the sign ∫ is an elongated S as in “sum”

lim_n→∞

the more pieces, the narrower the rectangles and the less they stick out above the curve or fall short below it; the exact area is what the sums approach

∑_i=1ⁿ

adding up all the pieces — the area of the whole is the sum of the areas of the bits

f(x_i)

the height of the rectangle — on a narrow piece the function hardly changes its value, so we replace it by a single value inside the piece

Δx

the width of the piece — height times width is the area of the rectangle; in the integral notation it becomes dx

The definition is at the same time a recipe for computing an integral on a computer — the midpoint rule takes the height at the middle of each piece:

def integral(f, a, b, n):
    dx = (b - a) / n
    plocha = 0
    for i in range(n):
        x = a + (i + 0.5) * dx    # midpoint of the i-th piece
        plocha += f(x) * dx       # height times width
    return plocha

Example — the area under the waiting-time density between 1 and 3 with four rectangles

f(x) = 0.5 · e^−0.5x,   a = 1,   b = 3,   n = 4,   Δx = 3 − 1/4 = 0.5

midpoints of the pieces x_i = 1.25; 1.75; 2.25; 2.75   →   heights f(x_i) = 0.2676; 0.2084; 0.1623; 0.1264

(0.2676 + 0.2084 + 0.1623 + 0.1264) · 0.5 = 0.7647 · 0.5 ≈ 0.3824

with 10 pieces 0.3832, with 100 pieces 0.3834 — the exact value (we compute it below) is 0.3834   (colours as in the formula above)

Theorem — the Newton–Leibniz formula

Let a, b ∈ ℝ, a < b, let f : [a, b] → ℝ be a continuous function and F : [a, b] → ℝ a function with F′(x) = f(x) for every x ∈ [a, b] (a one-sided derivative at the end points); such an F is called an antiderivative of f. Then f is integrable on [a, b] and

∫_a^b f(x) dx = F(b) − F(a).

4. Probability density function

Definition — probability density

Let X be a random variable with distribution function F_X : ℝ → [0, 1] and f : ℝ → [0, ∞) a function integrable on every bounded interval. f is the density of X if and only if

∀x ∈ ℝ :   F_X(x) = ∫_−∞^x f(t) dt,

where the right-hand side is the improper integral from the definition above. A variable that has a density is called absolutely continuous; its distribution function is continuous, so it is also continuous in the sense of the part Random variables and the cumulative distribution function. The density is not unique: if we change f at finitely many points, the integrals do not change.

Theorem — probability as area

Let X be a random variable with distribution function F and density f. Then

1. for all a, b ∈ ℝ, a < b, P(a < X ≤ b) = ∫_a^b f(x) dx = F(b) − F(a);
2. ∫_−∞^∞ f(x) dx = 1;
3. if f is continuous at a point x ∈ ℝ, then F is differentiable at x and F′(x) = f(x).

Proof. 1: F(b) − F(a) = ∫_−∞^b f − ∫_−∞^a f = ∫_a^b f, and by the theorem on the probability of an interval this is P(a < X ≤ b). 2: ∫_−∞^∞ f = lim_x→∞ F(x) = 1. 3: F(x + h) − F(x)/h = 1/h ∫_x^x+h f(t) dt is the average of f over an interval of length |h|, and by the continuity of f at x it tends to f(x). ∎

Example — the density of the exponential distribution and the area under it

For x > 0:   f(x) = F′(x) = (1 − e^−λx)′ = 0 − (−λ) · e^−λx = λ · e^−λx;   for x < 0, f(x) = 0.

Total area: ∫₀^∞ λ e^−λx dx = lim_b→∞ (F(b) − F(0)) = lim_b→∞ (1 − e^−λb) − 0 = 1

For λ = 0.5:   f(0) = 0.5,   f(2) ≈ 0.1839,   f(4) ≈ 0.0677 — every 2 minutes the density drops to e⁻¹ ≈ 37%.

The probability that the e-mail arrives between the first and the third minute is the area under the curve between 1 and 3. By the Newton–Leibniz formula we do not have to add it up rectangle by rectangle: F(3) − F(1) = (1 − e^−1.5) − (1 − e^−0.5) = e^−0.5 − e^−1.5 ≈ 0.6065 − 0.2231 ≈ 0.383.

5. Density is not probability

The value of the density at a point is not a probability — it is probability per unit. You only get a probability once you multiply it by the width of an interval: for a narrow interval, P(x < X ≤ x + h) ≈ f(x) · h. For x = 2 and h = 0.1 that gives 0.1839 · 0.1 ≈ 0.0184; the true value is 0.0179 (colours as in the formula below).

P(x < X ≤ x + h) ≈ f(x) · h

P(x < X ≤ x + h)

the true probability — exactly, it is the area under the density above the narrow interval

f(x)

the density — probability per unit of length; on a narrow interval it hardly changes, so we take it as the height of a rectangle

h

the width of the interval — only multiplying by a length turns “per unit” into a probability; that is why f can be large and the product still small

That is why a density can be greater than 1. If you arrive at a stop where a bus comes every half minute, your wait is uniform on the interval [0, 0.5] and the density there is 2 everywhere — the area of the 0.5 × 2 rectangle is still 1. It is like population density: 2,000 people per km² does not tell you how many people live in one house — you get a head count only when you multiply the density by an area.

Definition — uniform distribution

Let a, b ∈ ℝ, a < b, and let X be a random variable. X has the uniform distribution on the interval [a, b] if and only if it has the density f : ℝ → [0, ∞),

f(x) = 1/b − a   for x ∈ [a, b],    f(x) = 0   for x ∉ [a, b].

Integrating gives F(x) = x − a/b − a for x ∈ [a, b], 0 to the left and 1 to the right — the same distribution function as in the definition from the part Random variables and the cumulative distribution function, so the two definitions agree.

6. Estimating density from data: the histogram

Definition — histogram as a density estimate

Let n ∈ ℕ, n ≥ 1, let x₁, …, x_n ∈ ℝ be data, w > 0 the bin width and c ∈ ℝ the origin. The bins are the intervals

B_k = [c + kw, c + (k + 1)w),   k ∈ ℤ;

they do not overlap and cover the whole of ℝ, so every x ∈ ℝ lies in exactly one bin B(x). For a bin B, let n_B = |{i ∈ {1, …, n} : x_i ∈ B}| be the number of data points in it. The histogram (density estimate) is the function f̂ : ℝ → [0, ∞),

f̂(x) = n_B(x)/n · w   for x ∈ ℝ.

On each bin the histogram is thus constant and equal to the share of the data in the bin divided by the bin width.

f̂(x) = n_B/n · w

f̂(x)

the density estimate at the point x — the hat says “computed from data”, not the true f

n_B

the number of measurements in the bin B that x falls into — the more data near x, the higher the density there

n

the number of all measurements — it turns the count into the share n_B/n, an estimate of the probability of the bin; otherwise more data would give taller bars

w

the bin width — it turns the probability of the bin into a probability per unit of length, just like h in the ratio P/h; without it, wider bins would give taller bars

We made the data for the histogram ourselves. A computer can only draw uniformly from the interval [0, 1); we get exponential waits from that with the following transformation:

Theorem — inverse transform method

Let λ ∈ ℝ, λ > 0, and let U : Ω → [0, 1) be a random variable with the uniform distribution on [0, 1] (the value 1 has probability 0, so we may leave it out). Then the random variable

X = −ln(1 − U)/λ

has the exponential distribution with parameter λ.

Proof. Since 1 − U ∈ (0, 1], we have ln(1 − U) ≤ 0 and X ≥ 0, so for x < 0, P(X ≤ x) = 0. For x ≥ 0, −ln(1 − U)/λ ≤ x ⇔ 1 − U ≥ e^−λx ⇔ U ≤ 1 − e^−λx, so P(X ≤ x) = P(U ≤ 1 − e^−λx) = 1 − e^−λx = F(x). ∎

def histogram(n, lam=0.5, w=0.5, konec=10):
    pocty = [0] * int(konec / w)
    for _ in range(n):
        u = nahodne()                    # uniformly from [0, 1)
        x = -log(1 - u) / lam            # inverse transform
        if x < konec:
            pocty[int(x / w)] += 1       # which bin it falls into
    return [c / (n * w) for c in pocty]  # heights: the bars have total area 1

This is how we simulated 100, 1,000 and 10,000 e-mail waits (with a bin of 0.5 minutes) and drew the true density over the histograms:

With 100 waits a bar is off from the truth by as much as 0.098, with 1,000 by 0.017 and with 10,000 by 0.008. The share of waits up to one minute came out as 0.440, 0.397 and 0.390 — the true probability is F(1) ≈ 0.393. As data accumulate, the histogram hugs the density, just as the empirical distribution function hugged the true one.

Example — the first bar of the histogram

Bin [0, 0.5), w = 0.5. Of the 100 waits, 27 fell into it:   f̂ = 27/100 · 0.5 = 0.54

Of the 10,000 waits, 2,172 fell into it:   f̂ = 2,172/10,000 · 0.5 ≈ 0.434   (colours as in the formula above)

The histogram estimates the average density over the bin: F(0.5) − F(0)/0.5 = 1 − e^−0.25/0.5 ≈ 0.442

With 100 waits the first bar is 0.098 higher than it should be — that is the largest deviation from the text above.

7. Discrete and continuous side by side

8. Densities and probability mass functions in artificial intelligence

• Classifiers and language models output a probability mass function. The last layer (softmax) gives every class or every word a number between 0 and 1, and together they add up to 1 — exactly the properties of a probability mass function.
• Likelihood. A model learns so that the training data get the highest possible probability (for discrete data) or density (for continuous data). The negative logarithm of that number is a common loss function.
• Anomaly detection. A model estimates the density of normal traffic; a transaction or measurement in a place where the density is very low is suspicious.
• Generative models learn the density of images, sounds or texts — and a new image is a sample from that learned density.

Likelihood by example

Definition — likelihood and negative log-likelihood

Let Θ be a set of parameters and for every θ ∈ Θ let f_θ : ℝ → [0, ∞) be the density (or probability mass function) of some distribution — the model. Let n ≥ 1 and let x₁, …, x_n ∈ ℝ be data that we regard as values of independent random variables X₁, …, X_n with the same distribution. The likelihood is the function L : Θ → [0, ∞),

L(θ) = ∏_i=1ⁿ f_θ(x_i) = f_θ(x₁) · f_θ(x₂) · … · f_θ(x_n).

For θ with L(θ) > 0, the negative log-likelihood (NLL) is

NLL(θ) = −ln L(θ) = −∑_i=1ⁿ ln f_θ(x_i).

The maximum likelihood estimate is a θ̂ ∈ Θ for which L(θ̂) ≥ L(θ) for all θ ∈ Θ (if one exists). Because −ln is decreasing, this is the same as NLL(θ̂) ≤ NLL(θ) for all θ with L(θ) > 0 — the method picks the parameters that minimise the NLL.

We measured three e-mail waits: 1, 2 and 3 minutes. What rate λ do they have? For the exponential density, ln(λ e^−λx) = ln λ − λx, so

NLL(λ) = −n · ln λ + λ · ∑_i x_i

NLL(λ)

the loss — the smaller it is, the higher the density the model gives the measured data

−n · ln λ

the height of the density at zero is λ — this term pushes λ up, because a higher curve gives the data a higher density

λ · ∑_i x_i

a penalty for long waits — with a large λ the density falls quickly and the longer waits that were measured would be improbable; it pushes λ down

Example — the loss for three waits of 1, 2 and 3 minutes

n = 3,   ∑ x_i = 6

NLL(0.25) = −3 · ln 0.25 + 0.25 · 6 ≈ 4.159 + 1.5 = 5.659

NLL(0.5) = −3 · ln 0.5 + 0.5 · 6 ≈ 2.079 + 3 = 5.079

NLL(1) = −3 · ln 1 + 1 · 6 = 0 + 6 = 6.000   (colours as in the formula above)

minimum: λ̂ = 3/6 = 0.5 — one e-mail every 2 minutes, exactly the mean of the data

9. Summary

• A discrete variable is described by its probability mass function p(x) = P(X = x); the bars add up to 1.
• A continuous variable has probability 0 at every point; what makes sense is probability per unit of length — the density.
• The density f is the derivative of the distribution function, and the probability of an interval is the area under it — an integral.
• Density is not probability and can be greater than 1; the total area under it is always 1.
• A histogram divided by the number of data points and the bin width estimates the density from data.

Do you see it differently? Write to support@coden-app.com.