Derivatives, the gradient and the chain rule: how a model finds out which way to learn

The derivative as a rate of change, partial derivatives, the gradient and the chain rule — and backpropagation worked out number by number on a single neuron.

14 minutes Matěj Žalmánek 2026-10-15

Series AI basics Part 10 of 52

Full series syllabus 52 of 83 topics published

What you will take away

What a derivative is, how it is computed and why it comes out zero at a minimum.
What the gradient is and why it points in the steepest direction uphill — and −∇f downhill.
How the chain rule computes backpropagation, step by step on a single neuron.

Useful to know first: Functions and mappings: why every AI model is a function, Graphs and trees: the structure that both search and neural networks stand on

A model learns by changing its parameters in small steps. To know which way to change them, it needs the answer to a single question: if I turn this knob a little, how does the error change? The derivative answers that question. For millions of knobs at once, it is the gradient. And the chain rule tells you how to compute it across a whole neural network.

This part is the last piece of mathematics you need to understand how a neural network really learns. It builds on functions and on the computation graph of a neuron from the part on graphs.

1. The derivative: how fast a function changes

Definition — derivative

Let I ⊆ ℝ be an open interval, f : I → ℝ a function and x ∈ I. The derivative of the function f at the point x is the number

f ′(x) = lim_{h → 0} f(x + h) − f(x)/h,

if this limit exists and is finite (a real number); otherwise the derivative at the point x does not exist. The fraction is defined for h ≠ 0 with x + h ∈ I. By the limit we mean: lim_{h → 0} q(h) = L ∈ ℝ if and only if

∀ ε > 0 ∃ δ > 0 ∀ h : 0 < |h| < δ ∧ x + h ∈ I ⇒ |q(h) − L| < ε.

If f has a derivative at every point of I, the derivative of the function f is the function f ′ : I → ℝ, x ↦ f ′(x). Another (Leibniz) notation: f ′(x) = df/dx(x).

The fraction under the limit is the slope of the secant of the graph between the points (x, f(x)) and (x + h, f(x + h)); the derivative is therefore the limiting slope of the secants, the slope of the tangent at the point x.

In plain words

The derivative is a speed. If f describes where a car is at time x, the derivative is the speed on the speedometer at that moment. On a graph it is the steepness: a positive derivative — the function rises, negative — it falls, zero — it is flat.

f ′(x) = lim_{h → 0} f(x + h) − f(x)/h

f ′(x): the derivative — the slope of the tangent at the point x; what we want to compute
lim_{h → 0}: the limit — a secant over a finite stretch measures the average slope; only when we shrink the stretch to zero do we get the slope at a single point
f(x + h) − f(x): how much the value changed when we moved by h
h: the length of the move — dividing by it turns the total change into a change per unit, that is, a slope

The change of the output per unit change of the input, for an infinitely small step.

The secant of the graph of f(x) = x² at the point 3 has slope 8 for h = 2, slope 7 for h = 1, slope 6.5 for 0.5 and slope 6.1 for 0.1. The smaller the step, the closer the secant gets to the tangent — and the derivative f ′(3) = 6:

With smaller h the secants turn toward the tangent

Example — the derivative of x² straight from the definition

f(x + h) − f(x)/h = (x + h)² − x²/h = 2xh + h²/h = 2x + h (for h ≠ 0)

lim_{h → 0} (2x + h) = 2x, at the point 3: f ′(3) = 6

(colours as in the formula above)

The slope of the secant at the point 3 is thus exactly 6 + h — that is why it came out as 8, 7, 6.5 and 6.1 for h = 2, 1, 0.5 and 0.1. For x² the difference from the tangent is exactly h.

2. Rules and derivatives of functions from AI

In practice, derivatives are not computed with a limit but by rules. The most important ones:

Rules of differentiation

Derivatives of elementary functions: for a constant c ∈ ℝ and all x ∈ ℝ, (c)′ = 0 and (e^x)′ = e^x; for n ∈ ℕ, n ≥ 1 and x ∈ ℝ, (xⁿ)′ = nxⁿ⁻¹ (for a real exponent n the same holds for x > 0); for x > 0, (ln x)′ = 1/x.

Let I ⊆ ℝ be an open interval, let f, g : I → ℝ be differentiable at the point x ∈ I and c ∈ ℝ. Then at the point x

(f + g)′ = f ′ + g ′, (cf)′ = cf ′, (fg)′ = f ′g + fg ′,

and if moreover g(x) ≠ 0, then

(f/g)′ = f ′g − fg ′/g².

Definition — differentiable function

Let I ⊆ ℝ be an open interval, f : I → ℝ and x ∈ I. The function f is differentiable at the point x if and only if the limit

lim_{h → 0} f(x + h) − f(x)/h

exists and is finite. Equivalently: both one-sided derivatives f ′₋(x) (the limit over h < 0 only, secants from the left) and f ′₊(x) (the limit over h > 0 only, secants from the right) exist, are finite and f ′₋(x) = f ′₊(x). A function is differentiable on I if and only if it is differentiable at every point of I.

ReLU(x) = max(0, x) is not differentiable at zero: for h < 0, ReLU(h) − ReLU(0)/h = 0/h = 0, for h > 0 it is h/h = 1, so f ′₋(0) = 0 ≠ 1 = f ′₊(0). Learning libraries take 0 there by convention. Since the input comes out as exactly 0 only rarely, it does not hurt learning.

Function	Derivative	Where you meet it
`x`²	2`x`	mean squared error
ln `x`	1/`x`	cross-entropy
`σ`(`x`) = 1/1 + `e`^−x	`σ`(`x`)(1 − `σ`(`x`))	logistic regression, network output
tanh `x`	1 − tanh² `x`	recurrent networks
ReLU(`x`) = max(0, `x`)	0 for `x` < 0, 1 for `x` > 0	hidden layers of networks

Notice that the derivative of the sigmoid can be computed from the value of the sigmoid itself — that saves work, because the network has already computed it during the forward pass. And the derivative of ReLU is the simplest possible: either 0 or 1. That is one of the reasons ReLU is so popular.

3. The derivative shows the way to the minimum

Theorem — necessary condition for an extremum

Let I ⊆ ℝ be an open interval, f : I → ℝ and x* ∈ I. The function f has a local minimum at the point x* if and only if

∃ δ > 0 ∀ x ∈ I : |x − x*| < δ ⇒ f(x) ≥ f(x*),

and a local maximum if and only if the same holds with f(x) ≤ f(x*). Both together are called a local extremum; the extremum is strict if the strict inequality holds for x ≠ x*.

Theorem. Let f : I → ℝ be differentiable at the point x* ∈ I and have a local extremum at x*. Then f ′(x*) = 0.

The converse does not hold: f ′(x*) = 0 can also happen at a point where there is no extremum — for f(x) = x³, f ′(0) = 0, but f(x) < 0 for x < 0 and f(x) > 0 for x > 0 (an inflection point).

In plain words

At the bottom of a valley the ground is flat. The derivative also tells you which way is down: when it is positive, go left; when it is negative, go right. That is exactly what gradient descent does — only in millions of dimensions at once.

For f(x) = x², f ′(x) = 2x, which is zero only at the point 0 — and that is where the minimum is. For more complicated functions we cannot solve the equation f ′ = 0, so the minimum is searched for step by step: a little against the direction of the derivative, compute again, another little step.

Theorem — sufficient condition for a minimum

Let I ⊆ ℝ be an open interval and let f : I → ℝ be differentiable on I. The second derivative of f at a point x ∈ I is the derivative of the function f ′ at that point, f ″(x) = (f ′)′(x), if it exists.

Theorem. Let f : I → ℝ be differentiable on I, x* ∈ I, let f ″(x*) exist and f ′(x*) = 0. Then

if f ″(x*) > 0, f has a strict local minimum at the point x*;
if f ″(x*) < 0, f has a strict local maximum at the point x*;
if f ″(x*) = 0, the theorem does not decide (there may or may not be an extremum).

For x²: f ″(x) = (2x)′ = 2 > 0, so the point 0 really is a minimum. For x³: f ″(0) = 6 · 0 = 0 — the theorem is silent, and indeed there is no extremum at zero.

x_k+1 = x_k − η · f ′(x_k)

x_k+1: the new estimate of the minimum — one step closer to the bottom
x_k: where we are standing now — we start from the last estimate, we don’t start over
−: minus — the derivative shows where the function grows; we want to go down, so the opposite way
η: the step length (learning rate) — the derivative holds only near the point, so we go in small steps; too large an η overshoots the minimum
f ′(x_k): the derivative — it gives the direction (by its sign) and the size of the step: far from the minimum the slope is steep and we move fast, near the bottom the step shortens by itself

One step of gradient descent in one variable.

def gradientni_sestup(df, x, eta, kroky):
    for _ in range(kroky):
        x = x - eta * df(x)       # a little against the direction of the derivative
    return x

f(x) = x², f ′(x) = 2x, start x₀ = 3, η = 0.1

x₁ = 3 − 0.1 · 6 = 2.4, x₂ = 2.4 − 0.1 · 4.8 = 1.92, x₃ = 1.92 − 0.1 · 3.84 = 1.536

(colours as in the formula above)

Each step here multiplies the position by the number 1 − 2η = 0.8, so we approach zero geometrically. With η = 1 it would be −1 and the descent would jump between 3 and −3 forever.

4. Partial derivatives

Definition — partial derivative

Let n ≥ 1, f : ℝⁿ → ℝ, x = (x₁, …, x_n) ∈ ℝⁿ and i ∈ {1, …, n}; e_i ∈ ℝⁿ is the i-th unit vector (1 in the i-th place, 0 elsewhere). The partial derivative of the function f with respect to x_i at the point x is the number

∂f/∂x_i(x) = lim_{h → 0} f(x + he_i) − f(x)/h,

if this limit exists and is finite. Since x + he_i = (x₁, …, x_i + h, …, x_n), it is the ordinary derivative of the function of one variable t ↦ f(x₁, …, t, …, x_n) (t in the i-th place) at the point t = x_i: all the other variables are held constant.

In plain words

With a function of several knobs, you turn one and leave the others as they are. The partial derivative tells you how sensitively the result reacts to that one knob.

f(x, y) = x²y

∂f/∂x = 2xy, ∂f/∂y = x²

at the point (3, 2): ∂f/∂x = 12, ∂f/∂y = 9

Example — the partial derivative as a limit, and a check

By the definition above, ∂f/∂x_i(x) is the limit of the quotient f(x + he_i) − f(x)/h, where e_i is the unit vector in the direction of the i-th axis — we move along one axis only. For small h the quotient by itself gives a good estimate:

a move of h = 0.01 along the x-axis: f(3.01, 2) = 3.01² · 2 = 18.1202, f(3, 2) = 18

18.1202 − 18/0.01 = 12.02 ≈ 12

The remaining 0.02 is the error of the finite step; it vanishes with smaller h. During the computation y = 2 did not move at all — that is all there is to “partial”.

5. The gradient: the direction of steepest ascent

Definition — gradient

Let n ≥ 1, f : ℝⁿ → ℝ, and let x ∈ ℝⁿ be a point at which all partial derivatives ∂f/∂x_i(x), i = 1, …, n, exist. The gradient of the function f at the point x is the vector

∇f(x) = (∂f/∂x₁(x), …, ∂f/∂x_n(x)) ∈ ℝⁿ.

Theorem. Let f : ℝⁿ → ℝ have continuous partial derivatives and let x ∈ ℝⁿ be a point with ∇f(x) ≠ 0. Then for every vector u ∈ ℝⁿ with ‖u‖ = 1 the directional derivative (defined below) satisfies

−‖∇f(x)‖ ≤ D_uf(x) ≤ ‖∇f(x)‖,

the upper bound is attained exactly for u = ∇f(x)/‖∇f(x)‖ — the gradient points in the direction of fastest growth and its length is that greatest rate of growth;
the lower bound is attained exactly for u = −∇f(x)/‖∇f(x)‖ — −∇f(x) is the direction of steepest descent;
D_uf(x) = 0 if and only if u · ∇f(x) = 0 — the gradient is perpendicular to the level set {y ∈ ℝⁿ : f(y) = f(x)} (to its tangent directions) at the point x.

In plain words

You are standing on a slope in fog and feel the incline under your feet. The gradient is an arrow pointing uphill as steeply as possible. If you want to go down, go exactly the opposite way.

For the bowl f(x, y) = x² + 3y², ∇f = (2x, 6y); at the point (1, 1) it is (2, 6). The arrows in the picture show −∇f, that is, which way is down — and they all cross the level curves at right angles toward the bottom:

Level curves of the bowl x² + 3y² and arrows −∇f pointing downhill

The arrows do not point straight at the centre: in the y direction the bowl is steeper, so the gradient is larger there. This very “zigzag” is what later troubles gradient descent in narrow valleys.

Why does the gradient point exactly in the steepest uphill direction? The directional derivative gives the answer.

Definition — directional derivative

Let n ≥ 1, f : ℝⁿ → ℝ, x ∈ ℝⁿ, and let u ∈ ℝⁿ be a unit vector, ‖u‖ = 1 (the dot product and length are introduced in the part Linear algebra for machine learning). The directional derivative of the function f at the point x in the direction u is the number

D_uf(x) = lim_{h → 0} f(x + hu) − f(x)/h,

if this limit exists and is finite — the slope of the terrain when we step out of the point x in the direction u. Partial derivatives are a special case of it: ∂f/∂x_i(x) = D_{e_i}f(x).

Theorem. If f has continuous partial derivatives (is continuously differentiable), then for every x ∈ ℝⁿ and every unit u ∈ ℝⁿ the directional derivative exists and

D_uf(x) = ∇f(x) · u.

If ∇f(x) ≠ 0 and φ ∈ [0, π] is the angle between u and ∇f(x), by the definition of the angle this is the same as ‖∇f(x)‖ cos φ.

D_uf = ∇f · u = ‖∇f‖ cos φ

D_uf: the slope in the chosen direction — how fast the function grows when we go in the direction u
∇f: the gradient — it summarizes the slopes along all axes; the slope in any direction is put together from them by the dot product
u: the direction of the step, of length 1 — so that the result is a slope, not a slope times the step length
‖∇f‖: the length of the gradient — the highest possible slope at this point; it does not depend on where we go
cos φ: the angle between u and the gradient — the only thing the choice of direction affects; it is largest (1) for φ = 0, smallest (−1) for the opposite direction, zero along the level curve

Hence the theorem on the gradient: the steepest ascent is in the direction ∇f, the steepest descent in the direction −∇f, and along a level curve (perpendicular to the gradient) the height does not change.

Example — the slopes of the bowl at the point (1, 1) and one step down

∇f(1, 1) = (2, 6), ‖∇f‖ = √4 + 36 = √40 ≈ 6.32

along the x-axis, u = (1, 0): D_uf = (2, 6) · (1, 0) = 2; along the y-axis, u = (0, 1): D_uf = (2, 6) · (0, 1) = 6 (colours as in the formula above)

a step against the gradient with η = 0.1: (1, 1) − 0.1 · (2, 6) = (0.8, 0.4)

f(1, 1) = 1 + 3 = 4 → f(0.8, 0.4) = 0.64 + 3 · 0.16 = 1.12

No direction gives a slope greater than 6.32 — not even the y-axis, which is “almost” in the direction of the gradient.

6. The chain rule

Theorem — derivative of a composite function

Let I, J ⊆ ℝ be open intervals, f : I → ℝ and g : J → ℝ functions with f(I) ⊆ J, and x ∈ I. Let f be differentiable at the point x and g differentiable at the point f(x). Then the composite function h = g ∘ f : I → ℝ, h(x) = g(f(x)), is differentiable at the point x and

h ′(x) = g ′(f(x)) · f ′(x).

In Leibniz notation, with y = g(u) and u = f(x):

dy/dx = dy/du · du/dx.

In a longer chain all the links are multiplied: if h = f_k ∘ … ∘ f₁, u₀ = x and u_j = f_j(u_j−1) for j = 1, …, k, and each f_j is differentiable at u_j−1, then by induction h ′(x) = f_k′(u_k−1) · … · f₁′(u₀).

In plain words

Gears: if the first gear turns once, the second turns three times, and the third turns twice for every turn of the second, then the third gear turns six times. Sensitivities multiply along the chain.

h(x) = (2x + 1)², outer g(u) = u², inner u = 2x + 1

h ′(x) = 2u · 2 = 4(2x + 1), at the point 1, u = 3: h ′(1) = 6 · 2 = 12

(colours as in the formula below)

h ′(x) = g ′(f(x)) · f ′(x)

h ′(x): the sensitivity of the whole chain — how the final result changes when we move the input x
g ′(f(x)): the sensitivity of the outer function — evaluated at f(x), because g does not receive x but the intermediate result; in the example 2u = 2 · 3 = 6
f ′(x): the sensitivity of the inner function — how much the intermediate result moves per unit of x; in the example 2

A small change of x is enlarged f ′ times by the inner function and another g ′ times by the outer one — that is why the sensitivities multiply: 6 · 2 = 12.

Example — where the derivative of the sigmoid comes from

The sigmoid is a composite function: σ(x) = u⁻¹ with the inner u = 1 + e^−x.

outer: (u⁻¹)′ = −u⁻², inner: (1 + e^−x)′ = −e^−x (again by the chain rule: (e^−x)′ = e^−x · (−1))

σ ′(x) = −(1 + e^−x)⁻² · (−e^−x) = e^−x/(1 + e^−x)²

= 1/1 + e^−x · e^−x/1 + e^−x = σ(x)(1 − σ(x))

(colours as in the chain rule formula above: the outer and the inner derivative)

The last step: 1 − σ(x) = 1 + e^−x − 1/1 + e^−x = e^−x/1 + e^−x. For x = 0.5: σ ≈ 0.622, and so σ ′ ≈ 0.622 · 0.378 ≈ 0.235 — a number we will meet in a moment in the backward pass.

7. Backpropagation in a single neuron

Now let us put it all together. The neuron from the part on graphs computes ŷ = σ(wx + b) and its loss is L = (ŷ − y)². Take the input x = 2, the weight w = 0.5, the bias b = −0.5 and the correct answer y = 1.

The forward pass computes the values from left to right: wx = 1, z = 0.5, ŷ = σ(0.5) ≈ 0.622 and the loss L ≈ 0.143.

The backward pass goes through the same graph from right to left and at each vertex multiplies whatever arrived by the derivative of that operation:

∂L/∂ŷ = 2(ŷ − y) ≈ −0.755

∂L/∂z = ∂L/∂ŷ · σ(z)(1 − σ(z)) ≈ −0.755 · 0.235 ≈ −0.177

∂L/∂w = ∂L/∂z · x ≈ −0.177 · 2 ≈ −0.355, ∂L/∂b = ∂L/∂z · 1 ≈ −0.177

(colours as in the formula below)

∂L/∂w = ∂L/∂ŷ · ∂ŷ/∂z · ∂z/∂w

∂L/∂w: how much the loss grows per unit change of the weight — exactly what gradient descent needs
∂L/∂ŷ = 2(ŷ − y): how the neuron’s answer changes the loss — the larger, the further the answer misses the truth, and the sign says whether it is too large or too small (≈ −0.755)
∂ŷ/∂z = σ(1 − σ): how the sigmoid changes the answer — when the neuron is “saturated” (σ near 0 or 1), this link is almost zero and the signal does not get through (≈ 0.235)
∂z/∂w = x: how the weight changes the inner sum — the weight multiplies the input, so its influence is as large as the input (= 2); with a zero input the weight has nothing to learn from

−0.755 · 0.235 · 2 ≈ −0.355: three links of the computation graph, three derivatives, one product.

Forward pass and backpropagation through a single neuron values gradients ∂L/∂·

Both gradients are negative. That means that if we increase both w and b a little, the loss drops — the neuron then answers closer to one. Gradient descent does exactly this: it moves the parameters a little against the gradient. A whole neural network learns the same way, only instead of one neuron it has millions and the backward pass goes through hundreds of layers. By the way, the result can be checked by brute force — nudge w by a tiny amount, compute the loss again and divide. You get the same thing, just more slowly.

The whole procedure as an algorithm — forward pass, backward pass, step:

def krok_uceni(w, b, x, y, eta):
    # forward pass: values from left to right (and remember them)
    z = w * x + b
    y_hat = sigmoida(z)
    L = (y_hat - y) ** 2
    # backward pass: derivatives from right to left, the chain rule
    dL_dyhat = 2 * (y_hat - y)
    dL_dz = dL_dyhat * y_hat * (1 - y_hat)   # σ′ from the remembered value
    dL_dw = dL_dz * x
    dL_db = dL_dz * 1
    # gradient descent: a little against the gradient
    return w - eta * dL_dw, b - eta * dL_db

Example — one learning step and a gradient check

step with η = 1: w ← 0.5 − 1 · (−0.355) ≈ 0.855, b ← −0.5 − 1 · (−0.177) ≈ −0.323

new: z ≈ 0.855 · 2 − 0.323 ≈ 1.387, ŷ = σ(1.387) ≈ 0.800, L ≈ 0.040 (before 0.143)

check by central difference, h = 0.001: L(w + h) − L(w − h)/2h ≈ −0.355

A single step cut the loss to less than a third, and the answer moved from 0.622 to 0.800. The central difference (a move to both sides) is more accurate than the one-sided one from the definition: its error decreases with h², not with h.

8. Derivatives in artificial intelligence

Automatic differentiation — libraries such as PyTorch and JAX build a computation graph as they compute and work out the gradient with a backward pass on their own. Nobody writes the derivatives of networks by hand.
Numerical differentiation (small h) is far too slow for millions of parameters — one extra computation per parameter. Backpropagation computes all the partial derivatives for the cost of roughly one forward pass.
Vanishing gradient — the derivative of the sigmoid is at most 0.25. The chain rule through five sigmoids in a row multiplies by at most 0.25⁵ ≈ 0.001, so deep layers barely learn. That is why ReLU, with its derivative of 1, took over in hidden layers.
The gradient with respect to the input shows which pixels of an image most influenced the network’s decision — this is how “saliency maps” are drawn and how weaknesses of models are found.

Theorem — the derivative of the sigmoid is at most ¼

Let σ : ℝ → ℝ, σ(x) = 1/1 + e^−x, be the sigmoid. Then for all x ∈ ℝ

0 < σ ′(x) ≤ 1/4,

and the equality σ ′(x) = 1/4 holds exactly for x = 0.

Proof. Let p = σ(x); since e^−x > 0, p ∈ (0, 1). By the example above, σ ′(x) = p(1 − p) > 0 and p(1 − p) = 1/4 − (p − 1/2)² ≤ 1/4, with equality exactly for p = 1/2, that is e^−x = 1, x = 0. ∎

Through n sigmoids in a row the chain rule multiplies n such links, with weights between them. If no weight has an absolute value greater than 1, the result is at most (1/4)ⁿ in absolute value — for n = 5 that is 1/1024 ≈ 0.001.

9. Summary

The derivative is the rate of change — the slope of the tangent that the secants approach.
At a minimum the derivative is zero, and its sign tells you which way is down.
A partial derivative turns one knob; the gradient is the vector of all of them and points most steeply uphill.
The chain rule multiplies the derivatives of the links of a composite function.
Backpropagation is the chain rule on a computation graph — that is how the gradient of a whole neural network is computed.

Do you see it differently? Write to support@coden-app.com.

Derivatives, the gradient and the chain rule: how a model finds out which way to learn

1. The derivative: how fast a function changes

2. Rules and derivatives of functions from AI

3. The derivative shows the way to the minimum

4. Partial derivatives

5. The gradient: the direction of steepest ascent

6. The chain rule

7. Backpropagation in a single neuron

8. Derivatives in artificial intelligence

9. Summary

An Agent on your computer, driven from your phone