Neuro-fuzzy systems (ANFIS): fuzzy rules that learn from data

A Sugeno fuzzy system as a five-layer network, hybrid learning with least squares and gradient, learning a function from noisy data, choosing the number of rules and overfitting, the readability of learned rules and the link to RBF networks.

13 minutes Matěj Žalmánek 2026-10-15

Series AI basics Part 37 of 52

Full series syllabus 52 of 83 topics published

What you will take away

How a Sugeno fuzzy system is written as a network and which of its parameters are learned.
Why hybrid learning (least squares plus gradient) outpaces pure gradient descent.
Why adding rules risks overfitting and what is left of the readability of learned rules.

Useful to know first: Fuzzy control: Mamdani and Sugeno from a fan to a pole on a cart, Gradient descent: how a model learns

The fuzzy controllers from the previous parts had one weakness: someone had to come up with the rules and the membership functions. When an experienced operator exists, that is an advantage. When there is none — or when the system is supposed to reproduce measured data — it is a problem. Neural networks have the opposite nature: they learn anything from data, but nobody can read what they have learned. In 1993 Jyh-Shing Roger Jang combined the two in a system called ANFIS (Adaptive Neuro-Fuzzy Inference System): a Sugeno fuzzy system written as a network whose parameters are learned from data. This part closes the module on fuzzy logic.

It builds on the parts Fuzzy control: Mamdani and Sugeno and Gradient descent.

1. A fuzzy system as a network

Take a first-order Sugeno system with one input and four rules “IF x is A_i, THEN y = p_ix + r_i”. Every step of its computation is a layer of a network:

ANFIS with one input and four rules; layers with learned parameters in orange

Definition — ANFIS (Jang, 1993)

Let n ≥ 1 be the number of rules and for every i = 1, …, n let there be given premise parameters c_i ∈ ℝ, σ_i > 0 and consequent parameters p_i, r_i ∈ ℝ. ANFIS with one input and n rules “IF x is A_i, THEN y = p_ix + r_i” is the function y : ℝ → ℝ whose value at an input x ∈ ℝ is computed in five layers (for i = 1, …, n):

Membership: μ_i(x) = exp(−(x − c_i)²/2σ_i²) ∈ (0, 1] — the Gaussian membership function of the term A_i with centre c_i and width σ_i.
Firing strength: w_i = μ_i(x) (with more inputs, the product of the memberships of all conditions of the rule).
Normalization: w̄_i = w_i/∑_j=1ⁿ w_j (the denominator is positive, because every w_j > 0).
Consequent: w̄_i · (p_ix + r_i).
Sum: y(x) = ∑_i=1ⁿ w̄_i (p_ix + r_i).

So it is the output of a first-order Takagi–Sugeno controller with the product as the t-norm, only written as a network; layers 1 and 4 contain the learned parameters.

In plain words

It is exactly the Sugeno controller from the previous part, only drawn as a network. Each rule has an “area of influence” (a Gaussian hill) and a linear prediction inside it. The output is a weighted average of the predictions, according to how much the input lies in each rule’s area. The whole system is differentiable, so it can be trained just like a neural network.

y = ∑_i w_i/∑_j w_j · (p_ix + r_i)

y: the output of the whole system for input x — what we compare with the data and use to measure the error
∑_i: a sum over all rules — every rule contributes, none is “selected”; that is why the output is smooth and differentiable
w_i/∑_j w_j: the normalized strength w̄_i — dividing by the sum makes the weights add up to 1, so the output is a weighted average of the consequents and does not depend on how tall the hills are
p_ix + r_i: the rule’s linear consequent — a local line describing the function in the rule’s area; a line instead of a constant lets a few rules follow the slope as well

The output is an average of local lines, weighted by how much the input belongs to each rule’s area.

μ_i(x) = exp( − (x − c_i)²/2σ_i² )

μ_i(x): a membership degree in [0, 1] — how much x is “around c_i”; the exponential of a negative number is always between 0 and 1, and at the centre it is exactly 1
(x − c_i)²: the squared distance from the centre — membership falls off equally on both sides and stays smooth even at the centre (with an absolute value it would have a spike there and no derivative)
2σ_i²: the width of the hill — distance is measured in units of σ_i, so at a distance σ_i from the centre the membership is always exp(−1/2) ≈ 0.61; learning σ_i changes how far the rule “reaches”

Example — a forward pass at x = 3

Take the system after epoch zero of learning (see below): centres c = 0, 2, 4, 6, all widths σ = 1 and consequents from least squares. For x = 3 the distances from the centres are 3, 1, 1, 3, and so

membership and strength (colours as in the formula for μ_i): w₁ = exp(−3²/2 · 1²) = e^−4.5, w₂ = exp(−1²/2 · 1²) = e^−0.5 and symmetrically w₃ = e^−0.5, w₄ = e^−4.5; so w = (0.0111; 0.6065; 0.6065; 0.0111);
normalization (colours from this step on as in the formula for y): the sum is 1.2353, so w̄ = (0.0090; 0.4910; 0.4910; 0.0090), e.g. w̄₂ = 0.6065/1.2353;
consequents: the lines p_ix + r_i with the learned coefficients (9.54x + 0.01; 9.34x − 17.38; 8.56x − 36.61; 2.60x − 18.90) take the values (28.63; 10.66; −10.94; −11.09) at 3 — the coefficients are rounded, the values computed from the exact ones;
sum of the products w̄_i · (p_ix + r_i): y = 0.2574 + 5.2329 − 5.3698 − 0.0997 ≈ 0.021.

The true value is sin(3) = sin(3²/3) ≈ 0.141. Notice that the output comes from subtracting two large, nearly equal contributions — we will come back to that under readability.

2. Hybrid learning

We want to minimize the mean squared error on the training data. It could all be done by gradient descent. Jang, however, noticed that the output is linear in the consequent parameters.

Definition — mean squared error and RMSE

Let N ≥ 1, let (x₁, y₁), …, (x_N, y_N) ∈ ℝ × ℝ be data and ŷ : ℝ → ℝ a model (its output for input x is ŷ(x)). The mean squared error of the model ŷ on these data is the number

E = 1/N ∑_k=1^N (ŷ(x_k) − y_k)² ≥ 0

and the RMSE (root mean square error) is its square root, RMSE = √E. Minimizing one or the other is the same thing; RMSE is reported because it has the units of the output. All errors in this part are RMSE.

RMSE = √1/N ∑_k (ŷ(x_k) − y_k)²

ŷ(x_k): the system’s prediction at the k-th point — the only place where the parameters c, σ, p, r enter the error
− y_k, then squared: the deviation from the measured value, squared — the sign does not matter, large errors weigh more than small ones, and the square is smoothly differentiable (and for the consequents allows an exact least-squares solution)
1/N ∑_k: an average over the points — the error on 40 training points and on 121 test points is then comparable

The square root at the end brings the error back into the units of y: an RMSE of 0.1 means “typically 0.1 off”.

Algorithm — hybrid learning

With fixed premises, y(x) = ∑_i ((w̄_ix) p_i + w̄_i r_i) is a linear function of the consequent parameters. This algorithm is built on that.

Input: training data (x_k, y_k) ∈ ℝ × ℝ, k = 1, …, N; the number of rules n; initial centres c_i ∈ ℝ and widths σ_i > 0; a learning step η > 0; a number of epochs T ∈ ℕ; the smallest allowed width σ_min > 0. Output: the parameters c_i, σ_i, p_i, r_i, i = 1, …, n. For epochs t = 0, 1, …, T repeat:

forward: compute w̄_i(x_k) for all i and k and find the consequent parameters p_i, r_i exactly by the least-squares method (definition below);
stopping: if t = T, stop and return the parameters;
backward: with the consequents found this way, take a gradient descent step for the centres and widths (gradients by the theorem below): c_i ← c_i − η ∂E/∂c_i, σ_i ← max(σ_min, σ_i − η ∂E/∂σ_i).

In the example in this part η = 0.1, T = 300 and σ_min = 0.05.

In plain words

Where a problem can be solved exactly, solve it exactly; leave the gradient only for what cannot be solved exactly. Linear regression from the part on statistics finds the best line in a single computation — and here it finds the best linear consequents of all rules at once. The gradient then only shifts and widens the hills of the membership functions.

Definition — least squares for the consequents

Let (x_k, y_k) ∈ ℝ × ℝ, k = 1, …, N, be the training data, n ≥ 1 the number of rules, and let the premise parameters c_i, σ_i be fixed. Then the values w̄_i(x_k) are just numbers (w̄_i for short). Arrange the consequent parameters into a vector θ = (p₁, r₁, …, p_n, r_n) ∈ ℝ²ⁿ and for each point k build the row

a_k = (w̄₁x_k, w̄₁, …, w̄_nx_k, w̄_n) ∈ ℝ²ⁿ, so that ŷ(x_k) = a_k · θ.

The rows a₁, …, a_N form a matrix A ∈ ℝ^N×2n, and y = (y₁, …, y_N) ∈ ℝ^N. The least-squares solution is a vector

θ* ∈ argmin_{θ ∈ ℝ²ⁿ} ‖Aθ − y‖²/N,

i.e. the consequent parameters with the smallest mean squared error. Every such θ* satisfies the normal equations A^TA θ = A^Ty — a system of 2n linear equations (and conversely every solution of them is a minimum; it is unique if and only if A has linearly independent columns). For 4 rules and 40 points, A is a 40 × 8 matrix and the system has 8 equations. Our computation also adds 10⁻⁶ to the diagonal of A^TA (a small ridge term), so that the system is solvable even with nearly dependent columns, and solves it by Gaussian elimination.

Theorem — gradient with respect to centre and width

Let (x_k, y_k) ∈ ℝ × ℝ, k = 1, …, N, be data and ŷ an ANFIS with n rules and parameters c_i ∈ ℝ, σ_i > 0, p_i, r_i ∈ ℝ. Let E be its mean squared error on the data, viewed as a function of all the parameters, and for an input x write w_i = exp(−(x − c_i)²/2σ_i²), W = ∑_j w_j, w̄_i = w_i/W and g_i = p_ix + r_i. Then for every i = 1, …, n

∂E/∂c_i = 2/N ∑_k=1^N (ŷ − y_k) (g_i − ŷ) w̄_i x_k − c_i/σ_i²,

∂E/∂σ_i = 2/N ∑_k=1^N (ŷ − y_k) (g_i − ŷ) w̄_i (x_k − c_i)²/σ_i³,

where ŷ, g_i and w̄_i are taken at the point x = x_k.

Proof. From ŷ = ∑_j w_jg_j/W the quotient rule gives ∂ŷ/∂w_i = g_iW − ∑_j w_jg_j/W² = g_i − ŷ/W. The derivatives of a Gaussian hill are ∂w_i/∂c_i = w_i x − c_i/σ_i² and ∂w_i/∂σ_i = w_i (x − c_i)²/σ_i³, and the parameters c_i, σ_i affect ŷ only through w_i. By the chain rule ∂E/∂c_i = 2/N ∑_k (ŷ − y_k) · g_i − ŷ/W · w_i x_k − c_i/σ_i², and w_i/W = w̄_i gives the first formula; the second likewise. ∎

∂E/∂c_i = 2/N ∑_k (ŷ − y_k) (g_i − ŷ) w̄_i x_k − c_i/σ_i²

∂E/∂c_i: how the error changes when we shift the centre of the i-th hill — the learning step goes against this sign
ŷ − y_k: the error at the point — where the system hits the target, the point wants nothing and contributes nothing to the gradient
g_i − ŷ: how the rule’s consequent differs from the overall output — giving the rule more weight pulls the output towards its consequent; when the consequent says the same as the whole, shifting the hill is pointless
w̄_i: the rule’s share of the output — points far from the hill, where the rule has almost no say, barely affect its centre
x_k − c_i/σ_i²: the direction of the shift — a point to the right of the centre pulls the hill right, one to the left pulls it left; at the centre itself the hill is flat and a shift changes nothing

For the width the formula is the same, only the last term is (x_k − c_i)²/σ_i³: distant points want to widen or narrow the hill, and the direction does not matter.

The whole learning from the example below in pseudocode (step η = 0.1, 300 epochs):

c = centres evenly spaced on [0, 6]  # for 4 rules 0, 2, 4, 6
σ = half the spacing of the centres  # for 4 rules 1
for epocha in 0 … 300:
    # forward: layers 1–3, then the consequents exactly by least squares
    wn[k][i] = w̄_i(x_k)  for all points k and rules i
    A = rows (wn[k][1]·x_k, wn[k][1], …, wn[k][n]·x_k, wn[k][n])
    p, r = solve (AᵀA + 10⁻⁶·I)·θ = Aᵀy
    if epocha == 300: break
    # backward: a gradient step by the formulas above
    gc, gσ = ∂E/∂c, ∂E/∂σ
    c = c − η·gc
    σ = max(0.05, σ − η·gσ)  # the hill must not vanish

3. ANFIS learns a function

The task: from 40 noisy samples (noise with standard deviation 0.1), learn the function y = sin(x²/3) on the interval [0, 6]. The function oscillates faster and faster to the right, so four evenly spaced rules are not ideal. We also measure the error on 121 noise-free points outside the training data:

ANFIS learning with four rules: output on top, membership functions below

Right in epoch zero — with just least squares for the consequents and evenly spaced hills — the test error is 0.394 and the training error 0.198. Then the premises start learning. After 100 epochs the test error is 0.202 and after 300 epochs 0.173. On the training data it drops to 0.107 — below the noise level in the data (0.114); you cannot sensibly do better. The centres of the hills moved only a little; the main change is in the widths — the last rule on the right, where the function oscillates fastest, narrowed from 1 to 0.64.

Definition — noise level in the data

Let f : ℝ → ℝ be the true function, x₁, …, x_N ∈ ℝ the inputs, and let the training outputs be y_k = f(x_k) + ε_k, k = 1, …, N, where ε_k are random deviations (noise). The noise level in the data is the RMSE that the true function f itself would have on the training data as a model:

√1/N ∑_k=1^N (f(x_k) − y_k)² = √1/N ∑_k=1^N ε_k².

In our example N = 40, f(x) = sin(x²/3), and ε_k are independent random deviations from a normal distribution with mean 0 and standard deviation 0.1; the noise level came out as √1/40 ∑_k ε_k² = 0.114. It is slightly larger than 0.1 only by chance — in our 40 drawn deviations.

A model whose training error is smaller than the noise level (0.107 < 0.114) is already starting to copy the random deviations too. With such a small difference it does no harm yet — the test error keeps falling — but it is a first warning, which we will come back to when choosing the number of rules.

Test error: hybrid learning versus pure gradient hybrid gradient only dashed: noise in the data

Pure gradient descent for all parameters starts from zero consequents and learns much more slowly: after 300 epochs its test error is 0.387, and even after 3,000 epochs (0.188) it does not catch up with hybrid learning after 300. A larger learning step, moreover, quickly leads to divergence. Splitting the problem into a linear and a nonlinear part pays off.

4. How many rules?

Number of rules	2	3	4	6	8	12
Training error	0.510	0.181	0.107	0.098	0.089	0.085
Test error	0.590	0.177	0.173	0.172	0.566	0.319

As rules are added, the training error keeps falling. On the test data, however, it rises sharply from eight rules on — the system has started learning the noise: with 40 points and 32 parameters (8 rules × 4) it has enough freedom to oscillate wildly between the training points. That is overfitting, the same phenomenon as with an over-bent polynomial regression. A sensible number of rules here lies between three and six.

Example — how many parameters ANFIS has

With one input, each rule has two premise parameters (c_i, σ_i) and two consequent parameters (p_i, r_i), 4n in total. For 40 training points:

4 rules: 16 parameters, 8 of them linear — each linear parameter gets 40/8 = 5 points;
8 rules: 32 parameters, 16 of them linear — 40/16 = 2.5 points per linear parameter, so least squares has enough freedom to fit the noise as well;
12 rules: 48 parameters — more than there are points. Least squares determines “only” 24 consequents from 40 equations, but the other 24 premise parameters can move and narrow the hills exactly where the noise happened to stray.

A useful rule of thumb: there should be at least several times more points than parameters estimated from them.

5. What can be read

The promise of neuro-fuzzy systems is that the learned model stays readable. The premises are readable: four rules “around 0”, “around 2.2”, “around 4” and “around 6.2”. The consequents, however, are not. The rule “IF x is around 2.2, THEN y = 10.06 x − 13.83” would predict 8.25 at its centre, while the true value is 1.0. The individual rules balance each other out — the hills overlap a lot and the output only makes sense as a whole.

Example — how the rules balance each other at x = 2.195

The learned centre of the second rule is c₂ = 2.195. There the second rule has full membership (w₂ = 1), but its neighbours do not have zero: w = (0.140; 1; 0.331; 0.000), sum 1.471, so for example w̄₂ = 1/1.471 ≈ 0.680 (colours as in the formula for y in the first section, g_i = p_ix + r_i):

Rule	1	2	3	4
weight `w̄`_i	0.095	0.680	0.225	0.000
consequent `g`_i	12.64	8.25	−25.64	−31.68
contribution `w̄`_i`g`_i	1.203	5.609	−5.769	0.000

The sum is y = 1.203 + 5.609 − 5.769 = 1.043, the true value sin(2.195²/3) = 0.999. The output is good, but it arises as the difference of two contributions of around ±5.7: the second rule “overshoots” upwards and the third pulls it back down. None of the rules on its own says what y is.

Interpretability, then, does not come for free. Constraints help to keep it: less overlap between membership functions, regularization of the consequents, or zero-order Sugeno with constant consequents. The price is paid in accuracy. It is the same balance as everywhere in machine learning — between what a model can do and what we can say about it.

6. ANFIS and neural networks

Kinship with RBF networks. In 1993 Jang and Sun showed that radial basis function networks and zero-order Sugeno fuzzy systems are, under mild conditions, the same thing, only interpreted differently.
The curse of dimensionality. With n inputs and k terms per input, the rule grid has kⁿ rules — over 59 thousand for 10 inputs and 3 terms. That is why ANFIS is used for few inputs, or with rules chosen by clustering (for example fuzzy c-means from the previous part).
Applications. Time series forecasting (the classic benchmark is the chaotic Mackey–Glass series), modelling and control of systems where we have measured data but little theory.
Today. Deep networks have long since taken over tasks with many inputs. The idea of “trainable but readable” lives on in explainable AI, though — and fuzzy layers are occasionally added even to deep networks where the output needs to be explained by rules.

Definition — radial basis function (RBF) network

Let d ≥ 1 be the number of inputs, n ≥ 1 the number of hidden neurons, c₁, …, c_n ∈ ℝ^d centres, v₁, …, v_n ∈ ℝ output weights and φ : [0, ∞) → (0, ∞) a decreasing radial basis function (typically Gaussian, φ(t) = exp(−t²/2σ²) with σ > 0). A normalized RBF network is the function y : ℝ^d → ℝ,

y(x) = ∑_i=1ⁿ v_i φ(‖x − c_i‖)/∑_j=1ⁿ φ(‖x − c_j‖),

where ‖·‖ is the Euclidean norm; the denominator is positive because φ has positive values. So the output is a weighted average of the output weights, where the weight of neuron i decreases with the distance of the input from its centre c_i. A zero-order Sugeno system with Gaussian memberships and the product as “and” is y = ∑_i w̄_i r_i — after substituting w_i = φ(‖x − c_i‖) and v_i = r_i, exactly the same formula. Put simply: rule = neuron, membership function = basis function, consequent = output weight. (Jang and Sun, 1993, list the conditions precisely: the same number of rules and neurons, constant consequents, Gaussian memberships with the same width within one rule, the product as “and”, and the same way of combining the output — weighted average or weighted sum.)

Example — the curse of dimensionality in numbers

With n inputs and k terms per input, the full grid has kⁿ rules. A first-order Sugeno consequent y = p₁x₁ + … + p_nx_n + r has n + 1 parameters, so there are kⁿ(n + 1) linear parameters. There are only 2kn premise parameters — each input has its own k hills, shared by all rules.

For n = 10 and k = 3: 3¹⁰ = 59,049 rules, 59,049 · 11 = 649,539 consequent parameters and 2 · 3 · 10 = 60 premise parameters. Least squares would be solving a system with 649,539 unknowns — and far more data still would be needed.

7. Summary

ANFIS is a Sugeno fuzzy system written as a five-layer network: membership, firing strength, normalization, consequent, sum.
Hybrid learning finds the linear consequents exactly by least squares and fine-tunes the centres and widths of the membership functions by gradient.
On the function sin(x²/3), learning the premises reduced the test error from 0.394 to 0.173; pure gradient needed ten times more epochs for a worse result.
Too many rules lead to overfitting — from eight rules on, the test error rises sharply.
The premises stay readable, but the consequents balance each other out; interpretability has to be watched.

This part closes the module on fuzzy logic: from the word “warm” through logic and controllers to a system that learns its rules by itself. Want to get hands-on? Have an AI agent train ANFIS on electricity consumption versus temperature from public data — and then look at which rules make sense.

Do you see it differently? Write to support@coden-app.com.