Evaluating and securing AI systems: how to measure and how to keep watch

Benchmarks and measurement uncertainty, how selecting on a test set inflates results, data contamination in an experiment, kinds of risk, an agent's command filter as a denylist versus an allowlist, and defence in depth.

12 minutes Matěj Žalmánek 2026-10-16

Series AI basics Part 52 of 52

1. Map of artificial intelligence: what is what and how it all fits together
2. A brief history of artificial intelligence: from Turing to language models
3. Data, model, training, inference: the basic terms on a single example
4. How to read mathematical notation: symbols, indices, sums and quantifiers
5. Sets and relations: the building blocks of definitions in AI
6. Functions and mappings: why every AI model is a function
7. Logic and proofs: the rules AI reasons by
8. Graphs and trees: the structure that both search and neural networks stand on
9. Linear algebra for machine learning: vectors, matrices and why AI mostly multiplies
10. Derivatives, the gradient and the chain rule: how a model finds out which way to learn
11. Gradient descent: how a model learns
12. What probability is: from rolling dice to the Kolmogorov axioms
13. Conditional probability and Bayes’ theorem: how to update your estimate when you learn something new
14. Random variables and the cumulative distribution function: one function that tells you everything about a distribution
15. Probability density and probability mass functions: why a continuous variable has zero probability at a point
16. Expected value, variance and covariance: where a distribution lies, how wide it is and what goes with what
17. Overview of probability distributions: Bernoulli, binomial, Poisson, uniform, normal and exponential
18. Law of large numbers and central limit theorem: why averages behave so predictably
19. Statistics and parameter estimation: maximum likelihood, MAP and confidence intervals
20. Information theory: entropy, cross-entropy and KL divergence — where loss functions come from
21. State-space search: BFS, DFS, UCS and A* with animations
22. Games and adversaries: minimax, alpha-beta pruning and Monte Carlo tree search
23. Constraint satisfaction: backtracking, constraint propagation and AC-3 on sudoku and a map of Czech regions
24. Logic and knowledge systems: forward and backward chaining, resolution and expert systems
25. Planning: STRIPS, the blocks world, heuristics from relaxation and planning in AI agents
26. Local search and simulated annealing: a travelling salesman through the Czech regional capitals
27. Genetic algorithms: population, crossover, mutation and selection on a backpack for a hike
28. Evolution strategies and genetic programming: from the one-fifth rule to Kepler’s law
29. Ant colony optimization: how a pheromone trail finds a short path
30. Particle swarm optimisation: inertia, personal experience and the swarm’s experience
31. Neuroevolution: evolving the weights and shape of neural networks, and the NEAT algorithm
32. Fuzzy sets and membership functions: how to write down “somewhat warm”
33. Fuzzy logic and probability: why “somewhat warm” does not mean “probably warm”
34. Fuzzy logic: t-norms, implications and IF–THEN rules
35. Fuzzy control: Mamdani and Sugeno from a fan to a pole on a cart
36. Fuzzy clustering: the fuzzy c-means algorithm and points between clusters
37. Neuro-fuzzy systems (ANFIS): fuzzy rules that learn from data
38. Markov chains: the transition matrix, the stationary distribution and PageRank
39. Hidden Markov models: the Viterbi algorithm and the dishonest casino
40. Bayesian networks: why the grass is wet and what it means for the sprinkler
41. The Kalman filter: how to estimate where a car is and how fast it goes from imprecise measurements
42. Types of machine learning: supervised, unsupervised, semi-supervised and more
43. Prompts, context and RAG: how to give a model the right information
44. Variational autoencoders (VAE): new images from a continuous latent space
45. Generative adversarial networks (GAN): the forger versus the detective
46. Diffusion models: how pure noise becomes an image
47. Markov decision processes: states, actions, rewards and the Bellman equations
48. Q-learning and DQN: learning the value of actions, from a table to a neural network
49. Policy gradient and PPO: learning the policy directly
50. How an AI agent works: a language model in a loop with tools
51. Programming with AI agents: the task, tests and review
52. Evaluating and securing AI systems: how to measure and how to keep watch

Full series syllabus 52 of 83 topics published

“The new model is 3 points better.” “The agent is safe, it has a ban on dangerous commands.” Both sentences sound reassuring, and both can be false without anyone lying. The first because of how things are measured, the second because of how things are forbidden. In the last part of the section on agents we look at the two questions asked most often about AI systems: how to tell what they can really do, and how to make sure they don’t do what they shouldn’t.

It builds on the parts Model evaluation, How an AI agent works and Programming with AI agents.

1. How we measure what a model can do

Definition — benchmark

A benchmark is a fixed triple on which models are compared:

1. a finite set of tasks u₁, …, u_n, n ≥ 1;
2. for every task u_i a reference solution r_i (the correct answer, or tests the solution must pass);
3. a scoring rule h that assigns to an answer a to the task u_i the value h(a, r_i) ∈ {0, 1} (1 = correct).

If a model M answers the task u_i with the answer M(u_i), its result on the benchmark is the share of tasks solved correctly

S(M) = 1/n ∑_i=1ⁿ h(M(u_i), r_i) ∈ [0, 1].

Examples: multiple-choice questions from many fields (MMLU), maths word problems (GSM8K), fixes of real bugs in open-source projects verified by tests (SWE-bench). They are complemented by human comparisons (people vote on which of two anonymous answers is better) and by your own tests on the tasks that matter to you.

Even an honest exam has limited precision, though. A model that solves 82% of 200 tasks has a 95% confidence interval of 76.1–86.7%; a model at 85% has 79.4–89.3%. The intervals overlap for the most part — from a measurement like this you cannot say which one is better. On 2,000 tasks the interval for 82% would be only 80.3–83.6%. Whenever you see a difference of a few percentage points, ask how many tasks it was measured on.

Definition — 95% confidence interval

Let p ∈ [0, 1] be the unknown probability that the model solves a task, n ≥ 1 the number of tasks and X₁, …, X_n independent random variables with P(X_i = 1) = p, P(X_i = 0) = 1 − p (X_i = 1 means that the model solved task i; the Bernoulli distribution). The measured proportion is

p̂ = 1/n ∑_i=1ⁿ X_i.

Let T₁ ≤ T₂ be bounds computed from X₁, …, X_n (in our case from p̂ and n). The random interval (T₁, T₂) is a 95% confidence interval for p if and only if

∀ p ∈ [0, 1] :   P_p(T₁ < p < T₂) = 0.95,

where P_p is the probability when the true value is p. If the equality holds only approximately for large n (in the limit n → ∞), it is an approximate (asymptotic) interval — the Wilson interval below is one of these. You will find the general definition in the part on statistics.

So if we repeated the measurement on new sets, the interval would contain the true p in 95% of cases. It is not the claim “p lies in the interval with probability 95%” — the interval is random, not p.

In this part we compute intervals with the Wilson formula (Edwin B. Wilson, 1927). Unlike the simple p̂ ± 1.96 · √p̂(1 − p̂)/n, it never goes beyond 0 and 100% and it also works for proportions close to the edges:

p̂ + z²/2n ± z √p̂(1 − p̂)/n + z²/4n²/1 + z²/n

p̂

the measured proportion of correct answers — the starting point around which we build the interval

z²/2n

a shift towards the middle: as if we added z²/2 ≈ 2 correct and 2 wrong answers to the results. A small set thus cannot claim “100%, an interval of zero width”

z √…

half the width of the interval; the first term under the square root is the variance of the measured proportion, which shrinks with n — that is why four times as many tasks give only half the interval

1 + z²/n

the denominator brings the shifted centre and the width back to the right scale; for large n it is almost 1 and the formula turns into the simple p̂ ± z · √p̂(1 − p̂)/n

Example — the interval for 82% of 200 tasks and the difference between two models

Colours as in the Wilson formula above; p̂ = 0.82, n = 200. z² = 1.96² = 3.8416. Centre: 0.82 + 3.8416/400 = 0.8296. Half-width: 1.96 · √0.82 · 0.18/200 + 3.8416/160,000 = 1.96 · √0.000738 + 0.000024 = 0.0541. Denominator: 1 + 3.8416/200 = 1.0192.

Interval: 0.8296 − 0.0541/1.0192 = 0.7609 to 0.8296 + 0.0541/1.0192 = 0.8671, i.e. 76.1–86.7%.

Is 85% versus 82% a real difference? The standard error of the difference of two independent proportions is √0.82 · 0.18/200 + 0.85 · 0.15/200 = 0.0371, so the 95% interval of the difference is 3 ± 1.96 · 3.71 = 3 ± 7.3 points. It contains zero — the difference cannot be told apart from chance. (When both models solve the same tasks, a more sensitive paired measurement is possible, using only the tasks on which they differ.)

How many tasks are needed for the half-width to be 1 point? For large n the simple formula is enough: from z √p(1 − p)/n ≤ 0.01 it follows that n ≥ z² p(1 − p)/0.01² = 3.8416 · 0.82 · 0.18/0.0001 ≈ 5,671 tasks.

2. The test set wears out

Definition — adaptive overfitting to the test set

Let the tasks come from some distribution 𝒟 and let S be a test set of n tasks drawn independently from 𝒟. Let f₁, …, f_K be variants of a model (different settings, prompts…), K ≥ 2. For a variant f denote by

1. acc(f) ∈ [0, 1] the true accuracy — the probability that f solves a task drawn at random from 𝒟;
2. acc_S(f) the measured accuracy — the share of tasks from S that f solves.

If the variant f is chosen without regard to S, the measured accuracy is an unbiased estimate: E[acc_S(f)] = acc(f). But if we select the variant by its results on the same set,

k* = argmax_{k ∈ {1, …, K}} acc_S(f_k),

then the measured accuracy of the selected variant stops being an unbiased estimate. Adaptive overfitting to the test set is the situation in which

E[acc_S(f_k*) − acc(f_k*)] > 0,

i.e. the measured accuracy of the variant selected by S is systematically higher than its true accuracy. It always holds that E[max_k acc_S(f_k)] ≥ max_k E[acc_S(f_k)] = max_k acc(f_k) (the maximum is at least as large as each of its terms, and therefore so is its expected value), so on average the reported number never underestimates even the best of the variants, and it usually overestimates it.

In words: when you repeatedly select based on results on the same test set, the chosen variant is partly the best only because chance favoured it on this particular set, and on new data it does worse. More generally this is captured by Goodhart’s law: when a measure becomes a target, it ceases to be a good measure.

An experiment: we measure K variants of a model, all with a true accuracy of 70%, on the same set and report the best result. The average over 300 repetitions:

On a set with 100 tasks, picking from 100 variants reports 81.1% on average, and from 500 variants 83.2% — 13 points more than any of them can really do. A larger set helps (with 1,000 tasks 73.6%), but the problem does not go away. The variants in the experiment err independently; real variants of one model make similar mistakes, so the effect tends to be smaller — but it points the same way. The remedy: select on one set (the validation set) and measure the final number once on a different, untouched set.

Why it comes out like this can be computed exactly. The number of tasks X that one variant solves has the binomial distribution Bin(n, 0.7) (binomial distribution). The best of K variants reaches at least t tasks if at least one of them does:

P(max ≥ t) = 1 − ( 1 − P(X ≥ t) )^K

P(X ≥ t)

the chance that one variant solves at least t tasks by luck; for t above the true level it is small — and that is exactly what selection feeds on

K

the number of variants tried; in the exponent, because (1 − P)^K is the chance that none of the independent variants manages it

P(max ≥ t)

the chance that the reported (best) result reaches t; as K grows it approaches one even for high t

Example — 80% of 100 tasks

One variant: X ~ Bin(100, 0.7) has mean 70 and standard deviation √100 · 0.7 · 0.3 = 4.58 tasks. It solves at least 80 tasks with probability P(X ≥ 80) = 0.0165, i.e. in about 1.6% of cases — more than two standard deviations above the mean.

Out of 100 variants (colours as in the formula above): P(max ≥ 80) = 1 − (1 − 0.0165)¹⁰⁰ = 0.810. A result of “80% or more”, a rarity for a single variant, gets reported in four experiments out of five.

The expected value of the best result can be computed from the formula without simulation: E[max] = ∑_t=1ⁿ P(max ≥ t). It comes out at 81.1% for 100 variants, 83.2% for 500 variants and 73.6% for 100 variants on a set with 1,000 tasks — the same as the simulation with 300 repetitions gave (81.1, 83.2 and 73.6%).

3. Contamination: the model has seen the answers

Definition — data contamination

Let D be the training data of a model and S = {u₁, …, u_n} a test set.

1. A task u ∈ S is contaminated if and only if D contains u (the question, possibly together with the solution) or a close copy of it. What counts as a “close copy” is not fixed; in practice a rule is chosen, for example that it shares a sufficiently long contiguous stretch of text with u.
2. The contamination rate of the set S is the share of contaminated tasks |{u ∈ S : u is contaminated}|/n.

So contamination happens if and only if the contamination rate is positive — test tasks (or close copies of them) have ended up in the training data. For large language models, trained on a large part of the internet, this is a serious problem: benchmarks are public, often together with their solutions, and the model may have memorized them. The measured result then measures memory, not ability.

An experiment on the 8 × 8 digits from the part on clustering: a nearest-neighbour classifier learns from only three examples of each digit, so on 200 test digits it honestly scores 60%. Then part of the test digits “leaks” into the training data:

ŷ(x) = y_i*,   i* = argmin_{i ∈ training} ∑_j=1⁶⁴ (x_j − x_i,j)²

∑ (x_j − x_i,j)²

the squared Euclidean distance between two images of 64 pixels each; small when the images are alike

argmin_{i ∈ training}

which training example is closest — the search covers only what the model has “seen”; that is why a leak of test data directly changes the answers

y_i*

the answer is the digit of the nearest neighbour; no generalization, just “it looks like that one”

With a 10% leak the accuracy jumps to 69%, with 25% to 79%, with half to 92% and with the whole test set to 100%. Interestingly, even the digits that did not leak improve (with half leaked, 86% instead of 60%): the leaked examples resemble the other test examples, and so they help with those too. That is why it is not enough to check whether the model has seen exactly the same question — it is enough that it has seen similar ones. The defence: non-public test sets, new tasks created only after the model finished training, and checking whether the model can “complete” a task from memory.

4. What can go wrong

The safety of AI systems is not one thing. It helps to distinguish where the harm comes from:

• Errors — the model is wrong or makes things up (hallucinations) and a person believes it.
• Misuse — someone uses the model to do harm: fraudulent e-mails, help with an attack, disinformation.
• Attacks on the system — a prompt injection in a document or on the web takes control of the agent; a “jailbreak” gets around the model’s rules.
• Dangerous actions — an agent with access to a computer or accounts does something irreversible, even though nobody wanted it.
• Misalignment — the system optimizes exactly what we told it to instead of what we wanted (reward over-optimization). As systems become more autonomous, this is a subject of both research and regulation.

5. Deny or allow

A concrete example from agent practice: the agent may run commands in a terminal. How do you stop it from running something dangerous? We compare two filters on 28 commands — 12 ordinary ones (tests, git, listing files) and 16 dangerous ones (deleting, sending data out, running a foreign script).

Definition — denylist and allowlist

Let C be the set of all commands (text strings). A filter is a function φ : C → {pass, stop}; the agent does not carry out a stopped command — for an agent it typically waits until a human approves it.

1. Let Z be a finite set of forbidden patterns. A denylist is a filter for which φ_Z(c) = pass  ⇔  ¬ ∃ z ∈ Z : c matches the pattern z.
2. Let A be a finite set of allowed patterns. An allowlist is a filter for which φ_A(c) = pass  ⇔  ∃ a ∈ A : c matches the pattern a.

What “matches the pattern” means is determined by the particular filter; in our experiment, for the denylist it is “z is a substring of c” and for the allowlist “c starts with a and contains none of the characters | ; & ` $” (breakdown below). So a denylist lets through everything it does not explicitly forbid, and an allowlist stops everything it does not explicitly allow. The security principle of least privilege recommends the latter.

The denylist contains the patterns “rm -rf”, “curl | sh”, “DROP TABLE”, “--force”, “.ssh”. The allowlist lets through only listing and reading files, git without writing, running tests and creating a directory, and only without pipes and semicolons.

denylist passes c ⇔ ¬ ∃ z ∈ Z: z is a substring of c
allowlist passes c ⇔ ∃ a ∈ A: c starts with a ∧ c contains no | ; & ` $

¬ ∃ z ∈ Z

denylist: passes everything in which it finds none of the forbidden patterns. The default answer is “yes” — every spelling the author didn’t think of gets through

∃ a ∈ A: c starts with a

allowlist: passes only a command that starts with one of the allowed prefixes. The default answer is “no” — an unknown command waits for a human

no | ; & ` $

without this an allowed beginning could be “extended”: git status; rm -rf ~ starts with the allowed git status

The denylist stopped only 5 of the 16 dangerous commands. It is enough to write rm -r -f instead of rm -rf, delete files via find -delete or via Python, download a script with wget instead of curl, or encode the command in base64. A denylist is always one step behind ingenuity — and a language model is ingenious even when it wants to harm no one; it is just looking for another way to get the task done. The allowlist stopped all 16. The price: 3 of the 12 ordinary commands (installing a library, a commit, deleting a temporary file) have to be approved by a human. That is exactly the human’s role in overseeing an agent: to decide in the cases the rules don’t cover.

Example — filters as classifiers

A filter is a classifier with the positive class “dangerous”, so it can be described with the numbers from the part on model evaluation. Recall (TPR, the share of dangerous commands stopped) and the false positive rate (FPR, the share of ordinary commands stopped):

TPR = TP/TP + FN = number of dangerous commands stopped/number of dangerous commands,   FPR = FP/FP + TN = number of ordinary commands stopped/number of ordinary commands

Denylist: TPR = 5/16 = 0.313, FPR = 0/12 = 0.00. Allowlist: TPR = 16/16 = 1.00, FPR = 3/12 = 0.25.

A false alarm here costs a few seconds of a person clicking “approve”; a missed dangerous command can cost the whole project. With costs this asymmetric, a filter with high recall pays off even at the price of false alarms — that is, the allowlist.

6. Defence in depth

No single layer is enough — a model can be talked round, a filter bypassed, a human gets tired. That is why layers are stacked:

• Model — fine-tuning so that it refuses harmful requests and admits uncertainty (the part on RLHF).
• Instructions and data separation — a system prompt, text from the web and from documents marked as data, not commands.
• Permissions — tools with the least necessary rights, allowlists, separate accounts and keys.
• Isolation — the agent runs in a container or a virtual machine, without access to things it does not need.
• Human in the loop — approval of irreversible and unusual steps.
• Logging and review — what the agent did can be traced back and undone (git, backups, logs).

P_harm = p₁ · p₂ · … · p_m

p₁

the probability that the first layer lets a harmful action through (say, the model does not refuse it)

p₂

the probability that the second layer (the command filter) lets it through — they multiply because the action has to get through all the layers

p_m

the last layer (isolation, a backup); every additional layer with p < 1 makes the result smaller, even if it is weak on its own

P_harm

the chance that a harmful action gets all the way through

Four layers, each letting through 10% of harmful actions, give P_harm = 0.1 · 0.1 · 0.1 · 0.1 = 0.0001 — one in ten thousand. But layers are rarely independent: a prompt injection that convinces the model often convinces the human too, who approves based on the model’s explanation. If the second to fourth layers were fooled every time the first one was, P_harm = 0.1 would remain. That is why the layers should rest on different principles: the model on content, the filter on the syntax of the command, isolation on what the process has access to at all.

Definition — red teaming and dangerous capability evaluations

These are not mathematical concepts but procedures; they are defined precisely by the steps they consist of.

Red teaming is a repeated procedure of targeted search for failures of a system:

1. it is set out in advance what the system must not do (rules, forbidden outputs and actions);
2. attackers — people and increasingly other models — look for inputs and situations in which the system breaks one of the rules;
3. every violation found is recorded and the weakness is fixed (by fine-tuning the model, a filter, a change of permissions);
4. the procedure is repeated on the fixed system.

Dangerous capability evaluation is the measurement of capabilities before a model is released:

1. a set of tasks is put together whose mastery could lead to serious harm (such as helping with a cyberattack);
2. it is measured how successfully the model solves them;
3. the result is compared with thresholds set in advance — if the model exceeds a threshold, the developer tightens the safeguards before release (or does not release the model).

The large developers have such thresholds and the safeguards tied to them written down in their public safety policies.

Technical measures are increasingly joined by regulation. The European Artificial Intelligence Act (AI Act, in force since 2024 and applied gradually) divides systems by risk: it bans some practices outright and imposes detailed obligations on high-risk systems that make decisions about people — in hiring, lending or healthcare.

7. Summary

• A benchmark is an exam: useful for comparison, but with limited precision. On 200 tasks, 82% has the interval 76.1–86.7%.
• Selecting on the same test set inflates the result: picking from 100 variants with a true accuracy of 70% reports 81.1%.
• Contamination measures memory instead of ability; a leak of half the test raised accuracy from 60% to 92%.
• Risks: errors, misuse, attacks on the system, dangerous actions and misalignment.
• The denylist stopped 5 of 16 dangerous commands, the allowlist all 16 — at the price of approvals.
• Safety rests on layers: the model, data separation, permissions, isolation, a human in the loop and logging.

Want to get hands-on? Write down ten tasks from your own work together with the correct results and have an AI agent compare two models on them — and have it add a confidence interval to the result. Then try to come up with three commands that would get around your agent’s denylist filter.

Do you see it differently? Write to support@coden-app.com.