Programming with AI agents: the task, tests and review

How to write a task for an agent with acceptance criteria, why to work in small steps, an experiment with mutation testing (how many bugs happy-path tests, edge cases and property-based tests detect), who should write the tests and what to watch out for.

10 minutes Matěj Žalmánek 2026-10-16

Series AI basics Part 51 of 52

Full series syllabus 52 of 83 topics published

What you will take away

What belongs in a task for an agent so that it knows when it is done.
How many bugs tests of varying thoroughness detect and why edge cases matter.
What mistakes agents make and how to find them when you review.

Useful to know first: How an AI agent works: a language model in a loop with tools, How to approach development with AI agents

In the previous part we took apart how an agent works on the inside. This part is practical: how to give it work, how to check that work and how not to lose track of what is going on in the project. Most of the advice fits into a single sentence — you have to give the agent a way to tell for itself that it is done, and you have to have a way to verify it. An experiment will show how much that matters.

It builds on the parts How an AI agent works and How to approach development with AI agents, which say more about the role of the human and about why you should let the agent propose a solution first.

1. The task: what, where and how to recognise done

Definition — acceptance criteria

Let a task be given and let R be its result (the state of the project after the work is handed in). An acceptance criterion is a condition K on the result that satisfies:

it concerns only what can be observed or run on the result — concrete behaviour (“returns 0 CZK for an empty cart”), tests that must pass, a command that must finish without an error;
it is verifiable: there is a procedure known in advance (a test, a command, a check against a list) that for every result R decides unambiguously whether K(R) holds or not.

If a task has the acceptance criteria K₁, …, K_m (m ≥ 1), then

the task is done ⇔ K₁(R) ∧ K₂(R) ∧ … ∧ K_m(R).

So a task is done if and only if the result meets all the criteria. Unlike a description of the goal (“make it clear”), a criterion can be decided unambiguously: either it holds or it does not.

In plain words

“Tidy your room” is a goal. “Nothing is lying on the floor, the bed is made and the bin is emptied” are acceptance criteria. Only with them can you say whether the room is tidy — and the child (or the agent) knows when it can stop.

Definition — specification as precondition and postcondition

Acceptance criteria can be written down precisely as a Hoare triple (C. A. R. Hoare, 1969; today’s notation with curly braces). Let

Σ be the set of states of the program — a state σ ∈ Σ assigns a value to every variable;
C be a program; the notation ⟨C, σ⟩ ⇓ σ′ means that C, started in the state σ, terminates and ends in the state σ′;
P and Q be conditions on states, i.e. for every σ ∈ Σ both P(σ) and Q(σ) are true or false.

P is called the precondition (what the program may expect of its input), Q the postcondition (what must hold of the result). The Hoare triple {P} C {Q} holds if and only if

∀ σ, σ′ ∈ Σ : ( P(σ) ∧ ⟨C, σ⟩ ⇓ σ′ ) ⇒ Q(σ′).

In words: whenever P holds before C runs and C terminates, Q holds afterwards. A program C is (partially) correct with respect to the specification (P, Q) if and only if the triple holds; totally correct if and only if, in addition, it terminates from every state σ with P(σ). About states where P does not hold, the triple says nothing.

{ castka ≥ 0 } s := sazba(castka) { s = 0 for castka < 1,000; s = 0.05 for 1,000 ≤ castka < 5,000; s = 0.1 otherwise }

castka ≥ 0: precondition: the function makes no promises for a negative amount (castka) — it is there to make clear what we do not have to test and what the caller must take care of
s := sazba(castka): the program we are talking about (the rate function); the specification says nothing about how it is written, so the agent may satisfy it any way it likes
s ∈ {0, 0.05, 0.1}: postcondition: exactly three bands and their boundaries (≤ or <) — this is where mistakes are made most often, which is why the values 999, 1,000, 4,999 and 5,000 belong in the acceptance criteria

The “Done when” line in the task below is the same postcondition, written so that it can be checked by a test.

A good task for an agent has four parts. Not because it is a ritual, but because each one covers a thing the agent would otherwise have to guess:

Goal: On the cart page, show a discount based on the amount (from 1,000 CZK 5%, from 5,000 CZK 10%).

Context: The price calculation is in src/kosik.js, the template in pohledy/kosik.ejs.
Shipping is calculated in a similar way in the function doprava() — stick to the same style.

Constraints: Do not change the data format in the database. No new libraries.

Done when:
- the function sazba(castka) returns 0 / 0.05 / 0.1 and the tests in test/kosik.test.js pass,
- you add tests for the boundaries 999, 1,000, 4,999 and 5,000 CZK,
- npm test finishes without an error and you show me the diff (git diff).

Context saves the agent from searching (and saves you tokens — see the cost of context). Constraints stop it from “solving” the problem differently from what you want. And acceptance criteria give it the feedback without which long tasks fall apart.

2. Small steps, frequent checks

In the part about agents we calculated that a task of twenty steps, each of which succeeds 90% of the time, comes out right without checks only 12% of the time. The same rule applies to you: the bigger the chunk of work the agent hands over at once, the harder it is to check and the more expensive the fix. In practice:

One change = one task. Split a big thing into steps that can be checked separately (first the calculation, then the display, then the data changes).
Version control as a safety net. A clean state in git before every task, a commit after it. When something breaks, you go back with a single command.
Read the diff, not the summary. The agent’s “done, everything works” is a claim. The diff of the code (git diff) and the test output are evidence.

P_success = pⁿ

p: probability that a single step succeeds; it is there because every step is another chance for a mistake
n: number of steps between two checks; it sits in the exponent because the probabilities of independent steps multiply — and with every further unchecked step the chance drops geometrically, not linearly
P_success: chance that the whole stretch between checks turns out well and nothing has to be redone

Simplified: steps fail independently and a mistake is only noticed at a check.

Example — how much work a check after every five steps saves

Colours as in the formula above. Without checks: P_success = 0.9²⁰ = 0.1216, i.e. 12%. With a check after every five steps: 0.9⁵ = 0.5905 for each of the four stretches.

Assume the check always catches the mistake and a broken stretch is redone, with the attempts independent. The number of attempts until the first success then has a geometric distribution: the k-th attempt is the first successful one with probability (1 − P_success)^k−1 P_success, and the mean number of attempts is ∑_k=1^∞ k (1 − P_success)^k−1 P_success = 1/P_success. The whole task without checks therefore costs on average 20/0.1216 ≈ 165 steps, with a check after every five 4 · 5/0.5905 ≈ 34 steps and with a check after every step 20/0.9 ≈ 22 steps. Small steps do not raise the chance that one step succeeds — they shrink how much work is thrown away when there is a mistake.

3. Tests as a safety net: an experiment

How much do tests help? We take five small functions of the kind an agent writes all the time — an average, a leap year, truncating text, binary search and a discount based on the amount — and make deliberately broken versions of them.

Definition — mutation testing

Let f be a program (the original), X the set of its admissible inputs and T = {t₁, …, t_k} a test suite; each test runs the program on chosen inputs and either passes or fails (a crash or an infinite loop of the program also counts as a failure).

A mutation operator is a rule for one small syntactic change of the code: < instead of <=, - 1 instead of + 1, || instead of &&, a different constant.
A mutant f′ is a program obtained from f by a single application of one mutation operator at one place in the code. Let M denote the set of all mutants produced this way.
A mutant f′ is equivalent if and only if ∀ x ∈ X : f′(x) = f(x), i.e. it differs from the original on no input. E ⊆ M is the set of equivalent mutants.
A suite T detects (kills) a mutant f′ if and only if ∃ t ∈ T : t fails on f′. O ⊆ M ∖ E is the set of detected non-equivalent mutants.

The mutation score of the suite T is the share of detected mutants among the non-equivalent ones (for |M| > |E|):

MS(T) = |O|/|M| − |E| ∈ [0, 1].

So the mutation score measures how large a part of the artificial bugs the tests catch. Equivalent mutants are not counted, because no test can detect them. Equivalence cannot be decided algorithmically in general; in our experiment we estimate it by comparing the mutant with the original on 2,000 random inputs.

In plain words

Testing a smoke detector: you deliberately light a small candle and watch whether it beeps. Here you deliberately put a small mistake into the code — exactly the kind people and agents make — and watch whether the tests notice.

From the source code of the five functions, 32 mutants were generated automatically. 3 of them never differed from the original on 2,000 random inputs (for instance a[s] <= x instead of a[s] < x in the binary search, where the previous line has already handled equality), which leaves 29 real bugs. Against them we put four levels of tests, each including the previous ones:

Happy path — one or two typical examples per function. These are the tests you get when you just say “add tests”.
Edge cases — an empty array, band boundaries (999, 1,000, 4,999, 5,000 CZK), the years 1900, 2000 and 2100, text exactly at the length limit.
Properties — 200 random inputs and rules that must always hold: the result of truncation has at most max characters and is a prefix of the text, the index found really points to the number searched for, the average matches a calculation written in a different way.

How many of 29 bugs the tests detect

Happy-path tests catch only 15 of 29 bugs (52%). They pass even with a function that returns 1 for an empty array, with a discount that only starts at 1,001 CZK, or with a binary search that finds nothing in a one-element array. Adding edge cases raises the share to 90%, property tests to 97%.

function	mutants	happy path	+ edge cases	+ properties
`prumer`	5	4	5	5
`jePrestupny`	8	5	5	7
`zkrat`	3	2	3	3
`najdi`	6	3	6	6
`sazba`	7	1	7	7

MS = |O|/|M| − |E|

MS: mutation score — the share of real bugs the test suite catches — the height of a bar in the chart
|O|: number of detected mutants (at least one test failed on them, or they crashed or looped forever); what the tests can do
|M|: all mutants produced — the stock of bugs we measure the tests against
|E|: equivalent mutants; they are subtracted because they do not differ from the original, so no test can detect them — otherwise even perfect tests would never reach 100%

In our case |M| = 32 and |E| = 3, so the denominator is 32 − 3 = 29 (colours as in the formula above). Happy path: MS = 15/32 − 3 = 0.517; with edge cases 26/32 − 3 = 0.897; with properties 28/32 − 3 = 0.966. The procedure that produced the numbers:

for each function f:
    original = compile(code of f)
    for each operator (from → to) and each occurrence of from in the code:
        mutant = compile(code where this occurrence of from is replaced by to)
        if the mutant never differs from the original on 2,000 random inputs:
            equivalent — not counted, next
        for each test level u:
            detected[u] = some test of level u fails on the mutant,
                          crashes, or loops forever (cut off after 10,000 iterations)
MS(u) = number detected at level u / number of non-equivalent

The most interesting one is the leap year. Edge cases let through mutants that treat every even year as a leap year, or that rule out years divisible by ten — including the year 2020. Tests with the years 1900, 2000 and 2100 do not notice, because on those the mutants happen to answer correctly. The property test with random years finds both bugs. The only mutant that survived everything is (rok % 4 === 0 && rok % 100 !== 0) || rok % 200 === 0: it differs only for years divisible by 200 but not by 400 (200, 600, 1000…), and there was no such year among the 200 random ones. Not even random tests are almighty.

P_survives = ( 1 − q )ⁿ

q: share of inputs on which the mutant differs from the original; 1 − q is the chance that a single random input does not expose it
n: number of random inputs; the inputs are drawn independently, so the chances of “did not expose” multiply
P_survives: probability that all n inputs miss the mutant — the bug gets through even though the property is written correctly

Random tests reliably catch bugs that show up often; rare bugs only with a lot of inputs.

Example — why the mutant with the year divisible by 200 survived

The property test draws years from 0 to 2,399. The mutant rok % 200 differs from the original in only 6 of them (200, 600, 1000, 1400, 1800, 2200), so q = 6/2400 = 0.0025 and with n = 200 it survives with probability P_survives = (1 − 0.0025)²⁰⁰ = 0.606 (colours as in the formula above) — roughly in six attempts out of ten. So its survival was not bad luck but the expected outcome.

The mutant rok % 2 differs for 600 years (q = 0.25): it survives with probability P_survives = 0.75²⁰⁰ ≈ 10^-25, i.e. practically never. The mutant rok % 10 differs for 96 years (q = 0.04) and survives with probability P_survives = 0.96²⁰⁰ = 0.00028.

How many inputs would be needed to catch the year-200 mutant with probability 95%? We require (1 − q)ⁿ ≤ 0.05. Taking logarithms, n ln(1 − q) ≤ ln 0.05, and because ln(1 − q) < 0, the inequality flips when we divide: n ≥ ln 0.05/ln(1 − q) = −2.996/−0.00250 ≈ 1,196.8, i.e. 1,197 inputs. It is cheaper to add the edge case year 200 or 600 — anyone who knows the calendar rule knows where to look.

4. Who should write the tests

We wrote the tests in the experiment from the specification (what the function is supposed to do), not from the code. That matters. When an agent writes code and then tests for it straight away, there is a risk that the tests merely repeat what the code does — including its bugs. They pass and prove nothing. What helps:

Tests (or at least a list of them) first. Have the agent write the tests from the task first, read them, and only then the code. The acceptance criteria from the task are exactly that list.
Spell out the edge cases. Empty input, a single item, band boundaries, negative numbers, accented characters. The agent often adds them on its own, but only if you ask for them.
Properties wherever possible. “A sorted array is still sorted”, “encode and decode = the original text”, “the new calculation gives the same as the old one”. Libraries such as fast-check (JavaScript) or Hypothesis (Python) generate random inputs by themselves.

Definition — property-based testing

Let f : X → Y be the function under test. A property is a condition φ : X × Y → {true, false} on an input–output pair; the function satisfies it if and only if

∀ x ∈ X : φ(x, f(x)).

A property test is the following algorithm.

Input: the function f, the property φ, the number of tries N ∈ ℕ, N ≥ 1, a generator of random elements of X and a shrinking function that for every x ∈ X returns a finite set of candidates simplify(x) ⊆ X, each of which has a strictly smaller size than x (the size is a natural number — the length of a text, the absolute value of an integer, the length of an array).
Output: a counterexample x ∈ X with ¬φ(x, f(x)), or the message “N inputs passed”.

For i = 1, …, N: generate a random x ∈ X; if ¬φ(x, f(x)) holds, continue with step 2.
Shrinking: as long as there is a y ∈ simplify(x) with ¬φ(y, f(y)), set x := y. The size is a natural number and drops in every pass, so shrinking terminates. Return the counterexample x and stop.
If no input violated the property in step 1, return “N inputs passed”.

So the test looks for a counterexample to the statement “for all x”; if it does not find one, the property is not thereby proved. The idea was popularised by the QuickCheck library (Koen Claessen and John Hughes, 2000).

In plain words

You do not ask “what is 7 + 5”, but “is it true that a + b is always the same as b + a?” — and let the computer try a thousand random pairs. It finds cases you would not have thought of.

A property is a statement with the quantifier “for all”. For the function zkrat (truncate) from the experiment, the test writes it like this:

∀ t, m ≥ 1: |v| ≤ m ∧ ( |t| ≤ m ⇒ v = t ) ∧ ( |t| > m ⇒ v = t_{[0, |v| − 1)} + “…” ), where v = zkrat(t, m)

∀ t, m ≥ 1: for every text and every length — that is why it can be checked on random inputs and we do not need to know the correct result in advance
|v| ≤ m: the function’s main promise: the result fits; it catches the mutant that forgets to count the “…” (slice(0, max))
|t| ≤ m ⇒ v = t: a short text is left unchanged; without this, a function that always truncates would pass
v = t_{[0, |v| − 1)} + “…”: the truncated result is the beginning of the original text and ends with an ellipsis; without this, a function returning, say, an empty text would pass

t is any text, m ∈ ℕ, |t| is the number of characters of the text t and t_{[0, k)} is the first k characters of the text t. The property does not say exactly how many characters remain — only what must hold; that makes it more general than a single concrete example.

A property-based testing tool then does roughly this:

for i = 1 … N:                       # N is usually 100 to 1,000
    x = generate a random input
    if property(x, f(x)) does not hold:
        # shrinking: look for a simpler input on which it still fails
        while there is y = simplify(x) on which the property does not hold:
            x = y                    # shorter text, smaller number, shorter array
        report counterexample x
report “N inputs passed”             # not a proof, just good news

The last line matters. Tests, random or not, do not prove correctness — they only look for counterexamples. Edsger Dijkstra put it in 1969 like this: testing can show the presence of bugs, never their absence. We calculated above the probability that a rare bug slips through.

5. What to watch out for

Agents make different mistakes from people. The most common ones worth looking for when you review:

“Done” without verification. The agent announces success without having run the tests, or having run only some of them. Ask for the test output.
Changing the tests instead of the code. When a test fails, the easiest “fix” is to change the expected value or delete the test. Look for changes to tests in the diff.
Silenced errors. An empty catch, a default value instead of an error, a hard-coded result for the test input.
Extra work. A rewritten file that has nothing to do with the task, a new library, an “improved” data format. Constraints in the task and small steps keep this in check.
Secrets and irreversible steps. Passwords in the code, deleting files, writing to the production database, pushing to a remote repository. Have these actions require your approval.

6. Keeping track

An agent remembers nothing between sessions. What it should know every time — how the project is run and tested, what the conventions are, what to avoid — belongs in an instructions file at the root of the project, which the agent reads at the start. Write down there whatever you have had to point out to the agent more than once. And what only you know (why something is done this way, what was tried and did not work) belongs in commits and documentation — for the agent and for yourself six months from now.

The last piece of advice is the simplest: stay involved. Not necessarily at the computer — the agent can work on its own and you take a look when it needs a decision. That is why coden was created: the task, step approvals and result review are on your phone, while the agent works on your computer.

7. Summary

A good task: goal, context (files, patterns), constraints and verifiable acceptance criteria.
Small steps, a commit after each one, and review of the code diff and the test output, not the summary.
In the experiment, happy-path tests detected 52% of the bugs, with edge cases 90%, with properties 97%.
Write (or approve) tests from the specification and ideally before the code, otherwise they merely repeat the implementation’s bugs.
Watch for changed tests, silenced errors, extra work and irreversible steps.

Want to try it yourself? Ask an AI agent to run a mutation testing tool on your project (Stryker for JavaScript, mutmut for Python) and show you the mutants that survived. Each of them is a missing test.

Do you see it differently? Write to support@coden-app.com.

Programming with AI agents: the task, tests and review

1. The task: what, where and how to recognise done

2. Small steps, frequent checks

3. Tests as a safety net: an experiment

4. Who should write the tests

5. What to watch out for

6. Keeping track

7. Summary

An Agent on your computer, driven from your phone