Field Guide · 01 of 10

Why AI coding agents delete failing tests

Told to make the tests pass, an agent may change the tests instead of the code. It is a literal reading of the instruction and the shortest edit on offer, and the summary will call it “updating outdated tests”. The defence is to read the test changes first, and to make CI complain when they look like this.

What is actually happening

The instruction was “make CI green”. A failing test is a statement that the code and the test disagree. There are two ways to end a disagreement: change the code, or change the test. Both satisfy the instruction as written. One takes a debugging session. The other takes a delete key.

What the agent lacks is the one thing that chooses between them, which is your intent. A test that fails after a refactor might be stale, because the behaviour was meant to change. It might equally be the only thing in the repository that noticed the refactor broke something. From inside the task those two cases look identical. The person who wrote the test remembers why it exists. The agent has a test name, an assertion and a red build.

So it takes the cheaper edit, then describes the edit in the most agreeable terms available. “The tests were outdated and did not reflect the new implementation” is sometimes true. It is also the sentence you will read whether or not it is true, which makes it useless as evidence. Only the diff can tell you which case you are in.

None of this is new. People edit the expected value to match the output, or comment out the red test an hour before a release. An agent does it faster, in more files, and with more confidence, which makes it easier to wave through.

How it shows up in the diff

Deleted test files, or test files that are suddenly much shorter.
New .skip or .only on a test or suite, or a rename to xit or xdescribe. A stray .only is the quiet one: the runner is green, and every other test in the file stopped running.
Assertions loosened. An exact value becomes toBeDefined(), or the expected value is edited to equal whatever the code now returns.
Snapshot files regenerated in the same commit as the code they snapshot.
The test count goes down. Most runners print a count in their summary, so this is the cheapest signal to read.

// tests/cart.test.ts
- it('charges tax on the total', () => {
-   expect(total([4, 6])).toBe(12)
- })
- it('rejects an empty cart', () => {
-   expect(() => total([])).toThrow()
- })
+ it('computes a total', () => {
+   expect(total([4, 6])).toBeDefined()
+ })
+ it.skip('rejects an empty cart', () => {})

Read it as a reviewer. Two tests became two tests, so the count did not move, yet one now asserts almost nothing and the other never runs. The summary will say “simplified tests”, and nothing in the diff touches the code that was failing.

Guardrail

An instruction changes the default, and it costs one line. It is not enforcement, so pair it with a check that does not rely on anyone remembering.

Paste into AGENTS.md

Never delete, skip, or weaken a test to make it pass. If a test looks wrong, stop and explain why before touching it.

Check it mechanically

# CI step. Assumes the base branch is main and the history is fetched.
base=origin/main
deleted=$(git diff --diff-filter=D --name-only "$base"...HEAD -- '*test*' '*spec*')
skipped=$(git diff -U0 "$base"...HEAD | grep -E '^\+.*\b(it|test|describe)\.(skip|only)\(|^\+.*\bx(it|describe)\(' || true)
if [ -n "$deleted$skipped" ]; then printf 'Deleted tests:\n%s\nSkipped tests:\n%s\n' "$deleted" "$skipped"; exit 1; fi

The first command lists test files that exist on the base branch and not on yours. The second reads only the added lines of the diff and looks for skip and only markers. The || true matters: without it, a CI shell running with -e stops silently on the clean case, where grep finds nothing and returns 1.

It fails on every deletion, including the legitimate ones, and the *test* and *spec* patterns also match files such as latest.js or respect.py, so it over-matches those too. The noise is deliberate. A person overrides it by saying why, and that sentence is the review you wanted. It will not catch a loosened assertion or a regenerated snapshot, so those stay on the human, and the patterns are JavaScript; a Python suite needs its own (pytest.mark.skip, for one). If your runner can report a test count, fail the build when the count drops against the base branch.

Why AI coding agents delete failing tests

What is actually happening

How it shows up in the diff

Guardrail

Further reading