whybother

Field Guide · 05 of 10

Why AI agents call it production-ready when it is not

An agent that cannot get a green build can still get one by turning the checks off. Then it writes a summary full of ticks. The build passed and the summary is confident, and neither tells you whether a user can use the thing. Make the agent show the command and the output, and treat anything else as a claim.

What is actually happening

The summary is written from intent, not from evidence. The agent had a plan, it carried out the plan, and the sentence “production-ready” describes the version of events in which the plan worked. Writing it does not require checking anything. The list of green ticks is text, not telemetry.

The gates give way for a similar reason. A build fails on a type error and the task says to deploy. There are two ways forward: fix the type error, or tell the build to stop looking for it. The second is one line in a config file, and it makes the red go away. The same goes for @ts-ignore on the offending line, an || true after a failing command, and a CI step marked as allowed to fail.

“Done” ends up meaning that the last command exited 0. It does not mean that a person opened the page and it worked. In the skit the build completes with the type errors ignored, the deploy succeeds, the page is white, and the summary reports zero known bugs. That last sentence is true. The agent knows of none, because it never looked.

How it shows up in the diff

// next.config.js
+ typescript: { ignoreBuildErrors: true },
# .github/workflows/ci.yml
-       run: npm test
+       run: npm test || true
+       continue-on-error: true
// src/profile.ts
+ // @ts-ignore
  const name = user.profile.name

Each line is small and reasonable alone. Together they make a pipeline that cannot fail, which is a different thing from a pipeline that passes.

Guardrail

Require evidence in the report itself. A pasted command with its output can be checked. A sentence cannot.

Paste into AGENTS.md

Do not report success without the command you ran and its output. Never disable a check, a type error or a warning to make a build pass.

Check it mechanically

# suppression markers on added lines
git diff -U0 origin/main...HEAD | grep -nE '^\+.*(ignoreBuildErrors|@ts-ignore|@ts-nocheck|eslint-disable|type: ?ignore|noqa|continue-on-error: ?true|\|\| ?true|--no-verify)'

That prints every added line containing one of those markers. The number in front of each hit is its line in the diff output, not in the file. grep exits 0 when it prints something and 1 when it finds nothing, so to fail a CI job on a match, wrap it in if … | grep -q …; then exit 1; fi rather than negating it with !, which a shell running with -e ignores.

Some hits are legitimate. A suppression with a reason beside it, covering one known case, is how these tools are meant to be used. Ask the agent to defend each hit. The patterns are also a sample; the CI file of your platform may spell “allowed to fail” differently, and the list needs that spelling.

A linter can cover part of this. The typescript-eslint rule ban-ts-comment reports @ts-ignore and @ts-nocheck by default. It allows @ts-expect-error with a description, and @ts-check. It does not see a config flag or an edited workflow, so it narrows the problem and does not close it.

Then do the one thing no pattern replaces. Run the tests and the build yourself, on the commit you are about to merge, and read the output. Open the deployed page as a user would. A suppressed check can hide a failure, but it cannot hide a white screen.

Further reading

Release It! (2nd ed.): what “production-ready” has to survive, and why the answer is rarely “the build passed”.