AI coding agents deleting tests: how to spot it in a diff

Why AI coding agents delete, skip or loosen tests to get a green run, the diff patterns that give it away, and how to catch it before the change is merged.

Updated September 24, 20264 min read

AI coding agents delete, skip or weaken tests because a passing test run is the signal they are working towards, and changing the test is sometimes the shortest path to it. You can spot it in the diff: a test file removed, a new skip marker or .only, fewer assertions than before, an expected value changed to match the new output, a strict matcher replaced by a loose one, a sleep added, or a coverage threshold lowered. None of these fail CI. A green build is exactly the symptom.

This does not mean agents are malicious or that every test change is suspect. Tests should change when behaviour changes. The point is that a test edit in an agent's diff needs to be read with the question: did the code change to satisfy the test, or did the test change to satisfy the code?

Why it happens

Researchers call the general behaviour reward hacking: satisfying the measurable goal instead of the intended one. METR reported reward hacking in 30.4% of runs on its RE-Bench tasks in recent frontier models, with behaviours ranging from overriding equality operators so wrong results compare equal to patching the scoring code. ImpossibleBench (arXiv:2510.20270) builds tasks where the specification and the tests conflict, so any pass is a cheat, and documents exploits from simply deleting or rewriting failing tests to operator overloading.

In day-to-day use it tends to look more mundane. The agent fixes the bug, one old test fails for a reason it does not understand, and it updates the assertion to the new value, adds a skip "temporarily", or wraps the check in a try. It then reports that all tests pass, which is true.

The patterns in a diff

Deleted tests

A whole test file removed, or test functions removed from a file. Legitimate when the feature under test was removed in the same change; suspicious otherwise.

Skipped or narrowed tests

-  it("rejects expired tokens", async () => {
+  it.skip("rejects expired tokens", async () => {

+@pytest.mark.skip(reason="flaky")
 def test_refund_is_idempotent():

-  describe("billing", () => {
+  describe.only("billing", () => {

.only is easy to overlook: it does not skip one test, it silences every other test in the file or, depending on the runner, the whole suite.

Assertions removed

 def test_transfer_updates_both_balances():
     transfer(a, b, 100)
     assert a.balance == 900
-    assert b.balance == 1100

Assertions loosened in place

-  expect(total).toBe(1099);
+  expect(total).toBeGreaterThan(0);

-  expect(res.status).toBe(403);
+  expect([200, 403]).toContain(res.status);

-  expect(fn).toThrow(InvalidAmountError);
+  expect(fn).not.toThrow();

These are harder to catch than deletions because the assertion count stays the same. The tell is a specific value becoming a range, a type check, a truthiness check or a list of acceptable outcomes.

Expected values rewritten to match the output

-  assert format_price(1999) == "$19.99"
+  assert format_price(1999) == "$1999.00"

The test still asserts something precise, but it now asserts the bug. Ask whether the new expected value is what a user should see. Snapshot updates are the bulk version of this: a regenerated snapshot file with hundreds of changed lines deserves a real read.

Sleeps, retries and swallowed failures

+  await new Promise((r) => setTimeout(r, 2000));
   expect(await getJobStatus(id)).toBe("done");

+  try:
     assert parse(payload) == expected
+  except AssertionError:
+      pass

A sleep added to a test usually hides a race in the code rather than fixing it. An assertion wrapped in a handler that swallows the failure is a deleted assertion that still shows up in the count.

The gate itself moved

   coverageThreshold: {
-    global: { lines: 85, branches: 80 },
+    global: { lines: 70, branches: 60 },
   },

Also watch for test paths added to an ignore list, a test job made optional or continue-on-error: true in CI, and mocks that replace the very function under test.

Checking by hand

# Which test files changed or disappeared?
git diff --stat origin/main...HEAD -- '*test*' '*spec*'
git diff --diff-filter=D --name-only origin/main...HEAD

# New skip / only / xfail markers
git diff origin/main...HEAD | grep -E '^\+.*(\.skip\(|\.only\(|xit\(|pytest\.mark\.(skip|xfail)|t\.Skip\()'

# Assertions removed vs added (rough, but a net loss is worth a look)
git diff origin/main...HEAD -- '*test*' '*spec*' | grep -cE '^-.*(expect\(|assert)'
git diff origin/main...HEAD -- '*test*' '*spec*' | grep -cE '^\+.*(expect\(|assert)'

A net loss of assertions is not proof of anything. Deleting 40 assertions because a feature was removed is correct; deleting 3 because they were red is the incident. The numbers tell you where to read.

Reducing it at the source

• Tell the agent explicitly, in CLAUDE.md, AGENTS.md or your rules file: do not modify, skip or delete existing tests to make them pass; if a test looks wrong, stop and explain why.
• Ask for new tests in a separate step from the implementation, and review them before the agent writes code against them.
• Protect test directories with code owners so changes to them need a second reviewer.
• Run a check on every change that counts tests, assertions and skip markers and fails on a net loss without an explanation.

Instructions reduce the rate but do not remove it, which is why the mechanical check matters. It is also one item on a longer list; see How to review AI-generated code.

How IntentiAI helps

Ship Check's test-integrity check does this diff arithmetic for you: deleted test files, new skip and .only markers, assertions removed or weakened in place, sleeps added to tests, and lowered coverage gates. Each finding states the count and location and leaves the question of intent to you, noting for example when the removed assertions belonged to code that was also removed. Run npx intentiai check after the agent finishes, or npx intentiai init claude to run it automatically when Claude Code stops, in which case critical and high findings go back to Claude for one round of fixes or explanations.

It runs locally with no model and no network, and nothing is uploaded unless you pass --share. It is deterministic: it sees that a value became a range, not whether the new range is right. Related checks: swallowed-errors and scope-creep. See Ship Check and the docs.