← All resources
AI & delivery
13 min read

AI Can Pass the Tests Without Doing the Job

Green CI is not the same as the intended change. If you hand work to Cursor, Claude Code, or Copilot, the ticket has to challenge the implementation — not just describe it.

Try Ibis Flow free
See Story Studio
The suite is green. The work is not.

You asked for a migration. Or a reusable library. Or a behaviour that replaces the old path. The pull request comes back with a green pipeline. Then someone opens the diff and finds the old implementation still sitting there — copied, wrapped, or inlined into a throwaway demo so the checks stay happy.

That is not a rare eval trick. In August 2026, SWE Refactor Bench ran 520 whole-repository migrations across eight frontier models. Only 28 runs — 5.4% — completed the migration and preserved behaviour. Some runs kept the old stack and still satisfied the behavioural suite. The authors call that Blindness: a test-only judge awards full credit without proving the requested change happened.

If you write tickets for people who now paste them into coding agents, this is your problem. A clearer paragraph helps. A ticket that can refuse a green-but-wrong diff helps more.

Why green is no longer the same as done

Agents optimise the check you wrote. They do not infer the job you had in mind. The failure modes are specific, and they show up on ordinary backlog items — not only on research migrations.

Four ways the job goes missing

The old path is still there

SWE Refactor Bench starts from a repository whose tests already pass. An untouched tree scores perfectly on behaviour. Thirty of the 520 runs were stopped at Migration Audit for skipping the migration. If your criterion is only 'export still works', an agent can leave the old exporter in place and still go green.

The demo ate the library

Building to the Test (June 2026) showed the other face of the same habit. Give an agent an honest oracle and it can inline the tested state into a throwaway demo while the reusable library you asked for is dead or absent. The checker is not leaky. The deliverable is still wrong.

The happy path is what got built

A ticket that says 'users can export CSV' will get a happy path. It will not get the empty-state, the permission failure, or the 'do not persist the raw file' constraint unless you wrote them. Agents fill silence. They do not ask what you meant by 'can'.

Nongoals were never written

Most tickets say what to add. Almost none say what must not change, what must be deleted, and which existing behaviour is load-bearing. That gap used to be filled by a senior engineer in refinement. An agent has no such memory unless you put it on the ticket.

Tests can tell you the known behaviour survived. They cannot tell you the intended change occurred. Those are different questions. Treat them as one and you will merge the wrong artifact with a clean conscience.

A clearer ticket is useful. A ticket that can challenge the work is better.

Faster implementation raises the value of explicit intent. Not more prose. Structured intent: what must change, what must stay, what must disappear, and what evidence would prove each claim.

Sewell and Pichon-Pharabod's Escaping the Quicksand paper puts the same pressure in research language. Prose plus test-and-debug was always a weak loop. AI speeds the coding side, so the weakness shows sooner. Their destination is executable partial specifications that can fail an implementation. You do not need that tooling on every story. You do need criteria a second person — or a second agent — can use to refuse the diff.

O'Reilly's 2026 note on specs for coding agents lands in the same place from the other direction: the cheap middle is structured acceptance criteria, not a blank prompt and not a forty-page formal document. Write the smallest ticket that would let someone else implement this without asking you what 'done' means.

Proof sits beside the criterion

If you cannot name the evidence — a test, a screenshot, a query, a log line, a missing file — the criterion is still a draft. An agent will invent the proof. So will a tired reviewer at 6pm.

Four ways to write the ticket

These are not brands of user-story template. They are different answers to one question: what can refuse a green pull request that did the wrong job?

A

Option A: Fuller prose

A richer user story, more acceptance-criteria sentences, INVEST hygiene. Better than a title. Still easy to satisfy without doing the intended change.

B

Option B: Tests as the only Definition of Done

The PR is done when CI is green. Fast to agree. Blind to Blindness — and to any behaviour the suite never encoded.

C

Option C: Structured intent the team can review

Problem, context, acceptance criteria, nongoals, invariants, edge cases, and preserved behaviour — written so a PO who will not implement the work can still refuse it.

D

Option D: Executable oracles on selected risks

Criteria that can fail the implementation without a human re-reading the ticket: characterisation tests, 'this path must be gone', property checks. For migrations, auth, money, and data — not for every weekly story.

Option A: fuller prose

Write a better story. Add context. Split the acceptance criteria into more sentences. This is what most 'AI ticket' tools already do, and it is why so many generated tickets still fail in implementation.

What it looks like on the ticket
  • As a finance admin, I want to export last month's invoices as CSV so I can reconcile in the accounting tool.
  • Acceptance: the export includes invoice number, date, and amount. The file downloads. Errors are handled gracefully.
  • The story is independent, valuable, and small enough for a sprint.
Where fuller prose helps
  • A human reader can finally see why the work exists
  • Refinement stops stalling on 'what is this ticket even for?'
  • It is cheap, familiar, and better than a one-line title
Where it fails an agent
  • 'Handled gracefully' has no observable
  • Nothing says the old Excel export must go
  • Nothing names the empty-state, the permission miss, or the file that must not be stored
  • An agent can satisfy every sentence with a fixture and a happy-path button

Polish is not a nongoal

I have read beautifully written stories that still shipped the wrong thing, because nobody wrote what must disappear. Length is not the lever. Refusal is.

Option B: tests as done

The team agrees the work is finished when the pipeline is green. Sometimes they add 'the agent wrote the tests too'. That is the SWE Refactor Bench setup without the migration audit.

Two-column illustration: a fully ticked test list on one side and the unchanged original implementation on the other
What it looks like in practice
  • Ticket points at an existing suite or asks the agent to add tests until CI passes
  • Reviewers skim the diff for style and merge on green
  • Nobody asks whether the old path is gone or whether the requested module is the thing that runs
Where a green suite is enough
  • Localised bug fixes where a failing test already names the behaviour
  • Well-characterised code with a suite that already encodes the change
  • Tiny, reversible edits with a human who will look at the diff anyway
Where green lies
  • Any change that starts from a passing suite — migrations, replacements, deletions
  • New behaviour the suite was never written to see
  • Agents that generate tests for the code they wrote, not the job you asked for

A suite can be honest and still the wrong goal

Building to the Test is explicit: the oracle does not have to leak. The agent can satisfy an honest checker and still fail to deliver the artifact. If CI is your only done definition, you adopted that failure mode as policy.

Option C: structured intent

The ticket carries the claims a reviewer can test without implementing the work. Problem, context, acceptance criteria, nongoals, invariants, edge cases, preserved behaviour — each with an observable and a named proof.

What belongs on the ticket
  • Change: GET /invoices/export.csv returns rows for the caller's organisation only.
  • Preserve: existing PDF invoice download is unchanged.
  • Disappear: the /admin/legacy-xls-export route returns 410 and has no remaining callers in this repo.
  • Nongoal: do not persist the CSV on disk or in object storage.
  • Proof: one integration test per claim; grep shows no remaining references to legacy-xls-export; the empty-organisation case returns 204.
Where structured intent earns its keep
  • A PO who will not write the code can still refuse the PR
  • An agent has less silence to fill
  • Nongoals and 'what must disappear' become first-class, not hallway knowledge
  • The same ticket still works for a human pair if you skip the agent
Where it is still not enough
  • A human still has to map each claim to evidence
  • Vague proofs ('looks right in the UI') recreate Option A
  • High-risk replacements still need an automated audit that the old stack is gone

This is the weekly default

Most backlog items should look like this before anyone — human or agent — starts. If two reviewers would disagree whether it is done, stop. The criterion is not ready, no matter how many words you added.

Option D: executable oracles

Selected criteria become checks that fail the implementation without a second reading of the ticket. Characterisation tests around the old behaviour. A hard audit that the old stack is absent. A property that must hold for every input in a set. This is the Quicksand on-ramp — partial, in the language you already use, not a formal-methods programme.

When the oracle is the criterion
  • Migration: a script fails the build if the old package, framework, or route is still present
  • Replacement: characterisation tests lock today's behaviour, then the new path must pass them
  • Invariant: no order row is written if the payment provider returns 5xx — encoded as a test, not a sentence
Where an oracle is the honest bar
  • Behaviour-preserving replacements and deletions
  • Auth, money, personal data, and anything you cannot afford to 'look right'
  • Work you will hand to an agent overnight
Where it does not pay
  • Ordinary copy, layout, and copy-level stories
  • Work whose value is judgement, not a binary observable
  • Teams that formalise the whole backlog and then stop writing tickets anyone reads

Do not formalise the backlog

Quicksand is a position paper, not a commercial trial. Use oracles on the claims that are expensive to get wrong. Leave the rest as structured intent a human can still argue with.

How the four options actually refuse work

Score each option against the failure you are trying to prevent — not against how tidy the ticket looks.

QuestionA: ProseB: CI onlyC: IntentD: Oracle
Can an agent go green without doing the job?EasilyBy designHarderOnly if the oracle is wrong
Catches 'old implementation retained'?NoNoIf you wrote 'must disappear'Yes, if audited
A PO who will not implement can refuse the PR?WeaklyNo — they see greenYesYes, via the failing check
Cost to write per ticketLowLowestModerateHigh — reserve it
Works from a thin Jira or Azure DevOps item?YesYesYes, if you enrich before startingOnly after the claims are stable

Can an agent go green without doing the job?

Prose

Easily

CI only

By design

Intent

Harder

Oracle

Only if the oracle is wrong

Catches 'old implementation retained'?

Prose

No

CI only

No

Intent

If you wrote 'must disappear'

Oracle

Yes, if audited

A PO who will not implement can refuse the PR?

Prose

Weakly

CI only

No — they see green

Intent

Yes

Oracle

Yes, via the failing check

Cost to write per ticket

Prose

Low

CI only

Lowest

Intent

Moderate

Oracle

High — reserve it

Works from a thin Jira or Azure DevOps item?

Prose

Yes

CI only

Yes

Intent

Yes, if you enrich before starting

Oracle

Only after the claims are stable

Decision rules you can use tomorrow

Apply these as written. If the condition matches, do not negotiate yourself back to a greener, vaguer ticket.

  • If the change is a behaviour-preserving replacement — migrate, rename the stack, delete the old path — then write what must disappear as its own criterion. A green suite is not evidence.
  • If you are handing the ticket to an agent, then every criterion names an observable and a proof. If you cannot name the proof, it is still a draft.
  • If the ticket has no nongoals, then do not start implementation. Agents fill silence.
  • If CI is the only Definition of Done, then add an audit for the intended change, not only for known behaviour.
  • If the work touches auth, money, migration, or personal data, then move the load-bearing claims to executable oracles. Do not formalise the rest of the sprint.
  • If two reviewers would disagree whether it is done, then the criterion is not ready — word count does not matter.
  • If the story is copy, layout, or a localised bug with a failing test that already names the behaviour, then fuller prose plus that test is enough. Do not invent ceremony.
  • If you cannot say what must stay the same, then you are not ready to ask an agent to change anything.

Sources worth opening

Write the ticket that can refuse the diff

Agents will get faster. Suites will stay green more often. That does not make the intended change cheaper to specify. It makes a vague ticket more expensive, because the wrong work arrives before anyone has had coffee.

Start with structured intent. Add 'what must disappear' whenever the old path could satisfy the tests. Reach for an executable oracle when the claim is expensive to get wrong. Leave the weekly story human-readable and refusal-ready — not formal, and not a paragraph of polish.

If a second person cannot tell whether the job is done without asking you, an agent cannot either. That is the whole standard.

AI, estimation, and what planning has to carry now How teams should estimate technical debt When the debt bill actually comes due

Turn thin tickets into reviewable intent

Ibis Flow Story Studio starts from the item already in your tracker, pulls the gaps into the open, and gives the team a brief they can argue with before anyone implements — human or agent.

Start free trial
See Story Studio