The bugs that hide between two files: 4 AI code reviewers, 1 pull request

A guest opens their invoice and sees one total. Their card is charged a different one. Nothing crashed, no test failed, and every function did exactly what it was written to do. The bug isn't in any single file. It sits in the gap between two.

That's the kind of bug we wanted to test AI code reviewers on, so we built one on purpose. We gave the same pull request to Codzee, Greptile, CodeRabbit and CodeAnt to see who would notice.

The setup, in 30 seconds

• One PR: 192 lines across two modules of a guest-billing system (invoices, payments, tax, cancellations).
• 13 planted bugs. We wrote the answer key before opening the PR, so nobody could move the goalposts afterwards, including us.
• 6 of them were cross-file. Each module looks correct on its own. The bug only shows up when you read both against the contract that ties them together.
• Scoring: every finding pulled from the GitHub API and matched to the key by hand.

This was round six of an ongoing series. It's also the first round Codzee won, so we've been careful to include the parts where it didn't do well.

The scoreboard

The bug at the centre of it all

Here's the idea behind the bug that everything else was built around:

invoice = build_invoice(booking)   # built...
charge  = rates.compute(booking)   # ...ignored, then recomputed
gateway.charge(guest, charge)      # guest sees one number, pays another

The payment code builds the invoice, throws it away, and works out the charge again from a different module, leaving out tax and any extra lines. You can only catch it by noticing what the code doesn't do.

Only Codzee and Greptile caught it. Codzee marked it as its only Critical finding in the PR, which is the right call, because this is the one bug that silently bills a customer the wrong amount. CodeRabbit flagged the missing tax in that same function but never noticed the invoice being thrown away a few lines up. It saw the symptom and missed the cause.

Three things only Codzee caught

1. A declined card that counts as paid

The payment code never checks the gateway's receipt. If a card is declined, the invoice is still marked as settled and no money is taken. We hadn't planted this one.

2. A retry that charges twice

If a payment times out and gets retried, the code builds a new invoice with a new number. That changes the idempotency key, so the guest is charged twice. We hadn't planted this one either.

3. Tests that vouch for bugs

Three tests in the PR checked for the wrong behaviour, so the bugs showed up green. Codzee was the only tool that opened the test file, and it wrote down the correct expected value for each test: the due date should be 15 June, not 1 July, and an unpaid invoice should show 22,300 pence outstanding, not zero.

Credit where it's due

• Greptile matched our 6 / 6 on cross-file bugs using half as many comments. It's the most consistent tool in the series and hasn't had a weak round yet. It also found two real issues we hadn't planted: a silent fallback to the 20% UK tax rate, and payment records that disappear on restart.
• CodeRabbit gave the clearest fix advice of the round, naming both places where the tax needed to be added.
• CodeAnt is sharp on bugs that sit inside one function, such as an unsafe counter and a getter that lets callers edit the ledger. It's weaker when a bug means comparing two files.

What nobody caught

Three bugs got past all four tools: a mutable default argument, an import tucked inside a function that hid the recompute, and a rounding cut that loses a penny on every invoice. The first is something any linter flags.

Lesson: an AI reviewer is not a replacement for your linter.

Where Codzee fell short

Placement. Only 10 of Codzee's 18 findings were posted inline on the code. The rest went into the summary, including all three test findings, which are the ones that most need to sit next to the line they describe. Every other tool posted inline.

No merge verdict. Codzee's summary said “Changes requested”, but its GitHub check came back neutral, even with a Critical billing bug flagged. A neutral check can't block a merge.

We're working on both. Running the same test round after round is how we'll know when they're fixed.

Run this test on any tool, including ours

• Plant a bug in the gap between two files, for example one module quietly ignoring what another one returns.
• Plant one wrong test and see whether the tool even opens the test file.
• Count the comments, then count the ones you'd actually act on.
• Check where comments land: inline on the code, or buried in a summary.
• Check whether “Critical” really means critical, and whether the tool can block a merge.

The fine print: this was one 192-line PR in one domain, with bugs designed by our own team, and findings matched by hand. Tools change quickly, so these results reflect the day we ran the test. It's an honest comparison, not a statistically significant benchmark. You'll also see a Qodo comment on the PR: its trial had ended, so it didn't review and isn't scored here. The raw reviews are on GitHub, and the full round-by-round write-up is on Medium.