Reviewing AI-generated code: a checklist for agent-written PRs

A pull request from a coding agent usually looks finished. The formatting is consistent, the names are descriptive, there are tests, and the description explains the change. When the code is wrong, the polish hides it. The call to a method that doesn't exist is as neatly formatted as the rest of the file, and so is the test that can't fail.

That changes what a reviewer should look for. Many of the signals reviewers use on human code, such as messy formatting, missing tests or odd naming, tell you little about agent output, because agents rarely get those things wrong. This post lists seven failure modes that do show up in agent-written code, with an example of each and what to check.

Why polished code needs a different review

Surface quality and correctness were loosely correlated in human code. Someone who writes careless code usually formats it carelessly too. Agent output breaks that link. The formatting is clean whether or not the logic is right.

Polished output also makes people overconfident. In a Stanford user study (CCS 2023), participants with an AI assistant wrote less secure code than those without one, and were more likely to believe their code was secure. Participants who questioned the assistant's output and reworked their prompts produced fewer vulnerabilities.

Volume matters too. The 2024 DORA report found that higher AI adoption was associated with a 7.2% reduction in delivery stability, and pointed to basics like small batch sizes and robust testing as what teams need to keep. Agents make large diffs cheap to produce, and a large diff is harder to review carefully.

The job in review is less about spotting sloppiness and more about spotting code that is plausible and wrong.

1. APIs and options that don't exist

Models learn APIs from a lot of code written against many versions of each library. They mix those versions up, or produce a call that fits the library's naming style but was never part of it.

import { promises as fs } from "node:fs";

// fs/promises has no exists(). This throws a TypeError the first time
// this line runs, not at import time.
if (await fs.exists(configPath)) {
  config = JSON.parse(await fs.readFile(configPath, "utf8"));
}

In a TypeScript project the typecheck catches that one. Options objects are harder, because an unknown key is often ignored without an error:

// node-fetch v2 had a `timeout` option. Native fetch ignores it, so the
// 5-second limit never applies.
await fetch(url, { timeout: 5000 });

// Native fetch takes an AbortSignal instead:
await fetch(url, { signal: AbortSignal.timeout(5000) });

The same happens with configuration keys, environment variable names, CLI flags and HTTP endpoints, none of which a compiler checks.

New dependencies need a separate check. A USENIX Security 2025 study of 16 code-generating models found that on average at least 5.2% of the packages suggested by commercial models, and 21.7% of those suggested by open-source models, didn't exist. Anyone can register a package under a name models tend to invent.

What to check:

• For every call into a library, confirm the method exists in the version in your lockfile. Jump to the definition rather than trusting the name.
• For options objects, config keys and flags, look each one up in the library's docs for that version.
• For a new dependency, confirm the package is the one you think it is: the right name, a real maintainer, recent releases.

2. Tests that can't fail

Agents write tests readily, and the tests pass. Some of them would keep passing if the code under test were broken. Two versions come up often.

The expected value is computed with the code being tested:

it("applies the spring discount", () => {
  const total = applyDiscount(100, "SPRING");
  // If getDiscountRate returns the wrong rate, both sides change together.
  expect(total).toBe(100 - 100 * getDiscountRate("SPRING"));
});

Or the mock supplies the answer, so the test only checks the wiring:

it("returns the user's plan", async () => {
  jest.spyOn(billing, "getPlan").mockResolvedValue("team");
  // getPlan is the function this test is supposed to cover.
  expect(await billing.getPlan("user_1")).toBe("team");
});

A test that pins the behaviour uses a literal expected value from the spec: expect(applyDiscount(100, "SPRING")).toBe(85).

What to check:

• For each test, ask whether it would fail if you broke the implementation. If you're unsure, break it and run the test.
• Look for expected values derived from production code, mocks of the module under test, and snapshots generated from whatever the code currently outputs.
• Check that the edge cases in the ticket have tests, not only the happy path.

3. Patterns from someone else's codebase

An agent that hasn't read enough of your repository falls back on the most common way to do something, which may not be how your codebase does it. The code works but doesn't fit.

// Agent-written route handler
import { PrismaClient } from "@prisma/client";

export async function GET(req: Request) {
  const prisma = new PrismaClient();
  const orders = await prisma.order.findMany();
  return Response.json(orders);
}

This repository already exports a shared client from src/db/client.ts. Prisma's docs say to create one instance and reuse it, because each PrismaClient has its own connection pool. The version above opens a new pool on every request, and it also skips the repository's usual error handling and authorization helper.

What to check:

• Compare the new code with how the codebase already handles data access, errors, logging, configuration and auth.
• When the PR introduces a new way of doing something, ask whether there's a reason or whether the agent didn't know about the existing one.
• When the same mistake keeps coming back, write the rule into your instruction file. AGENTS.md and CLAUDE.md are where coding agents pick up repository conventions.

4. More abstraction than the feature needs

Asked for one feature, agents sometimes build a framework for it: an interface, a factory, a registry and one implementation.

interface NotificationTransport {
  send(message: Notification): Promise<DeliveryResult>;
}

class EmailTransport implements NotificationTransport {
  /* the only implementation */
}

class NotificationTransportFactory {
  static create(kind: "email"): NotificationTransport {
    return new EmailTransport();
  }
}

The ticket asked for an email when an invoice is paid. A sendInvoicePaidEmail() function would do the job, and it would be easier to read, test and delete.

What to check:

• Compare the number of new files, interfaces and layers with the size of the feature.
• Be skeptical of an interface with a single implementation, or configuration nobody asked for.
• Ask for the simple version. Adding a layer later, when a second case exists, is cheap.

5. Only the happy path

Agent code usually handles the case the prompt described. The inputs and failures around it get less attention.

// maxRetries: 0 becomes 3, because 0 is falsy.
const maxRetries = options.maxRetries || 3;

// One failed charge rejects the whole batch, and the charges that did
// succeed are never recorded.
const results = await Promise.all(invoices.map((inv) => charge(inv)));
await markPaid(results);

Neither line throws in a test that uses sensible inputs. The first should use ??. The second needs Promise.allSettled, or per-item handling that records which charges went through.

What to check:

• Trace the unhappy paths for each new function: missing or zero values, empty results, timeouts, retries, partial failure and concurrent calls.
• Check how failures are reported. Agents often catch an exception and return null or an empty array, which turns an error into wrong data.

6. Business logic that looks right

This one needs knowledge the agent doesn't have. The rules live in a spec, a ticket or a conversation, and an agent that doesn't have them fills the gap with a guess that looks reasonable.

const subtotal = sum(cart.items.map((i) => i.price * i.quantity));
const withTax = subtotal * (1 + taxRate);
return withTax * (1 - discountRate); // the spec applies the discount before tax

// Trials are 14 days, so day 14 is still in the trial.
if (daysSinceSignup >= 14) lockAccount(user); // should be > 14

If the agent also wrote the tests, they will usually encode the same assumption, so a green CI run tells you nothing here.

What to check:

• Read the change against the ticket or spec, not against your sense of what looks right.
• Pay particular attention to the order of operations, boundaries (> or >=), units (per second or per minute), rounding, and anything involving money, permissions or limits.

7. Outdated APIs and libraries

Models favour what was common in their training data, and some of that has since been deprecated or removed.

import moment from "moment";
import ReactDOM from "react-dom";

const trialEnds = moment().add(14, "days");
ReactDOM.render(<App trialEnds={trialEnds} />, document.getElementById("root"));

Moment.js describes itself as a legacy project in maintenance mode, and React 19 removed ReactDOM.render in favour of createRoot from react-dom/client. On React 19 this code fails. On an older version it works and adds a migration you'll have to do later.

What to check:

• For new dependencies, check the maintenance status and the latest release date.
• For framework APIs, check the current docs for the version you run, and watch for deprecation warnings in the test output.
• Be strictest in security-sensitive code, where an outdated API can mean a known vulnerability.

How much to trust the PR

Not every agent PR needs the same depth of review. These signals are worth looking for before you start. They're judgement calls, not rules:

• Was it verified? An agent that ran the build, the typecheck and the tests, and fixed what failed, has already removed a lot of item 1. A PR where the agent never ran anything needs a full read.
• Did it have context? Agents that can search the repository and read an instruction file match existing patterns more often. That helps with item 3 but does nothing for item 6.
• How big is it? Review a 50-line change line by line. For a 2,000-line change, the most useful comment may be a request to split it.
• What does it touch? Billing, auth, permissions, migrations and anything that deletes data deserve a full review whoever wrote the code.

The checklist

Keep this open during reviews:

1. APIs and options: Does every method, option, config key and flag exist in the version you run? Is every new package real and maintained?
2. Tests: Would each test fail if the code were broken? Are expected values literals from the spec, and are the mocks outside the unit under test?
3. Patterns: Does the code use the repository's existing helpers and conventions?
4. Abstraction: Is the structure proportional to the feature?
5. Edge cases: What happens with zero, empty, missing, slow, failing and concurrent inputs?
6. Business logic: Does it match the spec, including order, boundaries, units and rounding?
7. Freshness: Are the APIs and libraries current for the versions you run?

Checking whether your review process works

A few things are worth tracking over time. None has a threshold we'd call right for every team; look at the trend.

• Defects traced back to agent PRs, compared with human PRs, using whatever you already use for incidents or change failure rate. This is the metric that tells you whether review is catching problems.
• Revision requests tagged by failure mode. After a month you'll know which of the seven items above your agents actually get wrong, and where to spend review time.
• Review time and PR size together. Agent PRs getting bigger while review time stays flat means each line gets less attention.

When one failure mode keeps showing up, fix it at the source: add a rule to the repository's AGENTS.md or CLAUDE.md so the next agent run doesn't make the same mistake.

Where an AI reviewer fits

An automated reviewer reviews every agent PR the same way, however tidy the code looks. It doesn't replace the human pass on business logic, because it doesn't have the spec unless you give it one. Very large PRs can exceed its context window, which is one more reason to keep agent PRs small.

Tenki Code Reviewer is a GitHub App that reviews pull requests. With Automatic Review on, it runs when a PR is opened or updated, and you can trigger it on any PR by mentioning @tenki-reviewer in a comment. It posts a summary with findings grouped by severity and a review comment that ends in a recommendation: approve, approve with minor suggestions, or changes requested. On our own benchmark of real bugs from 50 open-source pull requests, it caught 84 of 122 bugs (68.9% recall), the highest of the seven tools tested. It also posts the most comments, and its precision is about 0.30. The severity threshold filters which findings appear; all levels are on by default.

It reads your repository's AGENTS.md or CLAUDE.md as custom context, on top of its standard bug, security and quality checks. That means you can turn parts of this checklist into standing instructions:

## Always flag

- New dependencies in `package.json`; say what they're for and whether they're maintained
- `new PrismaClient()` anywhere outside `src/db/`
- Tests that mock the module they are testing, or compute the expected value with production code
- `Promise.all` over external calls where one failure would drop the other results

## Focus areas

- `src/billing/**`: check the order of discount and tax, rounding, and currency handling

Coding agents that read the same file get the rule too, so it can prevent the mistake as well as flag it.

Reviews cost $1.00 each from workspace credits. The reviewer doesn't post a status check. If you want its result to affect merging, turn on Approve on Clean Review, which approves a PR when the review finds nothing blocking, and combine it with a branch rule that requires an approval. For agent-written code, also require a human approval, for example with the rule's "Require review from Code Owners" option. Setup takes a few minutes with the quickstart.