Automating Code Review with AI: What Worked and What Didn't

Code review takes time. Senior engineers burn hours switching between their work and PR reviews. I started using AI to handle the repetitive stuff—security checks, style issues, obvious bugs—so humans could focus on the parts that actually need human judgment.

#Where AI helps

I've been running this for about three months now. AI is pretty good at catching mechanical issues:

• Security problems: SQL injection, XSS, hardcoded credentials, insecure deserialization
• Common bugs: null pointer risks, off-by-one errors, obvious race conditions
• Style consistency: naming conventions, import order, dead code
• Missing documentation: public APIs without JSDoc, error descriptions that don't exist
• Dependency risks: known CVEs in packages you're adding, license issues

What it's bad at: architectural decisions, business logic validation, understanding performance implications in your specific context.

#The setup

name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]
 
jobs:
  ai-review:
    runs-on: ubuntu-latest
    permissions:
      pull-requests: write
      contents: read
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
 
      - name: Get PR diff
        id: diff
        run: |
          git diff origin/${{ github.base_ref }}...HEAD > pr.diff
          echo "diff_size=$(wc -l < pr.diff)" >> $GITHUB_OUTPUT
 
      - name: Run AI review
        if: steps.diff.outputs.diff_size < 2000
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
        run: node scripts/ai-review.js

I skip AI review on large diffs (2000+ lines). Accuracy drops on huge PRs, and honestly if your PR is that big you should probably split it up anyway.

#The review script

import Anthropic from "@anthropic-ai/sdk";
import { readFileSync } from "fs";
import { execSync } from "child_process";
 
const client = new Anthropic();
 
const diff = readFileSync("pr.diff", "utf-8");
const changedFiles = execSync("git diff --name-only origin/main...HEAD")
  .toString()
  .trim()
  .split("\n");
 
const prompt = `You are a senior software engineer reviewing a pull request.
 
## Changed files
${changedFiles.join("\n")}
 
## Diff
${diff}
 
Review this PR for:
1. **Security issues** — injection, XSS, hardcoded secrets, insecure patterns
2. **Bugs** — null/undefined risks, incorrect logic, edge cases
3. **Performance** — obvious N+1 queries, unnecessary re-renders, memory leaks
4. **Best practices** — error handling, naming, code organization
 
Rules:
- Only comment on issues that are clearly wrong or risky
- Do NOT suggest stylistic preferences or minor refactors
- Do NOT comment on unchanged code
- Be specific: reference the file and line number
- If the PR looks good, say so briefly
 
Format each issue as:
**[SEVERITY]** \`file:line\` — description`;
 
const response = await client.messages.create({
  model: "claude-sonnet-4-6-20250414",
  max_tokens: 2000,
  messages: [{ role: "user", content: prompt }],
});
 
const review = response.content[0].type === "text"
  ? response.content[0].text
  : "";
 
// Post as PR comment via GitHub API
execSync(`gh pr comment ${process.env.PR_NUMBER} --body "${
  review.replace(/"/g, '\\"')
}"`);

#Prompt engineering matters more than I expected

The prompt went through maybe 15 iterations before it worked well. A few things that helped:

#Tell it what NOT to flag

Without "Do NOT suggest stylistic preferences," the AI generates dozens of comments about variable naming and formatting. It clutters the review and developers start ignoring everything.

#Severity levels cut the noise

**[CRITICAL]** — Must fix before merge (security, data loss)
**[WARNING]**  — Should fix, potential bug or risk
**[INFO]**     — Suggestion, take it or leave it

Developers learned to always address [CRITICAL] and usually ignore [INFO]. Without severity, every comment felt equally urgent which meant nothing felt urgent.

#Managing context window

For large PRs (that still fit under the 2000-line limit), the script sends just the diff plus the full content of modified files. Not the entire repo.

function buildContext(changedFiles: string[], maxTokens: number): string {
  const fileContents: string[] = [];
  let estimatedTokens = 0;
 
  for (const file of changedFiles) {
    const content = readFileSync(file, "utf-8");
    const tokens = Math.ceil(content.length / 4);
 
    if (estimatedTokens + tokens > maxTokens) break;
 
    fileContents.push(`--- ${file} ---\n${content}`);
    estimatedTokens += tokens;
  }
 
  return fileContents.join("\n\n");
}

#Results after three months

Some rough numbers:

The main benefit isn't speed. It's that human reviewers focus on higher-level feedback now. The AI already caught the mechanical stuff.

#Things that go wrong

#False positives kill trust

If the AI flags too many non-issues, developers stop reading its comments. I track the false positive rate and tune the prompt when it goes over 15%. Trust is what makes this work.

#Cost management

Each review costs about $0.03 for average PRs on Sonnet. Large PRs with full file context can hit $0.50+. The 2000-line diff limit keeps costs predictable.

I checked last month's costs:

# Monthly cost tracking
echo "Reviews this month: $(gh api /repos/myorg/myapp/actions/runs \
  --jq '[.workflow_runs[] | select(.name=="AI Code Review")] | length')"

We spent maybe $40 total. Worth it.

#Don't review generated code

Auto-generated files produce noise. Exclude them:

exclude_patterns:
  - "*.generated.ts"
  - "*.lock"
  - "prisma/migrations/**"
  - "__generated__/**"
  - "*.min.js"

#What I've learned

The AI is good at catching obvious stuff—security issues, style problems, common bugs. Humans are better at architectural decisions and business logic. That split makes sense.

Prompt engineering matters way more than I expected. Same model, different prompt, completely different results.

Watch your false positive rate. If the AI flags too many non-issues, developers will start ignoring all its comments. I keep ours under 15%.

Skip AI review on large diffs. The accuracy just isn't there, and huge PRs should be broken up anyway.

Track costs and set limits. Context windows can get expensive if you're not careful.