Why AI code review is a different job
Code review as most teams practise it was built around assumptions about human authors. Mistakes cluster in predictable places: the code written at the end of a long day, the unfamiliar corner of the framework, the feature rushed for a deadline. A developer who handled authentication carefully on one endpoint has probably handled it carefully on the next. The polish of the code carries information about the care behind it. Reviewers lean on all of this without noticing, and for human-written code it mostly works.
None of it holds for AI-generated code. A model produces its best and its worst code in the same style, in the same minute, with no fatigue signature to guide a reviewer's attention. Getting a control right in one file says little about the next file, because each generation is a fresh attempt shaped by whatever context happened to be nearby. The volume changes the economics too. An assistant can produce in an afternoon what a team once wrote in a month, so reading every line is no longer a realistic control. Review has to become risk-targeted: know where this class of code typically fails, and check those places deliberately.
There is also a bias baked into the training data. Models learn from public code, and a large fraction of public code is example code, written to demonstrate a concept in the smallest number of lines. Example code disables certificate verification to keep the demo simple, allows every origin in its CORS configuration, hardcodes the API key so the reader can see where it goes, and skips the error handling that would obscure the point. That style transfers. AI-generated applications routinely ship with tutorial-grade defaults in exactly the places where production code needs the opposite.
The pattern of confident wrongness
A human developer who is unsure signals it. The code turns tentative, comments appear asking whether an approach is right, questions surface in the pull request. A model has no such register. It produces an insecure implementation with the same fluent assurance as a secure one, and often decorates it with comments announcing the very property it lacks. It is common to find a comment reading "securely hash the password" above a fast unsalted hash, or a function named validateInput whose body accepts nearly anything.
This creates a specific reviewer trap. The instincts that make a senior engineer good at reviewing human code, noticing when something looks careless, sensing when the author was out of their depth, fail here because the surface always looks careful. The code reads well, the names are right, and the structure is idiomatic. Whether the authorisation check actually binds to the object being fetched is invisible at reading speed, and reading speed is how most review happens.
The pattern extends to the conversation around the code. Ask an assistant whether its output is secure and it will usually say yes, with reasons. Push back and it will often revise with equal confidence in the opposite direction. Its own assurance is testimony from the party under examination and carries no evidential weight. The practical consequence is that verification has to move from reading code to exercising behaviour. A control counts as present when it fires under a real attempt, and a review that stops at the text of the code has only verified that the text exists.
Automated scanners help less than teams expect here. A static analysis tool will catch some hardcoded secrets and some injection patterns, and it is worth running one, but the highest-severity rows in the table below are mostly invisible to it. Whether a query filters by the current user's ownership is business logic, and no scanner knows your business. The same applies to a session that never really dies or a bucket that was meant to be private. Automated tools narrow the field, but a person still has to walk it.
The eight places it goes wrong
Across the AI-assisted codebases we build and the ones we review, security defects concentrate in the same eight areas. The table below is the centrepiece of this guide. For each area it states what generated code typically does, what a pre-ship check should establish, and roughly how bad the miss is if nobody looks.
| Risk area | What AI-generated code typically does | What to check | Severity if missed |
|---|---|---|---|
| Authentication and session handling | Rolls its own login, long-lived tokens, weak session invalidation | Proven library or identity provider; logout and expiry actually revoke | Critical: account takeover |
| Authorisation and object references | Checks who you are, rarely what you may access | Every endpoint enforces ownership; try another user's record ID | Critical: cross-tenant data exposure |
| Secrets handling | Hardcodes keys, commits .env files with real values | Nothing sensitive in repo, history or client bundle | Critical: credential theft, cloud takeover |
| Dependency selection | Picks stale, abandoned or invented package names | Every package exists, is maintained and earns its place | High: supply chain compromise |
| Input validation and injection | Validates the happy path; string-builds queries and commands | Parameterised queries; server-side validation on every input | Critical: injection, wholesale data theft |
| Data storage defaults | Broad database roles, no encryption decisions, keeps everything | Least-privilege roles; know what is stored, where, how long | High: larger breach, Privacy Act exposure |
| Error handling and information leakage | Returns stack traces and internal detail to callers | Generic errors outward; detail stays in server logs | Medium: maps your internals for attackers |
| Infrastructure and deployment defaults | Copies permissive tutorial configs; open CORS, debug on | CORS locked, storage private, debug off, TLS enforced | High: direct public exposure |
Eight risk areas in AI-generated applications, what to check, and the cost of a miss
Two rows worth expanding
Dependency selection is the least intuitive row. Models sometimes suggest packages that do not exist, inventing plausible names from the patterns of real ones, and attackers have noticed: registering those hallucinated names on public registries is now an established squatting technique. A dependency added by an assistant deserves a minute of scrutiny a human-chosen one might not need. Confirm the package is the one you think it is, check when it was published and by whom, and prefer the widely used option over an unfamiliar package with a perfect name.
The severity column carries one Australian footnote. If your organisation is covered by the Privacy Act 1988 (broadly, businesses with annual turnover above three million dollars, plus some categories such as health service providers regardless of size), a miss in the critical rows is how an eligible data breach happens. The Notifiable Data Breaches scheme then requires notification to the OAIC and to affected individuals where the breach is likely to result in serious harm. The fact that a tool generated the code changes none of this. Accountability sits with the organisation that shipped it.
What a lightweight pre-ship review covers
None of what follows needs a security specialist. It needs a competent developer, a few focused hours, and the willingness to attack your own application before someone else does. The list is ordered roughly by return on effort.
Two habits make the checklist worth more than the hours it takes. First, run it at the right moments: before the first release, and again after any large generated change, because a regeneration can silently reintroduce a defect you already fixed. Second, write down what you checked and what you found, even as a few lines in the repository. A finding in the critical rows of the table is a stop-ship until it is fixed, and the written record is what lets you say so later with a straight face to a customer, an insurer or a regulator.
- Log in as one user and request another user's records by changing IDs in URLs and API calls. Every endpoint that returns data should refuse.
- Search the repository, its full git history and the built client bundle for keys, tokens and connection strings. Rotate anything you find, then move it to a secrets manager.
- Confirm every dependency exists on the public registry, is actively maintained and is actually needed. Commit the lockfile and build from it.
- Feed hostile input to every form and API: quotes, angle brackets, very long strings, unexpected types. Watch for errors that echo a query or crash the server.
- Trigger failures on purpose and read the response in the browser. Stack traces, framework versions and file paths should never leave the server.
- List every storage location the application touches and check who can reach it. Buckets and containers stay private, and the database role holds only what the application needs.
- Sign out, then replay the old session token against the API and confirm it is refused. Check how long tokens live before they expire on their own.
- Turn off debug mode, confirm TLS is enforced everywhere, and read the CORS configuration the way an attacker would.
What the checklist cannot tell you
This checklist catches the failure classes that dominate AI-generated code, and a team that runs it before every release will ship materially safer software than one that trusts the generation. It also has clear limits. It confirms that the common doors are locked. It does not tell you whether a determined person can chain three small findings into one large one, whether your tenant separation holds under authenticated attack from a paying customer, or what your specific data model makes possible for someone who studies it for days. Those questions need adversarial depth and time that a checklist cannot supply. Answering them is engagement work.
There is a useful way to frame the boundary. The checklist establishes the presence of controls. An engagement establishes their strength under pressure, and pressure is the part you cannot apply to yourself, because you know where you already looked. The value of an external reviewer is partly skill and partly the simple fact that they arrive without your assumptions.
When to bring in a professional review
Some situations justify more than self-assessment. Bring in a professional review when the application handles personal information at meaningful scale, when it moves money or stores card data (PCI DSS obligations attach to cardholder data wherever it lives), when it serves multiple tenants whose separation is a commercial promise, or when a regulator has a view: APRA-regulated entities carry information security obligations under CPS 234, and organisations pursuing ISO 27001 or SMB1001 certification will need evidence that secure development practice exists in fact. An enterprise customer's security questionnaire is often the forcing event, and it costs less to review before the questionnaire arrives than after it.
A professional review starts where the checklist stops. Reviewers work with the codebase and a running environment, exercise the application as an authenticated attacker across roles and tenants, and chase the findings a scanner labels informational until they either chain into something real or are ruled out. Exploitation depth is the difference: demonstrating that the object reference flaw reaches another customer's records, or that a leaked path, a stale dependency and a permissive CORS policy combine into an account takeover. That demonstration turns a list of observations into decisions a business can rank and fund.
Black Shard builds with AI coding tools daily and reviews the code they produce, ours and other people's. The checklist in this guide is the layer we think every team should run for itself, and nothing in it is secret. The depth behind it, the exploitation work, the architecture review and the remediation that follows, is engagement work, because it changes with every codebase. If you have shipped something an assistant wrote and the table above raised doubts, run the checklist first, then weigh what the application handles and decide how much assurance it deserves.