Reviewing AI-generated code before it ships

Assistants hallucinate packages. Ask for an integration and the generated code may import a library that has never existed, under a name plausible enough that nobody blinks. The developer installs whatever makes the import resolve, or the assistant helpfully adds it to the manifest. Attackers have noticed: researchers have shown that models repeat the same invented package names often enough that registering those names on npm or PyPI is worthwhile, so the package that resolves can be a payload an attacker registered and left waiting. Because npm runs install scripts by default, the first machine to execute it is a developer laptop or a CI runner holding deploy tokens.

The subtler version is staleness. A model's knowledge of the ecosystem is a snapshot, so it recommends the library that was popular in its training data instead of the one your codebase already uses, pins versions that carry since-patched vulnerabilities, and writes against APIs that were deprecated a year ago. None of this fails loudly. The build passes and the dependency tree quietly grows another maintainer you have never heard of. In an AI-heavy diff, every new dependency needs a human to confirm three things before merge: the package exists and is the one you think it is, checked against its repository and publish history; the version is current and maintained; and the codebase had not already solved the same problem with something you own.

Example code on the public internet puts credentials inline, because examples are written to run. Assistants learned from that corpus, so they scaffold the same way: a connection string in the constructor, an API key as a default parameter, a token pasted into a test fixture. The placeholder becomes a real value the moment someone needs the code to work, and once committed it lives in git history whether or not a later diff removes it. Any live credential found in a generated diff should be rotated on discovery, because scrubbing history is far harder than issuing a new key.

Boundary defences suffer the same way. Asked to fix a CORS error, an assistant reaches for a wildcard origin or for code that reflects whatever Origin header arrives, because those are the most common answer shapes in its training data. A reflected origin combined with credentialed requests lets a hostile page act as your logged-in user, and the change reads like harmless configuration to anyone skimming.

Authorisation is worse, because it is invisible by omission. Ask for an endpoint that returns an invoice by id and you will get exactly that: a handler that fetches by id, with no check that the caller owns the invoice. The assistant will copy authentication middleware if neighbouring routes show the pattern, and it will still skip the object-level check, because who may see what is a fact about your business that no model can infer from a prompt. Input validation goes missing for the same reason. Generated code parses the input it expects, and the attacker sends the input it does not.

Cryptography is where generated code is most confidently wrong. Common output includes passwords hashed with MD5 or a bare SHA-256 instead of a dedicated password-hashing function such as bcrypt or Argon2, encryption in ECB mode or with a hardcoded IV, session tokens minted from a general-purpose random number generator, secrets compared with ordinary string equality that leaks timing, and hand-rolled JWT handling that never verifies the signature. Each of these patterns appears constantly in public example code, so the model reproduces them fluently, usually with a reassuring comment attached.

The workable rule is short. Generated cryptography does not ship until someone who can explain each primitive choice has read it, and the default fix is to replace hand-assembled constructions with the platform's high-level library, which encodes those choices correctly. If nobody on the team can perform that review, that gap is itself the finding, and it marks the point where outside review stops being optional.

Treat generated code the way you already treat a dependency: third-party code that nobody in the organisation has read yet, arriving inside your own pull request. Provenance is a legitimate review input, so make it normal for authors to say which parts of a diff were generated, and route effort accordingly. Attention goes first to trust boundaries: authentication, authorisation, input parsing, secrets handling, cryptography, and anything that touches money or personal information. Style in generated code matters far less than the checks that guard data, and review time should follow that ranking.

Generated tests deserve separate suspicion. The same model that wrote the code writes tests that assert what the code does, so a green suite proves the code agrees with itself. Read the tests against the requirement instead of the implementation, and add at least one case the assistant was never told about, ideally a hostile one. The checklist below is the floor we would expect any team shipping assistant-written code to stand on.

Scanners are worth having, and they catch the known shapes: leaked key formats, vulnerable dependency versions, some injection sinks. The distinctive failures of generated code sit outside those shapes, because the code is syntactically valid and often locally correct while being wrong for your system. A scanner does not know that invoices belong to customers, that a price field must never be trusted from the client, or that this particular permissive origin fronts an authenticated API. Catching those requires a reader who holds your data model and your threat model in their head at the same time.

This note gives you the posture and the checklist. The deeper work, tracing authorisation through a real codebase, exercising the boundaries a diff touched, and confirming that generated dependencies and cryptography hold up under scrutiny, is what a secure code review engagement covers, and it is where Black Shard now spends much of its review time, because AI-heavy codebases concentrate exactly these defects. The practical step this week is small: pull up the last few merges you know leaned on an assistant and run the checklist above over them. Fix what it finds before the next release goes out.