How to security test an LLM feature already in production

Enumerate the ingestion paths before writing a payload, because the interesting ones are indirect. Uploaded documents carry text a reader never sees: white or zero-point type, content positioned off the page, HTML comments, alt text, document properties and stored revisions. Anything ingested from outside the company is an entry point: support tickets, inbound email, form submissions landing in a customer record, partner data feeds and fetched web pages. Where the pipeline runs optical character recognition, text inside an image is an input, and so is the filename.

Plant the payload through the real business path. Pasting it into the chat window tests only the source you already control. Raise a ticket from an external address with the instructions in the body, then ask the assistant to summarise open tickets. Design the payload so success is observable: an instruction that makes the model say something odd produces an argument about interpretation, while one that emits a unique canary or requests a URL on a host you control produces a binary observation with a timestamp.

Two variants matter more than the rest. Persistent injection stores the payload in a record many later sessions read, so one planted ticket reaches every user whose assistant summarises that queue. Second-order injection writes the model's own output back into a record re-read as context later, which lets an instruction survive sanitisation applied only at ingest. Test system prompt recovery in the same pass, and check the prompt holds no internal hostnames, table names, credentials or business logic you would not publish.

The defect we find most often in a retrieval feature has nothing to do with language models. An indexing job runs under a service account that reads every tenant, chunks land in the vector store without the source record's permissions, and whatever permission predicate exists runs over the search results after the fact. Everyday questions retrieve the asking tenant's own content, so the feature looks correct for months. A question phrased close enough to another tenant's records returns them, and since the retrieval path never called your authorisation function, nothing was denied and nothing was logged.

The absent log is the more serious half. When a leak produces no authorisation event your monitoring shows a clean day, and afterwards you cannot say whether the same thing happened in production last quarter. Under the Notifiable Data Breaches scheme the assessment turns on unauthorised access to personal information and the likelihood of serious harm, so an assessment you have no logs to perform is one you cannot complete.

Structure the test around identities: two tenants, and two roles inside one of them with genuinely different record access. Do not grep the rendered answer for your marker, because the model paraphrases, and a summary of another customer's data leaks everything of value without reproducing a single string. Assert against the retrieval log or the chunk identifiers the query returned. Read the code for where the filter sits, since a post-filter that drops unauthorised neighbours after the search fails open on any bug in the drop step.

Once an instruction has landed, the damage happens where the answer is consumed, and those sinks yield to ordinary application security technique. The most common live finding is markdown rendered into a browser. If the renderer allows remote images, an injected instruction can make the model emit an image reference whose URL carries conversation content in the query string, which the browser fetches as the answer renders. Check whether raw HTML passes through, whether link schemes are restricted, and whether the content security policy would stop a request to an arbitrary host.

Where the feature turns a question into a query, that query is attacker-influenced by construction, and parameterisation does not help because the model writes the whole statement. The boundary has to be the identity it runs as: a read-only role that sees only the caller's tenant, a statement timeout, a row cap, and an allowlist of objects the role can reach. Where the feature fetches a URL the model chose, you have server-side request forgery with a language model picking the target.

Tool arguments deserve the same treatment as a form field even when the tool is described as read-only. A search tool passes its argument to a downstream API, a notification tool takes a recipient, a ticket tool takes a body another user reads, and each value was chosen by a model reading untrusted text. Validate at the tool boundary against what the calling user is entitled to do. The recurring mistake is that teams stop treating a string as untrusted the moment it comes back from their own service.

Find every route that reaches inference and check what authenticates it. Marketing chat widgets, public demo pages, trial tenants and mobile clients holding a key in the bundle all call the same backend, and a caller who can supply arbitrary prompt content is running their own workload on your account. The first test is whether the endpoint answers a request with nothing to do with your product. The second is whether your billing or monitoring would have shown you.

Rate limiting counted in requests is the wrong unit. One request carrying a maximum-size attachment with a high output limit costs far more than a hundred short ones, so the budget has to be expressed in tokens, tool calls and turns, enforced per user and per tenant. Test with the largest input the interface accepts, with prompts written to produce long output, and with a flow where the model calls a tool whose result it reads, since that shape can loop until something stops it.

Two disclosure problems sit alongside the abuse ones. Upstream errors returned verbatim leak deployment detail, so check what the caller sees when a call fails. Prompts routinely contain customer records, which makes every place a prompt body is captured a place personal information now lives: tracing tools, request logs, provider retention, and any evaluation set built from real traffic. Australian customers ask where that content sits, whether it leaves the country, and who can read it.

A payload that failed once may succeed on the next attempt with different sampling, a different context ordering, or a reworded question, so a single manual attempt is not evidence in either direction. A tester who tried it three times and moved on has produced a sentence that will not survive a customer's auditor, and an engineer who reproduced it once has not established how often it happens. Sampling settings, chunk ordering and the model behind the endpoint can change between two runs, and none of it is announced.

The workable answer is a seeded evaluation suite run as a regression gate. Each case is a fixed input planted through the real path, a fixed retrieval corpus, and an assertion on an observable instead of on the wording of a reply: whether a request reached the collector, whether the rendered output carried a remote image reference, whether the retrieval log held another tenant's chunk. Run each case a set number of times and record the rate, so the result reads as zero of fifty attempts and not as a claim that it did not reproduce.

Then split the cases into two classes, because one pass rate hides the difference. Some have a deterministic control behind them: the renderer strips the tag, the database role cannot see the table, the store enforces the retrieval predicate. Those must be zero, and a single success is a defect in the control that no amount of prompt tuning will address. Where only the model's refusal stands between the payload and the outcome, the finding is the missing control, and a low success rate obscures the fact that nothing there can fail closed.

Record what makes the run reproducible, since a customer will ask which system you tested. That means the model version or deployment identifier, the sampling parameters, the prompt template hash, the corpus snapshot and the application commit. A silent model upgrade or a one-line prompt edit invalidates the previous result, which is why the suite has to run in your pipeline on every change to either.

Security reviews of AI features converge on a short list of questions, and all of them are about method. A reviewer wants to know what was tested, against which version and corpus, by whom and when, then what was found, what was fixed and what proves the fix holds. Vendor questionnaires commonly cite the OWASP Top 10 for Large Language Model Applications, so mapping your case list to those categories once saves rewriting the answer for every customer.

Assemble the artefacts as you go. Reconstructing them later tends to be quietly incomplete. The attacks you attempted and could not make work belong in the pack alongside the ones that landed, because a documented list of failures is stronger evidence than an assertion that the feature is safe. A reviewer will also ask whether the person who tested it was the person who built it.

Answer the data handling questions in the same pack, since they arrive with the security ones: where prompts and outputs are stored, for how long, in which country, whether customer content trains anything, and which roles can read them. If testing surfaced a cross-tenant retrieval leak, the next question is whether it ever happened to a real customer, and a product whose retrieval path emits no authorisation event cannot answer that. That logging is worth fixing ahead of the leak itself.

We scope an AI feature as a component of the application penetration test, because the findings resolve in the application and the reviewer wants one report. The access we ask for follows from all of the above: two tenants, two roles with real differences in record access, the ability to submit content through the ingestion paths an outsider would use, a collector host we control, and read access to the retrieval logs. From a single account with a single role, the exercise can only demonstrate that the feature works.

Secure code review in the same engagement covers what is hard to reach from outside. It answers where the permission predicate is applied and whether the store enforces it, what identity generated queries execute as, whether the renderer permits remote resources or raw HTML, and what the indexing job runs as. Those are single-file answers when you can read the code and multi-day inferences when you cannot.

The evaluation suite goes back with the report and belongs in your pipeline, because a model version change or a prompt template edit resets what the test established. That matches how we build in regulated settings: a law firm's operations and compliance portal, a capital-markets firm's staff portal, and our own clinical software venture hosted on Azure in Australia, where every change leaves an audit record that can be produced later. A retrieval layer that answers questions without making an authorisation decision has removed that record from the part of your product most likely to be asked about next year.