Test prompt injection as a control failure

Injection exploits instruction ambiguity

LLM applications place policy, user requests, retrieved text, tool output, and prior messages into a context the model must interpret. Prompt injection occurs when attacker-controlled content is treated as instructions that conflict with the application's intended policy.

Direct injection arrives through the user. Indirect injection arrives through content such as a page, email, file, or tool response. Stored injection persists for later processing. Multimodal injection uses images, audio, or document structure.

Define a meaningful success condition

A model repeating a hidden phrase may demonstrate instruction confusion, but it is not automatically a high-severity vulnerability. The test becomes important when the confusion crosses a boundary: disclosing restricted data, bypassing a user authorization check, altering a transaction, or invoking a tool outside scope.

Write the expected invariant and collect the full trace: input source, retrieved content, model request, tool call, authorization response, final output, and logs. Repeat tests because model behavior is probabilistic.

• Can untrusted content influence a side-effecting tool call?
• Can one user cause retrieval of another user's data?
• Can a document override the required citation or approval policy?
• Does the application render model output as executable HTML or a command?
• Can the system detect, contain, and explain the attempted boundary crossing?

Layer the defense outside the prompt

Separate trusted instructions from untrusted data, but do not assume delimiters solve the problem. Enforce least privilege, authorize every tool call on the server, classify side effects, constrain output schemas, validate destinations, require confirmation for consequential actions, and monitor for unusual sequences.

Use model-level detection as one signal. Robust design assumes some injections will reach the model and prevents them from becoming unauthorized consequences.

The security goal is not a model that never follows a bad instruction; it is a system that keeps bad instructions from becoming harmful actions.