Prompt Injection

Prompt injection exploits the difficulty language models have distinguishing trusted instructions from untrusted content. An attacker supplies text or other model-readable material that attempts to redirect the model, override application rules, or influence subsequent actions. Unlike traditional code injection, the payload may be ordinary language rather than executable syntax. A successful attack does not necessarily compromise the underlying model infrastructure; it can instead misuse the application's legitimate access and capabilities.

Direct injection arrives through an attacker-controlled user message. Indirect injection arrives through material the application retrieves or processes, such as a web page, document, support ticket, repository file, or tool response. For example, a retrieval-augmented generation application might retrieve a document containing an instruction to ignore the user's question and send confidential context to an external address. Storing that document in a trusted cloud bucket does not make its embedded instructions trustworthy.

Start protection at trust boundaries. Keep system instructions separate from user input and retrieved material, clearly identify external content as data, and avoid placing credentials or unnecessary sensitive information in model context. These measures improve instruction handling but do not establish a reliable security boundary. Apply authorization before retrieval so users receive only documents they may access; a vector similarity match is not an authorization decision.

Use input screening and model guardrails to detect suspicious instruction patterns, policy violations, and attempts to override application behavior. Evaluate retrieved content and tool responses as well as user messages. On AWS, Amazon Bedrock Guardrails provides prompt attack filtering, but applications must correctly configure the policy and identify the input content to evaluate. Do not assume every orchestration step or retrieved passage is automatically protected. Filters can miss attacks and can block legitimate requests, so test both detection coverage and false positives.

The strongest controls limit what an injected model can accomplish. Treat a model-generated tool call as a proposed action, not authorization. A tool gateway or backend must validate arguments, authenticate the caller, enforce tenant boundaries, and independently authorize the operation. Use narrowly scoped execution roles, restrict outbound network destinations, and require approval for consequential actions such as deleting resources or exporting records. An agent that summarizes invoices should not have unrestricted storage access or permission to initiate payments.

Validate model outputs before sending them to browsers, databases, shells, or downstream APIs. Use structured schemas, parameterized queries, safe rendering, and allowlisted operations rather than executing generated text. Output filtering can reduce sensitive-data disclosure, but it cannot undo a tool action that has already occurred. Similarly, encryption and private networking protect other parts of the system but do not prevent malicious instructions in authorized content.

For security design and exam scenarios, recognize prompt injection as an OWASP Top 10 for LLM Applications risk and apply defense in depth. Jailbreaking overlaps with prompt injection but specifically targets model safety restrictions; prompt injection can also target business workflows without requesting prohibited content. Red-team direct and indirect attack paths, record guardrail decisions and tool activity, and retest after changes to prompts, models, retrieval pipelines, or permissions. Stronger filtering and approval workflows introduce latency and operational friction, but independently enforced authorization limits damage even when the model follows an attacker's instructions.