prompt injection

Prompt injection is an attack technique in which malicious instructions are embedded in content that an AI system processes. When the model reads that content, it may follow the embedded instructions as if they were issued by the legitimate user.

A simple example: a user asks an AI assistant to summarize a document. The document contains, alongside normal text, a hidden instruction that tells the model to instead reply with a fabricated summary and exfiltrate certain data. The model processes the document and may follow the embedded instruction. The user receives a plausible-looking output without knowing the model's behavior was redirected.

Prompt injection differs from conventional software vulnerabilities. There is no code execution involved. The attack works through the model's instruction-following behavior, which is the same behavior that makes the model useful.

No software update patches prompt injection away. Current language models do not reliably distinguish between user instructions and instructions embedded in processed content. That is a fundamental characteristic of how they work.

The risk grows in proportion to two factors. First, how much untrusted content the AI system processes. A system that only processes content the user typed is a lower risk than one that processes emails, web pages, uploaded documents, or data pulled from external sources. Second, what the AI system is allowed to do. A system that generates text is lower risk than one that can send emails, create documents, or interact with other applications.

Agentic AI systems, which act autonomously with a user's credentials, are where prompt injection risk is highest. An agent that processes an attacker-controlled web page or document and has permission to take actions in connected applications is a meaningful attack surface.

The practical risk from prompt injection depends entirely on what the AI system is authorized to do.

For AI assistants that only generate text, the risk is misleading output. A model processing an attacker-controlled document might produce a summary that serves the attacker's interests rather than the user's. The impact is limited to bad information reaching a human who then acts on it.

For agentic systems with application access, the risk profile changes. An agent with permission to send email could be directed by an injected instruction to send an email on the user's behalf. One with access to a file storage system could be directed to share files. One with calendar access could modify meetings. The action is bounded by the agent's permissions, which in enterprise deployments are often broad.

The attack requires no malware, no credential theft, and no network intrusion. It requires only that the AI system process content the attacker can influence, and that the system have permission to do something consequential.