In 2022, the researcher Simon Willison coined the term 'prompt injection' to describe a specific failure mode: an AI system reading a piece of content that contains, buried inside it, an instruction designed to look like part of the material but intended for the system processing it rather than for a human reader. His original example was almost exactly what security teams are now finding in production — a customer service agent reading an email that contains a hidden instruction to forward internal data or override a stated policy.
The basic shape of the attack is hiding an instruction inside content a system is expected to process as data, not as a command, and exploiting the fact that many current systems do not reliably distinguish between the two. It is closely related to social engineering, except the target is a system that can be relied upon to follow plausible-looking instructions exactly and without the skepticism a trained human employee might apply.
The risk scales directly with what these systems are allowed to do, which is why OWASP's Top 10 for Large Language Model Applications lists prompt injection as its number one risk category. A chatbot that can only respond with text presents a limited version of this problem — a bad response is embarrassing but bounded. An agent that can send emails, modify records, or execute transactions on someone's behalf turns the same vulnerability into a mechanism for real action taken under false pretenses.
Security researchers have already demonstrated versions of this against production tools, including documented cases showing Microsoft Copilot could be steered by hidden instructions embedded in a document or email it was asked to summarize. Every additional data source, tool, or permission granted to an AI agent is also an additional surface where content like that — a malicious email, a poisoned document, a compromised web page an agent is asked to read — can steer behavior nobody intended.
Current defenses are partial and mostly additive: input filtering, permission scoping that limits what an agent can do regardless of what it is told, and human approval steps inserted before consequential actions. None of these fully closes the gap, because the underlying issue is architectural — these systems are built to follow instructions found in the content they process, and drawing a hard line between trusted instructions and untrusted content remains unsolved.
Willison has said publicly that he does not expect this to be solved by the current architecture, only managed. Every company connecting a model to a new inbox, a new document store, or a new permission is making a choice, whether it names the choice that way or not, to accept his second word over his first.