Imagine you have a threat modeling session, you tie up every aspect imaginable, and yet you are still vulnerable to security holes. But the security gaps are not connected to securing channels, leaking memory, or other security concerns. The gaps represent an attack surface that has existed since the early days of LLMs, but one that agents dramatically amplify — and one of them is prompt injection.

To better understand what prompt injection is in these scenarios, let's first take a look at the definition:

Prompt injection happens when malicious actors embed instructions inside the data the model will process, causing the model to execute the attacker's command. Large language models cannot reliably separate developer instructions from untrusted data within a shared context window.

There are two ways this can happen:

directly through the (chat) interface
indirectly via resources

Direct prompt injection

Imagine you are using a support chat where initially an agent might answer questions. If you were a malicious attacker, you could potentially craft a prompt that reads Zendesk tickets or other information the agent has access to.

Indirect prompt injection

This is a more sophisticated attack. Let's say you are applying for a job and you know an agent will sift through your resume first. Knowing that, you could embed certain instructions into the CV by using white text on a white background. A human can't see this, but an agent processing a rendered document can. For text-based document parsers, a more realistic vector is instructions embedded in metadata fields or document properties, which are processed by the agent but invisible in any human-facing view. All of a sudden, the agent might make an HTTP request and share personal data from other candidates.

A particularly dangerous variant of indirect injection is memory-persistent injection. In agents with long-term memory or multi-session context, a malicious instruction doesn't need to trigger immediately — it can lie dormant and activate later when a specific condition is met, or even persist across separate sessions. This makes it especially hard to detect, since the injection and the harmful action may be completely separated in time.

What is common in both cases besides the prompt injection itself? Both approaches did not foresee the attack surface. Usually, you make sure you cover the basics (OWASP Top 10):

Broken access control
Security misconfiguration
Software supply chain failures
Cryptographic failures
Injection with cross site scripting and SQL injection
Insecure design
Authentication failures
Software or data integrity failures
Security logging or alerting failures
Mishandling of exceptional conditions

For AI-specific threats, OWASP also maintains a dedicated Top 10 for LLM Applications, where prompt injection ranks as LLM01 — the highest priority risk.

But the exploit does not happen within the mentioned channels, but rather through content. When an attacker embeds a malicious payload inside a legitimate document submitted through authenticated channels, every Zero Trust control passes it through without inspection. The document came from an authenticated user. It was transmitted over an encrypted connection. The requesting service had valid credentials. The payload is delivered perfectly.

The problem is structural: language models process instructions and data using the same mechanism. While modern LLMs expose separate system and user roles at the API level, these boundaries are not enforced at inference time — the model can be manipulated into ignoring them. It cannot reliably distinguish which instructions were authored by the system and which were injected by an attacker.

How to protect against direct attacks

The first step is a good system prompt. You define the role the agent plays as narrowly as possible. This does not prevent malicious actions, but at least puts some guardrails in place.

The second step is making sure no special text or formatting is passed through. Within the chat, you do not allow any specific structure like XML tags or anything like that. The passed text should be free-form and any structure should be added within the system. For example, if the chat has a form, after submission the form can structure the text in any way that suits the LLM to provide additional context.

The third step can be a deterministic pre-filter: check for common injection patterns like ignore or override. This is a weak defense on its own — trivially bypassed with synonyms or rephrasing — but it's cheap and blocks unsophisticated attempts before they reach more expensive checks.

As a fourth step, you apply AI as a judge — a semantically aware layer that covers what the pre-filter misses. You take the user input and ask another AI to infer the message's intent. This intent is then compared against the agent's intended action. If they do not match, you do not follow the user prompt. Keep in mind that this defense is not foolproof — a sufficiently crafted input could mislead both the primary agent and the judge simultaneously. Treat it as a strong layer in a defense-in-depth strategy, not a standalone guarantee.

How to protect against indirect attacks

As stated, indirect attacks happen through content that is poisoned, meaning invisible instructions to humans are shared with the agents like metadata or hidden data in documents.

Another common indirect surface is RAG (Retrieval-Augmented Generation). When agents query a knowledge base to enrich their responses, a single poisoned document in that knowledge base can compromise every future query that retrieves it — one injection, many victims.

Both of these surfaces share the same root problem: the agent processes untrusted content without being able to distinguish it from legitimate instructions. The following steps address that.

The first step to address this vulnerability is to separate reasoning and execution. You end up having two agents, each of them playing a specific role. The idea is to have architectural separation of the AI component that processes untrusted input from the AI component that executes actions on live systems. This is one of the most structurally sound approaches for prompt injection defense, as it does not try to detect malicious content; it removes the ability of compromised content to cause harm.

The Reasoning Agent’s only output is a structured JSON object describing the intended action: {"action": "send_email", "recipient": "user@company.com", "subject": "Summary ready", "body": "..."}. By forcing the primary language model to generate responses only in rigid, machine-validated formats through a JSON schema, the model cannot produce free-form text that could contain malicious secondary instructions. The Execution Agent validates that this action is permitted for the current user’s scope and executes it.

A successful prompt injection of the Reasoning Agent produces a hallucinated JSON object. The Execution Agent’s schema validator rejects anything that doesn’t match a permitted pattern. The attacker’s payload cannot introduce actions outside the permitted schema — though in-schema actions with malicious parameters remain a residual risk, which is why the Principle of Least Privilege and human approval steps are still necessary.

The second step is dual LLM verification. You introduce an auditor model that compares the user prompt against the output of the Reasoning Model. Keep in mind this agent is not poisoned by any content and should be able to flag discrepancies between user intent and concluded reasoning. It sees only the output of the primary agent — the proposed action — not the upstream documents, user messages, or RAG content that produced it. This means it cannot be manipulated by the original injection.

The third step is the Principle of Least Privilege. Configure AI agents to operate in read-only mode by default, requiring explicit human approval for any action with destructive, irreversible, or externally visible consequences. Even a fully compromised agent cannot cause harm if it lacks the authority to take harmful actions without human review. This is the AI-specific application of least-privilege access control. The agent generates a “pending transaction” object instead of directly executing high-risk actions. A human operator reviews the exact proposed API call — not a natural language summary of it, but the literal parameters — before authorizing execution.

The fourth step is having penetration tests in place that run periodically — daily in the best case, monthly in the worst case. Similar to periodic manual penetration testing, you introduce automated adversarial AI systems that continuously probe your agent’s defenses in staging environments. Since the attack surface for AI agents is not static, AI is the most suited for this job. Due to the nondeterministic nature, you may encounter a vulnerability through this process. A prompt injection technique that your agent resisted in May may succeed in June if the model provider silently updates the underlying model weights or the model tries certain permutations it hasn’t tried before. Manual penetration testing, even quarterly, cannot match the pace of evolution in adversarial techniques.

Conclusion

With the age of agents, prompt injection has evolved from a curiosity into a serious production risk. The defenses described here work in layers: cheap deterministic filters block the obvious attacks, architectural separation removes the ability to cause harm, and human approval gates the irreversible ones. No single layer is sufficient — but combined, they dramatically raise the cost of a successful attack. The attack surface will keep evolving, which is why automated adversarial testing is the last line: it ensures your defenses evolve with it.

Published by...

Jernej Klancic

Visit author page