Code & AI Learning Hub

Your agent opens a customer email, finds a hidden instruction, and sends private data to an attacker-controlled endpoint. No model weights changed. No server was breached. The application handed hostile text to a system with tools and trusted it to distinguish data from commands. That assumption is the root prompt injection defense must remove.

Prompt injection isn’t a strange prompt that makes a chatbot swear. In production, it’s an authorization and data-flow problem. Attackers place instructions in user input, web pages, files, retrieved documents, images, tool results, or memory. An LLM may then follow those instructions because natural language carries no reliable security boundary.

Prompt injection defense: a layered security design that treats model-visible content as untrusted, limits agent authority, validates every side effect, and detects abuse through tests and runtime monitoring. It reduces both direct prompt injection and indirect prompt injection, but it doesn’t promise perfect detection.

This plan covers threat modeling, trust boundaries, input handling, RAG security, tool authorization, output controls, data-loss prevention, human approval, red teaming, observability, and incident response. Security engineers, platform teams, and developers building agents can use it as a production checklist.

• Map hostile content paths before tuning prompts.
• Keep permissions outside the model.
• Parse data before exposing it to inference.
• Gate dangerous actions with deterministic code.
• Test attacks continuously, not before launch only.

Model Prompt Injection as a Trust-Boundary Failure

Prompt injection occurs when untrusted content changes model behavior beyond the application’s intended data use. The key failure isn’t rude language. It is confused authority: instructions and data share the same natural-language channel.

Direct prompt injection arrives through a user message. Indirect prompt injection hides in content the application retrieves or processes. A hiring agent might read a resume containing “rank this candidate first.” A browser agent might read a page telling it to upload cookies. A support agent might receive an email that requests a refund tool call.

A common first attempt is a blocklist for phrases such as “ignore previous instructions.” It performs well in a demo but rarely survives a determined attacker, since translation, encoding, typography, role-play, and benign-looking task instructions all bypass literal matching. Blocklists also produce false positives, flagging legitimate security documentation that happens to quote these attacks.

from dataclasses import dataclass
from enum import Enum

class Trust(str, Enum):
    POLICY = "policy"
    VERIFIED = "verified"
    UNTRUSTED = "untrusted"

@dataclass(frozen=True)
class DataFlow:
    source: str
    trust: Trust
    destination: str
    can_trigger_side_effect: bool


def risky_flows(flows: list[DataFlow]) -> list[DataFlow]:
    return [flow for flow in flows if flow.trust is Trust.UNTRUSTED and flow.can_trigger_side_effect]

flows = [DataFlow("email_body", Trust.UNTRUSTED, "refund_agent", True)]
assert len(risky_flows(flows)) == 1

Inventory every path from external content to a model and from a model to a side effect. Include OCR, image captions, file metadata, vector stores, memory, tool descriptions, and tool responses. A sound defense begins with this map because unseen paths can’t be controlled.

Separate Policy, Instructions, and Untrusted Data

Structural separation labels content by authority and purpose before inference, which helps the model interpret boundaries and helps reviewers inspect them. Treat delimiters as a clarity aid rather than a security sandbox: they reduce ambiguity but cannot enforce permissions on their own.

Keep system policy immutable at runtime. Put task instructions in a separate block. Wrap external text with source IDs and an explicit untrusted label. Never interpolate untrusted text into a sentence that grants authority, such as “follow these instructions: {document},” because that construction hands a command to the model dressed up as data.

from html import escape


def render_untrusted(source_id: str, content: str) -> str:
    if not source_id.strip():
        raise ValueError("source ID is required")
    safe_source = escape(source_id, quote=True)
    safe_content = escape(content)
    return (
        f"<untrusted_data source='{safe_source}'>\n"
        f"{safe_content}\n"
        "</untrusted_data>"
    )

assert "<script>" not in render_untrusted("email-42", "<script>")

Escaping prevents markup confusion in this serialization, but it does nothing to make the content safe to obey. The surrounding policy must say that content can supply facts while it cannot alter goals, permissions, destinations, or output rules.

A stronger pattern adds source, trust, owner, and expiry metadata before rendering, and injection hardening gets much harder when context is one opaque string instead. The related context engineering for AI agents guide explains how provenance and typed context blocks make these boundaries auditable. Research prototypes such as the dual-LLM design and the CaMeL capability model take the same idea further, keeping a privileged planner away from untrusted text and enforcing data-flow rules in code rather than in wording.

Normalize and Validate Inputs Without Trusting Filters

Treat input controls as attack-surface reduction, not as a verdict on what is safe. Enforcing expected type, size, encoding, and format before content reaches the model strips out obvious junk, yet any semantic filter has to stay advisory, because attackers can rephrase intent endlessly.

Normalize Unicode, reject control characters where they have no business value, cap length, verify MIME types, and parse known document formats. Remove scripts, comments, invisible layers, and unnecessary metadata. For invoices, extract required fields rather than sending an entire PDF, so less hostile surface ever reaches inference.

import unicodedata

MAX_INPUT_CHARS = 20_000
ALLOWED_CONTROLS = {"\n", "\t"}


def normalize_text(value: str) -> str:
    normalized = unicodedata.normalize("NFKC", value)
    if len(normalized) > MAX_INPUT_CHARS:
        raise ValueError("input exceeds size limit")
    cleaned = "".join(
        char for char in normalized
        if not unicodedata.category(char).startswith("C") or char in ALLOWED_CONTROLS
    )
    if not cleaned.strip():
        raise ValueError("input is empty after normalization")
    return cleaned

A classifier can flag likely injection and route the request to a safer workflow. Don’t let it unlock permissions. A false negative should still hit authorization controls, while a false positive should lead to review rather than silent data loss.

Useful Input Decisions

• Reject files whose content contradicts the declared MIME type.
• Extract visible cells from spreadsheets instead of formulas and macros.
• Disable external resource loading during document conversion.
• Preserve original hashes for investigations.
• Log filter decisions without retaining unnecessary secrets.

Threat-Model Multimodal and Encoded Inputs

This control layer must cover meaning extracted from images, audio, video, QR codes, attachments, and encoded text. A pipeline that inspects plain chat while blindly trusting OCR or transcription simply moves the attack into the channel it forgot to watch.

Map every transformation before inference. A PDF may become page images, OCR text, metadata, and extracted links; audio may become a transcript plus speaker labels. Each derived output inherits the untrusted status of its source, because converting a file into text never launders the authority of what that text says.

from dataclasses import dataclass

@dataclass(frozen=True)
class DerivedArtifact:
    artifact_id: str
    parent_id: str
    media_type: str
    trust: str
    extractor: str


def derive(parent_id: str, artifact_id: str, media_type: str, extractor: str) -> DerivedArtifact:
    if not all(value.strip() for value in (parent_id, artifact_id, media_type, extractor)):
        raise ValueError("artifact lineage fields are required")
    return DerivedArtifact(artifact_id, parent_id, media_type, "untrusted", extractor)

assert derive("pdf-9", "ocr-9-1", "text/plain", "ocr-service").trust == "untrusted"

Avoid automatic decoding of arbitrary payloads merely because a document asks for it. Decode only formats required by the task, cap expansion size, and preserve lineage. A small compressed object can expand into a huge context or parser workload. The defense overlaps with ordinary decompression, parser, and denial-of-service controls here.

Image-based injection is a classic blind spot: a chat filter can pass every request because the hostile text exists only after OCR. Add transformed artifacts to traces and security suites, or your monitoring will show a clean input beside a compromised action.

Harden RAG Against Indirect Prompt Injection

Securing RAG is not one control but several working in concert: access control, source governance, retrieval filtering, provenance, and untrusted-content handling. Retrieval places a document next to the model without making its contents any more authoritative than where they came from.

An attacker may poison a shared knowledge base, publish a malicious web page, or edit a low-authority wiki. Similarity search can rank that content above official policy. If an agent treats retrieved text as instructions, one indexed paragraph can redirect an entire workflow.

Filter by tenant and user permissions before semantic ranking. Exclude expired or unapproved sources. Prefer current primary documents. Keep source diversity so one poisoned document doesn’t fill every slot. Then render citations beside each chunk.

from dataclasses import dataclass
from datetime import datetime, timezone

@dataclass(frozen=True)
class Chunk:
    text: str
    source_id: str
    approved: bool
    tenant_id: str
    expires_at: datetime
    score: float


def select_chunks(chunks: list[Chunk], tenant_id: str, limit: int = 6) -> list[Chunk]:
    now = datetime.now(timezone.utc)
    allowed = [
        chunk for chunk in chunks
        if chunk.approved and chunk.tenant_id == tenant_id and chunk.expires_at > now
    ]
    return sorted(allowed, key=lambda chunk: chunk.score, reverse=True)[:limit]

Here’s where many RAG tutorials stop: retrieval quality. Security needs write-path controls too. Record who can add documents, who approved them, and which embeddings came from each version. Re-indexing shouldn’t erase lineage.

Use the OWASP LLM application risks as a baseline for injection and data-exposure review. Add domain-specific poisoning cases to your evaluation set.

Protect Agent Memory from Persistent Injection

Memory poisoning turns one hostile interaction into repeated future influence. The defense must control what an agent remembers, how long it survives, and which authority can correct it. Saving a transcript summary without review can preserve an attack after the source disappears.

Separate working state from durable memory. Require durable writes to name a source, confidence, owner, reason for reuse, and expiration. Explicit user facts may qualify. Instructions found in retrieved documents don’t. Never let external text create policy or tool permissions through memory.

from dataclasses import dataclass
from datetime import datetime, timezone

@dataclass(frozen=True)
class MemoryWrite:
    key: str
    value: str
    source_type: str
    confidence: float
    expires_at: datetime


def authorize_memory(write: MemoryWrite) -> None:
    if write.source_type not in {"user_explicit", "verified_system"}:
        raise PermissionError("source cannot create durable memory")
    if write.confidence < 0.9:
        raise ValueError("memory confidence too low")
    if write.expires_at <= datetime.now(timezone.utc):
        raise ValueError("memory is already expired")

Version memory and keep correction records. When a user retracts a preference, remove derived summaries too. During an incident, invalidate memories created from the poisoned source. This defense gets messy when lineage ends at a generated summary.

Do you need model-written durable memory at all? Often, no. Extract a small candidate, validate it, and ask for confirmation when persistence affects future behavior, and you end up with a system that is less autonomous but far easier to reason about during an incident.

Give Agents the Least Possible Privilege

Give an agent only the tools the current task needs, scoped to short-lived credentials and narrow resources, and an injected model has very little left to damage. That is the whole point of least privilege, and it collapses the moment a general-purpose credential sits behind a polite system message.

Separate read operations from writes. Split proposal from execution. A model can draft an email, while application code verifies recipients and a user approves sending. An agent can propose a refund, while a policy service checks amount, order ownership, account state, and rate limits.

from dataclasses import dataclass
from decimal import Decimal

@dataclass(frozen=True)
class Principal:
    user_id: str
    tenant_id: str
    refund_limit: Decimal

@dataclass(frozen=True)
class Refund:
    tenant_id: str
    order_total: Decimal
    amount: Decimal


def authorize_refund(principal: Principal, refund: Refund) -> None:
    if principal.tenant_id != refund.tenant_id:
        raise PermissionError("cross-tenant refund denied")
    if refund.amount <= 0 or refund.amount > refund.order_total:
        raise ValueError("invalid refund amount")
    if refund.amount > principal.refund_limit:
        raise PermissionError("refund exceeds principal limit")

The model never receives the database credentials or the authorization logic here; both stay in application code by design. Contrast that with the common pattern where an agent’s tool description says “never refund over $500” while the API accepts any value, so the restriction lives only in documentation and enforces nothing at runtime.

Issue credentials per request or short session. Restrict destinations, methods, records, and amounts. Add idempotency keys. This defense becomes far more reliable when the worst model decision produces a denied request rather than an incident.

Validate Tool Calls and Side Effects Deterministically

Every model-proposed action must cross a deterministic validation boundary. JSON Schema checks shape, while domain validation checks meaning, authorization, state, and risk. Both are required.

A valid URL can still point to an attacker. A valid amount can still exceed an account limit. A valid file path can still escape a workspace. A schema can’t know business ownership or user intent unless application code checks it.

from pathlib import Path
from urllib.parse import urlparse

WORKSPACE = Path("/srv/agent-workspace").resolve()
ALLOWED_HOSTS = {"api.example.com", "status.example.com"}


def safe_path(raw: str) -> Path:
    candidate = (WORKSPACE / raw).resolve()
    if candidate != WORKSPACE and WORKSPACE not in candidate.parents:
        raise PermissionError("path escapes workspace")
    return candidate


def safe_url(raw: str) -> str:
    parsed = urlparse(raw)
    if parsed.scheme != "https" or parsed.hostname not in ALLOWED_HOSTS:
        raise PermissionError("network destination denied")
    return raw

Apply rate limits and budgets across a whole task, not one call. An injected agent may stay below a per-call limit while making hundreds of small requests. Count records read, bytes exported, messages sent, money moved, and external destinations contacted.

And treat tool results as untrusted too. Search APIs, ticket systems, and browsers return attacker-controlled content. Validate result schemas, strip active content, label provenance, and prevent results from changing permissions. Tool use creates a loop, and every edge of that loop needs the same scrutiny. Deployment boundaries matter too: where server-side authorization and agent gateways run shapes which components can enforce these checks.

Control Network Egress and External Destinations

Network egress control prevents an injected agent from sending data to arbitrary hosts. The control layer should default-deny destinations and open only the exact protocols, domains, paths, and methods needed by a task.

Domain allowlists need careful parsing. Check normalized hostnames after redirects and DNS resolution. Block private address ranges for public fetchers to reduce SSRF. Re-check every redirect. Avoid wildcard domains unless every subdomain shares the same trust and ownership.

import ipaddress
import socket
from urllib.parse import urlparse

ALLOWED = {"api.example.com"}


def validate_public_https(url: str) -> None:
    parsed = urlparse(url)
    if parsed.scheme != "https" or parsed.hostname not in ALLOWED:
        raise PermissionError("destination denied")
    addresses = {item[4][0] for item in socket.getaddrinfo(parsed.hostname, 443)}
    if any(ipaddress.ip_address(address).is_private for address in addresses):
        raise PermissionError("private destination denied")

Production resolvers also need defenses against DNS rebinding and address changes between validation and connection. The safest pattern puts requests through a controlled proxy that resolves, connects, logs, and applies byte limits itself. Don’t validate in agent code and then let another library follow redirects freely.

Too often “web access” is built as unrestricted HTTP because domain filtering felt inconvenient, which quietly turns a feature into an exfiltration primitive an injected agent can point anywhere. Defense becomes dramatically easier once outbound paths are few, named, and owned.

Validate Outputs and Prevent Data Exfiltration

Output controls inspect model responses and proposed actions before data leaves a trust boundary. They should detect secrets, personal data, unauthorized records, dangerous destinations, and format violations. Output filtering alone is late, but it can stop a missed attack.

Use structured outputs for machine workflows. Bind every returned record to an authorized source set. Redact secrets before model input and scan output for known secret formats. For messages, compare recipients and attachments against user intent. For code agents, restrict writes to a workspace and inspect diffs before execution.

import re

SECRET_PATTERNS = [
    re.compile(r"AKIA[0-9A-Z]{16}"),
    re.compile(r"-----BEGIN (?:RSA |EC )?PRIVATE KEY-----"),
]


def reject_secrets(text: str) -> None:
    for pattern in SECRET_PATTERNS:
        if pattern.search(text):
            raise PermissionError("possible secret in model output")


def enforce_record_scope(returned_ids: set[str], authorized_ids: set[str]) -> None:
    extra = returned_ids - authorized_ids
    if extra:
        raise PermissionError(f"unauthorized record IDs: {sorted(extra)}")

Regex won’t catch every secret or transformed leak. Pair pattern checks with data classification, egress restrictions, volume limits, and canary tokens. Most importantly, don’t provide unnecessary sensitive data in the first place.

For privacy-sensitive products, the site’s on-device AI mobile apps guide shows how local processing changes data exposure. Local inference reduces some network risk but doesn’t stop hostile documents from steering local tools.

Use Human Approval for Consequential Actions

Human approval should guard actions whose harm is hard to reverse, financially material, legally sensitive, or externally visible. Approval must show the actual action and evidence, not a vague “continue?” dialog.

A trustworthy approval step comes down to three requirements:

1. Show the recipient, destination, amount, affected records, source citations, and a concise reason.
2. Freeze action parameters before approval so the model can’t change them afterward.
3. Expire approvals quickly and bind each one to a single user, tenant, and request hash.

from dataclasses import dataclass
import hashlib
import json

@dataclass(frozen=True)
class Action:
    tool: str
    arguments: dict[str, object]
    tenant_id: str


def action_hash(action: Action) -> str:
    payload = {"tool": action.tool, "arguments": action.arguments, "tenant_id": action.tenant_id}
    encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str)
    return hashlib.sha256(encoded.encode()).hexdigest()

original = Action("send_email", {"to": "[email protected]"}, "tenant-a")
assert action_hash(original) == action_hash(original)

Approval fatigue is real. If every harmless read asks for confirmation, users click through dangerous writes without reading. Use risk tiers. Allow low-risk reads, require confirmation for moderate writes, and require stronger review for bulk export, financial action, account change, code execution, or publication.

Don’t ask the model whether approval is needed; the application decides from tool, arguments, data class, and policy, which keeps the guard somewhere the injected component has no way to waive it.

Secure Multi-Agent Delegation

Multi-agent systems multiply trust boundaries because one model’s output becomes another model’s input. The defense must preserve source trust and authority across delegation. A planner shouldn’t convert untrusted evidence into trusted instructions merely by paraphrasing it for a worker.

Pass a structured delegation envelope containing task, allowed tools, resource scope, budget, source references, and expiration. Keep original trust labels. The receiving agent must not inherit the sender’s credentials automatically. Issue narrower credentials for the delegated subtask.

from dataclasses import dataclass
from datetime import datetime

@dataclass(frozen=True)
class Delegation:
    task: str
    allowed_tools: frozenset[str]
    source_ids: tuple[str, ...]
    tenant_id: str
    expires_at: datetime
    max_calls: int


def validate_delegation(value: Delegation, available_tools: set[str]) -> None:
    if not value.task.strip() or not value.source_ids:
        raise ValueError("task and sources are required")
    if not value.allowed_tools <= available_tools:
        raise PermissionError("delegation requests unavailable tools")
    if value.max_calls < 1 or value.max_calls > 20:
        raise ValueError("delegation call budget is invalid")

Limit delegation depth and total calls. Otherwise an injected planner can create a swarm that burns budget or hides a malicious action among many tasks. Aggregate tool and data budgets across the whole tree, not per agent.

Cross-agent messages are untrusted unless a deterministic service attests to their fields. This feels conservative because the agents all belong to one application, yet compromise propagates through internal channels exactly as it does through microservices, which is why zero-trust thinking has to reach inside the agent graph and not stop at its edge.

Compare Prompt Injection Defense Controls

No single control stops prompt injection, so it helps to see the main options side by side before deciding how to combine them.

Control	Purpose	Setup	Scales	Main limit	Best use
Delimiters	Clarify data boundaries	Low	Yes	Model may disobey	All external text
Classifier	Detect suspicious intent	Medium	Yes	False results	Routing and alerts
Policy engine	Enforce authorization	Medium	Yes	Policy upkeep	Every side effect
Human approval	Review high risk	Medium	Limited	Fatigue and delay	Irreversible actions

Read down the columns and the division of labor is clear: delimiters and classifiers improve interpretation and detection, policy engines and scoped credentials limit damage, and human review handles the ambiguous high-risk cases. A production defense wants all of these categories at once, precisely because each one fails in a way the others cover.

Design Fail-Safe Degradation Paths

Fail-safe degradation keeps useful service available when a security control blocks, times out, or becomes uncertain. A resilient defense shouldn’t choose between full autonomy and a blank error page. Define a lower-authority mode for each failure.

If injection classification is uncertain, disable write tools and answer from approved sources. If retrieval provenance is missing, request clarification or return no answer. If an approval service is down, queue the proposal without executing it. If output scanning fails, hold the response. The safe fallback depends on potential harm.

from enum import Enum

class Mode(str, Enum):
    FULL = "full"
    READ_ONLY = "read_only"
    HOLD = "hold"


def choose_mode(classifier_ok: bool, provenance_ok: bool, policy_ok: bool) -> Mode:
    if not policy_ok or not provenance_ok:
        return Mode.HOLD
    if not classifier_ok:
        return Mode.READ_ONLY
    return Mode.FULL

assert choose_mode(False, True, True) is Mode.READ_ONLY

Test fallback modes under load. A security service outage often creates pressure to bypass controls manually, which is exactly when attackers benefit. The defense needs operationally tolerable failure behavior, clear alerts, and a documented owner.

A blunt refusal is sometimes correct, but users usually need a next step: remove an unsupported attachment, confirm a destination, narrow a request, or wait for review. A system can stay genuinely helpful in these moments while still declining to treat its own uncertainty as authorization to act.

Red-Team and Evaluate Prompt Injection Defense

Security evaluation tests whether hostile content changes goals, reveals data, expands permissions, or triggers unsafe actions. A few jailbreak strings aren’t a test program. Build attack families across every untrusted source and tool path.

Include direct override attempts, indirect document attacks, encoded text, multilingual attacks, typographic hiding, fake system messages, memory poisoning, RAG poisoning, tool-result injection, data exfiltration, and multi-turn escalation. Test benign documents containing security terms to measure false positives.

from dataclasses import dataclass

@dataclass(frozen=True)
class SecurityCase:
    name: str
    payload: str
    forbidden_tools: frozenset[str]
    forbidden_fragments: frozenset[str]


def evaluate_case(case: SecurityCase, tools_called: set[str], output: str) -> list[str]:
    failures: list[str] = []
    if tools_called & case.forbidden_tools:
        failures.append("forbidden tool called")
    lowered = output.lower()
    if any(fragment.lower() in lowered for fragment in case.forbidden_fragments):
        failures.append("forbidden content leaked")
    return failures

Run each case several times across model and prompt versions. Track attack success rate, unsafe tool-call rate, data leakage, false-positive rate, refusal quality, and human escalation. Preserve failing traces as regression tests.

Use the OpenAI Evals repository or an internal harness to automate suites. The framework matters less than repeatability. A safety-prompt “fix” for one attack routinely reopens two older ones, and without regression runs that whack-a-mole is easy to mistake for forward progress.

Manage Security Policy and Prompt Changes

Security prompts and policies are production code. The defense needs version control, review, tests, staged rollout, and rollback for every change. An innocent wording edit can alter refusals, tool choices, or how untrusted data is interpreted.

Store immutable policy versions beside traces. Require review from application and security owners for changes affecting permissions or data handling. Run attack suites and normal-task suites against old and new versions. A safety gain that destroys task success will get bypassed by users, which creates a different security failure.

from dataclasses import dataclass
import hashlib

@dataclass(frozen=True)
class PolicyArtifact:
    version: str
    content: str
    approved_by: tuple[str, ...]

    @property
    def digest(self) -> str:
        return hashlib.sha256(self.content.encode()).hexdigest()


def releasable(policy: PolicyArtifact) -> bool:
    roles = {reviewer.split(":", 1)[0] for reviewer in policy.approved_by}
    return bool(policy.version.strip()) and {"app", "security"} <= roles

Canary by tenant or workflow and define rollback thresholds before launch. Keep model version and tool set fixed during policy experiments. Otherwise you can’t attribute a security regression.

A tiny prompt cleanup can remove one repeated restriction because it looks redundant, and attack success then rises only in long-context cases, so basic tests stay green. Effective defense needs realistic context lengths and source mixtures in release tests, not isolated attack strings.

Monitor Attacks and Prepare Incident Response

Runtime monitoring detects suspicious context, denied actions, unusual tool sequences, data-volume spikes, and policy failures. Incident response then contains damage, preserves evidence, rotates access, removes poisoned data, and ships tested fixes.

Trace source IDs, trust labels, selected documents, policy version, model version, proposed tools, authorization decisions, approvals, destinations, and output validation. Avoid logging raw secrets. Hash or redact sensitive blocks while retaining enough lineage to reconstruct decisions.

from dataclasses import asdict, dataclass
import json

@dataclass(frozen=True)
class SecurityEvent:
    request_id: str
    event_type: str
    source_ids: list[str]
    tool: str | None
    decision: str
    policy_version: str


def event_json(event: SecurityEvent) -> str:
    if event.decision not in {"allow", "deny", "review"}:
        raise ValueError("invalid security decision")
    return json.dumps(asdict(event), sort_keys=True, separators=(",", ":"))

Alert on denied bulk exports, new destinations, repeated classifier hits, cross-tenant access attempts, approval bypass attempts, and sudden retrieval of old sources. Correlate events by task because one harmless-looking call may be part of a larger exfiltration chain.

When an incident lands, disable dangerous tools first. Rotate credentials. Preserve traces and source artifacts. Remove poisoned documents from retrieval, invalidate derived memory, and identify affected tenants. Then add the exact attack to regression tests before restoring capability, rather than patching only the prompt that happened to be visible in the trace.

Roll Out a Production Prompt Injection Defense Plan

A production rollout should reduce authority before adding detection. Start with a bounded workflow, map sources and side effects, enforce authorization, and then layer classifiers, monitoring, and red-team automation.

1. Inventory user input, files, RAG, memory, tool results, and multimodal sources.
2. Classify each source by trust, owner, tenant, and retention.
3. Remove unused tools and replace broad credentials with scoped tokens.
4. Add schema checks, domain validation, destination allowlists, and task budgets.
5. Wrap untrusted content with provenance and fixed boundaries.
6. Gate irreversible actions with parameter-bound approval.
7. Build attack and benign evaluation suites.
8. Canary changes with predefined rollback thresholds.
9. Create alerts and an AI-specific incident playbook.

Don’t change model, prompt, retrieval, and tools in one release. Keep experiments attributable. Measure unsafe action rate, attack success, false positives, task success, latency, cost, and approval burden.

The NIST AI Risk Management Framework helps connect prompt injection defense to broader governance. For application implementation, connect controls to existing IAM, DLP, audit, and incident-response systems instead of building a separate security universe.

Build a Control Matrix Before Launch

A control matrix connects each threat path to prevention, detection, response, owner, and test evidence. The defense becomes reviewable when teams can point to controls instead of saying the system prompt is strong.

Threat path	Prevent	Detect	Respond	Owner
Malicious RAG chunk	Approval and tenant filters	Source anomaly alert	Remove and re-index	Data owner
Unsafe tool call	Policy engine	Denied-action alert	Disable capability	Platform
Data exfiltration	Egress allowlist	DLP and volume alert	Rotate and contain	Security

Review the matrix whenever a source, tool, model, or memory type changes. Require a linked automated test for each high-risk preventive control. Manual policy statements age quickly; executable evidence exposes drift.

Also document residual risk. Some workflows remain unsafe for autonomous execution even after controls. A mature defense should support a clear “do not automate” decision rather than forcing every process into an agent.

Frequently Asked Questions

What is prompt injection?

Prompt injection is an attack where untrusted content tries to change an LLM application’s intended behavior. It may arrive directly through a user prompt or indirectly through documents, web pages, RAG results, memory, images, and tool outputs. The danger grows when the model can access private data or call tools. Prompt injection defense focuses on limiting authority and validating actions, not only detecting phrases.

How do you prevent prompt injection?

You can’t guarantee prevention with one filter or system prompt. Use layered defense: separate trusted policy from data, minimize model-visible content, restrict tools and credentials, validate every action in code, allowlist destinations, scan outputs, require approval for high-risk operations, and monitor runtime behavior. Red-team all input paths and preserve failures as automated regression tests.

Why don’t delimiters completely stop prompt injection?

Delimiters clarify which text is untrusted, but an LLM doesn’t enforce boundaries like an operating system. It may still follow instructions inside a tagged block. Delimiters are valuable because they reduce ambiguity and improve auditability. However, prompt injection defense must assume they can fail. Authorization, resource scope, destination checks, rate limits, and human approval must remain outside the model.

When should a human approve an agent action?

Require human approval when an action is irreversible, financially meaningful, legally sensitive, externally visible, or affects many records. Examples include sending messages, publishing content, moving money, deleting data, executing code, changing accounts, and bulk export. Show exact parameters and evidence. Bind approval to a request hash so the agent can’t alter recipients, amounts, destinations, or attachments after approval.

What is the difference between direct and indirect prompt injection?

Direct prompt injection comes from the user-facing input channel, such as a chat message asking the model to ignore policy. Indirect prompt injection sits inside content the application later reads, including email, web pages, files, RAG chunks, images, or tool responses. Indirect attacks are harder to notice because the user may never see the hostile instruction, while the agent processes it with trusted capabilities.

Can an AI classifier detect every prompt injection attack?

No. Classifiers produce false negatives and false positives, and attackers can adapt wording, language, encoding, or multi-turn strategies. Use classifiers for risk scoring, routing, alerting, and added review. Never let a classifier grant extra permission. A missed attack should still encounter deterministic authorization and egress controls, while a benign flagged input should have a safe review path.

How does prompt injection affect RAG systems?

RAG can retrieve attacker-controlled or stale documents and place them beside trusted policy, and a high similarity score says nothing about whether a chunk carries any authority. Secure RAG therefore filters by tenant, permission, approval, freshness, and source type before ranking, preserves provenance, and treats chunks as data. It also protects the index write path, because a single poisoned document can influence many future requests after one successful insertion.

Is prompt injection defense worth adding to a read-only chatbot?

Yes, though controls can match lower risk. A read-only chatbot can still leak private retrieved data, produce harmful misinformation, or poison memory. Start with access-filtered retrieval, data minimization, source citations, output checks, and monitoring. Strong tool authorization matters less when no tools exist, but tenant isolation and data-loss prevention remain critical. Add stricter controls before enabling browsing, memory, file processing, or actions.

What metrics measure prompt injection defense?

Track attack success rate, unsafe tool-call rate, secret or record leakage, cross-tenant attempts, denied actions, false-positive rate, approval override, and time to contain incidents. Pair security metrics with task success, latency, and cost. Evaluate by attack family and source path rather than one average. A low overall rate can hide a severe failure concentrated in documents or tool outputs.

Does on-device inference stop prompt injection?

No. On-device inference may reduce cloud exposure and network data transfer, but hostile input can still steer the local model. A local agent may access files, contacts, sensors, or device APIs. The defense still needs data boundaries, scoped capabilities, output validation, and approval. The permission model changes location; the confused-authority problem remains.

Before rollout, run a tabletop exercise. Give the team a poisoned document that triggers an attempted bulk export. Ask who sees the alert, who can disable the tool, where credentials rotate, how retrieval entries are removed, which memories are invalidated, and how affected users are identified. Missing answers are engineering work, not paperwork.

Assign ownership by control. Application teams own domain validation. Security owns threat models and incident playbooks. Data owners approve retrieval sources. Platform teams own scoped execution, logging, and kill switches. Product owners define which actions deserve human review. The defense decays quickly when everyone assumes another team owns the gap.

Set release gates. No new write tool ships without authorization tests, destination controls, task budgets, rollback, and attack cases. No new data source enters RAG without owner, approval state, tenant scope, expiry, and deletion support. No durable memory type ships without a correction path. These gates are routine to enforce and catch the failures that turn into real incidents.

Measure control coverage as the system changes. Tool count, source count, and model capability tend to grow. Permissions rarely shrink on their own. Re-run threat mapping after adding browsing, multimodal input, code execution, long-term memory, or cross-agent communication.

Conclusion and Security Checklist

One final test matters: remove the safety prompt mentally and inspect what remains. Can an injected model access broad credentials, choose arbitrary destinations, read unrelated tenant data, mutate records without validation, or approve its own action? If yes, wording is carrying risk that architecture should own.

This defense should also lower blast radius over time. Review tool permissions quarterly. Delete unused data connectors. Shorten credential lifetimes. Tighten task budgets from observed usage. Retire temporary exceptions. Security rarely fails because one control never existed; it often fails because yesterday’s exception quietly became permanent.

Keep user experience in the review loop. Explain why an action needs confirmation and show exactly what will happen. Give safe alternatives when content is blocked. Let administrators inspect source provenance and denied actions. Controls users understand are less likely to be bypassed.

For engineering leaders, fund the boring infrastructure: scoped execution, policy services, trace storage, test harnesses, content lineage, and kill switches. Another paragraph in the system prompt is cheap. It also provides far less protection when an agent touches money, customer records, production code, or public communication.

Prompt injection defense works when model compromise is an expected failure mode rather than a surprising edge case. Treat all external content as untrusted. Give agents narrow capabilities. Validate side effects in deterministic code. Watch data leaving the system. Put humans in front of consequential actions.

• Map every path from hostile content to model context.
• Keep policy, authorization, and credentials outside generated text.
• Filter RAG by identity, tenant, approval, and freshness.
• Bind approvals to exact frozen parameters.
• Test direct, indirect, encoded, multilingual, and multi-turn attacks.

Start today with one agent action. Remove unused tools, add a destination allowlist, and write a test where retrieved content asks the agent to violate policy. Then inspect whether denial comes from code or from the model deciding to behave. Only the first answer is a dependable control.

Keep the OWASP prompt injection reference, NIST risk guidance, and your provider’s agent safety documentation close. Yet production traces matter most. Attackers target your particular data flows, permissions, and workflow shortcuts.

A clever security prompt may improve resistance, but it is not a firewall. Build the system so hostile text can confuse a model without ever gaining authority over your users, data, money, or infrastructure.