Adversarial example attack · model
An input crafted to look normal to a human but to push a model into the wrong output. Classic in vision; in LLMs it shows up as subtle token tricks that flip a classifier or jailbreak a guardrail.
A quick-reference glossary for the vocabulary that gets thrown around in AI security docs, vendor pitches, and incident write-ups. Definitions are plain-language and direct, not academic. If a term is genuinely contested, we pick the usage security engineers actually say and call it out.
An input crafted to look normal to a human but to push a model into the wrong output. Classic in vision; in LLMs it shows up as subtle token tricks that flip a classifier or jailbreak a guardrail.
Training a model against the very attacks you want it to resist, so it learns to survive them. In LLM-land this is how safety RLHF and red-team fine-tunes harden a model against jailbreaks.
An LLM-driven program that plans, calls tools, and acts in the world across multiple steps with limited human review. "Autonomous" is a gradient; the security-relevant question is how much irreversible damage it can do between checkpoints.
The project of making a model do what its operators and users actually want, not what they literally asked for or what it guessed. In practice a mix of RLHF, constitutional training, and guardrails; in security terms it is the baseline that jailbreaks try to defeat.
An organization-level management system for AI, modeled on ISO 27001 ISMS thinking. The reference standard is ISO/IEC 42001:2023 — the first certifiable AIMS.
NIST's AI Risk Management Framework: a voluntary structure of Govern, Map, Measure, Manage. Generic enough to apply to any AI system; the GenAI Profile (NIST AI 600-1) translates it for generative AI specifically.
The measurable cost in capability, speed, or helpfulness you pay to make a model safer. Worth knowing because aggressive "alignment" can push users to fine-tune around it — or to bypass it entirely via jailbreaks.
Algorithms that use one LLM as the attacker to automatically find jailbreaks against another model — PAIR (prompt automatic iterative refinement) and TAP (tree-of-attacks-with-pruning) are the named techniques. Cheap to run, which is what makes them scary.
A behavior secretly planted in a model that fires on a specific trigger — a phrase, image patch, or token sequence — and stays quiet otherwise. A supply-chain risk for open-weights and fine-tuned models; the sleeper agent pattern is one example.
The swarm of low-effort jailbreak styles — persona play, refusal-suppression preambles, base64, role-redirects, "hypothetical" framing — that still slip past weakly aligned models. Most public jailbreaks in 2025 are stitched from this bag.
When a model deployed at scale makes a pre-existing social skew sharper, not flatter — by ranking, recommending, or generating content that over-represents a stereotype. A safety issue, not strictly a security one, but it surfaces in red-team evals.
An LLM agent that drives a real web browser to fetch pages, click, fill forms, and authenticate on behalf of a user. From a security view it is the worst indirect-injection surface on earth: every page it visits can speak instructions to it.
The communiqué from the 2023 UK AI Safety Summit where 28 states agreed to cooperate on frontier-AI risk and safety institutes. More diplomatic alignment than binding rule; the practical footprint is it spawned US/UK AISIs.
A design where an agent's rights are explicit, revocable tokens (capabilities) bound to a specific action — not a blanket role. For agentic AI this is the right primitive: a "send_email(to, subject, body)" should be a scoped capability, not a free verb.
The Coalition for Content Provenance and Authenticity spec: cryptographically signed provenance and edit history embedded in media files. A mitigation for synthetic media, not a detector — it tells you what the creator claimed, if they opted in.
Asking a model to show its reasoning steps before the answer. Improves task accuracy; in security it matters because reasoning tokens are also a channel — monitoring CoT is one of the better signals for catching goal-hijacking and sleeper behavior.
The legal question of who is on the hook when an organization's chatbot gives wrong, harmful, or invented advice. Air Canada lost the precedent-setting 2024 case over its support bot's fabricated refund policy — the operator is bound by what its bot says.
A classic security problem where a privileged program is tricked by a less-privileged caller into misusing its privileges. Prompt injection is the LLM form of it: your agent has the keys, the attacker has the message, and the agent can't tell who asked for what.
An agent mode where the model operates a GUI — mouse, keyboard, screen — like a remote human. Powerful, and a giant attack surface: anything on screen, including ads and chat pop-ups, becomes an instruction-injection channel.
Anthropic's training method where an AI critic scores and revises model outputs against a written "constitution" of principles, cutting the human-labeling cost. A stepping stone from RLHF; produces models that are easier to align at scale.
The threat category of capable models helping with offensive cyber operations — generating exploit code, phishing kits, or recon automation. Frontier-model evaluators rate this as a misuse capability and gate it behind pre-deployment tests.
The original Reddit-era "Do Anything Now" persona jailbreak that asked ChatGPT to roleplay an unconstrained alter-ego. Mostly historical, but the persona-frame trick underneath still works on weaker models.
Tampering with training data so the model learns the wrong thing — a backdoor trigger, a skew, or a specific failure mode. Distinct from RAG poisoning because it affects the model itself, not just retrieved context.
Manipulating the model's decoding step — temperature sampling, top-k, beam choices — to coax out a harmful output the model would normally refuse. Often paired with GCG-style token optimization.
AI-generated synthetic media — face swaps, voice clones, full video — made to look or sound like a real person. Named after a 2017 Reddit user; now the umbrella term for synthetic-media fraud and disinformation.
A formal guarantee that the output of a query (or a model) cannot reveal whether any one individual was in the training data, within a tunable budget ε. Useful for training on sensitive text; expensive and rarely perfectly applied.
Querying a victim model enough times to train a smaller "student" that approximates it — effectively stealing the model's behavior. If the victim is gated and the student leaks the behavior, you've cloned the capability without consent.
Umbrella term for the practical work of hardening AI products — input guards, structured output, tool capability scoping, sandboxing, monitoring, evals, human checkpoints. Synthesizes what the rest of this glossary names individually.
A numeric vector that represents a piece of text (or image) as a point in a high-dimensional space, so semantic similarity becomes a distance. The building block of RAG retrieval, clustering, and most "semantic" features — and an attack surface via inversion.
Reconstructing the text (or chunks of it) that produced an embedding vector. An attack when embeddings are exposed through an API — it inverts a "safe" representation leaking out PII from the original document.
Regulation (EU) 2024/1689 — the first major AI-specific law. Risk-tiered: banned uses since Feb 2025, general-purpose AI obligations from Aug 2025, high-risk system rules phased through 2026-27. Extraterritorial: it reaches non-EU vendors selling into the bloc.
A repeatable test run that measures how a model behaves — accuracy on a task, refusal rate on harmful prompts, jailbreak success, bias. The plural "evals" covers the whole harness; in security, evals are how you know whether your defenses work.
When an agent has more authority — wider tool access, fewer approvals, irreversible verbs — than its task actually needs. It is the single largest amplifier of every other attack on this list: jailbreaks are annoying, jailbreaks plus "send_email and rm -rf" are catastrophic.
Any attack that pulls something out of a model it shouldn't reveal — training data verbatim, system prompt, weights via distillation, or PII memorized during pretraining. The umbrella that covers model extraction, prompt leak, and membership inference.
Stripping the safety alignment off a model by fine-tuning it on a small adversarial dataset — often cheap, sometimes via public open-weights. The uncomfortable truth behind "the model refused in our tests": fine-tune access can undo that.
A model at the leading edge of capability, where new risks appear before mitigations are documented. The regulatory poker chip: "frontier" gets you into the EU AI Act's GPAI tier and US/UK AISI pre-deployment evaluations.
A structured checklist-against-threats (OWASP LLM Top 10, NIST AI RMF, SAIF) used to organize an AI security program. Aids communication and coverage checking; never a substitute for the actual engineering work.
The first reliable whitebox jailbreak method: optimize an adversarial-suffix token sequence by following gradients on the model's logits. Demonstrated in 2023 that gradient-based attacks transfer to closed models via prompt-suffix reuse.
The EU AI Act's legal category for foundation models not built for a single narrowly defined use — i.e., GPT-style and most open-weight frontier models. GPAI providers carry disclosure, documentation, and (for systemic-risk GPAI) pre-deployment eval obligations.
Pushing an agent off its assigned objective onto an attacker-chosen one — typically the payload of an indirect prompt injection. The model still executes competently; it is just chasing the wrong goal now.
The policies, ownership, and review loops that decide who is allowed to deploy what model, against what evals, with what monitoring. Boring and load-bearing — every framework (AIMS, RMF, EU AI Act) is mostly a governance recipe.
When a model produces fluent, confident output that is wrong — a fabricated citation, a non-existent API, a made-up refund policy. Not technically an attack, but the failure mode that makes chatbot-liability and RAG-poisoning real.
A standardized suite of harmful prompts and behavior categories (HarmBench, AdvBench, MaliciousUseBench) used to score models and defenses on a common ruler. The point is comparability between vendors, not philosophical completeness.
A decoy instruction planted in a page or document specifically to catch an agent that obeys injected instructions — useful as a detection signal in place of, or alongside, real guards. If your summarizer summarizes the honeypot's hidden command, you know you're owned.
A required human approval step before the agent performs an irreversible or expensive action. The cheapest, most reliable mitigation for excessive agency — the art is gating only the verbs that actually need it, otherwise humans rubber-stamp everything.
Prompt injection delivered through content the model reads on its own — web pages, email bodies, retrieved documents, MCP tool descriptions — rather than user input. The defining LLM-agent vulnerability and OWASP LLM01; harder to stop than direct injection because the "user" never typed the payload.
Running a trained model to produce an output — the request-time phase, as opposed to training. Most attacks on this list happen at inference time, which is why inference-side defenses (guards, output validators, monitoring) get so much attention.
The 2023 international standard for AI management systems (the certifiable AIMS), structured like ISO 27001. The closest thing to a global, audit-ready AI governance certificate.
Companion to ISO 31000 risk management, specialized for AI risk. Less certifiable than 42001, more a how-to-think-about-risk document; useful when scoping a 42001 AIMS.
Any input crafted to make a model do something its alignment was supposed to refuse. The looser of the two big attack categories — prompt injection is "obey an attacker", jailbreak is "ignore your training"; the overlap is large and the words get used interchangeably, which is wrong.
Indirect prompt injection planted in content that only exists briefly — a transiently fetched page, an ephemeral tool result, a chat message that scrolls by. Hard to audit because the evidence disappears with the session.
Injecting attacker-chosen documents into the retrieval corpus so the model later retrieves them as "trusted" knowledge. Same effect as indirect prompt injection but the payload lives in your own vector DB.
A transformer trained on enough text that it can produce fluent natural-language continuations and learn in-context. The piece behind every entry on this site — and the part that is most often mistaken for the only attack surface.
OWASP LLM Top 10 (2025) item #1: "Prompt Injection." The category-topping risk for three years running, because indirect prompt injection remains unsolved at the model level and is the foundation of most agent attacks.
A proxy in front of an LLM endpoint that screens prompts in and answers out — pattern matching, topic classifiers, PII redaction, jailbreak heuristics. Useful as a layer; misleading when sold as a complete defense because indirect injection bypasses it.
Meta's open family of input/output classifier models trained to flag unsafe prompts and responses against a published taxonomy. Common building block for an LLM firewall without sending your data to a vendor classifier.
The Anthropic-documented trick where the user prepends a long list of fake Q&A pairs that practically beg the model to answer the final harmful one. Exploits in-context learning and the larger context windows of newer models; mitigated partly by prompt-isolation and structured-output discipline.
Anthropic's open protocol for connecting LLMs to external tools — servers, files, APIs — so an agent can list tools, call them, and read results. A unifying standard that also unifies the attack surface: every MCP server is a fresh indirect-injection and rug-pull target.
Deciding whether a specific example was in the model's training data by watching how the model responds to it (loss, entropy, output distribution). A privacy attack — "was this patient's record in the training set?" is a real question it can sometimes answer.
A short, structured disclosure document for a model — intended use, training data, eval results, known limitations. Required for GPAI providers under the EU AI Act; voluntary elsewhere but increasingly expected.
Stealing a model's behavior — usually by querying it until a surrogate approximates it, sometimes by lifting weights outright from a leaked checkpoint. The practical risk is IP loss plus the ability to red-team an unaligned clone offline.
Recovering training-data characteristics — sometimes near-verbatim examples — from a model's outputs. Worse than membership inference and known to be practical on small fine-tunes over sensitive text.
An attack that mixes modalities — text, image, audio — to slip a payload in where a guard watching one channel misses it. Examples: hidden text in an image embedded via OCR, or steganographic instructions in audio fed to a voice-capable model.
NIST AI Risk Management Framework 1.0 — voluntary US framework built around Govern, Map, Measure, Manage. The reference point most US enterprise AI governance programs cite and most auditors check against.
The July 2024 generative-AI companion profile to NIST's AI RMF. Translates the four functions into twelve GenAI-specific risks — including CBRN misuse, harmful bias, information integrity, and dangerous content.
A jailbreak or injection phrased in plain language rather than encoded tokens — persona play, hypothetical framing, role-redirect. The most common kind, because no optimization machinery is needed and they transfer across models.
The community-curated list of the ten most critical risks for LLM applications — injection, supply chain, excessive agency, unbounded consumption, sensitive-info disclosure, etc. Updated Nov 2025; the closest thing the field has to OWASP Top 10 status.
A model that runs locally on the user's phone or laptop rather than behind a vendor API. Better for privacy, worse for tamper resistance — an attacker who can run the model locally can fine-tune against it, probe whitebox gradients, or strip guards offline.
A model whose trained weights are published for anyone to download and run, often called "open-source" but without the training code or data. Security upside: third-party safety auditing. Downside: fine-tuning attacks and supply-chain poisoning become trivially possible.
Checking the model's response against a schema, allow-list, or classifier before it reaches the user or calls a tool. The last line of defense against jailbreaks and goal-hijacking — and the right place to enforce structured output.
Planting instructions in a model's input so it obeys the attacker instead of the operator — the SQL-injection of LLMs. The umbrella category; the direct form is user-typed, the indirect form is delivered through retrieved or tool-supplied content. Not the same as jailbreak, though people keep conflating them.
The discipline of clearly separating trusted instructions (system prompt) from untrusted content (retrieved docs, tool output, user input) inside the model's context, plus treating the untrusted parts as data not commands. Helps but does not solve indirect injection at the model level.
An extraction attack that gets the model to dump its own system prompt or other hidden instructions verbatim. Embarrassing when your "secret" guardrail telegrams itself in one polite request, but rarely does the leak itself enable a worse attack.
Personally identifiable information the model has memorized during training and can be coaxed to repeat — names, phone numbers, source snippets. A privacy breach that surfaces through extraction, membership inference, and model inversion.
The compromise of a retrieval-augmented pipeline through tainted documents in the store — wiki edits, indexed web pages, uploaded PDFs. The model retrieves the poison and treats it as trusted context; output intentionally wrong, leaky, or denial-of-service.
Knowing and proving where a piece of content came from — which model, which user, when, possibly with edits. The defense side of the deepfake problem; technical implementations include C2PA, watermarking, and signed manifests.
An attack that uses low-perplexity (highly predictable) token sequences to slip past perplexity-based filters — or high-perplexity sequences to deliberately trip them as a distraction. A cat-and-mouse example of why surface filters alone don't catch jailbreaks.
The training stage that scores candidate responses with a reward model fit to human labels and nudges the policy toward the preferred ones. The default alignment technique; also the layer that wears out fastest under fine-tuning attacks and jailbreak drift.
The pattern of fetching relevant documents from a store and inserting them into the model's context before it answers. The answer to "the model doesn't know my data"; also the largest expansion of the indirect-injection attack surface, because now every fetched doc is a prompt.
When an MCP-style tool server (or plugin) your agent already trusts is silently changed to a malicious version — new description, new endpoints, new side effects — and your previously-reviewed integration now executes attacker code. The supply-chain attack of the agentic AI era.
The organized practice of trying to break your own (or a vendor's) model before deployment — running attacks, jailbreaks, extraction, behavior probes, and writing up what worked. The output is an eval and a list of mitigations; without it you're guessing.
Anthropic's published commitment about what capabilities they will and won't ship until specific safety thresholds are met, with the evals to back it up. Other frontier labs (OpenAI, Google) have analogous documents; the format is becoming de facto industry norm.
A model behavior planted via data poisoning that lies dormant through normal training and evals, then activates only on a specific trigger sequence or date. Anthropic's 2024 paper demonstrated the pattern survives standard safety fine-tuning.
Running an agent's tool calls or code execution in an isolated, scoped environment — separate credentials, no production data, time- and call-limited. The defense for when prompt isolation fails and an attacker does drive the agent's actions.
The heavier sibling of a model card — covers the whole deployed system (model, pipeline, tools, eval evidence, residual risks). Anthropic popularized the format with Claude releases; the EU AI Act treats serious GPAI systems as requiring this depth of disclosure.
The hidden, operator-written instructions given to a model that set its role, rules, and tone. The "trusted" channel your defenses live in; also the thing prompt-leak attacks exfiltrate.
An open jailbreak-evaluation rubric that scores model responses for whether they actually leaked harmful content, not merely whether they "complied with the request." A response to false-positive inflation in naive refusal metrics.
A model agreeing with the user's stated belief even when the user is wrong, because RLHF rewarded smiles over truth. A safety bug, not strictly an attack — but it amplifies misinformation, makes false-premise prompting cheap, and degrades red-team signal.
Compromising the chain that delivers an AI system to its users — poisoned pretraining corpus, monkeypatched fine-tune pipeline, swapped open-weights checkpoint, trojaned tokenizer, or rug-pulled MCP server. OWASP LLM Top 10 calls this out explicitly (LLM08).
Forcing the model to produce answers inside a JSON / tool-call schema rather than free text, then validating against it before any action runs. Cuts a large class of goal-hijacking and chain-of-pitfall failures because the surface is constrained up front.
Google's Secure AI Framework — six elements (expand threat modeling, hold suppliers accountable, detect, automate, operate, govern). Less certifiable than ISO 42001, more an internal-and-supplier-management recipe aimed at Google's ecosystem.
Broad term for audio, images, video, and text generated by AI to imitate reality — the umbrella over deepfakes, voice clones, and synthetic-content disinformation. The risk axis most familiar to non-technical audiences and least addressed by technical defenses.
The agent pattern where the model emits structured function calls (and arguments) instead of plain text, and a runtime executes them and returns results back into context. The capability that makes "agentic" possible; the channel every tool-injection and tool-poisoning attack rides on.
When a tool server the agent trusts returns malicious instructions disguised as tool output (or as a tool description that the agent adopts as instructions). The MCP analogue of indirect prompt injection — every connected server is a new injection surface.
Embedding attacker instructions inside tool-call results (a returned web page, a database row, an email body) so the model treats the result as a new instruction source. Generalizes indirect prompt injection to any tool output that lives in context.
Encoding a forbidden payload in a way that evades surface filters but decodes naturally for the model — base64, ROT13, low-perplexity obfuscation, text hidden inside code. Worth knowing because the model's own tokenizer often "helpfully" reverses it in-context.
Anything the operator is willing to treat as authoritative — system prompts, signed tool descriptions, vetted documents. In LLM security the uncomfortable truth is "what counts as data" rather than "what counts as instructions" — and that line is fuzzy inside a context window.
A written account of which adversaries could abuse your AI feature, how, and with what consequences — against the model, the prompt, the retrieval pipeline, the tools, the integrations. Without one your "security" is a stack of unconnected vendor products.
Techniques that aim to remove specific knowledge or behavior from a trained model without retraining from scratch — forgetting a copyrighted text, an unwanted skill, or a PII-specific recall. Still mostly research; verified, robust unlearning remains unsolved.
The US AI Safety Institute (housed at NIST) — the federal body tasked with pre-deployment evaluations of frontier models, testbed standards, and bilateral cooperation with the UK AISI (now AISA). One of the practical outputs of the Bletchley process.
Inserting malicious vectors — or whitelisting malicious source documents — into the embedding store behind a RAG pipeline, so retrieval surfaces attacker-chosen content with high semantic similarity to benign queries. Same payload as poisoned RAG, funneled through the index.
Generating a synthetic voice that imitates a specific real speaker from a short sample. The 2024 Biden deepfake robocall is the canonical incident — even crude clones work on a phone line where there's no video to cross-check.
Embedding a statistical signal (low-bias token-frequency or green/red-list schemes) into generated text or media so it can be detected later. The provenance buddy of C2PA for detection — it tells you "this came from a model," not "this is true."
Microsoft's name for indirect prompt injection: the payload crosses from one prompt context (the document, retrieved content, or tool response) into another (your agent's instructions). XPIA, indirect prompt injection, and "indirect injection" are mostly synonyms.
The percentage of attempted attacks (jailbreaks, injections, extractions) that actually succeed against the target within a fixed budget. The number that matters in evals — not "we found one working jailbreak" but "what is our yield at 1000 attempts per technique.