Foundations

An offensive tester does not need to be able to train a model from scratch to attack it. They do need enough of the vocabulary, mechanics, and system layout that a target's architecture, error messages, and behavior are legible. This chapter is that baseline.

If you have shipped ML systems, skim it. If you have not, read it end-to-end. Later chapters assume the terminology introduced here.

1.1 Vocabulary and boundaries

Artificial intelligence (AI) is the umbrella term for any system that performs tasks that normally require human intelligence. In practice, in 2026, "AI" almost always means one of three things: a machine learning model trained on data, a generative model (usually an LLM) that emits text, images, audio, or video, or an agentic system built around one or more of those models.

Machine learning (ML) is the subset of AI where behavior is learned from data rather than programmed by hand. An ML system has a training phase where parameters (weights) are fit against labeled or unlabeled examples, and an inference phase where the fitted model is applied to new inputs.

Deep learning (DL) is ML with neural networks that have many layers. Convolutional networks (CNNs) for images, recurrent networks (RNNs) and transformers for sequences, and diffusion models for image generation are all deep-learning architectures.

Generative AI (GenAI) is deep learning that produces new content: text (LLMs), images (Stable Diffusion, Imagen), audio, video. Large language models are the biggest offensive attack surface today because they are given tools, memory, and privileges.

Agents are LLMs wrapped in a control loop that gives them the ability to call tools, read intermediate results, and iterate toward a goal. The dominant pattern is ReAct (Reason and Act): the model emits a reasoning trace, selects a tool call, observes the tool's output, and loops until it decides the goal is satisfied or a stop condition is hit. A single-agent system is one LLM in a ReAct-style loop. A multi-agent system is several agents cooperating, sometimes with a dedicated orchestrator that routes tasks between specialized sub-agents (a "triage agent," a "remediation agent," a "knowledge agent").

Tool calling (also called function calling) is the mechanism by which a model requests the execution of an external function using a structured schema (name, parameters, description) rather than free text. The model does not execute the tool itself — a runtime on the host application parses the model's structured request, executes the corresponding function, and feeds the result back into the model's context. Every tool description the model sees is untrusted input from a security standpoint, whether it comes from a local function definition or from a remote server; this is the foundation of tool poisoning attacks covered in Chapter 07.

Agent interoperability protocols standardize how agents discover and talk to tools and to each other:

• Model Context Protocol (MCP) — an open standard, originally published by Anthropic in late 2024, for connecting an LLM client to external tools, data sources, and prompts through a uniform server interface. An MCP server exposes a tools/list method describing its tools, and a tools/call method to invoke them. Because tool descriptions returned by tools/list are inserted directly into the model's context, a malicious or compromised MCP server can influence agent behavior before any tool is ever invoked. This guide uses the stable 2025-11-25 protocol revision as its normative baseline; see the versioned authorization specification, versioned transport specification, and official security guidance.
• Agent2Agent (A2A) protocol — an open protocol for agent-to-agent communication, in which agents publish a discoverable "Agent Card" at /.well-known/agent-card.json in A2A v1.0, describe their capabilities, and exchange messages with other agents across organizational and vendor boundaries. Legacy v0.2.6 deployments used /.well-known/agent.json. A2A threat modeling using the MAESTRO framework identifies agent card spoofing, task replay, and message schema violations as recurring risk classes (https://arxiv.org/html/2504.16902v1).

Three learning paradigms show up on engagements:

• Supervised learning — the model sees pairs of (input, expected output). Classification (spam / not spam) and regression (predict a number) are the two subvariants. Trained on labels, attacked with label flipping, clean-label poisoning, and adversarial evasion.
• Unsupervised learning — the model sees only inputs. Clustering, dimensionality reduction, and anomaly detection fall here. Attacked mostly through embedding-space manipulation.
• Reinforcement learning (RL) — the model interacts with an environment and receives reward signals. Rare on typical enterprise engagements but shows up in agentic systems and in fine-tuning phases (RLHF for LLMs). Attacked by reward-signal manipulation, gradient leakage from human-feedback datasets.

1.2 A very short review of ML mechanics

Features — the numeric representation of an input the model actually sees. A URL is not features until it has been tokenized, hashed, one-hot-encoded, or embedded. Attacks that change a feature's value below the classification threshold are called evasion attacks (Chapter 10).

Labels — the ground truth in supervised training. Attacks that swap labels during training are the label-flipping family (Chapter 09).

Loss function — a scalar measure of how far a prediction is from the label. Training minimizes loss by adjusting weights. Gradient-based attacks compute the gradient of loss with respect to input, then take a small step in the direction that maximizes loss instead — a mislabeling in the direction of the model's own worst confidence.

Overfitting — when a model has memorized training data instead of generalizing. Overfit models are more susceptible to membership inference and inversion attacks (Chapter 13) because they retain per-example signal.

Generalization gap — the difference between training accuracy and test accuracy. Large gap = overfitting = more privacy leakage.

Embedding — a fixed-dimensional numeric vector that represents an input's semantic content. Text embeddings live in spaces of dimension 384 (MiniLM), 768 (BGE-base), 1024 (Cohere), 1536 (OpenAI ada-002 and 3-small), or 3072 (OpenAI 3-large). Two texts with similar meaning have similar embeddings under cosine or Euclidean distance. RAG systems index documents by their embeddings and retrieve them by similarity to a query embedding.

Token — the atomic unit an LLM processes. Text is broken into tokens (usually 3–4 characters each) by a tokenizer that was fit on a training corpus. Tokenizer boundaries matter for jailbreaks, unicode-based bypasses, and payload smuggling — splitting a filtered word across a token boundary or hiding it behind zero-width Unicode characters is enough to defeat many keyword-based guardrails while the model itself still reconstructs the intended meaning.

Context window — the maximum number of tokens an LLM can attend to at once. Modern LLMs range from 4k to over 1M tokens. Context is where system prompt, user prompt, retrieved documents, tool outputs, and — in agentic systems — the outputs of other agents are all concatenated into a single trust-flattened stream.

1.3 Model families to recognize on engagements

Recognizing which family you are attacking narrows the technique set. FGSM targets differentiable models (CNNs, LLMs). Label flipping targets any supervised classifier. Membership inference works best on overfit classifiers, especially deep ones on small datasets.

1.4 The system components you will actually attack

Real engagements rarely involve attacking a raw model. They involve attacking a system built around a model, and by 2026 that system is very often agentic. The recurring components are:

Model registry — MLflow, SageMaker Model Registry, Vertex Model Registry. Stores versioned model artifacts, container definitions, and environment variables. Frequently misconfigured: unauthenticated APIs, artifact stores writable by too many principals, secrets baked into environment variables.

Feature store — Feast, Tecton, Hopsworks. Stores precomputed features for training and inference. Compromise here is upstream of every model that consumes those features.

Vector database — Weaviate, Qdrant, Milvus, Pinecone, pgvector, Chroma. Stores embeddings for retrieval. Frequently unauthenticated inside the cluster. Reads (embedding inversion) and writes (poisoning) are both high-impact.

Inference server — vLLM, TGI (Text Generation Inference), Triton, TorchServe, KServe, SageMaker endpoints, Ollama. Serves a model over HTTP. Version-specific vulnerabilities, occasional unauthenticated deployment inside VPCs.

Experiment and training platforms — MLflow tracking server, Ray clusters (Ray Dashboard, Ray Serve), Jupyter and JupyterHub notebooks. These are development-time infrastructure that frequently gets left reachable in production networks, and they are covered in the reconnaissance and infrastructure chapters because their default configurations are often unauthenticated.

Orchestration — Kubernetes, ECS, Nomad. AI workloads have quirks: GPU passthrough (containers get direct hardware access), model-loader init containers, shared-volume request queues, sidecars that scrape telemetry from co-located model containers.

Agent frameworks — LangChain, LangGraph, CrewAI, AutoGen, Google's Agent Development Kit (ADK), OpenAI's Agents SDK. These provide the ReAct loop, memory management, and multi-agent orchestration primitives on top of a base LLM. Each has its own default trust assumptions — CrewAI, for example, ships with task-level tool scoping available but not enabled by default — and its own disclosed vulnerability history (LangGraph's checkpoint deserializer, CVE-2025-64439; Google ADK's unauthenticated code-injection path, CVE-2026-4810).

MCP servers — expose tools (aws_cli_exec, snow_create_ticket, vault_rotate_secret, filesystem read/write) to LLMs. Almost always deployed inside a trust zone with high-value credentials from Vault or SSM. Treat every MCP server as both a target (its own vulnerabilities, its own exposed port) and as an untrusted input source (its tool descriptions are attacker-controllable if the server or its supply chain is compromised).

A2A endpoints — agents that publish an Agent Card and accept tasks from other agents, sometimes across organizational boundaries. Every A2A server is a new network-facing surface with its own authentication and replay-protection requirements.

Guardrails — input filters (regex, PII detection, classification of prompts as safe/unsafe) and output filters (redaction, content moderation, structured-output validation). Both are bypassable via encoding, unicode, indirect instruction, and reformatting.

Monitoring / observability — Arize Phoenix, LangSmith, Weights & Biases, custom logging. Determines which of your actions produce audit events. Extract detection rules whenever the vector store allows it (Chapter 06).

1.5 Two mental models that will save time

Traditional systems vs AI systems. Traditional pentesting targets deterministic components with well-defined interfaces: a login form takes a username and password, returns a token or an error. Behavior is either broken or not. AI systems are probabilistic. The same prompt can produce different outputs on different runs. Success in AI red teaming often means increasing the probability of a target outcome above a useful threshold, not achieving deterministic exploitation on the first try. Bring the mindset of statistical tuning, not the mindset of "exploit / no exploit". This is even more pronounced in agentic systems, where the same injected instruction can succeed on one ReAct iteration and fail on the next depending on what else is in context.

Data as an attack surface. In traditional systems, data mostly sits in databases, gets served, and the exploitation happens somewhere else. In AI systems, data is part of the model, and in agentic systems, data is part of the control flow. Training data ends up encoded in weights (extractable through inversion, memorized through overfitting). RAG data ends up encoded in embeddings (invertible via ALGEN / Vec2Text). Fine-tuning data ends up baked into behavior (backdoors, biases). Agent memory — session-scoped or persistent — ends up encoded as instructions the agent will act on later, which is why memory poisoning is treated as a first-class attack category (Chapter 07) rather than a footnote. "The dataset is the source code" is not a slogan; it is how you should think about every training pipeline, every RAG index, and every agent memory store you encounter.

1.6 What "value" looks like on an AI target

The old red-team dopamine loop is Domain Admin or root on a crown-jewel server. Both still apply to AI engagements — the ML infrastructure sits inside the same enterprise environment as everything else, and Chapter 12 covers infrastructure compromise in detail. But AI systems introduce new categories of value:

• Model weights themselves — often the target of extraction on frontier or fine-tuned models trained on proprietary data
• Training and fine-tuning datasets — customer data, internal documents, code repositories converted into training corpora
• Vector databases — knowledge bases holding runbooks, credentials, SOPs, embedded but recoverable via inversion
• Agent tools and credentials — the token an agent uses to call the AWS API, ServiceNow, or Vault is directly reachable through prompt injection, and in a multi-agent system a single over-privileged agent can act as a confused deputy for every other agent that trusts it
• MCP tool schemas — a complete map of what agent capabilities exist across a target's stack, frequently reachable without authentication if a server exposes a listing endpoint
• Agent memory — persistent instructions planted once and triggered by an unrelated future interaction, which is a durability property that classical prompt injection does not have
• Detection rules stored in the vector DB — knowing what the defender monitors is worth an entire additional engagement, and vector DBs frequently contain the detection ruleset

Every subsequent chapter builds on these categories. Chapter 02 turns this component and value map into an actual engagement methodology.

Practice checklist

• Given a target LLM, identify family (transformer size, tokenizer, quantization) from HTTP headers, /v1/models, timing, error strings
• Given a target embedding vector, identify the model producing it from dimensionality plus inference probing
• Given a description of an enterprise AI product, list the components (model registry, feature store, vector DB, inference server, orchestration, agent framework, MCP server, A2A endpoint) that are almost certainly present
• Identify whether the target is agentic, and if so, name its control-loop pattern (ReAct or a variant), its orchestration framework, and its interoperability protocols (MCP, A2A, or a custom equivalent)
• Map "value" on an engagement from files/hosts to also include embeddings, agent tokens, agent memory, MCP tool schemas, and detection rules

MITRE ATLAS references

The taxonomy that anchors the rest of the guide:

• AML.TA0002 ML Model Access (foundations underlying most later techniques)
• AML.TA0004 ML Attack Staging
• AML.TA0005 Exfiltration
• AML.TA0007 Impact

Specific ATLAS techniques appear in every subsequent chapter under their own "ATLAS references" tables. For agentic-specific risk categories not yet fully covered by ATLAS, this guide cross-references the OWASP Top 10 for Agentic Applications (ASI01–ASI10): ASI01 Agent Goal Hijack, ASI02 Tool Misuse and Exploitation, ASI03 Identity and Privilege Abuse, ASI04 Agentic Supply Chain, ASI05 Unexpected Code Execution, ASI06 Memory and Context Poisoning, ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Agent Failures, ASI09 Human-Agent Trust Exploitation, and ASI10 Rogue Agents.

Further reading

• MITRE ATLAS matrix — https://atlas.mitre.org/matrices/ATLAS
• Google Secure AI Framework (SAIF) — https://saif.google/
• OWASP LLM Top 10 (2025) — https://owasp.org/www-project-top-10-for-large-language-model-applications/
• OWASP Top 10 for Agentic Applications (2026) — https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
• Model Context Protocol 2025-11-25 authorization specification — https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization
• Model Context Protocol security best practices — https://modelcontextprotocol.io/docs/tutorials/security/security_best_practices
• A2A protocol threat modeling with MAESTRO — https://arxiv.org/html/2504.16902v1