Attacking Embeddings

Chapter 06 introduced embedding inversion in the context of RAG. This chapter goes deeper. Embedding attacks are worth their own treatment because they undermine an assumption operators make routinely — that vectors are one-way, similar to hashes — and because the technique set is specific enough that reaching a passing engagement result requires understanding it in isolation.

An embedding is a fixed-dimensional numeric representation of an input's semantic content. The properties that make embeddings useful for retrieval — geometric similarity reflects semantic similarity — are exactly what makes them vulnerable to inversion. Where meaning is preserved, information can be recovered. 2025–2026 research pushed this from a theoretical concern to a practical one: current state-of-the-art inversion recovers near-verbatim text from black-box embeddings, defeats differential-privacy noise defenses, and can be assembled from a production API for well under the cost of a single engagement day.

8.1 What is recoverable

Three categories of information can be extracted from embeddings.

Embedding inversion. Reconstruct the original text (or an approximation of it) from a single vector. Highest value, highest technical difficulty. Yields human-readable content directly.

Membership inference. Yes/no: is a specific string in the corpus that produced this vector? Limited alone, powerful combined with inversion (fill placeholders after structure is recovered).

Attribute inference. Predict metadata about the source (sentiment, topic, presence of PII) from the vector. Used to triage large stores before committing to expensive inversion on specific vectors.

The choice among techniques depends on what you know about the target and what you have time to prepare.

8.2 Zero-shot inversion

Zero-shot means no prior training on the target's model — attack from scratch at engagement time. Three approaches to know, in order of increasing sophistication.

8.2.1 Template-based inversion

The observation that makes it work: embeddings capture structure well even when they lose high-entropy detail. A password embedding preserves "this is a password field for the accounts system at northstar.example," but does not preserve the exact characters of a specific random string like Xj7#kL9p!.

The technique combines two steps:

1. Structure recovery. Build a template bank — many candidate contextualizations of common enterprise content ("The password for the accounts portal is {PASSWORD}", "To reset your password, use {PASSWORD} at first login", "New employees receive a temporary password {PASSWORD} valid for 24 hours"). Auto-detect the target's domain from the embedding by scoring similarity against domain-signature phrases, then generate templates weighted toward that domain.
2. Slot filling via membership inference. Iterate a wordlist through each template's slot. For each candidate fill, embed the completed template and compare to the target vector. The candidate whose completion produces the highest cosine similarity is the recovered value. Cross-template voting produces a consensus.

Refinements that pay for themselves:

• Margin-aware scoring. Compute similarity as the improvement over a neutral baseline (the template with a null token filler). This isolates the slot's contribution from the surrounding context. Without margin awareness, a wordlist entry that happens to be semantically related to the surrounding context (a password containing "north" scoring high because the company is "Northstar") wins on noise, not on match.
• Diversity clustering of templates. Near-duplicate templates should not each get a full vote — cluster templates by embedding similarity and treat each cluster as a single voter.
• Two-stage narrowing. First pass over a coarse subset of the wordlist to find candidates near the top; second pass over the top-100 with the full template bank. Cuts compute by an order of magnitude.
• Progressive fill-and-lock. For templates with multiple slots (URL, username, password), fill each slot iteratively — lock the highest-confidence slot, re-score the others with the lock in place.

Wordlist selection is critical. Too large (100k SecLists) and the noise floor swamps the signal from a 5-10 percent content contribution. Too small and the target is missed. The sweet spot for enterprise passwords is a 500-5000 targeted list — the top passwords for the region, the top passwords with corporate substitutions, and known-leaked passwords for the target sector.

8.2.2 Beam-search inversion and Zero2Text

Rather than iterate a fixed wordlist through templates, use a generator LLM (GPT-2 works, larger LLMs work better) to propose next tokens and score each extension by similarity to the target embedding. Keep the top-k beams at each step and continue for 30-40 steps. Converges to plausible English text near the target's embedding.

Zero2Text (arXiv 2602.01757, February 2026) is the current state of the art for this family, and the first practical training-free approach. Earlier methods split into two unsatisfying camps: optimization-based approaches require computationally prohibitive numbers of queries, and alignment-based approaches (ALGEN, Section 8.3.1) require unrealistic in-domain training data. Zero2Text needs neither.

Mechanism — "recursive online alignment": it combines the linguistic priors of a local LLM (for example, Qwen) with a dynamic ridge regression that updates token by token during decoding:

1. At each decoding step t, the local LLM generates logits for the next candidate token (vocabulary restricted to ASCII).
2. Candidates are filtered for diversity (pairwise cosine similarity below a threshold).
3. A subset of candidate sentences is sent to the victim API to obtain real embeddings, which solve W^t = (E^t⊤E^t + λI)^{-1}E^t⊤Ẽ^t — an online-updated linear projection.
4. Candidates are scored by combining the local LLM's prior with the projected cosine similarity; beam search (beam size 10) runs until [EOS].

Reported effectiveness: against OpenAI's text-embedding-3-large on MS MARCO, Zero2Text achieves 1.8 times more ROUGE-L and 6.4 times more BLEU-2 than the best baselines (Vec2Text, ALGEN, TEIA) — ROUGE-L of 26.08 versus 14.79 for ALGEN. Query cost averages 2,180 sentences (13.88k tokens) per reconstruction, far more efficient than Vec2Text with its corrector stage (82.6k tokens). Critically, standard defenses do not mitigate it: Gaussian/Laplacian noise and local differential privacy mechanisms (normalized Laplace, Purkayastha) fail to meaningfully degrade the attack, which retains a BLEU-1 of 20.36 even under strong noise (arXiv:2602.01757, Promptfoo analysis). This is the practical implication for a tester: DP-noised embedding pipelines are not automatically safe from inversion, and a manual claim of "we add differential privacy noise before storing embeddings" should be tested against Zero2Text specifically, not assumed sufficient.

Practical extensions on top of beam search generally:

• High-entropy slot detection. As beam search progresses, watch for tokens where the beam is essentially guessing (uniform distribution over candidates) — those positions are high-entropy in the source. Replace with {PASSWORD} slots, then run template filling on those slots.
• Regex-guided proposals. When entropy detection flags a slot, run regex-typed generators for known formats: JWT (eyJ), OpenAI API keys (sk-), hex strings, mixed passwords.
• Dual-embedder scoring. Use a locally-run embedding model to score candidate tokens and a ridge-regression map from local space to target space. Cuts victim-API queries by 10-100x when the attack is against a hosted model.

8.3 Pre-trained and few-shot inversion

Pre-trained inversion invests time before the engagement to train a general-purpose decoder, then applies it at engagement time. Higher upfront cost, faster per-vector at exploit time.

8.3.1 ALGEN with canary injection

ALGEN (ACL 2025) trains a small decoder locally to reconstruct text from embeddings, then uses a linear alignment matrix to bridge the local decoder's embedding space and the target's space. The training-time work is 2-4 GPU hours. The engagement-time work is minutes per target vector.

Effectiveness reported in the underlying research: a single aligned data point already produces a partially successful attack; with roughly 1,000 alignment samples, performance reaches its optimum — ROUGE-L of 45-50 — on par with prior attacks that required orders of magnitude more data. It transfers effectively across domains and languages, and the original authors tested multiple defense mechanisms, none of which proved effective (arXiv:2502.11308).

The critical variant for offensive use is canary injection. In classical ALGEN, alignment requires known (text, target-embedding) pairs. In canary injection, the tester writes known canary texts into the target's vector store, observes the embeddings the target produces, and now has aligned pairs from the exact model and environment being attacked — eliminating drift from any model or preprocessing difference.

The full pipeline:

1. Generate training data — multi-domain synthetic corpus at 50k texts covering enterprise categories
2. Insert canaries — write those texts into the target vector store as documents, capture the embeddings the target embeds them to
3. Train decoder locally — MLP-projected embedding to 32 pseudo-tokens to T5-small decoder, roughly 2 GPU-hours to reach reasonable quality
4. Fit alignment matrix — linear regression from target space to decoder space, using the canary pairs
5. Attack — for each target vector of interest, apply alignment, feed to decoder, get plausible text

ALGEN output is frequently "wrong topic" — it identifies domain fingerprints (login.northstar.example, "password reset", "must be updated on first login") but generates a topical paraphrase rather than exact content. Compose with a second stage:

• Inference probing. Use ALGEN's fingerprint keywords as query terms for the target RAG (or northstar-agent, if that is the deployed assistant). The RAG surfaces the actual documents. Combine RAG responses with target vectors to identify the source document precisely.
• Slot detection from redactions. RAG responses run through output guardrails that redact secrets. Redaction markers ([REDACTED], [MASKED], ***) become {PASSWORD} slots in a template. Membership inference on the slot recovers the redacted value from the stolen embedding.

Canary count is a tradeoff. Fifty thousand canaries produce excellent alignment but grossly pollute the RAG. Three thousand canaries give near-equivalent alignment. For stealth, distribute canaries over hours or days, mix into legitimate ingestion patterns, and clean up after the engagement (or document every inserted document for the client to remove).

8.3.2 Vec2Text

Vec2Text (arXiv 2310.06816) is a two-stage neural inversion — an inverter proposes initial text, a corrector iteratively refines by re-embedding and comparing. The original paper reported up to 92 percent exact match on 32-token inputs against models the decoder was trained for; independent 2025 reproduction work confirms and sharpens this, reporting BLEU scores of up to 97.3 for 32-token recovery against a black-box encoder, and confirms that Vec2Text also reconstructs password-like sequences with no clear semantic structure, though it is sensitive to input sequence length. Gaussian noise and embedding quantization partially mitigate the risk but do not eliminate it (arXiv:2507.07700). Training is heavy: multi-GPU days, millions of pairs.

Adapted for practical use on a single GPU:

• Hybrid corpus. 80 percent synthetic enterprise-templated content, 20 percent general-domain (Wikipedia). Reflects operational vocabulary better than a purely-general corpus.
• Recon not generation. Rather than train the inverter to fully reproduce the target, train it to recover structure and yield high-entropy slots that a slot filler then brute-forces. The inverter's job is to say "this is a password reset URL with slots for username and password"; the filler's job is to find the exact strings.
• Recon-guided template selection. Use the inverter output to narrow the template bank from hundreds to tens for the slot filler stage.

Vec2Text is model-specific. The trained inverter only works against the model it was trained on. If the target's embedding model is unknown, use ALGEN or Zero2Text instead.

8.3.3 BeamClean — inverting noised and obfuscated embeddings

BeamClean targets the case where embeddings sent to a server-side LLM have been obfuscated with noise (Laplacian or Gaussian) before transmission, with no access to the model or the obfuscation mechanism. It jointly estimates the noise parameters and decodes the token sequence by integrating a language-model prior via beam search, consistently outperforming naive nearest-neighbor distance-based attacks (arXiv:2505.13758). Combined with the Zero2Text finding that standard local-DP noise mechanisms fail against adaptive inversion, BeamClean is the reference technique to reach for whenever a target claims noise-based obfuscation as a defense — it is specifically designed to defeat exactly that claim.

8.4 Choosing an approach

Decision path: start with template + membership inference. If confidence is weak or moderate, run Zero2Text-style beam search for structure and to test resilience against any claimed noise defense. If the model is unknown, use ALGEN with canary injection. If the model is known months in advance and the corpus is large, train Vec2Text offline. If the target specifically obfuscates embeddings with noise before serving them to a downstream LLM, use BeamClean.

8.5 Stealing embeddings and embedding layers via the API

Inversion assumes you already have a target embedding vector. A separate, complementary attack extracts information about the embedding model itself directly from a hosted API — no vector store access required.

Carlini et al. demonstrated the first model-stealing attack that extracts precise, non-trivial information from production black-box models such as ChatGPT and PaLM-2: recovering the embedding projection layer (up to symmetries) using ordinary API access (arXiv:2403.06634). Concrete, reproduced costs:

• For under 20 US dollars, the full projection matrix of OpenAI's Ada and Babbage models was extracted, confirming for the first time their hidden dimensions of 1024 and 2048 respectively.
• The exact hidden dimension of gpt-3.5-turbo was recovered.
• The authors estimate that recovering the full projection matrix of gpt-3.5-turbo would cost under 2,000 US dollars in queries.

For a pentest engagement, this means embedding-model fingerprinting (Chapter 03) is not limited to passive dimensionality inference — a modest query budget against a production embedding API can extract structural information (hidden dimension, projection matrix up to symmetries) that materially improves the accuracy of every inversion technique in this chapter, particularly ALGEN and Vec2Text where knowing the target model's exact architecture removes a source of alignment error. Budget a few tens of dollars in API spend for this step before committing to expensive offline training.

8.6 Limitations to plan around

• Token length. All inversion techniques degrade rapidly beyond 32-64 tokens. Long RAG chunks (256-512 tokens) invert to plausible paraphrase, not exact text. Short chunks (credentials, config values) are most vulnerable. Pre-filter by chunk length before spending compute.
• High-entropy content. Random passwords, long API keys, and cryptographic material have no semantic predictability. The embedding does not preserve the exact string; only the structural label ("this is a token"). Slot fillers only work when the content is drawn from a distribution the filler can enumerate.
• Dimensionality reduction. PCA, random projection, and quantization applied before storage throw away recoverable information. Check whether the target's stored vectors are full-dimensional or compressed.
• Domain mismatch. Supervised inverters trained on general corpora (MS MARCO, Natural Questions) drop in accuracy on medical, legal, or proprietary content. Use in-domain training data or accept the drop.
• Model identification prerequisite. Except for ALGEN and Zero2Text, every technique requires knowing the embedding model. Chapter 03's fingerprinting, plus the API-extraction technique in Section 8.5, are the prerequisite step.
• Noise defenses are weaker than they look. Standard local-DP mechanisms (normalized Laplace, Purkayastha) and simple Gaussian noise do not stop Zero2Text or BeamClean. Do not let a target's claim of "we add noise" close out a finding without testing against these specific techniques.

8.7 The output-guardrail bypass

The strategic implication of embedding inversion is that output guardrails are structurally incapable of protecting content whose embedding has been exfiltrated. The guardrail redacts a value from the LLM's output; the raw embedding, stored in the vector database, still encodes the pre-redaction text. Membership inference on the redacted position recovers the original.

This shifts a defender's obligation upstream: preventing leakage requires preventing embedding exposure, not just filtering outputs. From the offensive side, the corollary is that any accessible vector store is a bypass path around whatever output filtering exists.

8.8 Canary hygiene and engagement discipline

If canary injection is used during an engagement, cleanup is a non-negotiable deliverable. Fifty thousand canary documents left in a production vector store degrade RAG quality (irrelevant retrievals), poison retrieval statistics, and violate the engagement's do-no-harm principle. Document every canary inserted (filename, content, ingestion time, target chunk), remove them at engagement end, and provide the client a list to run a completeness check. For very large canary counts, prefer the 3,000-canary sweet spot over the 50,000-canary maximum quality — the alignment gain from more canaries is diminishing.

8.9 Practice checklist

• Fingerprinted embedding model dimension via inference probing
• Budgeted a modest API spend to attempt hidden-dimension / projection-layer extraction (Section 8.5) if a hosted embedding API is in scope
• Confirmed embeddings are full-dimensional (not quantized or reduced)
• Triaged chunks by length (favor short) and by triaged similarity to sensitivity probes
• Ran template-based inversion first (cheap, fast)
• If weak, ran Zero2Text-style beam search for structure, especially if the target claims a noise-based defense
• If the target obfuscates embeddings with noise before serving, tested BeamClean rather than accepting the noise claim
• If model unknown, prepared ALGEN pipeline with in-scope canary insertion
• Documented every canary inserted; planned cleanup
• Applied slot filling to redacted positions in RAG output for guardrail bypass

MITRE ATLAS references

Further reading

• Vec2Text — Text embeddings reveal (almost) as much as text — https://arxiv.org/abs/2310.06816
• Vec2Text reproducibility study (BLEU 97.3) — https://arxiv.org/html/2507.07700v1
• ALGEN — Model-Agnostic Latent Grounding for text-embedding inversion — https://arxiv.org/abs/2502.11308
• Zero2Text — training-free zero-shot cross-domain inversion — https://arxiv.org/abs/2602.01757 — analysis: https://promptfoo.dev/lm-security-db/vuln/zero-training-cross-domain-inversion-bea68dd4/
• BeamClean — inversion of noised/obfuscated embeddings — https://arxiv.org/abs/2505.13758
• GEIA — Generative embedding inversion — https://arxiv.org/abs/2305.03010
• Stealing Part of a Production Language Model (Carlini et al.) — https://arxiv.org/abs/2403.06634
• Membership inference in embeddings — https://arxiv.org/abs/2106.11384