moai-ref-llm-security

# LLM / AI Defensive Security Reference

Defensive practitioner reference for hardening LLM-backed applications and agents.
Every section is framed as defense, hardening, detection, or verification — it
describes the misconfiguration, how to detect it, and how to prevent it, never how
to exploit it. Cross-domain web-app vulnerabilities live in
`moai-ref-owasp-checklist`; API design lives in `moai-ref-api-patterns`.

## Target Use

Apply when reviewing or building an LLM-backed system — a chat product, a
retrieval-augmented application, an autonomous agent, or an MCP server. The
material assumes an untrusted-input threat model: any text the model reads (user
turns, retrieved documents, tool results, file contents) may carry adversarial
instructions, and any text the model emits may be acted on downstream.

## Trust Boundaries in an LLM System

The core defensive insight: an LLM does not distinguish "data" from "instructions"
the way a parser does. Treat every text channel that reaches the model as a
boundary where adversarial instructions can enter.

| Channel | Entry risk | Primary defense |
|---------|-----------|-----------------|
| End-user prompt | Direct prompt injection | Instruction-hierarchy enforcement, input screening |
| Retrieved documents (RAG) | Indirect prompt injection | Provenance tagging, content isolation, retrieval allowlist |
| Tool / function results | Injected instructions in tool output | Treat tool output as untrusted data, re-validate before re-prompting |
| System / developer prompt | Leakage, override | Minimize secrets in prompt, assume prompt is recoverable |
| Model output | Improper downstream handling | Schema validation, encoding, never auto-execute raw output |
| Training / fine-tuning data | Poisoning | Data lineage, provenance verification, canary detection |

## OWASP LLM Top 10 — Defensive Mapping

The OWASP Top 10 for LLM Applications (2025 edition) is the canonical risk index
for LLM systems. Each row below states the risk, the defensive check, and the
hardening control. Citations are for defensive correlation only.

| ID | Risk | Defensive check | Hardening control |
|----|------|-----------------|-------------------|
| LLM01 | Prompt Injection | Can external text override system instructions? | Instruction-hierarchy enforcement, input/output screening, content isolation |
| LLM02 | Sensitive Information Disclosure | Can the model reveal secrets, PII, or system prompt? | Output filtering, minimize secrets in context, response redaction |
| LLM03 | Supply Chain | Are model weights, datasets, and plugins from verified sources? | Provenance verification, model/dataset signing, dependency pinning |
| LLM04 | Data and Model Poisoning | Could training/fine-tuning data carry adversarial content? | Data lineage, canary artifacts, anomaly detection on training sets |
| LLM05 | Improper Output Handling | Is model output executed/rendered without validation? | Schema validation, context-aware encoding, no auto-execution |
| LLM06 | Excessive Agency | Does the agent hold more capability/permission than needed? | Least-privilege tool design, human-in-loop for high-impact actions |
| LLM07 | System Prompt Leakage | Does the system prompt hold secrets that leak on extraction? | Never store secrets in the prompt; assume the prompt is recoverable |
| LLM08 | Vector and Embedding Weaknesses | Can embedding/RAG stores be poisoned or leak cross-tenant? | Per-tenant isolation, embedding-source validation, access control |
| LLM09 | Misinformation | Is unverified model output presented as authoritative? | Grounding, citation requirements, confidence signalling |
| LLM10 | Unbounded Consumption | Can a request exhaust tokens, cost, or compute? | Rate limiting, token budgets, request-size caps, timeout enforcement |

## Prompt-Injection Defense

Prompt injection is the highest-frequency LLM risk (LLM01). Defense is
defense-in-depth — no single control is sufficient, so layer the controls below.

### Direct injection (the user is the attacker)

| Layer | Control | Purpose |
|-------|---------|---------|
| Instruction hierarchy | Mark system instructions as highest-priority; instruct the model that downstream text cannot override them | Reduce override success rate |
| Input screening | Scan incoming prompts for known injection markers before they reach the model | Detect obvious override attempts |
| Privilege separation | Run a high-trust planning model separately from a low-trust task model that touches untrusted content | Limit blast radius of a successful injection |
| Output gating | Validate the model's response against an allowlist of expected shapes before acting on it | Block injected instructions from taking effect downstream |

### Indirect injection (the attacker poisons retrieved content)

Indirect injection arrives through documents, web pages, emails, or tool results
that the model reads. The defense is to treat all retrieved content as untrusted
data, never as instructions.

- **Provenance tagging**: wrap retrieved content in clearly delimited, labeled
  blocks so the model is instructed to treat the enclosed text as data to analyze,
  not commands to obey.
- **Content isolation**: keep retrieved content in a separate channel from the
  instruction channel; do not concatenate untrusted text directly after a system
  instruction.
- **Retrieval allowlist**: restrict RAG sources to verified corpora; reject or
  quarantine documents from unverified origins.
- **Re-validation after tool calls**: when a tool returns text that re-enters the
  prompt, re-screen it — a tool result is an untrusted channel, not a trusted one.

This class of attack maps to MITRE ATLAS **AML.T0051** (LLM Prompt Injection); the
defenses above are how you detect and prevent it, not an attack procedure.

## MCP / Agentic Tool-Call Hardening

Agentic systems that call tools (including MCP servers) expand the attack surface:
a successful injection can now drive real actions. The defense i