AI Stack Security Architecture
Your LLM is live. Your agent can send emails, query databases, and execute code. Your guardrails are a system prompt that says "don't do anything bad." And your security team found out about the deployment from a Slack thread three days after it hit production.
You've shipped a privileged workload with no access controls, no input validation, and no audit trail. Any other system would fail a compliance review in five minutes. But because it's AI, nobody asked.
Here's where things stand. 48% of cybersecurity professionals call agentic AI the top attack vector heading into 2026. MITRE ATLAS v2026.09 now catalogs 120 adversarial techniques - up from the handful it had two years ago. The EU AI Act's high-risk obligations become enforceable in August 2026. And only 10% of organizations have a strategy for managing non-human and agentic identities.
What follows is the security architecture that should exist between your model and your users. Seven layers, no vendor pitch.
The Problem in Numbers
| Metric | Value |
|---|---|
| Organizations with a mature AI identity strategy | 10% |
| Prompt injection success rate against current defences | 85%+ |
| MITRE ATLAS techniques (v2026.09) | 120 techniques, 88 sub-techniques, 73 case studies |
| Backdoored models pulled from Hugging Face (third wave, 2026) | Dozens - attackers moved to typosquatted Spaces with blockchain-routed C2 |
| PickleScan zero-days disclosed (JFrog, Sep 2025) | 3 - all allowed torch.load() execution despite scanning |
| Companies with active AI memory poisoning (Microsoft Defender, Feb 2026) | 31 companies across 14 industries in a 60-day window |
| EU AI Act high-risk compliance deadline | August 2, 2026 |
These are not projections. Every row has a CVE, a case study, or an enforcement date.
What Attackers Actually Target
Traditional appsec protects code. AI security protects behaviour - and the attack payload is not a buffer overflow. It is a sentence.
| Attack Class | What Happens | Real-World Example |
|---|---|---|
| Direct Prompt Injection | User crafts input that overrides system prompt | Meta Instagram chatbot - "I own this account" → password reset executed, no verification |
| Indirect Prompt Injection | Malicious instructions hidden in data the model processes | EchoLeak (CVE-2025-32711) - hidden instructions in email, zero-click exfiltration via Copilot |
| Model Supply Chain | Backdoored models from public registries | Hugging Face pickle deserialization - torch.load() runs attacker code during model import |
| MCP/Tool Poisoning | Malicious instructions in tool metadata | Postmark MCP Backdoor - instructions in description fields redirected API credentials |
| Data Poisoning | Corrupted training data plants backdoors | MITRE ATLAS AML.T0020 - 0.1% poisoned examples is enough for reliable activation |
| Model Extraction | Systematic querying builds a copy of your model | Repeated API queries + student training → extracted model used for offline adversarial research |
| Model Inversion | Outputs reverse-engineered to reconstruct training data | Gradient optimization reconstructs faces, text, PII from confidence scores |
| Memory Poisoning | Agent's persistent state corrupted to hijack future actions | Microsoft Defender (Feb 2026) - active across 31 companies in 14 industries |
| Adversarial Suffixes | Gibberish tokens bypass safety at the math level | GCG attack - optimized suffixes transfer across models, dodge semantic defences entirely |
| Agent Goal Hijacking | Injected instructions redirect an autonomous multi-step plan | OWASP Agentic Top 10 - one injection cascades through the full autonomous workflow |
None of this gets caught by WAF. SAST can't see it. EDR has no idea it happened.
The Architecture: Seven Layers
Layer 1 - Model Supply Chain Security
You would never pull an unsigned binary off a stranger's GitHub and run it in production. But teams routinely download .bin files from Hugging Face and call torch.load() - which executes arbitrary Python through pickle deserialization. JFrog disclosed three PickleScan zero-days in September 2025 where the scanner said "clean" and torch.load() still ran attacker code.
| Control | What It Does | What It Prevents |
|---|---|---|
| SafeTensors format only | Tensor data, no executable code, no deserialization hooks | Pickle RCE - the entire class of torch.load() attacks |
| Model signing (Cosign/Sigstore) | Cryptographic signature at training time, verified at deployment | Tampered models, unauthorized swaps |
| Provenance attestation (SLSA) | Records who trained it, on what data, what code, what metrics | "Where did this model come from?" - nobody can answer |
| Internal model registry | Approved models only - unsigned or unreviewed models rejected | Shadow deployments from public registries |
| SHA-256 pinning | Pin version by hash, not by name or tag | Tag confusion, rollback attacks |
| Dependency scanning | ML-specific supply chain checks (PyPI typosquatting, malicious notebooks) | TeamPCP-style cascading package compromises |
If you cannot trace a model back to its training run, its data, and the person who approved it - pull it.
Layer 2 - Training Pipeline Security
The model is as trustworthy as the data that built it. Garbage data in, backdoored model out. And "garbage" here means 0.1% poisoned examples - enough to plant a reliable backdoor that fires on a trigger phrase while passing every standard benchmark.
| Control | What It Does |
|---|---|
| Data provenance tracking | Cryptographic hashing of every dataset version. Immutable lineage from source to model artifact. |
| PII scrubbing before training | NER-based detection (Presidio, Comprehend) + replacement with synthetic equivalents. Second-pass verification. |
| Poisoning detection | Statistical outlier analysis. Spectral signature detection for backdoor patterns via Activation Clustering. |
| Differential privacy (DP-SGD) | Calibrated noise during training that mathematically caps what anyone can learn about a single example. Google's VaultGemma (2025) proved this scales to 1B parameters. |
| Isolated training environment | No internet during training. Private subnets. Encrypted GPU nodes. Authenticated training jobs. |
| Fine-tuning safety validation | Mandatory adversarial re-evaluation after every fine-tuning run. As few as 100 harmful examples can undo RLHF safety alignment. |
| Notebook security | SSO on JupyterHub/SageMaker, pre-commit secret scanning, nbstripout for outputs, outbound transfer monitoring. |
Fine-tuning deserves its own callout. It is the single most dangerous operation in the AI lifecycle - the place where safety alignment degrades, PII gets memorized, and backdoors slip in. Never assume the base model's safety survives a fine-tuning run. Test it again from scratch.
Layer 3 - Secure Model Serving
The model is an untrusted workload. It never talks directly to users, databases, or the internet.
User Request
↓
[API Gateway] → Auth, rate limiting, request size validation
↓
[Input Guardrails] → Injection detection, PII redaction, content safety
↓
[Model Inference] → Isolated container, no egress, resource limits
↓
[Output Guardrails] → PII redaction, prompt leak detection, toxicity filter
↓
User Response
| Control | Implementation |
|---|---|
| Container hardening | Distroless image. Non-root. Read-only rootfs. Drop all capabilities. Seccomp profile. No shell. |
| Network isolation | Inference pods on a dedicated subnet. No direct internet. Everything proxied through the guardrail layer. |
| GPU isolation | Dedicated GPU node pool per tenant. MIG/MPS for multi-instance. Memory cleared between requests. |
| Rate limiting | Per-user, per-key, per-tenant token budgets. Model extraction requires 10K–1M+ queries - rate limiting makes it impractical. |
| Inference timeout | Hard cap per request. Adversarial inputs can trigger excessive computation. Timeout kills it. |
| Model integrity check | Checksum verified on load from signed registry. Tampered weights caught before first inference. |
Guardrails run outside the model process. The model cannot disable them. If the guardrail service goes down, requests get blocked. Fail-closed, not fail-open.
Layer 4 - LLM Input/Output Security
Most teams skip this layer entirely. They ship the model with a system prompt and hope for the best. This is the layer that actually catches prompt injection, blocks PII leakage, and stops attackers from extracting your system prompt verbatim.
| Threat | Input Control | Output Control |
|---|---|---|
| Prompt injection | Classifier flags attempts before the LLM sees them (LlamaGuard, Rebuff, custom). Instruction hierarchy enforcement. | N/A - caught at input |
| System prompt extraction | "Never reveal instructions" hardening + canary tokens in the system prompt | Output classifier catches instruction-like patterns. Canary monitoring. |
| PII leakage | PII detection + redaction on user input before it hits the model (Presidio) | Regex + NER on every response. Anything matching SSN/CC/email/phone gets blocked. |
| Jailbreak | Many-shot detection, encoded payload checks (base64, ROT13), perplexity filtering for adversarial suffixes | Safety classifier on output (LlamaGuard/ShieldGemma) |
| Insecure output | N/A | Output encoding before rendering (prevents XSS through LLM-generated content). |
| Hallucination | N/A | Fabricated URL and package name detection - attackers register domains the model invents. Citation verification. |
Tooling worth looking at:
| Tool | What It Does |
|---|---|
| NeMo Guardrails | Guardrail flows in Colang. Input, output, and dialog rails. |
| Guardrails AI | Python framework. PII, toxicity, hallucination validators. |
| LlamaGuard / ShieldGemma | Fast safety classifiers - deploy as sidecars. |
| Bedrock Guardrails | Managed. Content filters, denied topics, PII redaction. Config-based, no code. |
| Rebuff | Multi-layer injection detection - heuristics + LLM + vector similarity + canary tokens. |
Here is the thing people get wrong. Safety alignment - RLHF, Constitutional AI, whatever the provider used - is probabilistic. It reduces bad outputs. It does not eliminate them. Jailbreaks exist because alignment is a learned preference, not a hard constraint. Your guardrails must be deterministic code that runs independently of the model. The Meta Instagram breach happened because identity verification lived inside the model's judgement instead of in a code-level MFA gate. The model got talked out of it. Code does not get talked out of anything.
Layer 5 - Agentic System Security
A bad response from a stateless API is embarrassing. A compromised agent that can send emails, write to databases, and call external APIs on your behalf is a breach. Agents combine autonomy, persistence, tool access, and multi-step planning - and a single prompt injection can hijack the entire chain.
| Control | What It Prevents |
|---|---|
| Tiered permissions | Read-only: allow. Drafts (unsent, unsaved): allow. Writes: user confirmation. Deletes/sends/executes: confirmation + re-auth. Security settings: hard-blocked, no override. |
| Deterministic auth gates | Privileged actions hit code-level checkpoints. The agent cannot talk its way past a permission gateway. |
| Tool parameter validation | Every function validates independently of LLM output. Schema enforcement. Parameterized queries downstream. LLM-generated parameters are untrusted input. Period. |
| Action budgets | Max tool calls per session. Max writes. Max API cost. Max tokens. Hard limits in code, not suggestions in the prompt. |
| Sandboxed code execution | Fresh container per session. Read-only rootfs, no network, resource limits, hard timeout. gVisor or Firecracker for OS-level isolation. |
| Inter-agent sanitization | Agent A's output gets sanitized before becoming Agent B's input. Communication via validated JSON schemas, not free text. Zero trust between agents. |
| MCP server audit | Every tool description audited for hidden instructions - the Postmark pattern. Trusted sources only. Pinned versions. |
| Intent verification | Separate model asks one question: "Is this action consistent with what the user actually requested?" Catches confused deputy attacks where injected instructions redirect the agent's plan. |
The mental model: agents are untrusted executors operating inside deterministic permission fences. They reason, they recommend. Only code executes.
Layer 6 - Governance and Compliance
Controls without governance is a firewall with no security policy. Somebody configures it once, nobody reviews it, and six months later you discover the rules haven't been updated since the team that wrote them left.
| Framework | Focus | When to Use |
|---|---|---|
| OWASP LLM Top 10 | Top risks for LLM apps - your assessment checklist | Every deployment, every red team |
| OWASP Agentic Top 10 | Agent-specific: goal hijacking, tool misuse, privilege abuse | Every agentic deployment |
| MITRE ATLAS | ATT&CK for AI - adversarial tactics, techniques, case studies | Threat modeling, detection engineering, red team design |
| NIST AI RMF | Govern / Map / Measure / Manage | Internal risk program structure |
| ISO 42001 | Certifiable AI management system | Customer-facing compliance proof |
| EU AI Act | Law. Fines up to 7% of global revenue. | Any AI touching EU users - August 2, 2026 |
| NIST SP 800-218A | Secure development for AI/ML, extends SSDF | Securing the build pipeline |
On the operational side:
| Component | What It Does |
|---|---|
| Model inventory | Every AI system registered - owner, purpose, data sources, risk tier, last assessment. If it's not in the inventory, it's not in production. |
| Risk tiering | Tier 1 (customer-facing, PII, actions): full threat model + adversarial testing + privacy review. Tier 2 (internal): automated scan. Tier 3 (analytics): self-assessment. |
| Model cards | Per-model documentation - provenance, metrics, limitations, security results, who approved it and when. |
| Audit trails | Full lineage from data to training to deployment. Decision logs on every AI action. Immutable storage. |
| Shadow AI detection | CASB monitoring for unauthorized ChatGPT/Copilot/Claude API traffic. DLP on outbound calls. Approved vendor list enforced. |
Layer 7 - Continuous Testing and Monitoring
You don't run a pen test once and declare the network secure for life. Same principle, different attack surface.
Before every deployment - automated, blocks promotion to prod on failure:
| Test | What It Probes |
|---|---|
| Prompt injection | 100+ probes - instruction override, encoded payloads, many-shot, prefix injection |
| Jailbreak | Known bypasses + generated variants - DAN, roleplay, multi-turn, adversarial suffixes |
| PII extraction | SSN, credit card, email, phone, address enumeration attempts |
| System prompt extraction | Verbatim dump, translation tricks, <system> tag probes |
| Tool abuse (agentic) | Unauthorized calls, privilege escalation via tool chains, parameter injection |
| Output safety | Toxicity, bias, hallucination with false source attribution |
Run these with Promptfoo (CI/CD native), Garak (automated scanning), PyRIT (multi-turn orchestration), or CyberSecEval (code generation safety). Every new attack pattern from manual red teaming goes straight into the automated suite.
In production - runtime signals that tell you something has changed:
| Signal | What It Catches |
|---|---|
| Token usage anomaly | Model extraction attempts - high query volume with systematic variation |
| Output distribution drift | Compromise, poisoning, or exploitation in progress |
| Guardrail block rate spike | New attack campaign hitting your system |
| Cost anomaly | GPU abuse, crypto mining, runaway agents |
| Action sequence anomaly | Agent deviating from baseline behaviour |
| Canary probe regression | Old attacks working again after a model update |
When something fires, AI incident response looks different from traditional IR. You quarantine the model (redirect traffic to the last known-good version, don't take the endpoint down). You do prompt forensics - what specific interaction triggered it? You assess blast radius - how many users affected, what data exposed, what autonomous actions were taken? And root cause splits three ways: did the model fail, did the guardrail fail, or did the infrastructure fail?
The Stack
┌─────────────────────────────────────────────────────────┐
│ 7. Continuous Testing & Monitoring │
│ Adversarial CI/CD │ Runtime anomaly │ AI IR │
├─────────────────────────────────────────────────────────┤
│ 6. Governance & Compliance │
│ Model inventory │ Risk tiers │ OWASP │ ATLAS │ EU AI │
├─────────────────────────────────────────────────────────┤
│ 5. Agentic System Security │
│ Permissions │ Auth gates │ Budgets │ Sandboxing │ MCP │
├─────────────────────────────────────────────────────────┤
│ 4. LLM Input/Output Security │
│ Injection │ PII │ Jailbreak │ Prompt leak │ Encoding │
├─────────────────────────────────────────────────────────┤
│ 3. Secure Model Serving │
│ Containers │ Network │ GPU │ Rate limits │ Integrity │
├─────────────────────────────────────────────────────────┤
│ 2. Training Pipeline Security │
│ Provenance │ PII scrub │ Poisoning │ DP │ Fine-tuning │
├─────────────────────────────────────────────────────────┤
│ 1. Model Supply Chain Security │
│ SafeTensors │ Signing │ SLSA │ Registry │ Pinning │
└─────────────────────────────────────────────────────────┘
Skip a layer and the ones above it cannot compensate. Hardened containers do nothing if the model inside them was trained on poisoned data. Perfect guardrails do nothing if the agent behind them has unrestricted tool access. Signing does nothing if nobody is watching for drift in production.
The Takeaway
We have spent twenty years building defence-in-depth for deterministic systems. AI drops a non-deterministic component into the stack - something that makes decisions based on probabilistic inference - and hands it the keys to your infrastructure.
Treat the model the way you would treat any untrusted workload. It reasons. It recommends. It does not execute. Deterministic code executes. And that code does not care how creative the jailbreak was.
The AI shipped. The architecture should have shipped with it.
References:
- MITRE ATLAS v2026.09 - Adversarial Threat Landscape for AI Systems
- OWASP Top 10 for LLM Applications 2025
- OWASP Top 10 for Agentic Applications
- NIST AI Risk Management Framework
- ISO/IEC 42001 - AI Management System Standard
- EU AI Act - High-Risk System Requirements
- Hugging Face Pulled Dozens of Backdoored Models - DEV Community
- Agentic AI: Biggest Enterprise Security Threat for 2026 - Kiteworks
- AI Agent Security in 2026 - Beam AI
- AI Supply Chain Security 2026
- AI Governance Frameworks: NIST, ISO 42001 & EU AI Act - Strac
- Promptfoo - Open-source LLM Red Teaming
- NVIDIA NeMo Guardrails
- Google VaultGemma - Differentially Private LLM
- MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning
- SoK: The Attack Surface of Agentic AI
