AI Security Threat Modeling: Build a Defense Map with NIST, MITRE ATLAS, and OWASP
AI Learning Project
AI Security Threat Modeling: Build a Defense Map with NIST, MITRE ATLAS, and OWASP
A practical AI security threat modeling guide using NIST adversarial ML, MITRE ATLAS, and OWASP LLM Top 10 to map assets, risks, evidence, and controls.
AI Learning Project

AI Security Threat Modeling: Build a Defense Map with NIST, MITRE ATLAS, and OWASP

Reading info

Level: Professional Reading time: 12 min
  • AI Security
  • Threat Modeling
  • NIST
  • MITRE ATLAS
  • OWASP
Open knowledge map

AI security fundamentally diverges from classical application security. A defensible AI security program cannot rely on post-deployment generic vulnerability scans; it demands rigorous threat modeling grounded in the mathematical realities of high-dimensional optimization. This requires mapping assets, actors, trust boundaries, failure modes, evidence, and residual risk across the entire MLOps pipeline.

This article synthesizes the NIST Adversarial Machine Learning taxonomy (NIST.AI.100-2e2025), MITRE ATLAS, and OWASP LLM Top 10 to architect a production-grade engineering map for AI defense. The objective is to transition from abstract risk registers to mathematically rigorous, reviewable threat models for AI and MLSecOps engineers.

1. Beyond the Weights: An Expanded Asset Taxonomy

Conventional application security focuses on APIs, databases, and IAM. AI systems introduce a complex attack surface characterized by continuous optimization and stochastic outputs. The asset taxonomy must be expanded:

  • Training Data & Pipeline: Raw samples, high-dimensional manifolds, labels, annotation functions, provenance cryptomaterial, and data-filtering heuristics. Vulnerable to data poisoning and backdoor (BadNets) injection.
  • Model Artifacts: Learned parameter matrices ( theta in mathbb{R}^d ), tokenizers, embedding spaces, calibration hyperparameters, and evaluation set distributions.
  • Prediction Interfaces (Inference): Inputs ( x ), logits, softmax probabilities ( f(x) ), confidence scores, and local/global explanations. Vulnerable to Model Extraction and Membership Inference Attacks (MIA).
  • Context & Orchestration Systems: RAG document corpora, vector database index structures (e.g., HNSW), reranker weights, and ReAct agent tool permission schemas.
  • Feedback Loops: RLHF reward models, preference datasets, and active learning retraining queues.

2. The Mathematical Threat Landscape: A Three-Layer Architecture

We architect the threat model across three operational layers:

Layer 1: The Optimization Risk (NIST taxonomy). Categorizes attacks by their mathematical objectives. Evasion computes perturbations ( delta ) such that ( argmax f(x+delta) neq y ). Poisoning injects ( (x_p, y_p) ) into the training distribution ( mathcal{D} ) to shift the empirical risk minimizer ( hat{theta} ). Privacy attacks compute the likelihood ( P(x in mathcal{D}_{train} mid f(x, theta)) ).

Layer 2: The Tactical Execution (MITRE ATLAS). Maps adversarial objectives to execution chains, such as ML Supply Chain Compromise (e.g., malicious pickle serialization leading to RCE) or Discovering ML Artifacts.

Layer 3: Application Failure Modes (OWASP LLM Top 10). Translates these to runtime exploits: Prompt Injection (modifying the LLM’s conditioning context), Sensitive Information Disclosure, and Excessive Agency in automated reasoning pipelines.

3. Red/Blue Team Post-Mortem: Production Threat Records

A production-grade threat record must explicitly define the mathematical optimization of the attacker and the empirical defense threshold. Example:

Asset: Inference API (Softmax Output)
Attacker Goal: Membership Inference Attack (MIA)
Mathematical Vector: Exploit the divergence in prediction entropy between training and holdout sets. Attacker trains a shadow model to classify ( mathcal{H}(f(x)) ).
Red Team Validation: Shadow model achieves MIA AUC-ROC > 0.7.
Blue Team Control: Temperature scaling, logit suppression (Top-k only), and differential privacy (DP-SGD) during training with ( (epsilon, delta) )-bounds.
Residual Risk: Complete defense via DP-SGD severely degrades primary task accuracy. Boundary leakage remains possible via timing side-channels.

4. Mathematical Formalization of Evasion (Threat Modeling Entry Point)

Evasion attacks exploit the local linearity of neural networks in high-dimensional space. An attacker seeks a perturbation ( delta ) subject to an ( L_p ) norm constraint ( |delta|_p le epsilon ).

The objective is to maximize the loss function ( J(theta, x + delta, y) ):

[ delta^* = argmax_{|delta|_p le epsilon} J(theta, x + delta, y) ]

This constrained optimization problem is the foundation of the threat model for the inference phase. Defenders must evaluate the Jacobian matrix of the model to understand sensitivity: ( nabla_x f(x) ). If the spectral norm of the Jacobian is high, the model is highly susceptible to small ( delta ).

5. Engineering Controls and Evidence

Threat modeling must produce artifacts that integrate into CI/CD/CT (Continuous Training) pipelines:

  • Data Provenance Cryptography: Cryptographic hashing of datasets and validation of source signatures to prevent supply-chain poisoning.
  • Robustness Certificates: Lipschitz continuity bounds or randomized smoothing guarantees logged per model version.
  • Inference Telemetry: Monitoring the KL-divergence between rolling inference distributions and the training manifold to detect OOD (Out-of-Distribution) evasion attempts.

6. Threat Model Evidence Matrix

A useful AI threat model is reviewable only when each risk has a measurable artifact. The table below turns the narrative model into an audit surface that can be checked during design review, release approval, and incident response.

Risk surface Attacker objective Evidence to collect Control boundary
Training data Shift the learned decision boundary or implant a trigger Dataset hash, source signature, label audit sample, duplicate rate, poison scan result Quarantine untrusted sources and require provenance before retraining
Model artifact Replace weights, tokenizer, or calibration metadata Signed checkpoint, dependency SBOM, evaluation hash, model card version Only signed artifacts enter the registry and deployment pipeline
Inference API Extract the model, infer membership, or probe decision boundaries Query entropy, rate-limit events, confidence distribution, repeated boundary queries Limit logits, bucket confidence, and alert on active-learning-like traffic
RAG and agents Inject instructions through retrieved data or abuse tool permissions Retrieved document trust tier, tool call policy decision, denied action logs Separate data context from instructions and enforce tool RBAC outside the LLM

7. Conclusion

Threat modeling for AI is an ongoing exercise in bounding the adversarial optimization landscape. It transitions security from qualitative checklists to quantitative, empirical risk measurement.

8. References

Internal links

Related tutorials

Review foundation High-Entropy Traffic Defense Notes

Study encrypted metadata leaks, entropy, traffic classifiers, and a defensive Python chaffing prototype.

 Continue next Adversarial Examples and Robust Evaluation

Evaluate clean and perturbed accuracy with an FGSM-style digits experiment.

 Learning path AI Learning Path

Read AI foundations, machine learning, CNN work, traffic defense, and AI security in order.

 Resources Resource Library

Download companion code, datasets, diagrams, model files, and notes.

 Topic hub Student AI Projects

Connect handwritten digits, browser demos, and CNN projects into a portfolio path.

Search questions

FAQ

Who is this article for?

This article is for readers who want a professional-level guide to AI Security Threat Modeling. It takes about 12 min and focuses on AI Security, Threat Modeling, NIST, MITRE ATLAS.

What should I read next?

The recommended next step is Adversarial Examples and Robust Evaluation, so the article connects into a longer learning route instead of ending as an isolated note.

Does this article include runnable code or companion resources?

Yes. Use the run notes, resource cards, and download links on the page to reproduce the example or inspect the companion files.

How does this article fit into the larger site?

It is connected to the article context block, learning routes, resources, and project timeline so readers can move from concept to implementation.

Article context

AI Learning Project

A practical route from AI concepts to machine learning workflow, evaluation, neural networks, Python practice, handwritten digits, a CIFAR-10 CNN, adversarial traffic-defense notes, and AI security.

Level: Professional Reading time: 12 min
  • AI Security
  • Threat Modeling
  • NIST
  • MITRE ATLAS
  • OWASP

Your next step

Continue: Adversarial Examples and Robust Evaluation
Review the foundation Open resource
Other language version AI 安全威胁建模：用 NIST AML、MITRE ATLAS 和 OWASP 建立攻防地图
Share summary AI Security Threat Modeling

Build a defense map with NIST adversarial ML, MITRE ATLAS, and OWASP LLM risks.

Download share card Open share center

Companion resources

Leave a Reply

Project timeline

Published posts

  1. AI Basics Learning Roadmap Separate AI, machine learning, and deep learning before going into implementation details.
  2. Machine Learning Workflow Follow the practical path from data and features to training, prediction, and evaluation.
  3. Model Training and Evaluation Understand loss, overfitting, train/test splits, accuracy, recall, and F1.
  4. Neural Network Basics Move from perceptrons to activation, forward propagation, backpropagation, and training loops.
  5. Matrix Calculus for Neural Networks Derive dL/dW for y = Wx + b and verify it with finite differences.
  6. Backpropagation as a Computation Graph Trace local gradients through ReLU and softmax cross-entropy in a two-layer MLP.
  7. Gradient Descent and Optimizer Geometry Compare gradient descent, momentum, and Adam on a visible quadratic loss surface.
  8. Convolution and Receptive Field Math Compute convolution output size, receptive fields, channel mixing, and im2col layout.
  9. Transformer Attention Math Hand-calculate Q/K/V scores, softmax weights, masks, multi-head structure, and KV cache.
  10. Python AI Mini Practice Run a small scikit-learn classification task and read the experiment output.
  11. Handwritten Digit Dataset Basics Read train.csv, test.csv, labels, and the flattened 28 by 28 pixel layout before training the classifier.
  12. Handwritten Digit Softmax in C Follow the C implementation from logits and softmax probabilities to confusion matrices and submission export.
  13. Handwritten Digit Playground Notes See how the offline classifier was adapted into a browser demo with drawing input and probability output.
  14. CIFAR-10 Tiny CNN Tutorial in C Build and train a small convolutional neural network for CIFAR-10 image classification, then read its loss and accuracy output.
  15. High-Entropy Traffic Defense Notes Study encrypted metadata leaks, entropy, traffic classifiers, and a defensive Python chaffing prototype.
  16. AI Security Threat Modeling Build a defense map with NIST adversarial ML, MITRE ATLAS, and OWASP LLM risks.
  17. Adversarial Examples and Robust Evaluation Evaluate clean and perturbed accuracy with an FGSM-style digits experiment.
  18. Data Poisoning and Backdoor Defense Study poison rate, trigger behavior, attack success rate, and training pipeline controls.
  19. Model Privacy and Extraction Defense Measure membership inference signal and surrogate fidelity against a local toy model.
  20. LLM, RAG, and Agent Security Separate instructions from data and enforce tool permissions against indirect prompt injection.

Published resources

  1. Python AI practice code guide The article includes a runnable scikit-learn classification script.
  2. digit_softmax_classifier.c The C source for the handwritten digit softmax classifier.
  3. train.csv.zip Compressed handwritten digit training set with 42000 labeled samples.
  4. test.csv.zip Compressed handwritten digit test set with 28000 unlabeled samples.
  5. sample_submission.csv The official submission format example for checking the final output columns.
  6. submission.csv The prediction file generated by the current C project.
  7. digit-playground-model.json The compact softmax demo model and sample set used by the browser playground.
  8. digit-sample-grid.svg A small handwritten digit preview grid extracted from the training set.
  9. Handwritten digit project bundle Contains the source file, compressed datasets, submission files, browser model, and preview grid.
  10. cifar10_tiny_cnn.c source Single-file C tiny CNN with CIFAR-10 loading, convolution, pooling, softmax, and backpropagation.
  11. model_weights.bin sample weights Model weights generated by one local small-sample run.
  12. test_predictions.csv sample predictions Sample test prediction output from the CIFAR-10 tiny CNN.
  13. CNN project explanation PDF Companion explanation material for the CNN project.
  14. Virtual Mirror redacted code skeleton A redacted mld_chaffing_v2.py control-flow skeleton with secrets, node topology, and target lists removed.
  15. Virtual Mirror stress-test template A redacted CSV template for CPU, memory, peak threads, pulse rate, latency, and error measurements.
  16. Virtual Mirror classifier-evaluation template A CSV template for TP, FN, FP, TN, accuracy, precision, recall, F1, ROC-AUC, entropy, and JS divergence.
  17. Virtual Mirror resource notes Notes explaining why the public resources include only redacted code, test templates, and architecture context.
  18. AI Security Lab README Setup, safety boundaries, and quick-run commands for the AI Security series.
  19. AI Security Lab full bundle Includes safe toy scripts, result CSVs, risk register, attack-defense matrix, and architecture diagram.
  20. AI security risk register CSV risk register template for AI threat modeling and release review.
  21. AI attack-defense matrix Maps attack surface, toy demo, metric, and defensive control into one CSV table.
  22. AI Security Lab architecture diagram Shows threat modeling, robustness, data integrity, model privacy, and RAG guardrails.
  23. FGSM digits robustness script FGSM-style perturbation and accuracy-drop experiment for a local digits classifier.
  24. Data poisoning and backdoor toy script Demonstrates poison rate, trigger behavior, and attack success rate on digits.
  25. Model privacy and extraction toy script Outputs membership AUC, target accuracy, surrogate fidelity, and surrogate accuracy.
  26. RAG prompt injection guard toy script Uses a deterministic toy agent to demonstrate external-data demotion and tool-policy blocking.
  27. Deep Learning Math Lab README Setup commands, script entry points, generated outputs, and figure notes for the math series.
  28. Deep learning math full lab bundle Bundles NumPy scripts, CSV outputs, formula diagrams, loss contours, convolution figures, and attention heatmaps.
  29. Gradient check results CSV Stores MSE analytic gradients, finite-difference gradients, and error norms.
  30. Optimizer path CSV Step-by-step coordinates and loss for gradient descent, momentum, and Adam on a 2D quadratic.
  31. Attention weights CSV Scores, softmax weights, and context vectors for a three-token scaled dot-product attention example.
  32. Deep learning math figure set Includes matrix shapes, computation graphs, loss contours, convolution scans, and attention heatmaps.
  33. Deep learning math interactive visualizer Browser modules for gradient checking, optimizer paths, convolution output size, and attention heatmaps.
  34. Deep Learning topic share card A 1200x630 SVG card for sharing the Deep Learning / CNN topic hub.
  35. Machine Learning From Scratch share card A 1200x630 SVG card for the K-means, Iris, and ML workflow topic hub.
  36. Student AI Projects share card A 1200x630 SVG card for handwritten digits, C classifiers, and browser demos.
  37. CNN convolution scan animation An 8-second Remotion animation showing how a 3x3 convolution kernel scans an input and builds a feature map.

Current route

  1. AI Basics Learning Roadmap Learning path step
  2. Machine Learning Workflow Learning path step
  3. Model Training and Evaluation Learning path step
  4. Neural Network Basics Learning path step
  5. Matrix Calculus for Neural Networks Learning path step
  6. Backpropagation as a Computation Graph Learning path step
  7. Gradient Descent and Optimizer Geometry Learning path step
  8. Convolution and Receptive Field Math Learning path step
  9. Transformer Attention Math Learning path step
  10. LLM Visualizer Learning path step
  11. Python AI Mini Practice Learning path step
  12. Handwritten Digit Dataset Basics Learning path step
  13. Handwritten Digit Softmax in C Learning path step
  14. Handwritten Digit Playground Notes Learning path step
  15. CIFAR-10 Tiny CNN Tutorial in C Learning path step
  16. High-Entropy Traffic Defense Notes Learning path step
  17. AI Security Threat Modeling Learning path step
  18. Adversarial Examples and Robust Evaluation Learning path step
  19. Data Poisoning and Backdoor Defense Learning path step
  20. Model Privacy and Extraction Defense Learning path step
  21. LLM, RAG, and Agent Security Learning path step

Next notes

  1. Add more image-classification and error-analysis cases
  2. Turn common metrics into a quick reference
  3. Add more AI security defense experiment notes
Scroll down