AI Security Threat Modeling: Build a Defense Map with NIST, MITRE ATLAS, and OWASP

The most common mistake when threat modelling a machine learning system is reusing the traditional application security checklist — scan dependencies, check endpoint authentication, harden the server. All necessary, and none of it covers the attack surface specific to these systems: training data can be poisoned, the model weights are themselves an asset to protect, and the model’s outputs leak what it has seen.

The framework below draws on three public sources: the NIST Adversarial Machine Learning taxonomy (NIST.AI.100-2e2025), MITRE ATLAS, and the OWASP LLM Top 10. What this article adds is grounding them in a concrete asset inventory and verification evidence — every threat has to answer “how do I test this” and “what result counts as passing,” or it remains a noun.

1. Beyond the Weights: An Expanded Asset Taxonomy

Conventional application security focuses on APIs, databases, and IAM. AI systems introduce a complex attack surface characterized by continuous optimization and stochastic outputs. The asset taxonomy must be expanded:

• Training Data & Pipeline: Raw samples, high-dimensional manifolds, labels, annotation functions, provenance cryptomaterial, and data-filtering heuristics. Vulnerable to data poisoning and backdoor (BadNets) injection.
• Model Artifacts: Learned parameter matrices \( \theta \in \mathbb{R}^d \), tokenizers, embedding spaces, calibration hyperparameters, and evaluation set distributions.
• Prediction Interfaces (Inference): Inputs \( x \), logits, softmax probabilities \( f(x) \), confidence scores, and local/global explanations. Vulnerable to Model Extraction and Membership Inference Attacks (MIA).
• Context & Orchestration Systems: RAG document corpora, vector database index structures (e.g., HNSW), reranker weights, and ReAct agent tool permission schemas.
• Feedback Loops: RLHF reward models, preference datasets, and active learning retraining queues.

2. The Mathematical Threat Landscape: A Three-Layer Architecture

We architect the threat model across three operational layers:

Layer 1: The Optimization Risk (NIST taxonomy). Categorizes attacks by their mathematical objectives. Evasion computes perturbations \( \delta \) such that \( \arg\max f(x+\delta) \neq y \). Poisoning injects \( (x_p, y_p) \) into the training distribution \( \mathcal{D} \) to shift the empirical risk minimizer \( \hat{\theta} \). Privacy attacks compute the likelihood \( P(x \in \mathcal{D}_{train} \mid f(x, \theta)) \).

Layer 2: The Tactical Execution (MITRE ATLAS). Maps adversarial objectives to execution chains, such as ML Supply Chain Compromise (e.g., malicious pickle serialization leading to RCE) or Discovering ML Artifacts.

Layer 3: Application Failure Modes (OWASP LLM Top 10). Translates these to runtime exploits: Prompt Injection (modifying the LLM’s conditioning context), Sensitive Information Disclosure, and Excessive Agency in automated reasoning pipelines.

3. Red/Blue Team Post-Mortem: Production Threat Records

A production-grade threat record must explicitly define the mathematical optimization of the attacker and the empirical defense threshold. Example:

Asset: Inference API (Softmax Output)
Attacker Goal: Membership Inference Attack (MIA)
Mathematical Vector: Exploit the divergence in prediction entropy between training and holdout sets. Attacker trains a shadow model to classify \( \mathcal{H}(f(x)) \).
Red Team Validation: Shadow model achieves MIA AUC-ROC > 0.7.
Blue Team Control: Temperature scaling, logit suppression (Top-k only), and differential privacy (DP-SGD) during training with \( (\epsilon, \delta) \)-bounds.
Residual Risk: Complete defense via DP-SGD severely degrades primary task accuracy. Boundary leakage remains possible via timing side-channels.

4. Mathematical Formalization of Evasion (Threat Modeling Entry Point)

Evasion attacks exploit the local linearity of neural networks in high-dimensional space. An attacker seeks a perturbation \( \delta \) subject to an \( L_p \) norm constraint \( \|\delta\|_p \le \epsilon \).

The objective is to maximize the loss function \( J(\theta, x + \delta, y) \):

\[ \delta^* = \arg\max_{\|\delta\|_p \le \epsilon} J(\theta, x + \delta, y) \]

This constrained optimization problem is the foundation of the threat model for the inference phase. Defenders must evaluate the Jacobian matrix of the model to understand sensitivity: \( \nabla_x f(x) \). If the spectral norm of the Jacobian is high, the model is highly susceptible to small \( \delta \).

5. Engineering Controls and Evidence

Threat modeling must produce artifacts that integrate into CI/CD/CT (Continuous Training) pipelines:

• Data Provenance Cryptography: Cryptographic hashing of datasets and validation of source signatures to prevent supply-chain poisoning.
• Robustness Certificates: Lipschitz continuity bounds or randomized smoothing guarantees logged per model version.
• Inference Telemetry: Monitoring the KL-divergence between rolling inference distributions and the training manifold to detect OOD (Out-of-Distribution) evasion attempts.

6. Threat Model Evidence Matrix

A useful AI threat model is reviewable only when each risk has a measurable artifact. The table below turns the narrative model into an audit surface that can be checked during design review, release approval, and incident response.

7. Conclusion

Threat modeling for AI is an ongoing exercise in bounding the adversarial optimization landscape. It transitions security from qualitative checklists to quantitative, empirical risk measurement.

A threat model that changes no specific decision is just a document

The most common failure in threat modelling is not missing a class of attack. It is producing a thorough-looking document that changed no decision. It passes review, gets filed, and is never opened again.

A direct test for whether a threat model is useful: point at any line and ask “what specifically did we change because of this line.” If the answer is “raised awareness” or “increased monitoring vigilance,” that line produced no control. Useful answers look like: because of this, a tool moved from enabled-by-default to requiring explicit authorisation; because of this, an endpoint gained rate limiting; because of this, a dataset moved off shared storage.

The corresponding practice: every threat must resolve to an owner, a specific action, and a completion criterion. An entry without all three is an observation, not a control. Observations have value, but they belong on a separate list — mixed in with implemented controls, they make the document read as far stronger protection than actually exists.

The asset inventory’s most common gap: multiple paths to the same data

The asset taxonomy above is organised by asset type. In practice the most frequent omission is not a missing category but counting a piece of data as one asset without enumerating every path that reaches it.

I hit this on a machine of my own. After putting a two-factor gate in front of a file management interface, I asked at the end what the gate actually protected — and found it protected the interface and not the files. The directory those files lived in was simultaneously served as public static assets under a different hostname, so the same bytes had two reachable URLs and one of them never touched the gate. Both configurations were individually correct; the problem existed only at their intersection.

The full account is in Authenticated Is Not Protected. The reusable parts:

• The unit of modelling is “data plus a path to it,” not “data.” Two paths to the same data are two entries to be assessed separately, because their authentication, logging and rate limiting are quite likely different.
• Verification is measurement, not configuration review. Take an object you are certain should be private and try every entry point you can think of. Familiarity with the configuration is no substitute, precisely because the hole grows where every individual configuration is correct.
• Prove the control is in the request path first. Configuration that real traffic has never passed through is a document, not a control, however complete it reads. My gate was exactly that — the port it listened on was not the ingress forwarding target, and its access log contained nothing but loopback requests I had made myself.

How to tell whether a threat is real

Threat modelling generates many candidates and a substantial fraction do not survive scrutiny. Leaving unfounded entries in the document costs more than wasted attention; it dilutes the credibility of the real ones — when half a list is strained, readers start discounting all of it.

During that adversarial review, three accusations were ultimately rejected, each exposing a characteristic reasoning error worth keeping as a checklist:

• Inferring from types rather than verifying along the data path. “Citation links can carry a javascript: pseudo-protocol” sounds reasonable and was unreachable — the parsing regex already required https?://. Test: every step of the attack path must point at a specific location in code.
• Locating the boundary at the wrong layer. “The prompt has no anti-injection language” fails because what limits harm is the model’s lack of tool access, not the wording. Test: ask what happens if this control does not exist. If the answer is “nothing much,” it was not the defence.
• Looking for evidence in only one file type. Whoever raised “fetching has no concurrency limit” had searched application source and not the reverse proxy configuration — the rate limit lived in nginx all along. Test: a conclusion’s scope cannot exceed the scope of the search.

So before a threat enters the final list, run it past: can every step of the attack path be located concretely? If this control were removed, what specifically follows? Did my search cover every component the conclusion depends on?

8. References

• NIST AI 100-2e2025: Adversarial Machine Learning
• MITRE ATLAS
• OWASP Top 10 for Large Language Model Applications
• High-entropy traffic defense notes
• AI Security Lab README

For what a threat model looks like once it lands on a concrete mechanism, see this site’s own: the IP reputation scorer publishes the full scoring rules – which local signals are used, what each is worth, why country weighting is deliberately excluded, and why the score only sets a price rather than a verdict.