Threat-model AI across data, model, application, and action

Start with what must remain true

A useful threat model begins with security invariants: a user can retrieve only documents they are allowed to see; an agent cannot spend money without approval; a model update cannot enter production without provenance and evaluation; untrusted content cannot silently change tool policy.

Invariants are easier to test than broad goals such as make the chatbot secure. Each one connects an asset, trust boundary, control, and observable failure.

Use frameworks as complementary lenses

OWASP helps identify common application risks. MITRE ATLAS adds adversary tactics and techniques across predictive, generative, and agentic systems. NIST provides a consistent adversarial-ML taxonomy and lifecycle view. None of them replaces a system-specific analysis.

Map each relevant technique to an actual component and impact. If a technique cannot reach an asset in your architecture, record why and move on. If it can, identify the preventive control, detective evidence, and recovery path.

• Data plane: poisoning, leakage, unauthorized retrieval, and retention.
• Control plane: prompt, policy, configuration, identity, and deployment changes.
• Model plane: extraction, evasion, privacy, misuse, and unsafe capability.
• Action plane: tool misuse, excessive agency, cross-system movement, and transactions.
• Human plane: overreliance, social engineering, poor review, and ambiguous accountability.

Reframe reconnaissance as authorized attack-surface inventory

AI can summarize certificate records, asset inventories, code repositories, cloud configuration, and scanner output, but discovery must remain inside an owned or explicitly authorized scope. Public availability does not automatically grant permission to probe, correlate personal information, or profile employees.

A defender-first workflow starts with existing inventories and passive evidence, records provenance, removes personal data that is not needed, and sends uncertain ownership to validation. Use AI to cluster and explain observations; use authoritative asset and identity systems to decide what the organization owns.

• Begin with CMDB, cloud inventory, DNS ownership, certificate records, and approved scanner output.
• Separate passive collection from active probing in the rules of engagement.
• Do not turn employee names, social profiles, or breach data into targeting material.
• Verify exposed-service claims with the asset owner before escalation.
• A useful output is an owned remediation queue, not a list of attractive targets.

Prioritize by consequence and preconditions

A reproducible prompt that changes tone is not equivalent to a prompt that can retrieve another user's data or invoke a payment tool. Rank scenarios by impact, reachability, required access, detectability, and recovery cost.

Keep a separate list of unknowns. Threat modeling is valuable when it reveals a missing owner, unlogged decision, undocumented model, or tool that lacks an authorization boundary.

Test the path to impact, not the cleverness of the prompt.