Understand evasion, poisoning, privacy, and model extraction

Name the attacker model first

Adversarial-ML results make sense only after stating what the attacker can know and change. Can they alter training samples, query probabilities, submit inputs repeatedly, inspect model weights, or influence labels? Are they targeting integrity, availability, privacy, or misuse?

A defense evaluated against one perturbation budget or one query interface does not automatically generalize to another.

Four useful risk families

Evasion changes inference-time inputs so a model makes a chosen mistake. Poisoning changes training or adaptation data. Privacy attacks seek information about training records or model behavior. Extraction and abuse use access to reproduce capability, steal intellectual property, or repurpose the system.

Generative systems add prompt, retrieval, and misuse risks, while agents add tool and action paths. Keep classic application security in the same model: an exposed notebook or unsafe deserialization may be easier to exploit than the model itself.

• Evasion: test the exact input constraints and decision threshold.
• Poisoning: protect provenance, review, lineage, and rollback.
• Privacy: minimize sensitive training data and test memorization risk.
• Extraction: rate-limit, monitor, watermark where useful, and protect model artifacts.
• Supply chain: verify model, dataset, code, and package origin.

Mitigations have limits

Adversarial training can improve robustness against attacks represented during training, but may reduce clean performance and may not cover new methods. Input filtering can be bypassed. Obscurity and hidden model details are not strong primary controls.

Use defense in depth: data governance, provenance, access controls, robust evaluation, monitoring, human escalation, and recovery. Report the tested threat model and known gaps.

Robust to what, under which access, at what cost, and for how long?