CAMBRIDGE, Mass. — At the 2004 MIT Spam Conference, a machine-learning spam filter was trained to defeat another filter, automatically learning which words to add to an email to get it classified as not spam. The demonstration, by researcher John Graham-Cumming, stands today as the earliest documented adversarial machine-learning attack. That same year, Nilesh Dalvi and colleagues noted that linear classifiers in spam filters could be defeated by simple evasion attacks — for instance, spammers inserting “good words” into spam emails.
By 2007, some spammers had moved to random noise, fuzzing words in “image spam” to defeat OCR-based filters. In 2006, Marco Barreno and others published “Can Machine Learning Be Secure?”, outlining a broad attack taxonomy.
The field of adversarial machine learning had begun to organize itself, studying both attacks on and defenses for machine learning algorithms. Until 2013, many researchers had hoped non-linear classifiers — such as support vector machines and neural networks — were inherently robust to adversaries. That assumption cracked in 2012–2013, when Battista Biggio and others demonstrated the first gradient-based attacks on those very models.
The Deep Learning Shift
Starting in 2014, Christian Szegedy and others showed that deep neural networks could be fooled by gradient-based adversarial perturbations. The finding arrived just after deep neural networks had begun dominating computer vision in 2012. Later work revealed a complicating factor: adversarial attacks are harder to produce in uncontrolled environments.
Small rotations or changes in illumination can destroy the adversarial properties of an image. Google Brain researcher Nick Frosst has noted that it is physically easier to remove a stop sign than to create reliable adversarial examples for self-driving cars.
Real-World Vulnerabilities
The core vulnerability is data poisoning, where attackers supply fabricated training data that violates the assumption that training and test data come from the same statistical distribution — the so-called IID assumption. That assumption breakdown occurs often in high-stakes applications. Common attacks include evasion attacks, data poisoning, Byzantine attacks, and model extraction.
Concrete examples have emerged across sectors: obfuscating spam via misspelled “bad” words or inserted “good” words; obfuscating malware in network packets to mislead intrusion detection; and using fake biometric traits to impersonate users or compromise template galleries. Researchers have shown that changing just one pixel can fool deep learning algorithms.
Others 3D-printed a toy turtle with a texture engineered to make Google’s object detection AI classify it as a rifle from any angle.
Industry Response
Major tech companies — Google, Microsoft, and IBM — now curate documentation and open-source code to help developers assess model robustness. Frosst argues that the adversarial ML community has incorrectly assumed that models trained on one distribution will perform well on
