Learning from inverse adversarial examples
We study how to transfer adversarial robustness from a large teacher model to a smaller student. Simply copying the teacher’s predictions can pass on its mistakes. Our method trains the student using inverse adversarial examples, formed by reversing the direction of adversarial perturbations.
What passes from teacher to student
- Training inputs
- Inverse adversarial examples, obtained by reversing adversarial perturbations.
- Model alignment
- Match input-gradient information between the robust teacher and the smaller student.
- Weight perturbations
- Jointly perturb both models to find directions that improve robustness transfer.
Results
Across the evaluated datasets, the method improves both accuracy on clean inputs and robustness under attack compared with the tested distillation methods. The paper also examines gains from generated training data and extensions to multimodal models.
Data and training dynamics
At Imperial College London, I investigated data pruning and input-space geometry across model architectures. In separate work, we study when SGD reduces reliance on spurious correlations and how a short small-batch warmup can improve worst-group accuracy.
Related paper
Allies Teach Better Than Enemies: Inverse Adversaries for Robust Knowledge Distillation
BibTeX
@article{Junhao_raoof,
title = {{Allies Teach Better Than Enemies: Inverse Adversaries for Robust Knowledge Distillation}},
author = {Dong, Junhao and Zare Moayedi, Raoof and Ong, Yew-Soon and Moosavi-Dezfooli, Seyed-Mohsen},
year = {2026},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
doi = {10.1109/TPAMI.2026.3660863}
}