Learning robust prompts
We investigated how prompt tuning could improve the adversarial robustness of vision–language models such as CLIP. Prompt tuning adapts a pretrained model through a small set of learned prompt parameters. It is commonly used to improve performance on a target task; our interest was in learning prompts that also make the model more resistant to adversarial perturbations.
Robustness and clean accuracy
Methods such as TeCoA [1] and FARE [2] show that adversarial adaptation can improve CLIP’s robustness, but it can also reduce accuracy on unperturbed inputs. FARE retains more clean accuracy than TeCoA in the reported comparisons, though it can still fall below the original CLIP.
This trade-off motivated our work. We explored whether learning prompts could improve robustness while retaining or improving clean accuracy, with only limited changes to the pretrained model. The question was how much of that balance could be achieved through the prompts themselves.
References
[1] TeCoA · Chengzhi Mao, Scott Geng, Junfeng Yang, Xin Wang, and Carl Vondrick. Understanding Zero-Shot Adversarial Robustness for Large-Scale Models. ICLR, 2023.
[2] FARE · Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein. Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models. ICML, 2024. Reported clean and adversarial results.