GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models

We express sincere gratitude for your constructive and detailed comments.

Weakness 1: Definition of adversarial robustness； Algorithm in appendix reduces coherence; Clarification on Figure 2.

（1） We apologize for the lack of definition. To clarify, adversarial robustness evaluation refers to the process of assessing a model's resilience against adversarial attacks, which are inputs intentionally designed to deceive the model. We will include it in the next version .

（2） Due to space limitations, we placed the algorithm in the appendix. However, we understand your concern about coherence and will move it to the main text in the next version.

（3）To clarify how RA and the GREAT Score are integrated into Figure 2, we provide the additional details:

1. Auto-Attack RA: RA at each perturbation level is the fraction of correctly classified samples under perturbation. GREAT Score RA: Cumulative RA at each perturbation level is the fraction of samples with GREAT Scores above that level.

2. Blue curve: RA from empirical Auto-Attack. Orange curve: RA from GREAT Score, providing a certified robustness guarantee.

3. Trend similarity suggests GREAT Score reflects empirical robustness. The gap indicates the difference between certified and empirical robustness, not necessarily GREAT Score's inferiority, but possible undetected adversarial examples at higher perturbations.

Weakness 2: Unknown data distribution.

Thank you for your insightful comment. By "unknown" data distribution, we mean the true data distribution is unknown (e.g., the data distribution of all nature images). In practice, one trains a generative model to learn this unknown function. The aim of the generative model to serve as a proxy for the true data distribution. In the Appendix A.2, we provide an explanation of how the generative model can effectively match the true data distribution. This justification supports our claim that the proposed metric can scale to unknown data distributions, which is also an aim for current state-of-the-art image generative models.

Weakness 3: No guarantee on producing benign samples.

We acknowledge that the performance of our metric depends on the generative model's ability to produce benign samples. Our approach assumes the use of high-quality GANs.

• Ablation Study: As demonstrated in our ablation study, using GANs with better generative quality improves the ranking coefficient, indicating stronger performance of the robustness metric.
• Theoretical Support: In Appendix 2, we discuss how GANs can provably match the data distribution, providing a theoretical foundation for their use in our approach.

Weakness 4: Training and online learning for Generative Models.

Thank you for your valuable suggestion. Our ablation study indicates that improving the quality of the generative model significantly enhances the ranking correlation. Therefore, we definitely need better generative models. Currently, we are using off-the-shelf generative models, but we will consider incorporating online learning techniques to further improve their performance.

Weakness 5: Limitation to white-box settings.

Thank you for highlighting this important distinction. However, we would like to clarify that our discussion on limitations pertains specifically to the evaluation of attack-independent robustness, which we stated is limited to white-box settings. We acknowledge that adversarial accuracy can indeed be assessed in black-box scenarios, as demonstrated by methods like Square Attack.

Question 1: Omission of works on robustness evaluation.

Thank you so much for recommending the related works we need , we will add them in next version. Below is the added content.

In addition to discussed works, several studies evaluate model robustness differently. [Olivier 2023] introduce adversarial sparsity, quantifying the difficulty of finding perturbations, providing insights beyond adversarial accuracy. [Robey et al. 2022] propose probabilistic robustness, balancing average and worst-case performance by enforcing robustness to most perturbations, better addressing trade-offs. [Guo et al. 2024] introduce the adversarial hypervolume metric, a comprehensive measure of robustness across varying perturbation intensities.

[1] "How many perturbations break this model? evaluating robustness beyond adversarial accuracy." Olivier, Raphael, and Bhiksha Raj. International Conference on Machine Learning. PMLR, 2023. [2] "Probabilistically robust learning: Balancing average and worst-case performance." Robey, Alexander, et al. International Conference on Machine Learning. PMLR, 2022. [3] "Exploring the Adversarial Frontier: Quantifying Robustness via Adversarial Hypervolume." Guo, Ping, et al. arXiv preprint arXiv:2403.05100 (2024).

Question 2: Results on the original test samples on ImageNet.

Following the reviewer's suggestion, we conducted an additional experiment using 500 test samples from the original ImageNet dataset, with image dimensions of 224x224 pixels, and reported their GREAT Scores in the table below. However, it is important to note that unless these test samples are generated by a generative model, they may not fully represent the data distribution captured by such a model. The correlation between the GREAT Score ranking and the RobustBench ranking remains consistent with the results observed on the generated samples.

We appreciate your detailed and constructive comments, and we are encouraged that you find our work “well written, easy to follow”.

Weakness 1: Generative model limitations: (a) valid class instance, (b) unambiguous class, (c) data approximation.

Thank you for your feedback regarding the assumptions on the generative model. We would like to specify how to ensure these points:

(a) We acknowledge that the performance of our metric relies on the generative model's capability to produce samples belong to the conditioned class. Recent studies ([Nicolas, 2021] and [Tengyuan Liang, 2021]) have shown GANs' convergence to true data distributions under specific conditions. Moreover, Figure 3 shows high-quality instances produced by the generative models, as evidenced by the inception score and the Spearman's rank correlation between the GREAT Score and RobustBench.

(b) We recognize that there may be instances where the true class is ambiguous, potentially impacting the performance. Given that our focus is on evaluating the robustness of classifiers, it is important to note that the labels we use are typically well-defined and distinctive. When two classes exhibit ambiguity, determining their robustness is a separate issue that needs to be addressed before applying our method. Consequently, we consider the issue of label ambiguity to be outside the scope of our method.

(c) We understand that the assumption of the generative model being a good approximation of the true data-generating distribution is crucial. Recent work has demonstrated the convergence rate of approaching the true data distribution for a family of GANs under certain conditions. Please refer to Appendix A.2 for more details.

References:

Nicolas Schreuder, Victor-Emmanuel Brunel, and Arnak Dalalyan. Statistical guarantees for generative models without domination. In Algorithmic Learning Theory, pages 1051–1071. PMLR, 2021. 14

Tengyuan Liang. How well generative adversarial networks learn distributions. The Journal of Machine Learning Research, 22(1):10366–10406, 2021. 14

Weakness 2: Importance of robustness guarantee.

We appreciate your feedback regarding the importance of our score. Our method is a form of certified robustness, a concept well-utilized by [Tramer, F. 2020] and [Pintor, M. 2022], and recognized as a secure way for validating defenses. Many empirical defenses were soon to be broken by advanced attacks because they were not certified to be robust. Additionally, our score can be used for robustness ranking across different models, as discussed in our paper. This ranking allows for comparison of the resilience of models against adversarial attacks.

1. Tramer, F., Carlini, N., Brendel, W., & Madry, A. (2020). On adaptive attacks to adversarial example defenses. Advances in neural information processing systems, 33, 1633-1645.

2. Pintor, M., Demetrio, L., Sotgiu, A., Demontis, A., Carlini, N., Biggio, B., & Roli, F. (2022). Indicators of attack failure: Debugging and improving optimization of adversarial examples. Advances in Neural Information Processing Systems, 35, 23063-23076.

Weakness 3: Equation Clarification.

Thank you for your insightful comment. We would like to clarify that, as defined in Equation 9, $g'$ is inherently linked with the Gaussian distribution through the generative model $G(\cdot)$. Therefore, the formulation $(g' \circ G) * N(0, I)$ is indeed consistent with our definitions. The inclusion of $g'$ with the Gaussian reflects the model's behavior in the context of our robustness analysis. We believe this captures the relationship and does not undermine the integrity of our Lipschitz constant estimation.

Weakness 4: Small fonts.

We appreciate your feedback regarding the small font size. We will increase the font size in the next version to enhance readability.

Question 1: Gap between empirical attacks and GREAT score?

We thank the reviewer for bringing up this discussion. We believe your concern can be addressed by comparing empirical versus certified robustness. AutoAttack does not guarantee the absence of undiscovered adversarial examples when it fails, a common limitation of empirical robustness evaluation. In contrast, our score provides a certified robustness guarantee, ensuring no adversarial examples exist within the certified perturbation range. Therefore, the gap between our certified curve and AutoAttack's empirical curve does not imply our method's ineffectiveness; it could indicate undiscovered adversarial examples at higher perturbation radii. Without sound and complete attacks that confirm no adversarial examples exist if they fail, we cannot rule out the possibility of hidden adversarial examples in these high-radii regimes, as highlighted by certified robustness analysis.

Question 2: Label distribution?

We appreciate the reviewer's insightful idea. We utilize a conditional generative model to generate samples based on class labels. The class labels are uniformly distributed.

Question 3: Class-specific robustness?

Thank you for the insightful question. We acknowledge that deriving guarantees for subsets of the data distribution is important to explore.

Our current work focuses on evaluating model robustness over the entire data distribution. Given our capability to perform class-specific generation, we can indeed measure class-specific robustness guarantees. By generating samples specific to each class, we can evaluate how robust the model is against adversarial attacks targeting particular classes, thereby evaluating class-specific robustness .

Actually, we have performed a similar analysis to evaluate group-level robustness for facial recognition. This method can be extended to derive class-specific robustness guarantees. Table 4 in our paper displays the group-level GREAT Score results. Our evaluation reveals interesting observations: Most APIs exhibit a large discrepancy in scores between "Old" vs. "Young" groups.

We appreciate your detailed and constructive comments.

Weakness 1 & Question 1: Ablation studies for other norms (e.g., L∞)?

We thank for bring the limititions of our theorem. As stated in Section 5, our framework currently focuses on the $\mathcal{L}_2$ norm due to limitations in extending Stein's Lemma to other $\mathcal{L}_p$ norms (see Lemma 4 in Appendix A.3). We agree that generalizing beyond $\mathcal{L}_2$ robustness would be beneficial, but it is challenging without a generalized Stein's Lemma for other $\mathcal{L}_p$ norms. As of now, we are not aware of any results that support such an extension.

Weakness 2 : Generative model quality and sample set bias.

The reviewer's intuition is correct. In our ablation study of generative models (GMs), we do find that better GMs give higher ranking. In Figure 3, we show that increasing the Inception Score of GMs can significantly increase the Spearman's rank correlation. Intuitively, a higher inception score means better learning of GMs to the underlying data distribution, resulting in improved ranking efficiency in our case. Recent works such as [Nicolas, 2021] and [Tengyuan Liang, 2021] have proved theoretically that GANs can learn the true generating distribution under certain conditions. In summary, the effectiveness of GREAT Score is positively correlated with the generation capabilities of the GM in use. Please refer to Appendix A.2 for details

References: Nicolas Schreuder, Victor-Emmanuel Brunel, and Arnak Dalalyan. Statistical guarantees for generative models without domination. In Algorithmic Learning Theory, pages 1051–1071. PMLR, 2021. 14

Tengyuan Liang. How well generative adversarial networks learn distributions. The Journal of Machine Learning Research, 22(1):10366–10406, 2021. 14

Weakness 3: Detailed analysis on group variability.

We appreciate your suggestion to provide a more detailed analysis among different demographic groups to add depth to our analysis.

Exploring robustness variability among demographic groups is crucial for mitigating biases in facial recognition systems. Studies like [Klare, 2012] and [Deb, 2020] have shown that factors such as age and race can affect recognition accuracy.

We have found an evidence for the facial recognition system has less elder faces in training data. [Meade, R. 2021] found that the majority of models that perform gender classification are trained on the most popular actors and actresses sourced from Wikipedia and IMDB. As this is a celebrity dataset, most celebrities appear much younger than someone of the same age who is not a celebrity - creating a bias towards classifying older people.

We will add this discussion in the revised version.

References:

[1] Klare, Brendan F., et al. "Face recognition performance: Role of demographic information." IEEE Transactions on Information Forensics and Security. 2012.

[2] Deb, Debayan, et al. "Face Recognition Performance: Role of Demographic Information on Consumer- to Organizational-Level Applications." IEEE Transactions on Information Forensics and Security. 2020.

[3] Meade, R., Camilleri, A., Geoghegan, R., Osorio, S., & Zou, Q. (2021). Bias in machine learning: how facial recognition models show signs of racism, sexism and ageism.

Weakness 4 & Question 4: Stability of grid search in calibration ?

Thank you for your feedback on the calibration process. While grid search for optimizing temperature parameters might seem ad-hoc, it is widely used and accepted in state-of-the-art research.

Several recent studies have used grid search for calibration due to its simplicity and effectiveness. [Guo et al., 2017] optimize temperature scaling for neural network outputs, forming a foundational reference for modern calibration techniques. Similarly, [Kumar et al., 2019] use grid search to optimize calibration parameters, demonstrating its utility for reliable uncertainty estimates.

Grid search is simple and effective under time constraints. Advanced techniques require significant resources and may not yield much better results. A systematic study with these techniques could strengthen our framework for future work.

References:

[1] Guo, Chuan, et al. "On calibration of modern neural networks." International Conference on Machine Learning. 2017.

[2] Kumar, Ananya, et al. "Verified uncertainty calibration." Advances in Neural Information Processing Systems. 2019.

Weakness 5 & Question 3: Scalability and computational efficiency on large datasets/models?

To address the question on the scalability of the GREAT Score framework, we conducted experiments using three ResNets, varying dataset sizes from 500 to 2000 images. We recorded the computation times in ms without employing any attack mechanisms.

The results show a linear increase in computation time with increasing dataset size and model complexity. More complex models like ResNet152 took proportionally more time than simpler ones like ResNet50.

This linear scalability demonstrates that the GREAT Score efficiently handles larger datasets and more complex models, making it suitable for large-scale applications.

Question 2: Impact of generative model quality ?

Yes, we thank the reviewer for pointing out this question, actually, we have done an Ablation study on GANs and DMs. Evaluating on CIFAR-10, Figure 3 compares the inception score (IS) and the Spearman’s rank correlation coefficient between GREAT Score and RobustBench on five GANs and DDPM. One can observe that models with higher IS attain better ranking consistency.

We express sincere gratitude for your valuable feedback and constructive comments.

Question 1: GAN reliability and distribution coverage?

Thank you for your feedback on the reliance on GANs as a proxy for the true data distribution. We acknowledge the concerns about the method's accuracy, particularly the issues of bias and model collapse inherent in GANs. We would like to address these concerns and explain our approach in detail.

Firstly, we would like to clarify that we utilized conditional generative models in our framework. Conditional generative models, such as Conditional GANs (cGANs), allow for more controlled generation of samples by conditioning on specific labels or attributes. This helps in generating more representative and varied samples, which can mitigate some of the biases and issues associated with traditional GANs.

Recent works such as [Nicolas, 2021] and [Tengyuan Liang, 2021] have proved theoretically that GANs can learn the true generating distribution under certain conditions. Besides GANs, we have diffusion models in our analysis, and as the reviewer expected, it gives a better robustness analysis, as shown in Fig. 3. Please also see Appendix A.2 for details.

References:

Nicolas Schreuder, Victor-Emmanuel Brunel, and Arnak Dalalyan. Statistical guarantees for generative models without domination. In Algorithmic Learning Theory, pages 1051–1071. PMLR, 2021. 14

Tengyuan Liang. How well generative adversarial networks learn distributions. The Journal of Machine Learning Research, 22(1):10366–10406, 2021. 14

Question 2: Experiments with other estimators?

Thank you for the insightful question. Following the reviewer's suggestion, in our experiments below, we included two types of local robustness estimators to evaluate model robustness more comprehensively: Input Gradient Norm (as a proxy of sensitivity) and Entropy of the classifier output (as a proxy of confidence). We note that unlike our proposed GREAT Score, these estimators do not provide any certified robustness guarantee. Here's why we chose these estimators and how they affect our global measure:

1. Input Gradient Norm:

   • Rationale: This estimator measures the sensitivity of the model to small perturbations in the input by calculating the norm of the gradient of the loss with respect to the input. A higher gradient norm indicates greater sensitivity to input changes, suggesting lower robustness.
   • Reference: The use of input gradient norm for robustness evaluation is well-documented. [ Finlay and Oberman, 2021] proposed a method using input gradient regularization to improve adversarial robustness.
   • Implementation: We computed the gradient norm for input samples and took the average to represent the model's sensitivity.

2. Entropy:

   • Rationale: Entropy of the classifier's output probabilities reflects the uncertainty of the model's predictions. Higher entropy indicates that the model is less confident in its predictions, which can be a sign of lower robustness.
   • Reference: The relevance of entropy as a measure of model uncertainty and robustness is supported by [Smith and Gal ,2018], who explored various uncertainty measures, including entropy, for detecting adversarial examples .
   • Implementation: We calculated the entropy of the output probabilities for input samples and took the average to represent the model's uncertainty.

To validate the reliability of our global measure, we computed the Spearman correlation coefficient between our averaged estimation scores and the rankings from RobustBench. Besides the local robustness estimator, the experiment setup is the same as Table 1.

Experimental Results

This table summarizes the correlation results for the local robustness estimators and the GREAT Score with respect to AutoAttack and RobustBench. These results demonstrate the effectiveness of GREAT Score in evaluating model robustness and their alignment with established benchmarks.

References

Finlay, C., & Oberman, A. M. (2021). Scaleable input gradient regularization for adversarial robustness. Machine Learning with Applications, 3, 100017. Smith, L., & Gal, Y. (2018). Understanding measures of uncertainty for adversarial example detection. arXiv preprint arXiv:1803.08533.

Question 3: Theoretical bound on distribution gap and its effect on utility?

We separated out the convergence of our edtimate to the true estimate versus the learned data distribution to the true data distribution because the latter has been proved in the literature. Recent works such as [Nicolas, 2021] and [Tengyuan Liang, 2021] have proved theoretically that GANs can learn the true generating distribution under certain conditions. Besides GANs, we have diffusion models in our analysis, and as the reviewer expected, it gives a better robustness analysis, as shown in Fig. 3.

The reviewer's intuition is correct. In our ablation study of generative models (GMs), we do find that better GMs give higher ranking. In Figure 3, we show that increasing the Inception Score of GMs can significantly increase the Spearman's rank correlation. Intuitively, a higher inception score means better learning of GMs to the underlying data distribution, resulting in improved ranking efficiency in our case. We also had a discussion on the approximation guarantee of some GMs to the true data distribution in Sec. 6.2. In summary, the effectiveness of GREAT Score is positively correlated with the generation capabilities of the GM in use.