Constructing Semantics-Aware Adversarial Examples with Probabilistic Perspective

Thank you for taking the time to review our work. Here is our response:

some adversarial examples are vague compared to the original images.

Yes, indeed, this phenomenon is particularly evident in adversarially trained classifiers, which are notably challenging to attack.

..., it is unknown whether these examples are harder for the defense methods to detect.

Through reject sampling and sample refinement, the samples presented to human annotators are already capable of deceiving the victim classifier. Due to the length constraints of our submission, we detailed reject sampling and sample refinement in Appendix D, which might have led to some misunderstanding. We apologize for any confusion this may have caused.

It would be better if the authors can also validate the efficacy of the proposed method on larger network such as ResNet-50, and more difficult task such as CIFAR-100 and ImageNet.

We have observed a notable decline in classification performance in multi-class classifiers with a large number of classes after adversarial training, and the improvement in defense is not substantial. Consequently, we did not select CIFAR-100 as our target, following the precedent set by works [1] and [2]. To showcase the capability of our method with larger images, we are currently experimenting with ResNet-50 on a restricted version of ImageNet, aligning with the methodology in [1]. Updates on the progress of this additional experiment will be provided as soon as it is complete.

What is the perturbation radius used in Figure 1?

In Figure 1, the perturbation radius ($\epsilon$) for PGD with $L_{\infty}$ norm are set to 0.4, 0.15, and 0.1 for MNIST, SVHN, and CIFAR10, respectively. For PGD with the $L_2$ norm, the corresponding $\epsilon$ values are 5, 4, and 3 for the same datasets.

[1] Tsipras, Dimitris, et al. "Robustness may be at odds with accuracy." arXiv preprint arXiv:1805.12152 (2018).

[2] Madry, Aleksander, et al. "Towards deep learning models resistant to adversarial attacks." arXiv preprint arXiv:1706.06083 (2017).

The results of the ImageNet experiment have been compiled. Kindly refer to Appendix N in the present version of the PDF, located at the document's end. As I am in the process of revising the document, I will inform you should there be any changes to this reference.

Please review the ImageNet results at your earliest convenience and do not hesitate to reach out with any questions you might have. Thank you!

Thank you for your review. Below is our response:

It provides detailed background knowledge on paper writing and is very reader-friendly.

I am greatly pleased to hear your acknowledgment. We have put effort into making this section clear, as our goal is to bridge the gap in understanding between those focused on probabilistic modeling and those specializing in adversarial attack applications.

there is no essential difference between the previous adversarial example generation processes (they all use gradients to find optimization targets，the objective function used in this paper is still the C&W attack.).

We respectfully disagree with this point. As detailed in formula (1), the objective function comprises two components: $\mathcal{D}$ and $f$. While it's true that we selected $f_{CW}$ as a common choice for $f$, the unique contribution of our probabilistic approach lies in the data-driven aspect of $\mathcal{D}$. This means that $\mathcal{D}$ is not just a geometric distance anymore. The distance distribution $p_{\text{dis}}$ can be modeled by any probabilistic generative model. In our case, we found that the Energy-Based Model (EBM) provides an elegant mathematical formulation, allowing us to use minus energy to represent distance. We believe we have already highlighted this point in Section 4 of our original submission.

Moreover, we believe that the version you mentioned is identified as the 'probabilistic CW attack' (prob CW), wherein $\mathcal{D}$ is the L2 norm and $f$ aligns with $f_{CW}$, as depicted in Figure 3 (c). The primary distinction between prob CW and the conventional CW attack is the implementation of Langevin dynamics for the optimization process.

The method has only been verified on some small datasets.

We have tested our proposed method on the MNIST, SVHN, and CIFAR10 datasets, which is consistent with the experimental scale in [1]. As highlighted in Section 8, attacking digit datasets, despite their lower resolution, is practically more challenging. We believe the scope of our current experiments sufficiently demonstrates the strengths of our method. The primary limitation in extending to higher resolution images stems from the capabilities of the Energy-Based Model (EBM), which we used as one possible representation of $p_{\text{dis}}$. It's plausible that choosing a different $p_{\text{dis}}$ capable of handling higher resolution images could overcome this limitation. In this study, we chose EBM for its elegant mathematical form in the probabilistic context, presenting it as a foundational implementation while considering alternative $p_{\text{dis}}$ for future research.

Nevertheless, we will endeavor to include higher resolution experiments in this rebuttal and will keep you updated with any new experimental results.

Can defenders train a classifier to filter adversarial samples generated based on the energy model?

Certainly, if defenders know about the set of transformations $\mathcal{T}$ beforehand, they might integrate these into their training as data augmentation. However, our study is focused on victim classifiers that have undergone standard adversarial training, a prevalent method consistent with the approach in [1]. We haven't tested attacks on classifiers trained with such transformations, operating under the assumption that defenders are not privy to the attacker’s specific transformations in advance.

[1] Song, Yang, et al. "Constructing unrestricted adversarial examples with generative models." Advances in Neural Information Processing Systems 31 (2018).

Thanks for your reply!

However, It is unacceptable that an attack method currently cannot be applied to large-resolution images, especially in 2023.

My understanding is that the primary goal of academic conferences is to propose and exchange innovative ideas, rather than to engage in benchmarking or competition. I encourage you to focus on the originality and innovation of our ideas. It's unreasonable to expect the first steam locomotive to outperform horse-drawn carriages in every performance aspect. Similarly, a data-driven $p_{dis}$ represents a novel concept with considerable flexibility and future potential, particularly as probabilistic generative models continue to evolve.

Furthermore, considering that the popular resolution for ImageNet classification is 224x224, we believe that our resolution of 128x128 is comparable in scale.

... and did not provide the attack success rate. The author showed some attacked images, which was not convincing.

As previously mentioned, all the images generated by our method successfully deceive the classifier, achieving a 100% success rate. However, the challenge with unrestricted adversarial attacks is that they may be perceptible to human observers, as they are not assessed under an objective geometric distance. Therefore, a human subjective evaluation becomes crucial. We recognize, as demonstrated in Appendix N, that while our attack method generates adversarial examples with less noticeable tampering traces, the images tend to be relatively blurred. This blurring effect could be attributed to the current limitations in the implementation of energy-based models.

Thank you for your insightful review and for recognizing the contribution of our paper.

Here are our responses to your comments:

Since this paper proposes a semantics-aware method, the human evaluation is important to verify the validity of adversarial examples.

Yes, human evaluation is crucial. Our experiments conform to the evaluation methods detailed in [1].

It lacks a comparison on the number of queries.

Could you clarify what is the number of queries you're referring to?

There is a lack of an ablation experiment. In p_\{\text{adv}}, what will happen if p_\{\text{dis}} uses L2 or other methods to calculate Figure 1 or Table 1?

Refer to Figure 3 (c), showcasing the Prob CW (probabilistic CW attack).

Figure 3 (c) reveals that the adversarial examples represent intersections between the original image and the target class, making them easily identifiable by humans. Therefore, we omitted them from Table 1 to conserve human annotation efforts. In response to your suggestion, we will incorporate adversarial examples generated by the prob CW attack in the format of Figure 1 into our revised draft and will notify you once it's complete.

[1] Song, Yang, et al. "Constructing unrestricted adversarial examples with generative models." Advances in Neural Information Processing Systems 31 (2018).

Thank you for taking the time to review my submission. Below are our detailed responses to your comments.

The paper presents a probabilistic approach to construct semantically meaningful adversarial examples by interpreting geometric perturbations as distributions. The approach relies on training an energy based model (EBM) to emulate such distributions.

In addition to the concrete approach, we also want to highlight the significance of our probabilistic perspective presented in Section 3. The distance distribution $p_{dis}$ can be modeled using various probabilistic generative models. This probabilistic framework holds considerable potential for extension, with Energy-Based Models (EBM) being just one of many possible options. In this paper, we opted for EBM because its mathematical formulation aligns perfectly with Langevin dynamics, resulting in clear and concise derivations.

The paper is missing several references, and is relatively sparse with regards to related work. Similar works include ...

We contend that, with the probabilistic modeling perspective, the works cited are not similar to our approach. While some of these studies may employ Generative Adversarial Networks (GANs), it's important to note that GANs do not explicitly model the probability distribution $p(x)$. Our work is pioneering in rigorously deriving the distribution $p_{adv}$ from which adversarial samples are generated.

We acknowledge that from an application perspective, these works bear similarities to ours. In fact, under this viewpoint, any study not employing geometric constraints on the original image could be considered similar to our approach. However, our focus is on the novelty of the probabilistic perspective and modelling. Our contribution lies in introducing new concepts to the academic community, rather than solely advancing application benchmarks.

Thank you for introducing these related works to us. We will ensure to incorporate them into the revised version of our paper.

The experiments are limited to just two adversarially trained networks.

In the main text, we discuss three adversarially trained networks: two are presented in Table 1 for MNIST and SVHN, and another is detailed in Section 6.2 for CIFAR10. Furthermore, Appendix F features 12 additional networks, demonstrating our model's transferability. We limited the number of structures in the main text for two reasons: firstly, to maintain alignment with the settings in [1], and secondly, to efficiently illustrate our method's advantages without unnecessarily utilizing human annotators.

Furthermore, the adversarially trained models do not appear to have been trained under the semantic attack threat model and hence unlikely to be robust to such attacks.

The objective of our experiment is to demonstrate the effectiveness of our attack method against adversarially trained models. While some studies in this field have incorporated experiments where generated adversarial examples are used in training robust classifiers, we have not pursued this approach. We believe our current experimental setup sufficiently highlights the strengths of our attack method, in line with [1] and [3], and we aim to avoid unnecessary use of human annotators.

The attack also relies on adversarially trained models learning semantically meaningful features.

This is not correct. I encourage a careful review of the draft, particularly Figure 2 (a) and (b). These figures demonstrate that when the victim classifier is adversarially trained, $p_{vic}$ possesses generative capabilities, meaning it tends to produce images resembling those from the target class, which in turn makes an adversarial attack more challenging. Our proposed attack method is not dependent on adversarially trained models. In fact, attacking a victim classifier that is not adversarially trained would be significantly easier.

However, the images shown in the figures appear to be distorted which could be a consequence of the models overfitting on a subset of semantic features.

Could you please specify which distortion you're referring to and in which figure it occurs?

Could the authors clarify the training setup for the adverarially trained models?

We use the setting introduced in [2], as introduced in Section 2.2.

[1] Song, Yang, et al. "Constructing unrestricted adversarial examples with generative models." Advances in Neural Information Processing Systems 31 (2018).

[2] Madry, Aleksander, et al. "Towards deep learning models resistant to adversarial attacks." arXiv preprint arXiv:1706.06083 (2017).

[3] Xiao, Chaowei, et al. "Spatially transformed adversarial examples." arXiv preprint arXiv:1801.02612 (2018).

Thank you for your reply! This response will be divided into two text boxes.

1

In Section 3, we present the abstract form of the probabilistic perspective on adversarial attacks. The $p_{\text{dis}}$ can represent any probabilistic generative model. Given the abstract nature of this framework, it's not essential to exhaustively explore all possible choices for $p_{\text{dis}}$. Instead, we view it as a promising research direction for future exploration. In this work, we use the EBM as a case to demonstrate the framework's potential. Our current draft aims to introduce a new perspective and a model based on it, rather than achieving the ultimate performance in this task.

2

it is equally important to compare with similar approaches in order to evaluate the proposed attack.

We agree that comparing with similar approaches is crucial for evaluating our proposed attack. However, aligning comparisons in unrestricted adversarial attacks is challenging. In restricted attacks, we can standardize comparisons by fixing the L-p distance and comparing success rates. For unrestricted attacks, while we can achieve a 100% success rate in deceiving the classifier, it's unclear whether the adversarial examples deviate significantly from the original image in human perception. Therefore, we rely on human annotators to determine if the original class of the image is recognizable. This method, aligned with [1], is reflected in our Table 1 results.

Contrastingly, [2] proposes a method that doesn't guarantee a 100% success rate in deceiving the victim classifier. Consequently, they employ a different evaluation method, assuming their adversarial examples are imperceptible to humans. Their success rate is thus based on the classifier's deception rate. To validate their assumption, they conducted an A/B test assessing human perception, focusing on ImageNet. However, this method may lack rigor, as indicated by the discernibility of their MNIST adversarial samples in Figure 2 of [2], which might be more noticeable to humans than suggested.

[3] similarly assumes that their method produces adversarial examples undetectable by humans and compares the accuracy of the victim classifier, where a lower accuracy indicates a more effective attack. However, upon examining their figures, I find their assumption about human indistinguishability unconvincing.

[4], being a workshop paper, lacks comparative analysis.

[5] assumes that alterations in color and texture remain undetected by humans. Subsequently, they assess their attack's success rate in comparison with conventional methods.

In summary, our approach and [1], guarantee that our adversarial examples successfully deceive the victim classifier, with human annotators then evaluating their distinguishability. In contrast, [2], [3], and [5] proceed on the assumption that their generated adversarial examples are indistinguishable to humans, focusing instead on comparing the success rates in compromising the victim classifier.

Therefore, in our submission, we limit our comparison to [1]. In the revised version, we will discuss [2], [3], [4], and [5], but will not include them in our comparison.

how accurately do EBMs model the adversarial distribution in comparion to GANs

[1]'s approach uses GANs to implicitly model the distribution of adversarial examples, whereas our method explicitly models this distribution. Direct comparison of these distributions is challenging, leading us to select human annotators' success rate as our sole evaluation metric.

even uniform distributions over the transformations

I may have misunderstood your question, but if we were to use a uniform distribution for $p_{\text{dis}}$, then $p_{\text{adv}}$ would essentially reduce to $p_{\text{vic}}$, as depicted in Figure 2 (a) and (b).

3

Thank you for your suggestion. We will relocate the training and sampling details of the EBM to the Appendix and shift the table showing transferability results to the main text.

Reference

[1] Song, Yang, et al. "Constructing unrestricted adversarial examples with generative models." Advances in Neural Information Processing Systems 31 (2018).

[2] Xiao, Chaowei, et al. "Spatially transformed adversarial examples." arXiv preprint arXiv:1801.02612 (2018).

[3] Joshi, Ameya, et al. "Semantic adversarial attacks: Parametric transformations that fool deep classifiers." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[4] Hosseini, Hossein, and Radha Poovendran. "Semantic adversarial examples." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2018.

[5] Bhattad, Anand, et al. "Unrestricted adversarial examples via semantic manipulation." arXiv preprint arXiv:1904.06347 (2019).

I appreciate the quick reply and the clarifications as well as the new Imagenet results. However, I still disagree with the authors on comparisons. Perhaps it is harder to exhaustively compare all semantic adversarial methods fairly due to the varying setups. However, [1] provides a very similar threat model. Also, while the authors claim that [1] does not have a 100% success rate and therefore cannot be compared, Figs 3 and 4 from the main paper show similar failures. It might be useful to compare the failure modes to evaluate how semantic divergence is a better approach. In addition, given the significant progress in our understanding of adversarial robustness, and the standard evaluation approaches, evaluating a robust model under an attack with a different threat model does not make sense to me. Perhaps the other reviewers could chime in and present their opinions on this.

However, I do find the idea of learning an adversarial distribution interesting by itself as well as of interest to the community if it can be scaled to larger models and complex datasets. As of now, I am willing to increase my score to 5 (marginally below the acceptance threshold). I am also open to changing my mind if the other reviewers or authors have additional thoughts to support the paper.

[1] Xiao, Chaowei, et al. "Spatially transformed adversarial examples."

Thank you for your suggestion! We have successfully replicated the StAdv method as proposed by [1], and the detailed results are now included in Appendix O (Update: now is P) of the latest version of our document.

[1] Xiao, Chaowei, et al. "Spatially transformed adversarial examples."