Adversarial Guided Diffusion Models for Adversarial Purification

We greatly appreciate the comments of the Reviewer xPEV. Below are our responses to the questions raised.

Q1: The contributions of this paper are limited. The guidance technique is well explored under the training-free conditional diffusion such as FreeDom.

A1: Thank you for sharing an interesting paper. We think there are some significant differences between our work and [1]. Firstly, the problems we focus on are different: The adversarial perturbations are carefully designed, and their harmful impact on the generated images far exceeds the shortcomings of the generative model itself. When we use training-free conditional diffusion for standard classification tasks, it indeed achieves better performance, but it does not learn robust representations, resulting in poor performance on robust classification tasks. Secondly, due to the difference in problems, there are differences in our algorithms as well. As you mentioned, we can directly minimize the distance between adversarial examples and clean examples as one of the multi-conditional guidance. Wang et al. [2] propose a similar guidance following this idea, but they find that in the training-free mode when the distance between the adversarial and clean examples is too small, it is easy to preserve the perturbations, which is also highlighted and discussed in our paper. Therefore, we propose a new adversarial guidance for robust classification tasks. Although there is a similarity in the high-level concepts, the problems and specific algorithms are different.
[1] Yu et al. FreeDoM: Training-Free Energy-Guided Conditional Diffusion Model. ICCV, 2023.
[2] Wang et al. Guided Diffusion Model for Adversarial Purification. Arxiv, 2022.

Q2: Does AGDM really alleviate the trade-off for AP?

A2: AGDM can really defend against unseen attacks. Firstly, we would like to explain why AP can defend against unseen attacks, but AT cannot. From the pipelines, the answer is straightforward. AT, by 'overfitting' to specific adversarial examples, enables the classifier to effectively defend against those attacks, which obviously cannot defend against unseen attacks. Pre-trained generator-based AP methods first utilize the random transforms to destroy the perturbations, followed by a denoising operation using a generative model. Therefore, once the perturbations can be effectively destroyed, the entire system can defend against all attacks. There is more discussion in section 3.1, and the experiments support that AGDM can defend against unseen attacks in Tables 5, and 8.

Q3: The experiments are not enough: It lacks the BPDA attacks. The performance of AGDM is not better than ZeroPur [3], which even drops the diffusion models.

A3:
3.1: In evaluating diffusion-based AP method, considering the robustness misestimation caused by obfuscated gradients of the purifier model, the methods should not be evaluated by non-adaptive attacks. Nie, Lee, and Lin et al. [4,5,6] have all discussed this issue in the paper and have employed adaptive attacks to evaluate the methods. On the other hand, although BPDA+EOT is used as a robust evaluation of AP methods in Nie et al., in the latest research, Lee & Kim find that BPDA+EOT is unsuitable for diffusion-based AP methods tasks. The table below compares the robust accuracy of BPDA+EOT with the lowest robust accuracy between PGD+EOT and AutoAttack. The evaluation results of BPDA+EOT are far less reliable than those of PGD+EOT / AutoAttack. Therefore, the latest evaluation paradigm has been established with AutoAttack, StAdv, and PGD+EOT. We strictly adhere to the established pipeline to evaluate our experiments and follow the same settings to maintain consistency, ensuring the reliability of our results.

[3] Bi et al. ZeroPur: Succinct Training-Free Adversarial Purification. ArXiv, 2024.
[4] Nie et al. Diffusion Models for Adversarial Purification. ICML, 2022.
[5] Lee and Kim. Robust Evaluation of Diffusion-based Adversarial Purification. ICCV, 2023.
[6] Lin et al. Adversarial Training on Purification (AToP): Advancing Both Robustness and Generalization. ICLR, 2024.

3.2: Focusing on the community surrounding this paper, which aims to improve the robustness of the pre-trained generative model for adversarial purification, our method (AGDM) is highly efficient compared to existing defense methods.

Although ZeroPur can achieve better performance even drops the diffusion models, we believe our work is still meaningful. As described by Wang et al., with the continuous development of diffusion models, using better DMs can achieve superior performance [7]. In the future, the latest DMs are likely to surpass the performance of ZeroPur again, and our method will further optimize these DMs to achieve state-of-the-art results. At the same time, as ICLR (https://iclr.cc/Conferences/2025/ReviewerGuide) claims that: A lack of state-of-the-art results does not by itself constitute grounds for rejection. Submissions bring value to the ICLR community when they convincingly demonstrate new, relevant, impactful knowledge.

[7] Wang et al. Better Diffusion Models Further Improve Adversarial Training. ICML, 2023.

Q4: Is there a way to make the auxiliary classifier better defend against unseen attacks?

A4: As described in A2, the generalization ability to defend against unseen attacks primarily comes from random transforms, not the auxiliary network.

Q5: How does the influence of $s$? Could we use the different weights for the MSE and the logit?

A5: We have conducted this experiment in Figure 3c. When we use different weights, we can adjust the balance between standard accuracy and robust accuracy.

We greatly appreciate the comments of the Reviewer bRNc. Below are our responses to the questions raised.

Q1: The illustrations are not clear and there is no experimental support.

A1: For Figure 1, if only 70 steps are used, the modifications to the input image are minimal, and there is no significant difference in the output purified images. Therefore, in order to make a clearer comparison between our method and existing work, we set a larger step in this conceptual diagram. For Figure 2, we respectfully point out that we have extensively described it in Section 3.1 and provided the experiments to support this idea in Tables 4, and 6.

Q2: There is no theoretical analysis to support the proposed method.

A2: We respectfully point out that according to the reviewer guidelines provided by ICLR (https://iclr.cc/Conferences/2025/ReviewerGuide), the requirement of 'Does the paper support the claims?' is that 'This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.' Given our paper is not a theoretically oriented paper, we believe empirical evaluation is acceptable, and lack of theoretical analysis cannot be the reason for rejecting the paper.

Q3: Lack of innovation and contribution, which do not bring a new perspective on diffusion-based purification.

A3: Focusing on the community surrounding this paper, which aims to improve the robustness of the pre-trained generative model for adversarial purification, our method (AGDM) is highly efficient and completely different from the existing DM-based AP methods. In the original manuscripts, we have extensively discussed the paper [1] you provided, Wang et al. [1] proposed a training-free conditional DM that may result in preserving perturbations. To address this issue, we introduced a trainable guidance, which is completely a new perspective compared with [1]. In summary, 1. We propose a new robust guidance to optimize the reverse process to generate robust purified examples. 2. We are the first to consider the accuracy-robustness trade-off in pre-trained DM-based AP, which might be a significant contribution to advance the development of this field. 3. We conduct the experiments on two most important evaluation methods and recommend that we should consider the worst-case in evaluation of DM-based AP.
[1] Wang et al. Guided Diffusion Model for Adversarial Purification. Arxiv, 2022.

Q4: The results are not convincing. I recommend all results in this work be reported following the settings in [2].

A4: We strongly disagree with your claim that our results are not convincing, and your view that all results should follow the settings in [2].

1. Chen et al. [2] claim that the robustness is over-estimated when evaluating the diffusion-based model on AutoAttack, and share the same idea with Lee & Kim [3], using PGD+EOT to evaluate the diffusion classifier. We have cited these papers [2, 3] and conducted the experiments on both PGD+EOT and AutoAttack, using the worst-case robust accuracy as a more reliable evaluation.
2. Furthermore, Chen et al. [2] focus on classifier architecture, and we focus on adversarial purification defense methods. Our paper and [2] address completely different tasks of robustness, whereas the paper [3] that we follow aligns exactly with our task. Although many evaluation methods are universal, it is unreasonable to insist that we should not follow an AP evaluation paper [3] and forcefully recommend that we follow [2].
3. As you mentioned, evaluating AP methods only using AutoAttack would lead to overestimating the robustness, however, evaluating AT methods using PGD+EOT would also lead to overestimating, as shown in Figure 3. Therefore, we conduct the experiments using PGD+EOT and AutoAttack, regarding the worst-case robust accuracy as the metric for assessing robustness of methods.
   [2] Chen et al. Robust Classification via a Single Diffusion Model. ICML 2024.
   [3] Lee and Kim. Robust Evaluation of Diffusion-based Adversarial Purification. ICCV, 2023.

Q5: The chaotic formulations. Please follow the symbolics in DDPM or Score-SDE. For example, $**x**$ instead of $x$ should be used to represent an image.

A5: We strongly disagree with this weakness. For an independent paper, we need to ensure that within this paper, all symbols are self-contained, not necessarily the same as those in another paper. Also, if I maintain the same symbolic notation as in DDPM, there may be another reviewer who says: Please follow the symbols in DM-based AP, using $x$ instead of $**x**$. We believe this weakness is baseless.

We greatly appreciate the comments of the Reviewer QfvL. Below are our responses to the questions raised.

Q1: The notations and interpretations in Section 3.2 can be confusing. $p_\phi(x' \mid x_t)$ should be written as something like $p_\phi(s(x') \mid x_t)$.

A1: The parametric form of $p_\phi(x' \mid x_t)$ used in our model is explicitly defined in Eq. (2). Line 212-213 and Eq. (1) are general descriptions about the guided purification.

Q2: In Lines 277-278, it is stated that the auxiliary network is not required to be a robust classifier but to have adversarially robust representations. It seems that a non-classification model may not suffice for the guidance of AGDM.

A2: Thanks for the sharp insights and let us explain further. With the development of AI, various non-classification tasks have emerged and more research has begun to discuss whether only studies on robustness in classification tasks are limited. We believe that AP is a promising technology that is not limited to classification. Here, we want to clarify that the auxiliary network is also not limited to classification. Of course, if used for classification tasks, a classifier-based model is required; for other tasks, different models are required. The constant factor is that AP needs to provide robust representations or data. But we agree with you that this sentence may be an over-claim in this paper, and we will revise it accordingly.

Q3: Is the architecture of the auxiliary network the same as the classifier?

A3: The architecture of the auxiliary network is not required to be consistent with the classifier. All results use the WRN28-10 architecture to train the auxiliary network. In the paper, we evaluate it on various classifiers (ResNet-50, WRN28-10, WRN70-16), and all achieve good performance.

Q4: What are the number of PGD and EOT steps?

A4: The EOT iteration is 20 and PGD step is 200, the same settings in [1,2].
[1] Nie et al. Diffusion Models for Adversarial Purification. ICML, 2022.
[2] Lee and Kim. Robust Evaluation of Diffusion-based Adversarial Purification. ICCV, 2023.

Q5. How does the performance of different architectures as the auxiliary neural network?

A5: We tried the ResNet-18 and WideResNet-28-10 architectures in the early stages of experimentation. WRN-28-10 performed better, but there was no significant difference compared to ResNet-18. However, this is not the main goal that the paper aims to solve, so we did not conduct an in-depth study of these experiments. We will consider adding this part of the experiment to the supplementary materials.

Q6: The proposed method may be questionable. Both the training and the inference are time-consuming. There is also a lack of evidence for the transferability for different architectures or tasks.

A6:
6.1: Focusing on the community surrounding this paper, which aims to improve the robustness of the pre-trained generative model for adversarial purification, our method (AGDM) is highly efficient compared to existing defense methods.
The additional training is for the auxiliary neural network (ANN), which only consumes a small cost. Within one A5000 GPU, training the ANN on CIFAR-10 takes only ~3.17 hours, which is an entirely acceptable training cost. Compared to pre-trained diffusion-based AP, we indeed increase the cost of training since the fine-tuning cost for the former is 0. However, as Lin et al. [3] proposed AToP and have pointed out that additional training is necessary to improve the robustness of the current pre-trained generator-based AP. Furthermore, due to the significant training costs, they claim that AToP cannot work on the DM-based AP. In contrast, we only need to train a small neural network, which significantly reduces the training costs compared to AToP. Moreover, it is worth noting that while our method significantly improves performance, it has almost the same inference time as the vanilla diffusion-based AP method, as shown in the table below.

[3] Lin et al. Adversarial Training on Purification (AToP): Advancing Both Robustness and Generalization. ICLR, 2024.

6.2: We respectfully point out that we have conducted extensive experiments on multiple tasks (CIFAR-10, CIFAR-100, ImageNet) across various attacks (PGD, EOT, AutoAttack and StAdv), architectures(ResNet-50, WRN28-10, WRN70-16), norms($l_2$, $l_\infty$, non-$l_p$) to demonstrate the strong generalization ability of our method.

Q7: In Lines 209-210, why can we assume that $y$ and $x'$ are conditionally independent given $x_t$?

A7: Since $x_t$ contains information about $y$ and $x'$, we assume they are conditionally independent given $x_t$.

We greatly appreciate the comments of the Reviewer MYy2. Below are our responses to the questions raised.

Q1: The proposed method may have very low efficiency. Both the training and the inference are time-consuming.

A1: Focusing on the community surrounding this paper, which aims to improve the robustness of the pre-trained generative model for adversarial purification, our method (AGDM) is highly efficient compared to existing defense methods.
The additional training is for the auxiliary neural network (ANN), which only consumes a small cost. Within one A5000 GPU, training the ANN on CIFAR-10 takes only ~3.17 hours, which is an entirely acceptable training cost. Compared to pre-trained diffusion-based AP, we indeed increase the cost of training since the fine-tuning cost for the former is 0. However, as Lin et al. [1] proposed AToP and have pointed out that additional training is necessary to improve the robustness of the current pre-trained generator-based AP. Furthermore, due to the significant training costs, they claim that AToP cannot work on the DM-based AP. In contrast, we only need to train a small neural network, which significantly reduces the training costs compared to AToP. Moreover, it is worth noting that while our method significantly improves performance, it has almost the same inference time as the vanilla diffusion-based AP method, as shown in the table below.

[1] Lin et al. Adversarial Training on Purification (AToP): Advancing Both Robustness and Generalization. ICLR, 2024.

Q2: The paper did not report the training time and the inference time.

A2: We conduct experiments on training and testing times in the above table. In summary, compared to existing methods, our method can save a significant amount of training costs and has similar inference costs.

Q3: What is the seed during training? What are the iteration numbers for PGD+EOT and AutoAttack? What is the object of the attack?

A3: All experiments in this paper keep the same settings in [2,3]. The code will be available upon acceptance to clearly show details.
3.1: The seed is 777, a random lucky number. In the ablation experiments (Table 4), we used the same seed to ensure reliability.
3.2: The EOT iteration is 20, the PGD step is 200, and the AutoAttack version is rand, the same settings in [2,3].
3.3: All experiments are conducted on attacking the whole process, i.e., Attack{classifier(AGDM(x))}.

[2] Nie et al. Diffusion Models for Adversarial Purification. ICML, 2022.
[3] Lee and Kim. Robust Evaluation of Diffusion-based Adversarial Purification. ICCV, 2023.

Q4: The paper lacks ablation studies for the auxiliary network. How does the performance of different architectures as the auxiliary neural network?

A4:
4.1: We have conducted ablation experiments in Table 4.
4.2: We tried the ResNet-18 and WideResNet-28-10 architectures. WRN-28-10 performed better, but there was no significant difference compared to ResNet-18. In the paper, we use WRN-28-10 architecture for all experiments.

Q5: What's the motivation for comparing DM-based AP methods with AT methods?

A5: Our ultimate goal is to create a robust system to achieve better robust accuracy, not just to optimize the AP issue solely. The existing studies have shown that AT and AP have their unique advantages, none of them has an absolute advantage across various settings. Therefore, our method represents a high-level integration of AT and AP and it is necessary to compare it with both AT and AP methods.

Q6: What do you think is the most fair way to do so?

A6: It's a good question that we also mentioned in the paper. However, this paper does not delve into the evaluation method, so we cannot provide a definitive answer, we can only offer a preliminary idea is that: we should consider the worst-case results of the robust accuracy across PGD+EOT and AutoAttack as shown in Figure 3.