ADBM: Adversarial Diffusion Bridge Model for Reliable Adversarial Purification

Q2.2. Comparison with AToP

Since AToP is not a diffusion-based purification method, we did not include it in our initial submission. According to your suggestion, we have contacted the authors of AToP for the trained checkpoint of their work. Using their released code, we successfully replicated the results presented in their paper.

However, by a thorough review of their code implementation, we discovered that the used attacks in the original implmentation of AToP are, in fact, weak attacks. Their evaluation does not employ the full gradient of the entire adversarial perturbation process; instead, it relies solely on BPDA or generates adversarial samples on the original classifier. We further evaluated AToP's robustness under our reliable evaluatiion (linf, 8/255 on CIFAR-10, full gradient, EOT=20, steps=200) and regrettably found that their method loses robustness. Even $RT_2$ with GAN (the best implementation according to Table 8 of the AToP paper) achieved only 1.3% robust accuracy.

Regarding AToP's use of diffusion models, our preliminary findings indicate a 6.7% drop in robustness compared to DiffPure. This might be attributed to AToP's requirement for a diffusion model capable of handling masked images. But overall, this aligns with previous attempts to improve diffusion-based purification methods. As shown in Table 3, many works [2,3,4] tried to improve diffusion-based purification but also exhibited negative performance compared to the original DiffPure under our reliable evaluation. In contrast, ADBM is, to our best knowledge, the only method that provides actual improvements compared with DiffPure.

Q3. The assumption for the proofs is the DDIM sampler. What happens if we change the DDIM sampler to the original DDPM sampler?

Thanks for pointing out this. We would like to clarify that besides Theorem 1, Theorem 2 indicates that with infinite reverse timesteps, adversarial examples purified with ADBM are more likely to align with the clean data distribution than those with DiffPure, which is independent of the DDIM sampler.

Using the original DDPM sampler will incur significant inference costs proportional to the reverse steps. Our intention of mainly reporting the results using DDIM sampler is to investigate whether robustness can be maintained with minimal cost (i.e., one-step reverse) for both DiffPure and ADBM, which cannot be achieved by DDPM (see Table 2).

But as per your suggestion, we have also evaluated ADBM under the 100 reverse steps using the original DDPM sampler and the results are shown below (CIFAR-10, $l_\infty$ threat, EOT=20, steps=200):

We can see that ADBM also has better robustness over DiffPure. However, it is important to consider the practical implications of this approach. The increase in inference cost associated with 100 reverse steps gives significant practical constraints in real-world applications. In contrast, one of our contributions is demonstrating we can use the DDIM sampler to achieve similar robustness while significant reducing the inference cost by employing the DDIM sampler (line 101).

We hope you are satisfied with our response above. If you have any further comments, and we'd be glad to answer them in the discussion period.

[2] Jinyi Wang, et al. Guided diffusion model for adversarial purification. arXiv:2205.14969, 2022.

[3] Boya Zhang, et al. Purify++: Improving diffusion-purification with advanced diffusion models and control of randomness. arXiv:2310.18762, 2023

[4] Boya Zhang, et al. Enhancing adversarial robustness via score-based optimization. NeurIPS, 2024.

Thank you for the supportive review. We are encouraged by the appreciation of the comprehensive evaluation, the novelty and the detailed proof of this work. We have uploaded a revision of our paper to improve the writing (revised part is shown in blue, additional results will be added soon). Below we address the detailed comments, and hope that you may find our response satisfactory.

Q1. About the writing for this paper.

Thank you again for your careful review, which helps us imporve the quality of this paper. We have checked our formulations detailedly and made a revision of our draft (see the updated pdf). In addition, we have the following clarifications:

Q1.1 Some math symbols are confused, for example $\hat{x}_0$. In line 229 $\hat{x}_0$ is used to denote the final point of the reverse process, thus having $\hat{x}_0 = x_0$, but in line 879 $\hat{x}_0$ is changed to the prediction of $x_0$ for ${x}_t$, thus causing the confusion.

We apologize for the confusion. We believe that the confusion may come from the facts that 1) the output of reverse process should be the clean sample, i.e., $\hat{x}_0:={x}_0$, and 2) only one reverse step is used and thus the reversed output $\hat{x}_0$ is indeed the prediction of $x_0$ from $x_t^a$.

More specifically, $\hat{x}_{t:T \to 0}$ is denoted as the different states in the reverse Markov chain (line 226). In line 229, $\hat{x}_0$, the final point of the reverse process (the reverse Markov chain), is expected to directly predict $\hat{x}_0$, i.e. $\hat{x}_0:={x}_0$, so as to achieve the goal of purification. And in line 879, since we employ a one-step reverse process, it is appropriate to reuse the symbol of $\hat{x}_0$, the final point of the reverse process, to denote the prediction of $x_0$ from $x_t^a$.

The situation is as same as $x_t^d$. It is the final point of the forward process of ADBM, as both adversarial noise and Guassian noise are added. On the other hand, we purify them simultaneously in the reverse process, which begin with $x_t^d$. Given $\hat{x}_t := x_t^d = x_t^a-k_t\epsilon_a, k_0=1, k_T=0$ (line 231), we have $x_t^d=x_t^a$ when $t=T$, which is the end of the forward process and the start of the reverse process.

We have made a revision to make it clearer.

Q1.2 One-step DDIM is confusing since this paper includes the discrete and continuous VP-SDE.

Thanks for pointing out this. In the discrete case, ${\tau_0,\ldots,\tau_s}$ is a linearly increasing sub-sequence of $\{0,\ldots,T\}$, $\tau_0=1,\tau_s=T$. In the continuous case, $\{\tau_0,\ldots,\tau_s\}$ is a linearly increasing sequence of $[0,T] \sub [0.0,1.0]$, $\tau_0=1,0\leq\tau_s=T\leq1$. $T$ is determined in the forward process. We have made a revision to discuss the two cases respectively.

Q2. About experimental results

Q2.1. About the BPDA results.

Thanks for your valuable suggestion. We additionally evaluated DiffPure and ADBM on CIFAR-10 in the BPDA attack setting, and the results are shown below (200 iterations with 20 EoT steps):

We can see that 1) ADBM obtains improved robustness over DiffPure; 2) BPDA is a weak attack for diffusion-based purification compared with the evaluation using full gradient (Table 3). The second point has also been validated in RQ3 of [1]. We will add these results to the revised version.

[1] Minjong Lee and Dongwoo Kim. Robust evaluation of diffusion-based adversarial purification. ICCV, 2023

Dear Reviewer pfC6:

We sincerely thank you for your timely response and for recognizing the significant error. This is indeed another irrelevant anonymous file due to our negligence. We have updated the correct version of the manuscript, which includes revisions to improve the writing (the revised sections are highlighted in blue). Please recheck the revised version.

Once again, we appreciate your efforts and recognition of our work.

Sincerely,
Authors

We appreciate your recognition of our theoretical results and the effort devoted to checking this paper. We will add the additional results and clarifications to the revised version. Our responses to the weaknesses and questions are as follows:

Q1. The proposed method is only verified on datasets with small classification number, such as CIFAR10 and SVHN. The experiments on datasets with small sizes can not fully demonstrate the effects of the proposed strategy, since the applications in the real world will have large number of classification number. A more suitable dataset could be ImageNet with at least 1K class number.

Thanks for your question. On the one hand, we would like to clarify that beside CIFAR-10 and SVHN, we have evaluated ADBM on more datasets such as ImageNet-100 and Tiny-ImageNet, which have 100 and 200 classes respectively. The results in Table A5 and Table A6 show that ADBM still have good robustness on these cases with more classes (with at least 3% robustness improvements).

On the other hand, we apologize that a full comparison using ImageNet-1K with 1K classes is beyond our current capabilities. According to the report of [1], training a diffusion model with good generative ability on the ImageNet with 1K classes required 32xA100 (80G) training for 13 days, even on the resolution of $64 \times 64$. Although our ADBM only needs fine-tuning about 1/10 of the training steps of the original diffusion models, it is still a great burden to us. But as per your suggestion, we have tried our best to perform a preliminary experiments on ImageNet with 1K classes within the rebuttal period. Here we fine-tuned ADBM with just one epoch on ImageNet and used a ViT-B [2] as a classifier, and the results are shown in below:

These results show that even on complicated data distribution with 1000 classes, ADBM can obtain considerable improments over previous methods. And we believe that the gain will be more significant if more training resources can be provided (note that here we use just one epoch training).

In fact, according to recent advances of text-to-image diffusion models, diffusion models have quite good scalability even on the web-scale image dataset. And thus from this persepctive, the classes and the complicated data distribution could not be a main concern.

Q2. The experimental results are mainly built with the network of WRN, and the experimental results with more types of networks are not shown.

We use WRN because most of preivous works mainly used this network for evaluation and our intention is to perform a fair comparison. In addition, we would like to clarify that besides WRN, we have included the results on ViT (vision transformer) in Table A9 and the results on ResNet-50 in Table A8. On all these networks, together with WRN-70-16 and WRN-28-10, ADBM show better robustness than previous methods.

[1] https://github.com/NVlabs/edm

[2] https://pytorch.org/vision/stable/models/generated/torchvision.models.vit_b_16.html?highlight=vit+b#torchvision.models.vit_b_16

Q3. Compared with other methods, the accuracy on the adversarial examples are not improved a lot, while the accuracy on the clean examples are decreased clearly. The authors should analyze the factor which limits the accuracy on the clean examples.

First, we would like to clarify that compared with DiffPure and other AT methods, ADBM does not decrease the accuracy on clean examples. On CIFAR-10, the standard deviation of results are about $\pm 0.5$ and the difference between ADBM and DiffPure on clean images are also about $\pm 0.5$ (see Table 3). Besides, on Tiny-ImageNet (see Table A5) and ImageNet-100 (see Table A6), and the additional results on ImageNet (see our response to Q1), the accuracy on clean images even slightly increased.

Second, we respectfully disagree that adversarial robustness did not improve a lot. On the one hand, several attempts to improve DiffPure ( see Table 3) have resulted in negative effects compared to the original DiffPure, which means improve the diffusion-based purification is non-trivial. On the other hand, considering recent efforts to improve adversarial robustness, the 4.4% robustness gain of ADBM on CIFAR-10 is significant. For example, Kuang et al. [3] reported a 1.0% robustness gain under PGD on CIFAR-10 over the baseline method, Zhang et al. [4] achieved a 0.4% robustness gain under PGD on CIFAR-10, and Wu et al. [5] obtained a 1.0% robustness gain on CIFAR-10. These works [3, 4, 5] have all been published in recent ML conferences.

Q4. What is the performance on more datasets with more classification numbers?

Please see our response to Q1.

Q5. What is the performance of the proposed method with various networks?

Please see our response to Q2.

Q6. What is the performance of the proposed method towards different types of attack?

Thanks for this question. We have evaluated ADBM on three attacks with different threats in Table 3. In additon, we have evaluated ADBM on four attacks in Table 4 and Table A7 (configurations detailed in Section 5.3), including four unseen attacks: RayS, SPSA, Transfer, and Square attacks. Furthermore, we have conducted experiments on four additional unseen threats including patch-like attacks (PI-FGSM and NCF) and recent diffusion-based attacks (DiffAttack and DiffPGD), and the results are shown in Table A8 (configurations detailed in Appendix E.4).

Overall, we have evaluated ADBM on at least 11 different types of attacks. We can see that under all these attacks, ADBM demonstrates better performance and generalization ability than DiffPure and previous AT methods. We believe our evaluation is comprehensive.

If the reviewer agrees with our clarification above and kindly re-evaluates the value of our work, we would be very grateful. We are happy to address any further questions regarding our work.

[3] Kuang et al. Improving adversarial robustness via information bottleneck distillation. NeurIPS, 2023.

[4] Zhang et al. Improving Accuracy-robustness Trade-off via Pixel Reweighted Adversarial Training, ICML, 2024.

[5] Wu et al. Annealing Self-Distillation Rectification Improves Adversarial Training. ICLR, 2024.

After reading the response from the authors, the concerns about the dataset and network diversity have been solved. However, I still think the improvement on the robustness is not obvious. I will change my rating to marginally above the acceptance threshold.

Dear Reviewer H77B:

We sincerely thank you for the timely and positive feedback on our work. It is quite valuable for helping us strengthen this work. We believe that in many security-critical cases, ADBM, with favorable theoretical guarantees, good empirical improvements (+4.4% under reliable evaluation), and plug-and-play features, offers an alternative way to further enhance adversarial robustness.

We appreciate your recognition of our theoretical and empirical results and the effort devoted to checking this paper. We will add the additional results and clarifications to the revised version. Our responses to the weaknesses and questions are as follows:

Q1: Generalizability to unseen attacks like Square and StAdv attacks

Thanks for this question. Beyond different settings of white-box attacks in Table 3, we have evaluated ADBM on four unseen attacks in Table 4 and Table A7 (configurations detailed in Section 5.3), including four unseen attacks: RayS, SPSA, Transfer, and Square attacks. Furthermore, we have conducted experiments on four additional unseen threats including patch-like attacks (PI-FGSM and NCF) and recent diffusion-based attacks (DiffAttack and DiffPGD), and the results are shown in Table A8 (configurations detailed in Appendix E.4).

Here we additionally provide the results under StAdv according to your suggestion (see Table 4 for results under Square attack):

We can see that under all these unseen attacks, which cover various types adversarial attacks, ADBM demonstrates better performance and generalization ability than DiffPure and previous AT methods. We believe our evaluation on unseen attacks is comprehensive.

Q2: About Likelihood Maximization in RDC

Thank you for your suggestion. We have discussed the difference between RDC and ADBM in Appendix A.2 while ignoring the Likelihood Maximization (LM). Here we further clarify this:

We agree that LM is indeed a purification defense, which has the advantage that it does not require adversarial fine-tuning. But as shown in our response to Q1, ADBM can also handle various unseen attacks, with good generalizability.

Furthermore, compared with LM, ADBM has the following two advantages:

1. ADBM can perform purification with quite few purification steps, which requires quite fewer inference cost. Here we compared the performance of ADBM and LM under the same inference cost (purification step) against our attack evaluation (PGD200+EOT20, 8/255). We can see that ADBM can have good performance even with one purification step, while LM loses its robustness in this setting. ||||| |-|-|-|-| |Method \ Purification steps|1|2|3| |LM|13.1|25.0|26.6| |ADBM|45.7|47.7|47.7|

2. The evaluation of DiffPure has been extensively studied in recent years, establishing a comprehensive evaluation pipeline and backing it with theoretical guarantees [1, 2], such as certified robustness. In contrast, while LM has been evaluated with white-box attacks, it has not yet been fully supported by theoretical guarantees like certified robustness, which raises concerns about its vulnerability to stronger future attacks. For this reason, in our work, ADBM focuses on further enhancing DiffPure.

Overall, our ADBM constructs an elaborated diffusion bridge from the adversarial distribution to the clean distribution and has a well-defined mathematical structure, where both the forward and reverse processes are Gaussian. This approach fully inherits the desirable mathematical properties of DiffPure. ADBM requires only a few additional lines of code and one epoch of fine-tuning to directly improve DiffPure. We believe ADBM offers an easy-to-use, fine-tuning-based alignment method that improves robustness with affordable computational cost and implementation effort. We will add such discussion to the revised version.

Q3: About the property of ADBM and the selection of variables such as $\bar{\alpha}$.

Thank you for your suggestion. We would like to clarify that ADBM retains the property of converging to a standard Gaussian distribution as $t \to T$. ADBM is a new diffusion bridge model, in which the forward process remains identical to the standard diffusion process and therefore preserves this property. In the backward process, ADBM builds a Gaussian bridge from the diffused adversarial distribution to the clean distribution. ADBM modifies only the reverse process of diffusion models which has been proven to remain Gaussian with a slightly adjusted mean while the forward process remains unchanged. Therefore, ADBM maintains this convergence property of diffusion models. And thus we can reuse the scheduler ($\bar{\alpha}$) of DDPM without further adjustment.

If the reviewer agrees with our clarification above and kindly re-evaluates the value of our work, we would be very grateful. We are happy to address any further questions regarding our work.

[1] Carlini N, Tramer F, Dvijotham K D, et al. (Certified!!) Adversarial Robustness for Free! ICLR 2023.

[2] Xiao C, Chen Z, Jin K, et al. Densepure: Understanding diffusion models towards adversarial robustness. ICLR 2023.