Sample-specific Noise Injection for Diffusion-based Adversarial Purification

In the end, we want to thank you for providing this valuable feedback to us, which is always important to hear opinions from other experts. If you feel there are still unclear points regarding our paper, please discuss with us in the author-reviewer discussion phase. If you feel that your concerns are well-addressed, we do hope for an updated rating and new comments (if possible). Thanks again!

Thank you so much for your comments! It is our pleasure that the novelty of our method and the overall clarity can be recognized. Your thorough review and comments are very important to the improvement of our work! Please find our replies below.

W1. Presentation: The presentation of this work should be improved, especially for the visual presentation. For example, Figure 1 would be clearer to deliver the observations using the ImageNet dataset, because it has higher-resolution images which allow the audience to distinguish the objects.

Reply: Thanks for your suggestion! We agree with you that ImageNet-1K images are of higher resolution, which could make objects more distinguishable. However, the high resolution of ImageNet images, combined with the relatively small magnitude of adversarial noise, results in less visible differences between purified samples at different $t^*$ values compared to CIFAR-10. In our manuscript, Figure 1b can clearly demonstrate how the dog's semantics are purified into a frog's semantics, finally leading to misclassification. The aim of Figure 1 is to showcase such visible semantic shifts and highlight the motivation behind our method. However, unfortunately, using ImageNet-1K will make the semantic shifts less visible.

W2. Quality: I would highly recommend the authors to have a sufficient check on their lemmas and propositions, especially the proofs. Based on the coherent and flawless proofs, the lemmas and the propositions would be more convincing and solid. For example, the plus or minus sign in the proof of Proposition 1 when they apply the Cauchy-Schwarz inequality. More examples can be seen in Questions.

Reply: Thank you for your thorough review and for highlighting these concerns. We have carefully revised the proofs of the Lemma and Proposition in our paper. All of the updated content is highlighted in blue. Below we summarize the key fixes to the specific examples as you mentioned:

Cauchy-Schwarz inequality

The change from the minus to the plus sign, i.e., from

to

comes as the result of applying triangle inequality:

The derivation is fully detailed in the revised manuscript accordingly.

Q1. In terms of the significance and the novelty of this paper, could the authors compare the novelty of this work with Diffusion Models Demand Contrastive Guidance for Adversarial Purification to Advance [1] in ICML 2024 in terms of the t^* selection? It would be better to present the difference to highlight the contributions in this submitted paper. [1] Bai, M., Huang, W., Li, T., Wang, A., Gao, J., Caiafa, C.F. & Zhao, Q.. (2024). Diffusion Models Demand Contrastive Guidance for Adversarial Purification to Advance, Proceedings of the 41st International Conference on Machine Learning.

Reply: Thank you for bringing this interesting paper to our attention! We summarize some key differences between [1] and our work here:

• [1] aims to improve the performance of diffusion-based purification methods by guiding diffusion models with contrastive loss through the reverse process. In terms of the $t^*$ selection, [1] uses a constant $t^*$ for all samples.
• On the other hand, our method focuses on sample-specific reweighting on timestep $t^*$ (i.e., different sample may have different $t^*$), leading to different amounts of Gaussian noise being injected into the samples in the forward process.

For more detailed discussions, we will add a paragraph in the updated version of our manuscript!

[1] Diffusion Models Demand Contrastive Guidance for Adversarial Purification to Advance, ICML 2024.

Q2. For Lemma 3, there is a minor question on in the proof: it seems that given $K > 0, (1 - \epsilon - K\epsilon)||\mathbf{x}|| \geq (1 - 2\epsilon)||\mathbf{x}||$ may not hold. Would the authors please elaborate on it?

Reply: We are very grateful for pointing this out. This was indeed a flaw. The condition that $\epsilon$ should satisfy has been updated to finding the discriminant of the desired inequality $||s_t(\mathbf{x})|| \geq C || \mathbf{x} ||$. We have included the fix in the revision. Please kindly refer to Appendix A of the newly uploaded PDF.

Q3. For Proposition 1, could the authors explain why the derivative of $\bar{\alpha}$ is $\frac{\partial \bar{\alpha}_t}{\partial t} = -\beta_t \bar{\alpha}_t$, given $\bar{\alpha}_t = \prod^t(1-\beta_t)?$

Reply: Thank you for your question. We expanded the details of the approximations and simplifications used in deriving this derivative as follows:

Given that $\bar{\alpha}\_t = \prod\_{i=1}^{t} (1 - \beta\_i)$, we first consider a continuous approximation of $t$, where the product is approximated as an exponential of the integral

and apply first-order Taylor approximation to $\beta\_t$ (valid under small $\beta\_t$ values in common diffusion model practices), i.e., $\log( 1 - \beta\_i) \approx - \beta\_i$, leading to the approximated $\bar{a}\_t \approx \exp \left( - \int\_0^t \beta\_i \mathrm{d} i \right)$. This way we compute the derivative of $\bar{a}_t$ as

This expanded derivation, along with subsequent details, has been incorporated into the revised manuscript for completeness. To keep the response concise and targeted, we do not include them in this message.

We deeply appreciate your careful reading and insightful comments, which have allowed us to improve the rigor and clarity of our submission. We hope this response and the revised manuscript address your concerns.

Q4. BPDA+EOT as a white-box attack has been shown to achieve lower attack success rate than those utilizing direct gradients of the defense process when evaluating diffusion-based purification methods [2]. It would be better to elaborate the reason why BPDA+EOT is chosen to evaluate the proposed diffusion-based purification method. [2] Minjong Lee and Dongwoo Kim. Robust evaluation of diffusion-based adversarial purification. In ICCV, 2023.

Reply: Thanks for your insightful comments! We would like to point out that our experiments follow the settings in [1], and our main results are presented under the evaluations of adaptive white-box PGD+EOT. The main reason for using BPDA+EOT is to make our experiments more comprehensive and to ensure diversity in the attacks (i.e., evaluate the robustness of our proposed framework under different attack scenarios). Specifically, BPDA+EOT and PGD+EOT attacks differ fundamentally in their methodologies. BPDA uses an identity function to estimate gradients, which can significantly increase the evaluation time for DBP methods. In contrast, PGD+EOT computes full backpropagation gradients, which can result in memory exhaustion due to high computational demands.

[1] Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV 2023.

Dear Reviewer iJtf,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer iJtf,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer iJtf,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer iJtf,

Many thanks for your valuable comments again!

Since the current recommendation is borderline instead of "a good paper", we are not sure if there is anything we can do to make our paper even better? We can further strengthen our paper based on your new comments until our paper is good to be accepted. ^^

We also want to share with you the current situation regarding the other reviews. Unfortunately, Reviewer mLed gave a rejection recommendation despite acknowledging that we addressed all the concerns.

This has placed our submission in a particularly challenging position (i.e., the averaged rating is around borderline). We would be deeply grateful for your consideration of further raising your confidence or score. Your support would play a crucial role in ensuring our paper can be fairly evaluated in this rebuttal.

Looking forward to hearing back from you.

Best regards,

Authors of Submission 10115

In the end, we want to thank you for providing this valuable feedback to us, which is always important to hear opinions from other experts. If you feel there are still unclear points regarding our paper, please discuss with us in the author-reviewer discussion phase. If you feel that your concerns are well-addressed, we do hope for an updated rating and new comments (if possible). Thanks again!

Thank you so much for your comments! It is our pleasure that the theoretical justification and the experiment results can be recognized. Your thorough review and comments are very important to the improvement of our work! Please find our replies below.

Q1. How to adapt bias in the reweighting function in different datasets and network?

Reply:

• Justification: The design principle of the reweighting function is to scale the timestep $t^*$ based on the score information. As we have observed that clean data often has a much lower norm value than adversarial one, we believe that incorporating a linear function or a sigmoid function would reweight the timestep properly. For realizations, we extract 5,000 validation clean examples from the training data (denoted as $\mathbf{x}_v$) and we use $\left\|\text{EPS}(\mathbf{x}_v)\right\|$ as a reference to indicate the approximate EPS norm values of clean data, which can help us reweight $t^*$. Thus, based on the EPS norm information, the reweighting function assigns adaptive timestep accordingly.
• How to select bias term b: When we select the bias term, we first split a validation set from each dataset and record the performance in above tables. We then choose the value based on the validation performance, then we use the term in our test data evaluations.

We apologize that our manuscript lacks of ablation study of the bias term. We present the result below. The bias term has the same magnitude with the timestep $t^*$. Also, there is a small mistake in Equation 9, the correct formula should be: $f_\sigma(\left\|\text{EPS}(\mathbf{x})\right\|,~t^*) = \frac{t^* + b}{1 + \exp\{-(\left\|\text{EPS}(\mathbf{x})\right\|-\mu)/\tau\}}$

Table 1: Ablation study on the bias term $b$. We report clean and robust accuracy (%) against white-box PGD+EOT adaptive attacks ($\ell_\infty, \epsilon=8/255$) on CIFAR-10. The classifier used is WRN-28-10.

Table 2: Ablation study on the bias term $b$. We report clean and robust accuracy (%) against white-box PGD+EOT adaptive attacks ($\ell_\infty, \epsilon=8/255$) on CIFAR-10. The classifier used is WRN-70-16.

Table 3: Ablation study on the reweighting function parameter $b$. We report clean and robust accuracy (%) against white-box PGD+EOT adaptive attacks ($\ell_\infty, \epsilon=4/255$) on ImageNet-1K. The classifier used is ResNet-50.

From the experimental results, we can derive the following conclusions: the selection of bias term does not impact the performance of our framework under CIFAR-10 and ImageNet-1K. Note that when bias increases, there is a general observation that the clean accuracy drops and the robust accuracy increases. This perfectly aligns with the understanding of optimal timestep $t^*$ selection in the previous DBP work, where large timestep would lead to drop of both clean and robust accuracy and small timestep cannot remove the adversarial perturbation effectively (see Figure 1 in [1]).

[1] Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV 2023.

Dear Reviewer mLed,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer mLed,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer mLed,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Thank you so much for your comments! Your thorough review and comments are very important to the improvement of our work! Please find our replies below.

Q1. How is the standard accuracy calculated?

Reply: Thanks for your queston! Standard accuracy refers to the classification accuracy of a model on clean (non-adversarial) test samples after undergoing the purification process. In our work, when a natural image is input into the defense system, the classifier's accuracy is referred to as the standard accuracy. The error margin following the accuracy values in each table is calculated based on testing with three different random seeds. Our experimental results demonstrate that DBP methods, through our proposed framework, achieve a better trade-off between standard accuracy and robust accuracy.

Q2. Are the clean samples also purified before applying the pre-trained classifier network?

Reply: Yes. This is mainly because we cannot assume what type of test data we have. In practice, the input data could either be adversarial examples or clean examples. Therefore, clean examples will also be purified before applying the pre-trained classifier network.

Q3. What is the value of t within s_\theta in eq (5)?

Reply: Thanks for your question! This $t$ is a noise level specified in [1], $t \in (0, T)$ in the following equation: $\text{EPS}(\mathbf{x}) = \mathbb{E} _{t \sim \mathcal{U}(0, T)} \nabla _{\mathbf{x}} \log p _t(\mathbf{x}).$ We use the same settings as [1] specified for both CIFAR-10 and ImageNet-1K, where $t=20$ for CIFAR-10 and $t=50$ for ImageNet-1K.

[1] Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score, ICML 2023.

Q4. It is a bit confusing to me how the theoretical results in sec 3 support the claimed relationship between score norm and the optimal noise level. It seems that Lemma 3 and Prop 1 are about the score norm of pt(x) where pt is the marginal distribution of xt, generated by injecting sequential Gaussian noise to clean data x0. In practice, the data may be attacked in different ways and contained by non-Gaussian noise. I think it would be more beneficial to present the theoretical insights for the score norm evaluated on contaminated data under certain attacks (maybe measured by the contamination level ϵ), and how this norm depends on the contamination level ϵ, thus showing that it will be significantly larger than the score norm of clean data.

Reply: Thanks for your comment! However, there might be some misunderstanding. In Lemma 3 and Prop1, injecting Gaussian noise refers to the forward process of diffusion models. This process is independent of how the adversarial examples are generated. Lemma 3 and Prop 1 aim to motivate that the score norm can serve as a quantitative criterion for estimating the optimal noise level $t^*$ specific to each sample. Empirically, we demonstrate that the cleaner (i.e., lower $\epsilon$) a sample is, the smaller its score norm tends to be, which establishes a connection between the score norm and the maximum perturbation budget $\epsilon$. For more details, please kindly check Figure 4 in Appendix B in our manuscript.

In the end, we want to thank you for providing this valuable feedback to us, which is always important to hear opinions from other experts. If you feel there are still unclear points regarding our paper, please discuss with us in the author-reviewer discussion phase. If you feel that your concerns are well-addressed, we do hope for an updated rating and new comments (if possible). Thanks again!

Dear Reviewer LRdc,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer LRdc,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer LRdc,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer LRdc,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer LRdc,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Thank you so much for your comments! It is our pleasure that the motivation of our paper can be recognized. Your thorough review and comments are very important to the improvement of our work! Please find our replies below.

In all, I feel the main insights of the paper are based on some misunderstanding of the current diffusion models. Also, a lot of the design choices feel questionable, and thus heuristic to me. In the end, I feel unconvinced by the paper's argument and proposal. However, I am happy to be corrected.

Reply: Thanks for your comments! First of all, although our work uses diffusion models, the problem setting in our work differs significantly from the traditional generative tasks typically associated with diffusion models. Meanwhile, we realize that our manuscript lacks details of diffusion-based adversarial purification and background knowledge of adversarial perturbations, which might lead to potential misunderstanding of our work.

Therefore, we would like to clarify the problem setting and the background knowledge of our work at the early stage of our response. To address this comprehensively, we have divided our response into several sections:

Section 1: the high-level objective of our work and some preliminaries

The high-level objective of our work is to remove adversarial perturbations from adversarial examples before feeding them into the underlying classifier for classification prediction.

Thus, the task here is image classification, instead of image generation. Diffusion model serves as a tool to remove adversarial perturbations. Then, we will introduce some preliminaries to better understand our work.

• Adversarial examples (AEs) are often crafted by adding imperceptible adversarial perturbations to clean examples (CEs), which can easily mislead a well-trained deep learning model to make wrong predictions. Usually, adversarial perturbations are obtained by maximizing cross-entropy loss and it is typically constrained within a specified norm (e.g., $\ell_p$​-norm) to ensure it remains imperceptible. Please note that adversarial perturbations are significantly different from Gaussian noise.
• Diffusion-based purification (DBP) has recently emerged as a prominent solution to remove the adversarial perturbations from AEs, leveraging the denoising capabilities of diffusion models. DiffPure [1] is a well-known framework in DBP, which uses diffusion models as purifiers to remove the adversarial perturbations. The philosophy behind this is that adding Gaussian noise to the AEs during the forward process of diffusion models can align the distribution of AEs towards a Gaussian distribution. Then, by iteratively denoising the noise-injected AEs at $t^*$ using a reverse diffusion process, the adversarial perturbations can be effectively 'washed out'.

[1] Diffusion Models for Adversarial Purification, ICML 2022.

Section 2: key differences between the role of diffusion models in DBP and image generations

Here, we summarize some key differences between the role of diffusion models in DBP and image generations

In the end, we want to thank you for providing this valuable feedback to us, which is always important to hear opinions from other experts. If you feel there are still unclear points regarding our paper, please discuss with us in the author-reviewer discussion phase. If you feel that your concerns are well-addressed, we do hope for an updated rating and new comments (if possible). Thanks again!

Q1. The sub-caption of Figure 1 can be simplified (shared)

Reply: Thanks for your suggestion! We will simplify the sub-caption of Figure 1 in the revised version of our manuscript!

Q2. Judging from Equation 7, are you suggesting to evaluate the pre-trained diffusion model on many ts? If so, this requires many network evaluations, right?

Reply: Thanks for your question! Indeed, this process does not require many network evaluations. With a reference to Eq.7 in our manuscript: $\text{EPS}(\mathbf{x}) = \mathbb{E} _{t \sim \mathcal{U}(0, T)} \nabla _{\mathbf{x}} \log p _t(\mathbf{x}).$ We have previously discussed how to estimate the score function $\nabla _{\mathbf{x}} \log p _t(\mathbf{x})$ through a score network $s _\theta(\mathbf{x _t}, t)$. Here, we first select a range $(0, T)$, and then use each perturbed data $\mathbf{x _t}$, $t \in (0, T)$ to input to the score network and it directly returns the corresponding score of $\mathbf{x _t}$. Hence, the EPS is computed by the expectation over these scores. This does not require many network evaluations. For CIFAR-10, we use $T = 20$ and for ImageNet, we use $T = 50$, which follows the same setting mentioned in [1].

[1] Detecting Adversarial Data by Probing Multiple Perturbations Using Expected Perturbation Score, ICML 2023.

Q3. On Line 344, I think you mean "the key idea is to scale the coefficient" right? Instead of scaling |EPS(x)|.

Reply: Thanks for your question and sorry for the confusion! There might be a potential misunderstanding here. We believe scaling $∥\text{EPS}(\mathbf{x})∥$ is more appropriate here. Specifically, the term $\frac{\left\|\text{EPS}(\mathbf{x})\right\| - \xi _{\min}}{\xi _{\max} - \xi _{\min}}$ is explicitly designed to normalize $∥\text{EPS}(\mathbf{x})∥$ to a fixed range (typically 0-1) based on the minimum and maximum $∥\text{EPS}(\mathbf{x})∥$ values. Therefore, the goal here is to scale $∥\text{EPS}(\mathbf{x})∥$ so that the coefficient of $t^*$ (i.e., $\frac{\left\|\text{EPS}(\mathbf{x})\right\| - \xi _{\min}}{\xi _{\max} - \xi _{\min}}$) is within the range of 0-1. To avoid confusion, we will clarify this sentence in the revised version of our manuscript!

Q4. The time overhead reported in Section 5.5 is not very meaningful. Some relative measurements are more desired.

Reply: Thanks for your comment! We will clarify the reasons why we report the inference time in our paper. DBP methods are inference-time adversarial defense methods (i.e., this kind of approach does not require any training process as it uses pre-trained diffusion models and pre-trained classifiers). Then, the consequence is that the inference time for DBP methods is very slow, which is a general limitation in this field. Given this limitation, our method further adds an additional module (i.e., the reweighting module) into the DBP framework, which will inevitably increase the inference time. Then, a natural question is: ''whether the increase in the inference time is affordable and acceptable given the improvements in the classification performance?" The answer to this question is affirmative. Compared to the DBP baseline methods, this reweighting process is relatively lightweight, ensuring that SSNI is computationally feasible and can be applied in practice with minimal overhead.

W3. The two realizations of the reweighting function have no justification at all. Also, how is the bias term b selected?

Reply:

• Justification: The design principle of the reweighting function is to scale the timestep $t^*$ based on the score information. As we have observed that clean data often has a much lower norm value than adversarial one, we believe that incorporating a linear function or a sigmoid function would reweight the timestep properly. For realizations, we extract 5,000 validation clean examples from the training data (denoted as $\mathbf{x}_v$) and we use $\left\|\text{EPS}(\mathbf{x}_v)\right\|$ as a reference to indicate the approximate EPS norm values of clean data, which can help us reweight $t^*$. Thus, based on the EPS norm information, the reweighting function assigns adaptive timestep accordingly.
• How to select bias term b: When we select the bias term, we first split a validation set from each dataset and record the performance in above tables. We then choose the value based on the validation performance, then we use the term in our test data evaluations.

We apologize that our manuscript lacks of ablation study of the bias term. We present the result below. The bias term has the same magnitude with the timestep $t^*$. Also, there is a small mistake in Equation 9, the correct formula should be: $f_\sigma(\left\|\text{EPS}(\mathbf{x})\right\|,~t^*) = \frac{t^* + b}{1 + \exp\{-(\left\|\text{EPS}(\mathbf{x})\right\|-\mu)/\tau\}}$

Table 1: Ablation study on the bias term $b$. We report clean and robust accuracy (%) against white-box PGD+EOT adaptive attacks ($\ell_\infty, \epsilon=8/255$) on CIFAR-10. The classifier used is WRN-28-10.

Table 2: Ablation study on the bias term $b$. We report clean and robust accuracy (%) against white-box PGD+EOT adaptive attacks ($\ell_\infty, \epsilon=8/255$) on CIFAR-10. The classifier used is WRN-70-16.

Table 3: Ablation study on the reweighting function parameter $b$. We report clean and robust accuracy (%) against white-box PGD+EOT adaptive attacks ($\ell_\infty, \epsilon=4/255$) on ImageNet-1K. The classifier used is ResNet-50.

From the experimental results, we can derive the following conclusions: the selection of bias term does not impact the performance of our framework under CIFAR-10 and ImageNet-1K. Note that when bias increases, there is a general observation that the clean accuracy drops and the robust accuracy increases. This perfectly aligns with the understanding of optimal timestep $t^*$ selection in the previous DBP work, where large timestep would lead to drop of both clean and robust accuracy and small timestep cannot remove the adversarial perturbation effectively (see Figure 1 in [1]).

[1] Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV 2023.

W1.1 I would like to point out that diffusion models are never trained to model the score directly ...Typically, the training objective of diffusion models is parameterized so that the model's prediction has a constant norm across noise levels (for the ease of training) ... Essentially, in terms of diffusion model training, they are not trained to differentiate the norm of the scores across noise levels. This property holds for the two mode ls the authors used in experiments. Thus, I am wondering what exactly the authors observe here in Figure 4 in Appendix B.

Reply: Thanks for your comments! We agree with what you describe about the training objective of diffusion models. However, in our paper, the score $\nabla_\mathbf{x} \log p_t(\mathbf{x})$ can be estimated through the score network $s_\theta(\mathbf{x}_t, t)$. that has a training objective of score matching [1], which is indeed different from what you mentioned in the question. The training objective minimizes:

Section 3: research gap in DBP and our motivation

To use diffusion models to remove adversarial perturbations, a natural challenge in this field is how much Gaussian noise should be injected into the input sample during the forward process of diffusion models [1], which is a trade-off problem:

• If the Gaussian noise is not enough, then adversarial perturbations cannot be fully removed after the reverse process.
• If the Gaussian noise is too much, then the purified samples will lose their original semantic meanings (i.e., in this case, it is more like generating new images instead of recovering original images).

In practice, the amount of Gaussian noise injected to the samples is controlled by the timestep $t^*$, and existing DBP methods manually select a constant $t^*$ for all samples. However, in this paper, we find that using a sample-shared $t^*$ may overlook the fact that an optimal $t^*$ indeed could be different at sample-level. For example, some adversarial examples only require a small $t^*$ to be recovered. Therefore, a sample-shared $t^*$ will destroy the semantic meaning of these examples, affecting the classification performance. Meanwhile, some adversarial examples require a large $t^*$ to be recovered. Therefore, a sample-shared $t^*$ cannot effectively remove the adversarial noise.

[1] Diffusion Models for Adversarial Purification, ICML 2022.

Section 4: contributions of our work

• We empirically observe that using a sample-shared $t^*$ may overlook the fact that an optimal $t^*$ could indeed be different at sample-level. This could be used as a strong motivation to address the above-mentioned challenges.
• Based on this motivation, we further propose Sample-specific Score-aware Noise Injection (SSNI), a new framework that leverages score norms estimated by a pre-trained score network to adaptively adjust $t^*$ on a sample-specific basis.
• Extensive experiments show that SSNI can boost clean accuracy and robust accuracy simultaneously by a notable margin against well-designed adaptive white-box attacks.

In general, the success of SSNI takes root in the following aspects: (1) an optimal noise level $t^*$ for each sample indeed could be different, making SSNI a more effective approach to unleash the intrinsic strength of DBP methods; (2) existing DBP methods often inject excessive noise into clean samples, resulting in a degradation in clean accuracy. By contrast, SSNI injects less noise into clean samples, and thereby notably improving the clean accuracy. Meanwhile, SSNI can effectively handle adversarial samples by injecting sufficient noise on each sample; (3) SSNI is designed as a general framework instead of a specific method, allowing it to be seamlessly integrated with a variety of existing DBP methods.

W2.1 I am also confused by the logic that the further "away" the sample is from the "clean" distribution, the more noise we need to inject for purification. In particular, why are we adding a proportional amount of noise estimated in the sample again?

Reply: Thanks for your question and sorry for the confusion!

• Firstly, the "sample" in the sentence "the further away the sample is" refers to adversarial examples, instead of the Gaussian noise-injected samples after the forward process.
• Secondly, the "noise" in the sentence "the more noise we need to inject for purification" here refers to the Gaussian noise added in the forward diffusion process.

The logic/philosophy behind it is: Adding Gaussian noise to adversarial examples is widely recognized as an effective defense method in diffusion-based purification. In DiffPure, for instance, the theoretical foundation suggests that both adversarial and clean samples are aligned towards a Gaussian distribution when Gaussian noise is injected. This point is also discussed in a recent work [1]. The purpose of injecting Gaussian noise during the forward process is to effectively "cover" the adversarial perturbations, and thus they can be 'washed out' through the reverse process.

[1] Towards Understanding The Robustness of Diffusion-Based Purification: A Stochastic Perspective, arxiv 2024.

[2] Diffusion Models for Adversarial Purification, ICML 2022.

W2.2 For example, if the procedure estimates the model is at time t, Algorithm 1 essentially forwards the sample to t+t’ time right? I don't understand the logic behind this choice.

Reply: Thanks for your question! However, there might be some misunderstanding. Indeed, we do not introduce an additional $t'$. The detailed explanation of Algorithm 1 is as follows:

• Given an input $\mathbf{x}$ (e.g., it could be an adversarial example or clean example), we use a pretrained score network to calculate the score norm of $\mathbf{x}$.
• Then we feed the score norm into a reweighting function, which outputs $t(\mathbf{x})$. This $t(\mathbf{x})$ serves as the sample-specifc $t^*$s for purifying the sample $\mathbf{x}$.

Intuitively, we are making the optimal timestep $t^*$ adaptive for each sample based on its score norm.

W2.3 Also, now when you run the backward sampling process, do you start at t+t’ or t’ ?

Reply: Thanks for your question! Our proposed framework assigns an independent timestep $t(\mathbf{x})$ to each sample image (i.e., sample-specific $t^*$ for each sample). Therefore, each image starts the backward sampling process from a sample-specific $t^*$ that can be different from other images.

W2.4 Essentially, when you do the forward and backward, are you treating the test (start) sample x as a clean or a noisy one in terms of diffusion?

Reply: Thanks for your question! In terms of diffusion, every input begins as the initial image (i.e., $x_0$ at t = 0). Different from the image generation task, here the input is not random noise. Instead, $x_0$ here could either be a clean example or an adversarial example. This is because we cannot assume the type of test data we have beforehand. Also, it is important to note that the noise in adversarial examples (i.e., adversarial perturbations) is fundamentally different from the Gaussian noise used in the diffusion process.

Dear Reviewer 2MWZ,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer 2MWZ,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer 2MWZ,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Dear Reviewer 2MWZ,

We appreciate the time and effort that you have dedicated to reviewing our manuscript.

We have carefully addressed all your queries. Could you kindly spare a moment to review our responses?

Have our responses addressed your major concerns?

If there is anything unclear, we will address it further. We look forward to your feedback.

Best regards,

Authors of Submission 10115

Responses to Reviewer LRdc

1. Experiment Metric

• Concerns: How is the standard accuracy calculated?
• Responses: We explained that standard accuracy refers to performance under natural data evaluation and is critical for understanding the trade-off between clean accuracy and robustness.

2. Purification Process

• Concerns: Are the clean samples also purified before applying the pre-trained classifier network?
• Responses: We clarified and explained that test samples are treated uniformly $(t=0)$ regardless of whether they are adversarial or natural.

3. EPS Configuration

• Concerns: What is the value of $t$ within $s_\theta$ in eq (5)?
• Responses: We detailed hyperparamter settings for EPS computation in lines 332-335.

4. Connection between Prop 1 and Motivation

• Concerns: How does the theoretical result in sec 3 support the claimed relationship between score norm and the optimal noise level?
• Responses: We revise and reorganize Section 3 to clarify the connection between Prop 1, adversarial perturbation budget, and optimal denoise efforts (noise level). Prop 1 establishes a relationship between different samples' score norms and noise levels and motivates the adoption of score norms to reweight $t^*$ and align the purification process with the sample-specific perturbation budget, based on each sample's score norm. We have also lowered the claim of theoretical contribution of Prop 1 and highlighted the contribution as the proposed SSNI framework itself.

Responses to Reviewer mLed

1. Ablation Study

• Concerns: How to adapt bias in the reweighting function in different datasets and networks?
• Responses: We supplemented experiments to demonstrate the adaptability of the reweighting function across multiple classifiers (WRN-28-10, WRN-70-16 on CIFAR-10, and RN-50 on ImageNet-1K) in Appendix J.

Responses to Reviewer iJtf

1. Discussion of Relevant Literature

• Concerns: How does the study demonstrate novelty compared to the reference?
• Responses: We append the discussion and highlight the key difference between our study and the referenced study in Section 2.

2. Elaboration on Derivations:

• Concerns: Elaborate some intermediate steps of Lemma 3 and Proposition 1.
• Responses: We revise Appendix A to fully detail the intermediate steps in the proofs of Lemma 3 and Proposition 1. Specifically:
  • We update the condition under which Lemma 3 holds;
  • We clarify the use of key tools, e.g., triangle inequality in conjunction with the Cauchy-Schwarz inequality, and the continuous approximation and first-order Taylor expansion when deriving the derivative of $\bar{\alpha}_t$.

3. Attack Types

• Concerns: Why BPDA+EOT is chosen to evaluate the proposed diffusion-based purification method?
• Responses: We clarified that our main Experiment Results are evaluated under the Golden-standard of DBP methods, which is PGD+EOT white-box adaptive attack. Also, we would like to comprehensively evaluate our proposed framework under diverse attack types, where BPDA+EOT is significantly different from PGD+EOT attack in the fundamental principle.

We believe these responses have comprehensively addressed all reviewers' concerns, including follow-up questions raised by reviewer LRdc. The revision incorporates additional experiments, clarifications, and corrected derivations. The reviewer (mLed) acknowledged that we addressed all the concerns. The reviewers (2MWZ, iJtf) have not raised further issues and have indicated scores leaning toward acceptance.

Once again, we appreciate all reviewers' efforts and time, which have helped us to enrich the quality and clarity of our submission.