One Stone, Two Birds: Enhancing Adversarial Defense Through the Lens of Distributional Discrepancy

1. Batch-wise Processing

In our humble opinion, the practicality of a method should be evaluated in the context of specific scenarios and application requirements, which means there is no absolute 'practical' or 'impractical' method. The key message we want to deliver here is: batch-wise evaluation is not impractical, but it will have some costs:

• Proposed solution: For user inference, single samples provided by the user can be dynamically stored in a queue. Once the queue accumulates enough samples to form a batch, our method can then process the batch collectively using the proposed approach.
• Costs for this solution: The main cost is waiting time to accumulate enough samples (e.g., 50). However, in high-throughput scenarios (e.g., Google's terminals), this delay is minimal (often <2 seconds). For applications with stricter latency requirements, the batch size can be dynamically adjusted based on the incoming data rate to minimize waiting time. For instance, if the system detects a lower data arrival rate, it can process smaller batches to ensure timely responses.
• Comparison with SOTA AP methods: Diffusion-based AP methods can handle single-sample inputs but suffer from slow inference speeds (e.g., DiffPure [1] takes ~4 seconds per CIFAR-10 image on an A100 GPU). In contrast, our method averages only ~0.003 seconds per image. Assuming there are 1000 images, DiffPure would take 4000 seconds to complete the inference, while our method only takes 3 seconds. Therefore, if the waiting time to form a batch is less than 3997 seconds, our method is more time-efficient than DiffPure. Thus, diffusion-based AP methods can hardly be applied to a system where data arrives quickly. Instead, our method can handle it, demonstrating that batch-wise evaluation is not impractical.

Overall, it is a trade-off problem: using our method for user inference can obtain high robustness, but the cost is to wait for batch processing. Based on the performance improvements our method obtains over the baseline methods and the fact that current SOTA AP methods are generally slow at inference, we believe the cost is feasible and acceptable.

[1] Diffusion Models for Adversarial Purification, ICML 2022.

2. Training and Inference Efficiency

We provide comparisons of training time with 3 representative AT-based methods [1][2][3] in Table 1. Notably, the current SOTA AT method [4] requires generating 50M synthetic images, making it extremely time-consuming. The point we want to highlight is: even compared to simpler AT methods, our method demonstrates significantly better efficiency. We also provide comparisons of inference time with 2 SOTA diffusion-based AP methods [5][6] in Table 2.

Table 1: Training time (hours: minutes: seconds) of different methods on CIFAR-10 with 2 x NVIDIA A100. The target model is RN-18.

Table 2: Inference time per image (seconds) of different methods on CIFAR-10 with 1 x NVIDIA A100. The target model is WRN-28-10.

[1] Towards Deep Learning Models Resistant to Adversarial Attacks, ICLR 2018.

[2] Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019.

[3] Improving Adversarial Robustness Requires Revisiting Misclassified Examples, ICLR 2020.

[4] Better Diffusion Models Further Improve Adversarial Training, ICML 2023.

[5] Diffusion Models for Adversarial Purification, ICML 2022.

[6] Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV 2023.

3. Adaptive Attack

According to [1], PGD+EOT is considered the strongest attack against diffusion-based AP methods, while AutoAttack is strongest against AT-based methods. Our proposed adaptive attacks (both PGD+EOT and AutoAttack) additionally target DDAD's detection mechanism (i.e., MMD-OPT), and adaptive PGD+EOT proves most effective in breaking DDAD. Moreover, Table 3 in our paper demonstrates DDAD achieves the best average performance against adaptive BPDA+EOT attack across various baselines. Finally, please kindly check the results against adaptive AutoAttack in Section 2 & 3 of Reviewer 3bTb's responses.

[1] Robust Evaluation of Diffusion-based Adversarial Purification, ICCV 2023.

4. Applicability of DDAD

Thank you for your concern! As specified in our problem setting, we primarily focus on robust classification, which is the standard setting adopted by most existing defense methods. In our humble opinion, even though classification is relatively straightforward, achieving fully robust model predictions remains challenging, indicating that robustness would likely be even harder to attain in more complex tasks. However, we agree that generalizing DDAD to other tasks/domains is an interesting direction and we leave it as future work.

1. Rendering Errors of LaTeX

Thank you for pointing out this unexpected rendering issue caused by LaTex! The entire sentence in line 112 is: '...the ground-truth labelling functions for the clean and adversarial domains are equal in our problem setting.' This statement is supported based on Assumptions 1&2. We will fix this issue in the updated version of our paper!

2 & 3. Clarification of Experimental Settings

Sorry for the confusion! Indeed, we did not omit AutoAttack results intentionally. Instead, we follow the principle of evaluating each method under its worst-case scenario (i.e., the strongest attack against that method). Therefore, 'Robust Accuracy' in Tables 1 and 2 refers to:

• For AT-based methods, we measure the robust accuracy of AutoAttack, since it is empirically the strongest attack for AT-based methods, as demonstrated by [1].
• For AP-based methods, we measure the robust accuracy of PGD+EOT attack, since it is empirically the strongest attack for AP-based methods, as shown in [2].
• For DDAD, we measure the robust accuracy of adaptive PGD+EOT attack, which additionally targets both the denoiser and the detector modules (see Algorithm 3). We empirically observed that adaptive white-box PGD+EOT provides the worst-case scenario for DDAD compared to adaptive white-box AutoAttack.

Thus, each defense method is evaluated under its own strongest attack setting, ensuring a relatively fair comparison since it mitigates evaluation bias (e.g., AT-based methods perform well on PGD+EOT because they have seen PGD examples during training). Similarly, if we only consider attacking the denoiser and the classifier (i.e., the white-box setting for AP-based methods), DDAD can achieve around 77% robustness on PGD+EOT and 81% on AutoAttack, which is clearly not fair for both AT and AP methods. We provide the full results in Table 1 below.

Table 1: Clean and robust accuracy (%) against adaptive white-box PGD+EOT ($\ell_\infty, \epsilon = 8/255$) and adaptive white-box AutoAttack ($\ell_\infty, \epsilon= 8/255$) on CIFAR-10. * means this method is trained with extra data. We show the most successful defense in bold.

[1] RobustBench: a standardized adversarial robustness benchmark, NeurIPS D&B 2021.

[2] Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV 2023.

4. Training and Inference Efficiency

We provide comparisons of training time with 3 representative AT-based methods [1][2][3] in Table 1. Notably, the current SOTA AT method [4] requires generating 50M synthetic images, making it extremely time-consuming. The point we want to highlight is: even compared to simpler AT methods, our method demonstrates significantly better efficiency. We also provide comparisons of inference time with 2 SOTA diffusion-based AP methods [5][6] in Table 2.

Table 1: Training time (hours: minutes: seconds) of different methods on CIFAR-10 with 2 x NVIDIA A100. The target model is RN-18.

Table 2: Inference time per image (seconds) of different methods on CIFAR-10 with 1 x NVIDIA A100. The target model is WRN-28-10.

[1] Towards Deep Learning Models Resistant to Adversarial Attacks, ICLR 2018.

[2] Theoretically Principled Trade-off between Robustness and Accuracy, ICML 2019.

[3] Improving Adversarial Robustness Requires Revisiting Misclassified Examples, ICLR 2020.

[4] Better Diffusion Models Further Improve Adversarial Training, ICML 2023.

[5] Diffusion Models for Adversarial Purification, ICML 2022.

[6] Robust Evaluation of Diffusion-Based Adversarial Purification, ICCV 2023.

5. Sensitivity of DDAD to the Threshold Values

The performance degradation occurs in ImageNet-1K, and this sensitivity can be attributed to 2 major reasons: (1) the evaluation of ImageNet-1K uses AEs with lower perturbation budgets compared to CIFAR-10, which makes it reasonable to use a smaller threshold value for ImageNet-1K. (2) ImageNet-1K is a large-scale dataset, so it may require more samples to optimize MMD-OPT. It is possible that the current MMD-OPT for ImageNet-1k has not been fully optimized, leading to such sensitivity (for now, we only use 1000 training samples to train MMD-OPT for ImageNet-1K). Luckily, we still have a range of threshold values that can produce stable and competitive results on ImageNet-1K. In the future, it would be interesting to see whether MMD-OPT will be more stable to threshold values on ImageNet-1K if we increase the training samples.

1. Evidence of 'Minimizing distributional discrepancy can reduce the expected loss on AEs.'

In our paper, we derive a theoretical bound to support our claim, i.e., $R(h, f_\mathcal{A}, \mathcal{D_A}) \leq R(h, f_\mathcal{C}, \mathcal{D_C}) + d_1(\mathcal{D_C}, \mathcal{D_A})$. In previous literature [1] [2], the upper bound of the risk on the target domain is always bounded by one extra constant, e.g., $R(h, f_\mathcal{A}, \mathcal{D_A}) \leq R(h, f_\mathcal{C}, \mathcal{D_C}) + d_1(\mathcal{D_C}, \mathcal{D_A}) + C$. If $C$ is large, minimizing $d_1(\mathcal{D_C}, \mathcal{D_A})$ can hardly reduce $R(h, f_\mathcal{A}, \mathcal{D_A})$. In contrast, we derive an upper bound without any extra constant C, which means minimizing $d_1(\mathcal{D_C}, \mathcal{D_A})$ can more effectively reduce $R(h, f_\mathcal{A}, \mathcal{D_A})$. This is a major contribution of our work. We will clarify this more in the updated version of our paper!

[1] Domain adaptation: Learning bounds and algorithms.

[2] A theory of learning from different domains.

2. Recent SOTA Baseline Comparison

We compare several most recent SOTA defense methods on RobustBench with DDAD. We will include the results in the updated version of our paper.

Table 1: Clean and robust accuracy (%) of recent SOTA methods on CIFAR-10. We show the most successful defense in bold. [3]* uses an ensemble of networks (ResNet-152 + WRN-70-16) and 50M synthetic images.

[1] Decoupled Kullback-Leibler Divergence Loss, NeurIPS 2024

[2] Better Diffusion Models Further Improve Adversarial Training, ICML 2023

[3] MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers, TMLR 2024

3. Plot Other Methods in Figure 2

Thank you for your suggestion! The main reason is that including all baseline methods will make Figure 2 look very messy. Therefore, we use a table (please see Table 5 in our paper) in Appendix E.3 to clearly compare our method with baseline methods under different proportions of AEs in a batch.

4. Discuss Other Distributional Discrepancy Metrics

We discuss two representative statistical alternatives of MMD here: Wasserstein distance and energy distance.

• Compared to Wasserstein distance, MMD has two major advantages:
  • The estimator of MMD is unbiased, while the estimator of Wasserstein distance is biased. Therefore MMD estimator is more accurate than Wasserstein estimator, especially when the dimension of the data is large.
  • Wasserstein distance requires solving a transportation problem, which is computationally more expensive than MMD, which means using Wasserstein distance will be slow for large datasets.
• Compared to energy distance, MMD offers greater flexibility due to its use of kernel functions, which allow it to capture intricate differences between distributions and adapt to various tasks by selecting appropriate kernels. It is particularly well-suited for high-dimensional data, as its reliance on embeddings in a reproducing kernel Hilbert space mitigates issues like the "curse of dimensionality" that can affect energy distance. Moreover, MMD is sensitive to higher-order statistics of distributions, enabling it to capture subtle discrepancies beyond mean and variance.

We will add a short section to discuss these statistical measurements in the updated version of our paper!

5. Assumptions for Training Setting

To train the MMD-OPT, we only require access to the clean training data. AEs used for MMD-OPT are generated based on the clean training data. This assumption is reasonable and commonly used in adversarial training-based methods and other defenses requiring AEs during training.

6. Ablation study of $\alpha$ in Eq.(8)

Increasing $\alpha$ in Eq.(8) means focusing less on the effect of MMD-OPT, and thus the robust accuracy will decrease, as supported by our theoretical analysis. We will include the results in the updated version of our paper!

Table 1: Our clean and robust accuracy (%) under different $\alpha$ against adaptive white-box PGD+EOT $\ell_\infty (\epsilon = 8/255)$ on CIFAR-10.

7. AEs in the Latent Space

Thank you for your insightful question! The philosophy is grounded in the findings of [1]: Regardless of the type of latent space (Hilbert space or otherwise), if adding perturbations into the latent representation of CEs does not increase the MMD value significantly, it indicates minimal distributional discrepancy between AEs and CEs. Consequently, these AEs can hardly deceive classifiers.

[1] Maximum Mean Discrepancy Test is Aware of Adversarial Attacks, ICML 2021

1. Comparison to MagNet

We acknowledge that MagNet [1] is a very good work. The main reasons we did not include MagNet as a baseline are: (1) MagNet is outdated since it was published 8 years ago, and (2) MagNet cannot defend against adaptive attacks, placing it significantly behind current SOTA methods. Please kindly check Table 1.

Table 1: Clean and robust accuracy (%) against adaptive white-box AutoAttack and PGD+EOT attacks on CIFAR-10. We show the most successful defense in bold.

[1] MagNet: A Two-pronged Defense against Adversarial Examples.

2. Novelty of Theoretical Analysis

The theoretical bound we derive is tighter than previous work [1] [2]. Previously, the upper bound of the risk on the target domain is always bounded by one extra constant, e.g., $R(h, f_\mathcal{A}, \mathcal{D_A}) \leq R(h, f_\mathcal{C}, \mathcal{D_C}) + d_1(\mathcal{D_C}, \mathcal{D_A}) + C$. If $C$ is large, minimizing $d_1(\mathcal{D_C}, \mathcal{D_A})$ can hardly reduce $R(h, f_\mathcal{A}, \mathcal{D_A})$. In contrast, we derive an upper bound without any extra constant C, which means minimizing $d_1(\mathcal{D_C}, \mathcal{D_A})$ can more effectively reduce $R(h, f_\mathcal{A}, \mathcal{D_A})$. This is a major novelty of our theoretical analysis.

[1] Domain adaptation: Learning bounds and algorithms.

[2] A theory of learning from different domains.

3. Novelty of DDAD

In our humble opinion, our proposed method is significantly different from existing SOTA AP methods. We summarize the novelty of DDAD from several key perspectives:

• Philosophically, DDAD focuses on minimizing distributional discrepancies, which fundamentally differs from existing AP methods relying primarily on density estimation.
• Theoretically, we derive a tighter and more informative theoretical bound to support the design of DDAD, which is a major contribution of our work.
• In terms of training efficiency, precise density estimation typically requires powerful models (e.g., diffusion models) that are computationally intensive and time-consuming to train. In contrast, learning distributional discrepancies is inherently simpler and more feasible, making DDAD both effective and efficient.
• In terms of inference efficiency, diffusion-based AP methods suffer from slow inference speeds due to repeated calls to the forward process of diffusion models. Our method, however, achieves significant performance improvements without compromising inference speed.

4. Adaptive Attack

According to [1], PGD+EOT is considered the strongest attack against diffusion-based AP methods, and AutoAttack is strongest against AT-based methods. Our proposed adaptive attacks (both PGD+EOT and AutoAttack) additionally target DDAD's detection mechanism (i.e., MMD-OPT), and adaptive PGD+EOT proves most effective in breaking DDAD. For gradient obfuscation checks, DDAD achieves the best average performance against adaptive BPDA+EOT attack across various baselines (see Table 3 in our paper). Also, please kindly check the results against adaptive AutoAttack in Section 2 & 3 of Reviewer 3bTb's responses.

[1] Robust Evaluation of Diffusion-based Adversarial Purification.

5. Transferability Across Different Defenses

• Thank you for your concern! We would like to clarify that the purpose of conducting transferability experiments is not to claim SOTA performance, but rather to ensure our method maintains good robustness against unseen attacks, as it relies on AEs to train MMD-OPT and the denoiser. Table 4 demonstrates our method's strong transferability across different attacks, architectures, and perturbation budgets.
• Due to the time-consuming nature of these baseline experiments, we will upload the corresponding results once completed. In the meantime, we provide some intuitions here: AT-based methods are expected to exhibit weaker transferability, as they encounter specific AEs during training [1], while AP-based methods are likely to demonstrate better transferability since they are independent of AE types [2].

[1] Geometry-aware Instance-reweighted Adversarial Training.

[2] Diffusion Models for Adversarial Purification.

6. Clarification of MMD-OPT

MMD-OPT builds upon MMD, which has a learnable kernel $k_w$. We can obtain optimized $k_w$ (we denote it as $k^*_w$) by maximizing Eq.(5) in our paper. Then, MMD-OPT is the MMD estimator with an optimized kernel $k^*_w$. During training, MMD-OPT serves as an objective to update the denoiser's parameters. During inferencing, MMD-OPT serves as a metric to measure the distributional discrepancies between two batches. We will clarify it in the updated version of our paper!

---

Thank you for your time! If our response has resolved your major concerns, we would appreciate a higher score :)