Improved Diffusion-based Generative Model with Better Adversarial Robustness

Thanks for your valuable comments and suggestions. Here we address your concerns as follows.

Q1: More NFEs should also be verified, although this method can improve efficient sampling, whether is adaptable and robust for more denoising steps should also be verified.

A1: Following your suggestion, we report the results with more NFEs (100, 200) below and add them in Appendix F.3 of the revised paper.

Table 5. Sample quality measured by FID $\downarrow$ of various sampling methods of DPM under 100 or 200 NFEs on $`CIFAR10`$ 32x32.

Table 6. Sample quality measured by FID $\downarrow$ of various sampling methods of DPM under 100 or 200 NFEs on $`ImageNet`$ 64x64.

As can be seen, our method is still effective with hundreds of NFEs.

Q2: Some complex derivations in supplementary material are too brief to understand, such as Eq(30) and Eq(59-62), I'm not sure if there are any typos in them, I suggest checking the equations carefully and modifying them.

A2: Thanks for pointing out this, for these complex equations, we have revised them to be more clear in the revised version. Please check them accordingly.

Q3: Consistency models on benchmark datasets such as CIFAR10 and ImageNet, which can be more common and convincing?

A3: Following your suggestion, we conduct the consistency model experiments on ImageNet 64x64. Note that since the limited computational resources, we can't directly use the hyperparameters as [1], instead we train the models for 300K iterations with a batch size of 512. Compared with the baseline method CM, our proposed AT method improves the one-step FID from 7.80 to 7.23. We will continue the training process and revise them into the final version.

Q4: Derivations in supplementary material should be checked carefully and written with more details.

A4: Thank you for your advice, we have carefully checked the derivations in the revised version, and made them more readable.

Q5 Why efficient AT can improve performance compared with PGD is a bit confusing.

A5: As found in [2], for adversarial-type training, in classification, AT takes a balance between accuracy and robustness, i.e., obtaining robustness may sacrifice accuracy. We speculate this phenomenon also holds in the diffusion model, i.e., too strong perturbation may sacrifice the noise prediction accuracy. That explains why strong PGD has slightly worse performance than our efficient AT in some situations.

Reference

[1] Consistency Models. Song et al., 2023.

Thanks for your valuable comments and suggestions. Here we address your concerns as follows.

Q1: My main concern in this paper is the evaluation. Currently, the proposed method is only evaluated using the ADM model. I wonder whether the effectiveness of more advanced model such as the stable diffusion still holds?

A1: Thanks for your valuable suggestion. We apply our method to LCM under Stable Diffusion v1.5 in Section 6.3, and the experimental results show the effectiveness of our method. To further address your concern, we finetune stable diffusion v1.5 under the framework of DPM by our proposed adversarial training. The evaluation results are summarized below.

Table 5. Comparison of FID on MS-COCO dataset with DDIM sampler.

As can be seen, our method still has better performance, compared with the baseline method.

Q2: The authors only use FID score as the evaluation metric, while it is easy to evaluate the results using other metrics such as IS, sFID, precision, recall, as done in the ADM paper. Why these metrics are not included?

A2: Thanks for your suggestion. We list the IS, sFID, Precision, and Recall of CIFAR10 32x32 with DDIM sampler as a representation as below. The results of our ADM-AT also outperform the baseline across metrics overall. More results of various samplers and datasets can be found in Appendix F.4 of the revised paper.

Table 5. Comparison of sFID $\downarrow$ and IS $\uparrow$ on $`CIFAR10`$ 32x32.

Table 6. Comparison of Precision (P) $\uparrow$ and Recall (R) $\uparrow$ on $`CIFAR10`$ 32x32.

Thanks for your valuable comments and suggestions. Here we address your concerns as follows.

Q1: In general, the algorithm developed in this paper is motivated by the distribution mismatch along the diffusion path. However, there is no experimental results to justify the motivation, there are also no experimental results to verify that the DRO framework can indeed help mitigate the distribution mismatch problem.

A1: Thanks for your valuable comment, we follow your suggestion to verify the quality of the intermediate $x_{t}$, as the mismatching problem is revealed by this. Concretely, we evaluate the FID between the generated $x_{t}$ by our and baseline methods, and compare them with the ground-truth ones. We adopt the IDDPM sampler under 200 NFEs and report the last 10 steps FID-10K in Table 1. We can observe that the ADM-AT achieves better FID on these steps.

Table 1. Comparison of FID of intermediate $x_t$.

Note that the Step 200 is the endpoint of the sampling process.

Q2: Proposition 2 has already been discovered in existing theoretical papers [1], see their section 3.1. The authors should comment on this point around Proposition 2.

A2: Thank you for pointing out the important reference. The results in section 3.1 are similar to our Proposition 2, as both of us quantify the KL divergence between the generated samples and the ground-truth ones. However, their results focus on the generated target data $x_{0}$, while ours focuses on all $x_{t}$ ($0\leq t\leq T$), though our techniques are similar. We have mentioned this comparison in the revised version.

Q3: The advantage of ADM-AT is not that significant compared with the ADM method, a more detailed ablation study or theoretical analysis on using adversarial noise or random Gaussian noise should be added.

A3: Our ADM-AT significantly outperforms the baselines ADM or ADM-IP across various samplers, especially under the setting of fewer NFEs (more practical and efficient). As shown in Table 1 of our paper: on CIFAR-10 32x32, with the IDDPM sampler, ADM-AT improves FID from 10.52 to 6.60. With the DDIM sampler, ADM-AT improves FID from 11.66 to 9.30 under 10 NFEs. With the DPM-Solver, ADM-AT improves FID from 8.00 to 5.84. These improvements are recognized as significant and practical.

Firstly, the proposed adversarial noise is induced by our theoretical framework under Distributional Robust Optimization to avoid distribution mismatching. As for the comparison with Gaussian noise, the baseline method ADM-IP adds Gaussian noise during training. According to our empirical results, our proposed adversarial noise is significantly better than the Gaussian noise.

Q4: Some statements are not clearly presented. For instance, the description of ADM is not given.

A4: For the description of ADM, it is a UNet-type [2] (with self-attention layer) neural network proposed by [3]. It is a standard architecture for diffusion model under image generation task. We have added this description in the revised version.

Q5: The norm $\|\cdot\|$ notations are abused, should that be $\ell_1, \ell_2$ or $\ell_\infty$.

A5: Without specifying, we use $\|\cdot\|$ to denote $\ell_{2}$-norm. We have added this description in line 137 of the revised version.

Q5. Some ablation studies for different perturbation levels $\alpha$ should be given.

A5: We explore the proposed under different perturbation level $\alpha$ on $`CIFAR10`$ 32x32 and $`ImageNet`$ 64x64 below. IDDPM is adopted as the inference sampler.

Table 2. Comparison of different adversarial learning rate α on $`CIFAR10`$ 32x32.

Table 3. Comparison of different adversarial learning rate $\alpha$ of our AT framrwork on $`ImageNet`$ 64x64.

We observe that $\alpha = 0.1$ is better on $`CIFAR10`$ 32x32 and $\alpha=0.5$ is better for $`ImageNet`$ 64x64. We observe that image in larger size corresponds to larger optimal perturbation level $\alpha$. We speculate this is because we use the perturbation measured under $\ell_{2}$-norm, where the $\ell_{2}$-norm of vector will increase with its dimension. We have added this ablation study in Appendix G.3 of the revised paper, please check it.

Q6: Some discussions about different perturbation methods ($\ell_1, \ell_2$ or $\ell_\infty$) should be discussed.

A6: These three perturbations are actually mathematically equivalent e.g., for vector $x\in \mathbb{R}^{d}$, it holds $\|x\|_{\infty}\leq \|x\|_{2} \leq \sqrt{d}\|x\|_{\infty}$. Therefore, we select $\|\cdot\|_{2}$ as representation in our paper. To further address your concern, we explore our method under three different perturbation methods ($\ell_1, \ell_2$ or $\ell_\infty$). The results are summarized below.

Table 4. Comparison of different perturbation norms on $`CIFAR10`$ 32x32.

During our experiments, we found that our method under $\ell_2$-perturbation is more stable and indeed has better performance, thus we suggest to use $\ell_2$-perturbation as in our paper. This discussion has been added in the Appendix G.3 of the revised paper.

Reference

[1] Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions. Chen et al., 2023.

[2] Convolutional Networks for Biomedical Image Segmentation. Ronneberger et al., 2015.

[3] Diffusion Models Beat GANs on Image Synthesis. Dhariwal et al., 2021.

Thanks for your valuable comments and suggestions. Here we address your concerns as follows.

Q1: There is no qualitative comparisons. It would be better to put some images for more direct comparisons.

A1: Thanks for your suggestions. The revised paper adds the qualitative comparisons between ADM-AT with baseline ADM/ADM-IP on CIFAR10 32x32 ( Figure 4)and ImageNet 64x64 (Figure 5). We also add comparisons between LCM-AT and LCM (Figures 6 and 7). Overall, models trained with our proposed AT method demonstrate superior performance in generating images for both class-conditional image generation and text-to-image generation tasks. Our AT method generates more realistic and higher-fidelity samples, which is consistent with the results in Tables 1, 2, and 3 in our paper.

Q2: In addition, the code is not provided.

A2: We will release our code, but currently the code for the consistency model and ImageNet 64x64 is still polished. We provide the diffusion model adversarial training code on CIFAR10 32x32 in the following anonymous link: code.

Q2: The efficiency comparison with AT and the classical denoising objective.

A2: Yes, as you mentioned, solving the proposed adversarial objective (14) with standard adversarial training technique (e.g., PGD in [1]) is computationally expensive, as we have explored in Table 8 of Appendix G.1. However, as mentioned in Section 6.1, we refer to efficient adversarial training [2] algorithm (Algorithm 1) to resolve this. By doing so, every update of the model under the proposed (14) has a similar computational cost to the classical objective. For example, we evaluate 10K updates of the model under methods: ADM/ADM-IP/ADM-AT(Ours). The spending time is 960/961s/970s, respectively.

Reference

[1] Towards deep learning models resistant to adversarial attacks. Madry et al., 2018.

[2] Adversarial training for free! Shafahi et al., 2019.