Elucidating the design space of classifier-guided diffusion generation

Thanks for the helpful comments. Below, we address the raised concerns.

How about the performance of the proposed methods for the general case with text prompts as conditions? In particular, in section 5.4, the authors also provide a simple analysis of this case with the CLIP model. However, the results are not good.

Thanks for the question. We agree that the more general text prompt condition is very important. The class-conditional setting, although simple, does provide a solid test bed for different kinds of guidance methods, with well-accepted quantitative measurements. If one method is not good at class-guided generation, we wouldn't expect it to excel at the more complicated text prompts. Therefore, we think our evaluations on the more typical class-guided generation case are valuable and the findings can potentially be generalized to more general settings.

Nevertheless, we have explored the CLIP-guided generation task. In the added Figure 6 of our revised paper, we present a comparison of CLIP scores between the original guidance schedule and our proposed schedule. Notably, our updated schedule consistently achieves significantly higher CLIP scores throughout the sampling process. The objective of our CLIP-guided sampling approach is to showcase the capability of a traditional off-the-shelf CLIP model in guiding the ImageNet-trained diffusion model (ADM [Dhariwal et al. 2021]). By leveraging this guidance, our model generates images that surpass the limitations of the ImageNet dataset and align with the provided prompt.

We aim to investigate more complicated discriminative models such as BLIP-VQA, as well as discriminative models in other modalities as guidance in the future.

Human-level metrics should be involved for clearer comparisons... CLIP score should also be considered to evaluate the performance.

Given the subjectivity of human-level metrics, we primarily focus on utilizing quantitative metrics to evaluate the performance of the classifier-guided diffusion (including our extensive experiments in evaluating the classifier-guided diffusion with quantitative metrics across diverse diffusion architecture: DDPM, EDM, DiT). For a more comprehensive evaluation, we add more quantitative metrics to the experiments. There are some values missing because the authors did not provide them.

Table 1: ImageNet 64x64 evaluation: 250 steps of DDPM sampling results.

Table 2: ImageNet 128x128 evaluation: 250 steps of DDPM sampling results.

Table 3: ImageNet 128x128 evaluation: 25 steps of DDIM sampling results.

In the added Figure 6 of our revised paper, we present a comparison of CLIP scores between the original guidance schedule and our proposed schedule. Notably, our updated schedule consistently yields significantly higher CLIP scores throughout the sampling process.

It is weird that the final images are merely the same as each other in Figure 4 with different settings of logit temperature, while the FID score varies with different settings in Table 4.

In Figure 4 of our paper, the random seed was fixed to ensure a consistent and strict comparison, thereby minimizing dramatic changes. However, it is worth noting that some significant differences still exist between the two settings. For example, in the left-hand side figure representing the traditional guidance setting, the generated dog gradually loses the depiction of its feet from time 50 to 0. In contrast, in the right-hand side figure where the joint guidance was amplified, the dog's feet remain clear, and there is a noticeable enrichment of environmental details.

Thanks for the helpful comments. Below, we address the raised concerns.

"The comparison is not so comprehensive. Since the main baseline should be the noise-aware classifier, the work provides little comparison in Table 5. More metrics such as IS, sFID, Precision and Recall should be included."

In our research, we aim to demonstrate the potential of the more accessible traditional ResNet classifier and prove that even without specifically noise-finetuned, the off-the-shelf classifier can guide the diffusion to generate high-quality sampling: FID: 2.19 in ImageNet128x128) not only largely surpass the noise-aware fine-tuned classifier guided diffusion [Dhariwal et al. 2021] (FID: 2.97) but also the classifier-free trained diffusion (FID: 2.43) [Ho et al. 2022].

Due to the computation resource and time limits, we prioritize testing the ImageNet64x64 setting and we plan to include the ImageNet 256x256 results in our future research to provide a comprehensive analysis across different image resolutions. There are some values missing because the authors did not provide them.

Table 4: ImageNet 64x64 evaluation: 250 steps of DDPM sampling results.

Table 5: ImageNet 128x128 evaluation: 250 steps of DDPM sampling results.

Resolution: "In Table 5, results for IMN64x64 as well as IMN256x256 for DDPM from Dhariwal and Nichol (2021) is missing... How off-the-shelf models Resnet can be used with ImageNet64x64 or ImageNet128x128 although these models are trained on ImageNet224x224? Besides, the generated images are clip to be in range of [-1; 1], how does it fit with model trained with input images normalized by ImageNet datasets?"

Inside the ResNet, there is an average layer: torch.nn.AdaptiveAvgPool2d((1,1)), which is capable of reshaping the different pixel input into the fixed size. During guided sampling, we primarily adhere to the guidance definition outlined in ADM [Dhariwal et al. 2021]. In this approach, a gradient is computed with respect to the generated samples since the guidance is incorporated into the generated images. Additionally, the guidance is computed as the gradient of the logarithm of softmax classifier logits, which is then normalized through an exponential operation across all classes.

"Does off-the-shelf classifier guidance provide conditional information for the unconditional diffusion model? Classifier guidance vs. conditional diffusion. "

According to the findings presented in Table 4 of ADM [Dhariwal et al. 2021], classifier-guided unconditional diffusion falls short in terms of performance (FID: 12.0) compared to classifier-guided conditional diffusion (FID: 4.59). Consequently, the subsequent experiments in Table 5 of ADM [Dhariwal et al. 2021] specifically focus on classifier-guided conditional diffusion. To ensure a fair comparison with SOTA results, we have adopted the guided conditional diffusion setting in this study.

Additionally, DiT is trained in a classifier-free manner, which encompasses both class-conditional and unconditional conditions. Table 9 of our revised paper demonstrates that off-the-shelf classifier guidance can also be employed in classifier-free diffusion. However, we acknowledge that there are further settings and configurations to explore, which will be addressed in future research endeavors.

Dear Reviewer wqgP,

Thank you for your detailed review and valuable feedback. We sincerely hope our previous response has addressed your questions and concerns. We look forward to hearing back from you and addressing any further questions or concerns you may have. Thank you for your time.

Thanks for your time in reviewing our work. Below, we address the raised concerns. We hope you can give our work the benefit of the doubt and look at it as a whole, rather than singling out each of its elements.

Sec 4.2 "PREDICTED DENOISED SAMPLES" seems trivial.

The idea of using predicted denoised samples is not hard to think of and it has already been applied by other methods, e.g., universal guidance [Bansal et al. 2023]. However, why it is a better choice still lacks justification. We do not claim the proposal of it as one of our novelties, but we did offer additional empirical results including integral calibration error $\overline{\text{ECE}}$ and FID as justification, as shown in Table 2 of our paper. Such a detailed comparison has not been made in previous work.

Sec 4.3 "SMOOTH CLASSIFIER" seems trivial.

The smoothness of the classifier is linked to its calibration. This may seem trivial but to the best of our knowledge, we are the first to point out calibration as one of the guidelines for classifier-guided diffusion generation. Further, we take the diffusion process into consideration and propose to look at the expected calibration error along the diffusion process.

In Figure 1 of our paper, surprisingly, we found that off-the-shelf classifiers are more calibrated than the specifically fine-tuned ones when the noise levels are high. We followed up with this observation with guided generation experiments. In Appendix A.1, we found that the integral calibration error is strongly positively correlated with the performance This has been one of our biggest motivations for improving the classifier calibration. We have extended this part in Section 4.1 to make a more convincing case.

On the practical side, we propose to augment the activation function for calibration, from ReLU to softplus, and choose the hyperparameter $\beta$ by examining the ECE error. This type of modification has been explored in [Zhu et al. 2021] for improving adversarial transferability but not for guiding diffusion generation. With this design choice, we are able to achieve better performance than well-established classifier guidance and classifier-free guidance on ImageNet benchmarks, which is not known to be possible before.

Sec 4.4 "JOINT VS CONDITIONAL DIRECTION", the author mention "we reduce the value of marginal temperature", which seems kind of manual tuning parameters. Is there some validation metric the authors use to determine the optimal temperature?

Empirically, we determine the hyperparameter for the marginal temperature based on a lightweight guided sampling of 1000 generated samples. However, we emphasize that the true value lies in the underlying insight. This is supported by the ablation study presented in Table 4 of our paper (we put it below), where all reduced values of the marginal temperature exhibit significant improvements over the baseline($\tau_2=1$), differing only in magnitude.

Table: Ablation study of marginal logit temperature $\tau_2$ with respect to FID. $\tau_1$ is fixed as 1. The evaluation process involves generating 10,000 samples from the 128x128 ImageNet dataset using off-the-shelf classifier-guided ADM[4], utilizing 250 DDPM steps.

Sec 4.5 "GUIDANCE SCHEDULE" seems trivial.

The inspiration for the proposed new guidance schedule comes from the observation of CLIP scores' evolution in the CLIP-guided experiment, as depicted in the added Figure 6 of our revised paper. We observed that the CLIP scores only increased at the beginning of the guided sampling process (from 1000 to 800 time steps), but plateaued afterward. Consequently, we introduced a sine function to modify the original guidance schedule. This choice was motivated by two reasons: firstly, the sine function is continuous and smooth, allowing for a seamless transition in guidance; secondly, it amplifies the guidance, particularly during the early to middle stages of the guided sampling process, shown in Figure D.1 of the appendix.

Sec 5.3 "OFF-THE-SHELF GUIDANCE FOR DIT", how do the authors choose a proper gamma_t for a specific case?

When it comes to selecting the appropriate choices for $\gamma_t$, we typically employ a lightweight guided sampling of 1000 generated samples for evaluation.

Thanks for the helpful comments. Below, we address the raised concerns.

Connection between Proposition 4.1 and Eq. 1.... It is unclear to me how Proposition 1 and ECE inform the design choices of the method.

We apologize for the confusion. Please allow us to clarify.

In score-based generative modeling, accurate estimation of the score function is of critical importance. However, for off-the-shelf classifiers, the second-order score information is difficult to access. Proposition 4.1 states that if the calibration error is controlled, so is the fisher divergence. Since the calibration error can be estimated by the so-called expected calibration error (ECE), we turn to it as our criterion for tuning the classifier. Equation 1 is the definition of the expected calibration error, which counts for a certain range of predicted confidence, how much does it agree with the actual accuracy.

Taking the diffusion process into consideration, we propose to look at the integral calibration error and it has been one of our biggest motivations for the design choices.

I think the storyline of this paper is a little bit scattered.

The integral calibration error, denoted as $\overline{\text{ECE}}$, plays a crucial role in our research. It assesses the calibration of the off-the-shelf classifier throughout the diffusion process. Proposition 4.1 establishes a connection between classifier guidance and probability estimation, emphasizing the significance of achieving a lower calibration error. In Figure 1, we analyze $\overline{\text{ECE}}$ values and demonstrate that the off-the-shelf ResNet achieves lower $\text{ECE}_t$ values than the fine-tuned classifier when the noises are large. This finding provides a strong basis for asserting that the off-the-shelf classifier has the potential to match the performance of fine-tuned classifier guidance.

To leverage this potential, we utilize the predicted denoised image $\hat{x}_0(t)$ as the input to the classifier, as suggested in universal guidance [Bansal et al. 2023]. To achieve improved calibration, we propose three key design elements: smooth classifier guidance, consistent guidance direction, and an updated guidance schedule. These design choices aim to calibrate the off-the-shelf classifier with the diffusion models and enable the generation of high-quality guided samples. By incorporating these elements, we enhance the calibration of the off-the-shelf classifier and facilitate the production of superior samples.

We have refined the writing at the end of Section 4.1.

"Ablation study is missing"

Thanks for the question. From Section 4.2 to Section 4.5, we have provided ablation studies for each of the design choices. We agree that it's important to look at all of them together in one task. Hence, we have provided additional ablation experiments.

The table below (Table 6 of our revised paper) showcases the results of our off-the-shelf classifier guided sampling in sequential ablation experiments, where we explore different design choices. The sequential parameter rule is employed to identify the selection that yields the best FID value. The percentage number within the brackets of each row indicates the improvement achieved by each design choice compared to the baseline setting, which is represented in the first column of each row. The evaluation process involves generating 10,000 samples from the 128x128 ImageNet dataset using ADM[4], utilizing 250 DDPM steps.

Table: Sequential ablation:

We have added the ablation study in Section 4.6 of our revision.

Although the authors answered many of my questions, I would still like the authors to clarify how they choose their design choices based on ECE before I raise my score. But overall I am inclining to raise my score and also adjust my confidence to 3 in case there is any further misunderstanding from my side.

Thank you for your response. These answers clarify all my questions, and although I still think the story of the paper can be better structured since not all the tricks/methods developed are related to the main claimed contribution of the paper (i.e. using calibration as the general guideline to analyze the design space of classifier guidance diffusion), the analysis presented in this paper is significant enough to warrant an acceptance. Therefore, I am increasing my score to a 6.

Thanks for the helpful comments. Below, we address the raised concerns.

Related work: the omission of closely related prior works, [1, 2], is significant... Although the authors conducted experiments comparing various design choices of off-the-shelf guidance, it is not clearly presented how these relate to previous research, and there is no direct comparison of the final FID scores with prior methodologies.

Thanks for pointing out the related works [1] and [2]. Both works investigated guided diffusion generation. However, their focus is different from ours, and hence their proposed methodologies are fundamentally different. Below we provide a detailed comparison to each of them.

The goal of [1] is to demonstrate the flexibility of conditional diffusion generation given specified constraints by introducing a regularization term $\log c(x, y)$ in the denoising process. Specifically, it explores constraints to control the digit shape in MNIST generation, facial specified attributes, infer semantic segmentations of city images, and solve a continuous relaxation of the traveling salesman problem.

The take-home message of this work is that diffusion models, trained on domain-specific data and under carefully designed conditions, are capable of generating samples that conform to specified constraints. While this showcases the flexibility of diffusion models in producing constraint-compliant samples, it doesn't test the limit of the performance in each of the categories.

In comparison, we specifically investigate class-conditional image generation and demonstrated that with proper designs, off-the-shelf classifiers can beat existing state-of-the-art methods. Through comprehensive experiments on different architectures of diffusion models, including DDPM, EDM, and DiT, our off-the-shelf classifier-guided sampling consistently surpasses the baseline performance across various diffusion model architectures.

Specifically, in the table below (Table 7 of our revised paper), our FID result of 2.19 on ImageNet128x128 not only surpasses the fine-tuned classifier-guided performance (FID: 2.97) [4], but also outperforms classifier-free trained diffusion models (FID: 2.43) [5]. This highlights that off-the-shelf guidance is not merely a substitute for fine-tuned guided diffusion to save training costs, but rather a better choice for generating high-quality samples.

Table: ImageNet 128x128 generation results.

[2] introduces a multi-expert strategy that utilizes multiple expert classifiers, each trained to handle a specific range of noise values. These "experts" guide the diffusion sampling process at their corresponding time steps. The authors propose a parameter-efficient knowledge transfer strategy to reduce the computational cost. Further, they proposed a teacher-student training approach to avoid the need for labeled data.

However, the experiments conducted in [2] do not provide conclusive evidence that multi-expert-guided sampling can compete with the fine-tuned classifier-guided diffusion ADM-G [4]. Moreover, though they employed parameter-efficient fine-tuning strategies like LORA, their computation cost is still non-negligible and cannot be used in a plug-and-play fashion.

In contrast, our off-the-shelf guided sampling not only eliminates the need for further training but also delivers significantly higher sampling performance. As demonstrated in Table 7 of our revised paper, our FID result of 2.19 on ImageNet128x128 not only surpasses the performance of the fine-tuned classifier-guided approach (FID: 2.97) [4], but also outperforms diffusion models trained without the use of classifiers (FID: 2.43) [5].