Score-based Conditional Generation with Fewer Labeled Data by Self-calibrating Classifier Guidance

We appreciate your thoughtful feedback on our paper. We have carefully addressed every concern of yours, as discussed below.

• [Weakness 1] One of the two score estimators in the proposed framework is redundant
  • We agree that the existence of two score estimators is somewhat redundant. The redundancy that resulted from the auxiliary task of score matching, however, should not be considered a burden to the design. Other studies [1] that introduce auxiliary tasks to deep learning models are considered effective even though the extra task is not performed during testing time and is thus redundant. The auxiliary task introduced in our work should be considered beneficial since the performance of the main task is improved significantly.
• [Weakness 2] Compare with JEM using "Learning Energy-Based Models by Diffusion Recovery Likelihood" (Gao et al, 2021)

  • Thank you for the suggestion. We have started running experiments using this method after we received the review. However, since it is really slow to train EBMs as MCMC sampling is needed, we only have some preliminary results by now (i.e. images generated using early checkpoints). We will include complete results in future revisions when the experiments are finished. Here are the preliminary results for CG-JEM using the suggested method:

    Dataset	% Labeled data	intra-FID	Acc	FID	IS
    CIFAR-10	5%	25.91	0.519	2.803	10.00
    CIFAR-10	20%	21.02	0.590	2.75	10.10
    CIFAR-10	100%	28.78	0.451	2.13	9.87

    Dataset	% Labeled data	intra-FID	FID	IS
    CIFAR-100	5%	121.97	4.33	12.11
    CIFAR-100	20%	104.02	3.74	12.47
    CIFAR-100	100%	93.99	3.68	11.86

• [Weakness 3] The method is only evaluated on datasets with 32x32 resolution and $\leq 100$ classes (CIFAR-10 and CIFAR-100)
  • We agree on the importance of evaluating our method on more datasets with higher resolution and more classes. Due to the limited computational resources in the academic environment, we were not able to do so previously. We are continuously using our available resources for additional experiments and will include them in future revisions upon completion. In the meantime, we hope to emphasize that the clear advantage of our proposed CG-SC in the fewer labeled data regime in CIFAR-10 and CIFAR-100 already provides strong support for the validity of the proposed method.
• [Question 1] Elucidate with an analytical perspective why the proposed method should outperform the baselines (JEM, DLSM)?
  • Advantage over JEM:
    • Time efficiency: Since the training of JEM requires MCMC sampling, further interpreting classifiers as SGMs significantly reduces training time.
  • Advantage over DLSM:
    • Utilization of unlabeled data: The calculation of DLSM loss requires ground truth labels, while the proposed loss can be applied to both labeled and unlabeled data.
• [Question 2] Could the authors theoretically justify the effectiveness of the proposed loss in a semi-supervised setup (e.g., defining the relationship of data-label pairs through optimal transport)? (Seguy et al. Large-Scale Optimal Transport and Mapping Estimation, ICLR, 2018)
  • To our understanding, the theoretical effectiveness of our method is not backed by optimal transport. We agree that given the promising results, further theoretical justifications would be beneficial. We hope our work could stimulate future theoretical analysis of the effect of the proposed regularization method.
• [Question 3] How were the hyper-parameters for the baseline methods, specifically the balancing factors of CG-DLSM and CG-JR, and the smoothing factor in CG-LS, chosen to ensure a fair comparison in Section 4?
  • Balancing factor of CG-DLSM is selected to be $1$, as suggested in the original paper.
  • Balancing factor of CG-JR is selected to be $0.01$, as suggested in the original paper.
  • The smoothing factor in CG-LS is tuned between $\{0.1, 0.05\}$ for the better intra-FID. Note that the performance of the two settings do not differ by much.

[1] Liebel et al. Auxiliary Tasks in Multi-task Learning, arXiv, 2018

We appreciate your thoughtful feedback on our paper. We have carefully addressed every concern of yours, as discussed below.

• [Weakness 1] Lack of novelty: depends on JEM and EBM
  • Although the proposed method utilizes the findings of JEM and EBM, we believe it is unfair to judge our novelty based on this. Our main contributions are that we proposed to further reinterpret classifiers as SGMs for regularization purposes, discovered the potential of CGSGM under semi-supervised settings, and designed experiments to verify the proposed method's effectiveness in improving CGSGM under both fully-supervised and semi-supervised settings.
• [Weakness 2] In instances where labeled data exceeds 50% (as shown in Sec 4.2 and Appendix A), the CFG method exhibits greater conditional generative power than the proposed CG-SC method and traditional CG methods.
  • We agree about the strength of CFG (and Cond) with more labeled data. Somehow we hope to re-emphasize that the paper does not focus on the regime of more labeled data. We believe it would be fairer if our proposed CG-SC could be discussed within the regime that we designed the method for. The results demonstrate CG-SC is more effective in the more realistic or common scenarios with less than 20% labeled data---the focus of our paper.
• [Question 1] Tuning $\lambda_{CG}$ and $\lambda_{CFG}$
  • Both parameters are tuned to obtain the best intra-FID
  • $\lambda_{CG}$ is tuned within $\{0.5, 0.8, 1.0, 1.2, 1.5, 2.0, 2.5\}$
  • $\lambda_{CFG}$ is tuned within $\{0, 0.1, 0.2, 0.4\}$
  • We will include this in the Appendix of future revisions for further clarification of the experimental details.
• [Question 2] Why is there a mismatch between conditional and unconditional metrics?
  • Although the generated images exhibit high fidelity and cover a significant portion of the input data distribution (good FID), they might fail to accurately capture the class-conditional distributions (poor intra-FID). This limitation can result from either poor generation accuracy or low generation diversity in certain classes, leading to poor intra-FID scores for those specific classes, as shown in Appendix A. For instance, consider an extreme case where the generator mistakenly swapped images between the 'cat' and 'dog' classes, the FID may remain high, but the intra-FID would significantly decline. After we swap the incorrect images back, the intra-FID would become substantially better while the FID remains the same.
  • Tuning the scaling factor introduces a trade-off between accuracy and diversity. In Appendix H, as the scaling factor becomes larger, the generation accuracy increases while the diversity decreases. The mismatch between FID and intra-FID came from the fact that FID only considers the diversity of images, while intra-FID additionally takes into account how well the generated images represent each class (generation accuracy).

Dear Reviewer 4JFE,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

We appreciate your thoughtful feedback on our paper. We have carefully addressed every concern of yours, as discussed below.

• [Weakness 1] Experiments with RandAugment and CutMix
  • Indeed augmentation-based regularizations are important. We wanted to avoid augmentation-based regularization since it is hard to compare those methods with regularizations that use auxiliary losses as they are done in different stages (pre-processing v.s. training stage).
• [Weakness 2] Experiments with semi-supervised approaches
  • In the current stage, we've tried incorporating Fixmatch [1] and EMA-teacher [2] to train the time-dependent classifier. While it leads to some improvement over vanilla CGSGM, it often leads to collapsed conditional generation results of some classes. For example, the images generated with the label "airplane" might contain images generated from random classes while the generative performance of other classes improves. Current results suggest that integrating semi-supervised methods into time-dependent classification is non-trivial and requires further exploration.
• [Weakness 3] Why doesn't Cifar-100 have generation accuracy evaluation?
  • As mentioned in Section 4.1, nearly-accurate classification is desired for meaningful evaluation of the generation accuracy, but we were unable to locate such a classifier for the CIFAR-100 dataset. Therefore, generation accuracy is not included for the CIFAR-100 dataset.
• [Weakness 4] Why not use DLSM on unlabeled data?
  • In the formulation of DLSM loss \mathbb{E}\left[\frac{1}{2}\lVert \nabla_\tilde{x}\log p(\tilde{y}\vert \tilde{x}; \theta)+s(\tilde{x};\phi)-\nabla_\tilde{x}\log p_\sigma(\tilde{x}\vert x)\rVert^2\right], ground truth labels are required for calculation (the $\tilde{y}$ in \nabla_\tilde{x}\log p(\tilde{y}\vert \tilde{x}; \theta)). As a result, it is not applicable to unlabeled data.

[1] Sohn et al. FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence, NeurIPS, 2020

[2] Cai et al. Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning, CVPR, 2021

Dear Reviewer zxWa,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

We appreciate your thoughtful feedback on our paper. We have carefully addressed every concern of yours, as discussed below.

• [Weakness 1] Our method is incremental (adding JEM and DSM to train the classifier)
  • Although the proposed method utilizes the findings of JEM and DSM, we believe it is unfair to judge our novelty based on this. Our main contributions are that we proposed to further reinterpret classifiers as SGMs for regularization purposes, discovered the potential of CGSGM under semi-supervised settings, and designed experiments to verify the proposed method's effectiveness in improving CGSGM under both fully-supervised and semi-supervised scenarios.
• [Weakness 2] The focus on datasets like CIFAR-10 and CIFAR-100 limits the assessment of the methodology's generalizability to more diverse or complex data scenarios.
  • We agree on the importance of evaluating our method on more datasets with higher resolution and more classes. Due to the limited computational resources in the academic environment, we were not able to do so previously. We are continuously using our available resources for additional experiments and will include them in future revisions upon completion. In the meantime, we hope to emphasize that the clear advantage of our proposed CG-SC in the fewer labeled data regime in CIFAR-10 and CIFAR-100 already provides strong support for the validity of the proposed method.
• Minor weaknesses
  • Thank you for the suggestions. We will improve the presentation of the mentioned point in future revisions.
• [Question 1] Does the author's argument also suggest that, compared to Deep Generative Models (DGMs), classifier architectures may be more prone to over-fitting?
  • Both classifiers and deep generative models have over-fitting problems, and it is hard to determine which is more prone to over-fitting. What we want to say is that over-fitting classifiers are harmful to the generative performance of CGSGM. Therefore, the classifier needs to be regularized for better generative performance.
• [Question 2] Why are FID in CFG-all and CFG-labeled notably poor?
  • As discussed in Section 4.2, this is because those methods generate samples with degraded diversity under semi-supervised scenarios. We observed that CFG-all and CFG-labeled tend to generate samples highly correlated to the labeled data. The lack of labeled data causes those methods to generate images with far less diversity, making their performance much worse when evaluated using FID and intra-FID.

We appreciate your thoughtful feedback on our paper. We have carefully addressed every concern of yours, as discussed below.

• [Weakness 1] Originality: The techniques are common in deep-generation models
  • Indeed, regularization methods are common in deep generative models, but whether a new regularization technique is effective needs verification through experiments. Our main contributions are that we proposed to further reinterpret classifiers as SGMs for regularization purposes, discovered the potential of CGSGM under semi-supervised settings, and designed experiments to verify the proposed method's effectiveness in improving CGSGM under both fully-supervised and semi-supervised scenarios. Furthermore, to our understanding, there is no other work that proposed to regularize classifiers as SGMs.
• [Weakness 2 & Question 1] Why didn't the authors test the classification accuracy of the classifiers?
  • The main goal of our study is to improve the generative performance of CGSGMs instead of improving classification or joint modeling. Related works on CGSGMs [1,2,3] also reported only generative performance.
  • Nevertheless, we did evaluate the classification accuracy (at timestep $t=0$) of CG and CG-SC-all for the CIFAR-10 dataset in the preliminary stage of our work. Despite observing improvement over CG, the classification accuracies for both methods are poor compared to the SOTA of ordinary classification for the CIFAR-10 dataset. Note that this is similar to the observation reported here by authors of [1]. It is hard to tell whether these classifiers perform well, given that the time-dependent semi-supervised classification task is substantially more challenging than ordinary semi-supervised classification.

    % labeled data	CG	CG-SC
    100%	0.808	0.925
    80%	0.803	0.909
    60%	0.752	0.905
    40%	0.680	0.731
    20%	0.571	0.738
    5%	0.420	0.658

  • Although improvement was observed through evaluating classification accuracy, we could not explicitly explain how well the time-dependent classifiers are performing, and how testing accuracies of such classifiers affect the generation quality of CGSGMs is still to be examined. We aimed to provide evaluation metrics that more directly reflect the generation performance of different methods. As a result, we chose to present the current evaluation metrics (FID, intra-FID, inception score, and generation accuracy) in our work.
• [Weakness 3] Little effect on other sub-fields of machine learning
  • We believe our work should already be considered significant as the sub-field we are studying ( conditional generation) is itself a significant sub-field of machine learning.
• [Question 2] Discuss the relationship with [4]
  • Their work focuses on improving the performance of ordinary semi-supervised classifiers and conditional diffusion models using dual training.
  • Our work focuses on improving classifier-guided SGMs through regularizing the time-dependent classifier using both labeled and unlabeled data.

Link 1: https://openreview.net/forum?id=AAWuCvzaVt#:~:text=We%20have%20not,traditional%20ImageNet%20classifiers.

[1] Dhariwal et al. Diffusion Models Beat GANs on Image Synthesis, NeurIPS, 2021

[2] Chao et al. Denoising Likelihood Score Matching for Conditional Score-based Data Generation, ICLR, 2022

[3] Kawar et al. Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance, TMLR, 2023

[4] Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels, NeurIPS 2023

Dear Reviewer b4HY,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair