Direct Discriminative Optimization: Your Likelihood-Based Visual Generative Model is Secretly a GAN Discriminator

Thank you for your positive comments. Below, we provide detailed responses to your concerns.

The motivation for introducing $\alpha$ and $\beta$ to address gradient-vanishing/numerical issues is not well-supported. It would be helpful if the authors could provide experimental evidence to show the problem and demonstrate how $\alpha$ and $\beta$ help mitigate these issues.

The training will simply collapse without introducing a small $\beta$. For example, on CIFAR-10, the data dimension is 3072, and the bits per dimension (BPD) is around 2.5, meaning the log-likelihood $\log p_\theta(x)$ is around $3072\times2.5\times\log 2\approx 5000$. The log-likelihood ratio $\log\frac{p_\theta}{p_{\theta_{ref}}}$ also scales linearly with the data dimension and can be as large as hundreds, making $\sigma(\log\frac{p_\theta}{p_{\theta_{ref}}})$ suffering from gradient-vanishing. If we use $\beta=1$, the gradient will vanish when $p_\theta$ only deviates a bit from $p_{\theta_{ref}}$, and the finetuning almost does nothing to the base model. We also observe in our experiments that the FID remains unchanged with a large $\beta$. As for $\alpha$, it is an empirical hyperparameter to oppose the fake samples and accelerate the FID decrease, which we have already ablated in Figure 5(c).

The ablation study for Multi-Round Refinement is missing. Could the authors provide experiments where the reference model remains fixed throughout the entire fine-tuning process? Meanwhile, could the authors provide ablation studies on the number of iterations per round? Is a smaller or larger number of iterations more effective?

We would like to clarify that (1) we already provided the ablation for multi-round refinement in Figure 5(a) and (2) Figure 5(b)(c) already provides ablation studies on the number of iterations per round, as we continuously monitor the FID score during the whole finetuning process in a single round (with the reference model remaining fixed), demonstrating performance improvement at different numbers of iterations.

Could the authors provide experiments on fine-tuning the EDM, EDM2, and VAR using a standard fine-tuning method to assess how much the proposed approach improves over it?

Good question! Reviewer GoSA also mentioned this question, but he/she also says "I have few reasonable doubts as most baselines are already extremely optimized". We first show our empirical results and explain the fundamental reason behind it.

The "standard fine-tuning method" of likelihood-based models is to continue training with MLE, which can be seen as extending the pretraining period. On unconditional CIFAR-10 with the EDM base model:

By continuing training with the MLE objective, the FID just fluctuates within a small range and never advances the performance like DDO. This phenomenon can be expected, not only because the baselines are already extremely optimized, but also for the traditional MLE's inherent nature. As stressed in the paper's abstract and introduction, MLE is mode-covering and will be penalized seriously if the model underestimates the likelihood of any training samples. This limits the MLE objective's upper bound performance under limited model capacity. Therefore, we can never expect significant generation quality improvement by continuing training with MLE, while DDO is a fundamental different way that breaks the limitation.

It would be useful to show whether the proposed method also benefits training the generative models from scratch.

We did not expect the method to training generative models from scratch. As stated in Section 3.1 and 3.2, DDO replies on good initializations for stable optimization. Intuitively (e.g., as seen in Figure 1), DDO mainly refines a well-trained density and concentrate it on main modes. Moreover, the MLE objective has proven to be a stable and scalable approach for pretraining of likelihood-based generative models. Therefore, the best practice is pretraining with MLE and refining with DDO.

Additionally, we would like to provide some new results on higher-resolution datasets. After submission, we further use DDO to finetune EDM2-L on ImageNet 512x512, successfully advancing the FID from 2.11 to 1.36, without any guidance. Some visualizations are provided in the anonymous link. We find DDO significantly enhancing the image quality/details without affecting the overall content/diversity. We will release all model checkpoints and code that reproduce the reported results upon acceptance.

Thank you again for your consideration, and for giving positive rating. We hope that our response can resolve your concerns, and we are happy to answer further questions.

Thank you for your positive comments. Though you do not have notable concerns, we would like to provide some new results on higher-resolution datasets.

After submission, we further use DDO to finetune EDM2-L on ImageNet 512x512, successfully advancing the FID from 2.11 to 1.36, without any guidance. Some visualizations are provided in the anonymous link. We find DDO significantly enhancing the image quality/details without affecting the overall content/diversity. We will release all model checkpoints and code that reproduce the reported results upon acceptance.

Thank you again for your consideration, and for giving positive rating. We hope that our additional results can further support our method, and we are happy to answer further questions.

Thank you for your positive comments. Below, we provide detailed responses to your concerns.

One potential weakness of the paper is the lack of discussion on diversity metrics Recall during the DDO finetuning process. Analyzing how recall evolves during the DDO distillation process could provide deeper insights into whether the method maintains or improves diversity while enhancing sample quality.

We did not report the Recall metric because (1) it is measured only on relatively large-scale datasets like ImageNet 256x256, while not on CIFAR-10/ImageNet-64 (2) it is not as importance as FID/IS metrics, and good generative models often have indistinguishable Recall scores (as reported in the VAR paper). Moreover, the FID metric itself is sensitive to the generation diversity. A model with low diversity will yield degenerated FID scores.

Additionally, we would like to provide some new results on higher-resolution datasets. After submission, we further use DDO to finetune EDM2-L on ImageNet 512x512, successfully advancing the FID from 2.11 to 1.36, without any guidance. Some visualizations are provided in the anonymous link. We find DDO significantly enhancing the image quality/details without affecting the overall content/diversity. We will release all model checkpoints and code that reproduce the reported results upon acceptance.

Why does DDO require such specific hyperparameter settings (large alpha and small beta) to weight fake samples, while GANs do not seem to need this additional weighting?

Though we show that a discriminator can be implicitly parameterized by a generative model to utilize the GAN loss, an implicit discriminator and an explicit discriminator (used in GANs) can have different expressiveness and inherent output space structure. As mentioned in the paper, the log-likelihood of a generative model scales linearly with the data dimention, so does the implicit discriminator. Therefore, we need a small beta to manually scale it. A large alpha is an empirically choice to oppose the fake samples and accelerate the FID decrease. For an explicit discriminator, the output is directly produced by a separate network (more expressive) and can be freely scaled internally, so fewer manual operations outside are needed.

Thank you again for your consideration, and for giving positive rating. We hope that our response can resolve your concerns, and we are happy to answer further questions.

Thank you for your positive comments. Below, we provide detailed responses to your concerns.

Can the few debatable claims be reformulated along with the title?

• I disagree with the title...it is not a discriminator per se, but can be used to parameterize a discriminator.

We agree with "it is not a discriminator per se, but a parameterization". Our use of the expression "your xx is secretly xx" mirrors the famous DPO [1]. In DPO, a language model is also not a reward model per se, but can be used to parameterized a reward model. We think the expression is a convention in the community and will not be misleading. Otherwise, we can also change the title to "your ... can implicitly parameterize ...".

[1] Direct preference optimization: Your language model is secretly a reward model

Other improper claims in abstract/intro/Section 3.1... Formatting issues...

we partially agree your comments but have to post discussion later due to character limitation of rebuttal stage

comparison to further regular training of the base model?

On unconditional CIFAR-10 with the EDM base model:

By continuing training with the MLE objective, the FID just fluctuates within a small range and never advances the performance like DDO. This phenomenon can be expected, not only because the baselines are already extremely optimized, but also for the traditional MLE's inherent nature. As stressed in the paper's abstract and introduction, MLE is mode-covering and will be penalized seriously if the model underestimates the likelihood of any training samples. This limits the MLE objective's upper bound performance under limited model capacity. Therefore, we can never expect significant generation quality improvement by continuing training with MLE, while DDO is a fundamentally different way. We will add this comparison in the revised paper.

Improvements on related work

• (Kingma & Gao, 2024)"

We think this is common sense for diffusion models: the minimizer of denoising score matching is the ground-truth score function at every $t$ regardless of specific choices of $p(t),w(t)$. We cite (Kingma & Gao, 2024) because it is a systematic discussion on how to understand different $p(t),w(t)$ and how they are theoretically equivalent.

• More stable GAN settings have been found

We will temper the "unstable" statements for GANs. The instability is not the only factor why we favor diffusion/autoregressive models over GANs. Diffusion/autoregressive models are proven scalable to high dimensional data. Besides, they have an iterative generation process, which intuitively provides a larger potential generation capability compared to GANs.

• discussion of Chen et al. (2024)

We acknowledge the similarities and will add a section in the appendix to discuss the relationship. In our opinion, DDO is distinguished from CCA philosophically. On a high level, DDO's utilization of self-generated data can be seen as a form of on-policy optimization that roots deeply in reinforcement learning (RL) in today's language models. Similar to RL, DDO can fundamentally improve the base model's ability, while CCA cannot.

• $f$-divergence GAN

Thank you for introducing the $f$-divergence GANs! We think these generalizations are applicable to DDO, while they are unable to directly address the numerical issues as the likelihood ratio $\frac{p(x)}{q(x)}=\exp(\log\frac{p(x)}{q(x)})$ can be explosive ($\log\frac{p(x)}{q(x)}$ scales with data dimension). Nevertheless, we will add a section in the appendix to discuss this possible extension and leave more detailed exploration to future work.

No code is provided

DDO is simple to implement. To show reproducibility, we now release some code for the EDM2 codebase at the anonymous link. The loss function is simple (DDO_EDMLoss in training_loop.py), and we only need to add an additional dataloader to load model-generated fake samples.

Additionally, we provide some new results on higher-resolution datasets. After submission, we further finetune EDM2-L on ImageNet 512x512, successfully advancing the FID from 2.11 to 1.36, without any guidance. Some visualizations are provided in anonymous link. We find DDO significantly enhancing the image quality/details without affecting the overall content/diversity. We will release all model checkpoints and code that reproduce the reported results upon acceptance.

Thank you again for your consideration, and for giving positive rating. We appreciate your responsibility in carefully reviewing our work and providing very contrustive suggestions that help us improve. We hope that our response can resolve your concerns, and we are happy to answer further questions.