PaGoDA: Progressive Growing of a One-Step Generator from a Low-Resolution Diffusion Teacher

We greatly appreciate the reviewer’s constructive feedback.

Weakness 1. LCM with lighter VAE may be more efficient than PaGoDA's upsampling blocks?

Ans.

[Lightweight PaGoDA] The smaller VAE [A] decoder can be integrated into PaGoDA's super-resolution framework. If LCM's VAE is made more lightweight, PaGoDA's network could be similarly lightweighted. The degree of simplification for heavy decoder (or upsampling network) is determined by the function complexity. As PaGoDA’s upsampling function is simpler than VAE's nonlinear transformation, PaGoDA is more suitable for model distillation. Also, a lightweight decoder sometimes fail at high-quality generation, illustrated in Figure C in the attached PDF.

Weakness 2. A smaller VAE requires single-stage training, while PaGoDA involves multiple sequential stages.

Ans.

[Smaller VAE is Sequential Training] The smaller VAE [A] separately distills a heavy encoder and decoder with lightweight counterparts to match latent distributions. In knowledge distillation, lightweight models distilled from heavy teachers tend to perform better than those trained without teacher. Likewise, the smaller VAE [A] requires to have the original VAE to minimize the performance sacrifice, which falls into a sequential training.

[PaGoDA Skips Intermediate Stages] PaGoDA's progressive upsampling can be compressed. Instead of doubling resolution jumps, they can be increased by factors of 4x or 8x, allowing the super-resolution network to be trained all at once. It was observed that even with an 8x jump, model remained stable and performance was retained.

Weakness 3. FID evalutaion could be biased due to the use of EfficientNet and DeiT pre-trained on ImageNet.

Ans.

[Frechet Distance] Following reviewer's suggestion, we additionally report FD using DINOv2 in Tables A and B of the attached PDF. On ImageNet 64x64, we have three implications:

• vs. EDM & CTM: All three models (PaGoDA, EDM, CTM) share the same architecture. CTM performs better than teacher EDM only in FID, a signal of FID bias. However, PaGoDA consistently outperforms teacher EDM in both FID and FD.
• vs. StyleGAN-XL: Direct comparison between PaGoDA and StyleGAN-XL is challenging due to differences in architectures and training details. Nonetheless, PaGoDA outperforms StyleGAN-XL in all metrics.
• vs. Validation Data: Given that PaGoDA's FID is better than the validation data while its FD is worse, FID may no longer be a reliable metric for model evaluation.

On ImageNet 512x512, we compare PaGoDA as follows:

• vs. EDM: PaGoDA significantly surpasses EDM in both FID and FD, even with the same architecture (EDM is trained on latent space).
• vs. EDM2: EDM2 outperforms PaGoDA in FD, regardless of the number of parameters used. However, please note that EDM2 requires $63$ steps to achieve good FDs, while PaGoDA requires only a single step.

Based on these findings, we will focus on comparing PaGoDA using FD in our paper revision. We plan to explore whether applying PaGoDA on EDM2 architecture would bring us a better quality in future work.

Question 4. CFG scale used in table 6 for PaGoDA: is the best performance point in Fig. 14 reported in Table 6?

Ans.

It is correct that we used the best performance in Figure 14, with a CFG scale of 1.15, to report the score in Table 6. Adopting reconstruction loss reduced the optimal CFG scale from 2-3 to 1-2.

Question 5. Why were different datasets used when collecting latents (CC12M) and during GAN training (COYO-700M)?

Ans.

[Prevent Overfitting] We used different datasets to mitigate overfitting and memorization issues. Let's suppose that $z$ is a latent of $x$ from CC12M. Then, the reconstruction loss pushes $G(z)$ towards $x$. If we utilize the same CC12M for GAN training, then the discriminator will push $G(z')$ towards CC12M, where $z'$ is a neighborhood of $z$. Therefore, to alleviate $G(z')$ being too close to $x$ in CC12M, we provide an effect of $G(z')$ diverging from CC12M by training discriminator with COYO-700M.

Question 6. Does the DDIM inversion in Section 5.4 needs a diffusion teacher for inversion?

Ans.

Yes, we utilized teacher diffusion in Section 5.4 for controllable generation.

<Final comments to Reviewer 3>

We respectfully request the reviewer to re-evaluate our paper with an emphasis on its upsampling capabilities. The components, though seemingly separate, are integrated to achieve efficient super-resolution generation.

[Upsampling: DDIM Inversion for Recon Loss] DDIM inversion allows obtaining a latent $z$ from high-dimensional input data $x_{\text{high}}$ by downsampling it to DM's resolution. By training a map from $z$ to $x_{\text{high}}$, PaGoDA provides upsampling capabilities alternative to SD's VAE and Cascaded Diffusion Models. When using DDIM, we cannot train such an upsampling map.

[Upsampling: Classifier-Free GAN for Adversarial Loss] Previous diffusion distillation methods used standard GANs with $\omega$-CFG teacher samples, limiting student performance and resolution extension. An easy fix could be using appropriately resized real-world data into GAN's real part, but this way is incompatible to T2I because the optimal student follows $p_{\text{data}}(x|c)$, not $p_{\text{data}}(x|c,\omega)$. Classifier-Free GAN solves this by predicting $\omega(x,c)$ and feeding it into the student network, ensuring alignment with the desired distribution $p_{\text{data}}(x|c,\omega)$ (see also L157-L159).

[Upsampling = Progressive Growing + DDIM Inversion + Classifier-Free GAN] By integrating three proposed components altogether, we achieved efficient super-resolution generation in both training and sampling, providing a new and effective alternative to Stable Diffusion and Cascaded Diffusion Models. We kindly ask the reviewer to reconsider our paper with this perspective in mind.

We sincerely express our gratitude for the reviewer to allow us compare PaGoDA with SD more deeply. We believe the review is highly helpful, and would like to revise the paper based on our discussion.

Q1-1. While training a new VAE requires sequential training, PaGoDA also necessitates an additional data generation process.

Ans.

[Minimal Overhead] It takes about one day with 8xH100 GPUs to converge in T2I at base resolution if training data pairs are synthesized online. The data collection overhead is minimal compared to the overall DM pretraining cost. For higher-resolution training, the data pairs collected at base resolution are reused, significantly reducing the need for additional DDIM inversion and compute resources.

Q1-2. For conditional generation, the model p(w∣x,c) must also be trained.

Ans.

[Use Released Checkpoint] We recognize that our approach involves higher initial costs due to the need to train the classifier. However, once the classifier is publicly available, the training pipeline will be as equivalent of training a smaller VAE.

Q1-3. I still doubt whether PaGoDA truly offers a more convenient training pipeline compared to training a smaller VAE.

Ans.

In terms of the number of training stages, PaGoDA is indifferent from SD. However, we would like to further discuss the discrepancy between SD and PaGoDA when we desire to increase the sample resolution, e.g., from 512x512 to 1024x1024.

[SD Needs Entire Retraining] In Stable Diffusion, the entire pipeline needs to be retrained from scratch, following these steps:

• Train VAE at 1024x1024 from scratch
• Train latent DM from scratch
• (Optional) Distill latent DM into one-step generator
• (Optional) (Optional) Distill VAE into smaller VAE

[PaGoDA Recycles Pretrained Network] In contrast, PaGoDA could reuse 512x512 generator and only necessitates training a upsampling network from 512x512 to 1024x1024, making it significantly more cost-effective than SD, as follows:

• Reuse pixel DM at base-resolution
• Reuse PaGoDA at base-resolution
• Reuse PaGoDA at super-resolution up to 512x512
• Train PaGoDA for upsampling from 512x512 to 1024x1024

Q2. Doesn't PaGoDA also experience reconstruction error for the input image in Figure C?

Ans.

PaGoDA exhibits less accurate reconstruction than generation quality. This is because PaGoDA's reconstruction quality solely depends on reconstruction loss. Below, we provide details on this respect.

[Different GANs for Different Purposes] We kindly remind that the model objectives are different: VAE in SD is for data compression, while PaGoDA is to assist generation. Accordingly, VAE and PaGoDA applied GANs for two different purposes:

• GAN for reconstruction
  • Real: a real image
  • Fake: its corresponding reconstructed image
• GAN for generation
  • Real: a real image
  • Fake: a randomly generated image from a random noise

PaGoDA prioritizes generation quality over reconstruction quality, thus using (II) type of GAN. Therefore, PaGoDA's reconstruction relies solely on LPIPS reconstruction loss. In contrast, SD's VAE uses option (I), specifically tailored for better reconstruction quality.

[Applying Option (I) in PaGoDA for Better Reconstruction] We observed that the further use of (I) in PaGoDA on top of (II) improves reconstruction quality, and we will include a figure in our final revision to illustrate this improved reconstruction quality. It is important to note that a direct comparison with Figure C is not feasible for PaGoDA, as DDIM inversion relies on the text prompt, which is not provided in Figure C.

(Side Remarks on Q2)

We presume that the reviewer may have raised concerns about PaGoDA's upsampling capability due to the loss of high-frequency signal during downsampling of the encoder. This information leakage seems fundamentally blocking higher-dimensional generation at the first glance. Below, we further discuss that it is not necessarily true in PaGoDA.

[PaGoDA: Lossless Compression] Although our encoder may seem to perform lossy compression, it can actually achieve lossless compression in principle, i.e., perfect reconstruction. If the downsampled versions of two different images $x$ and $y$ remain distinguishable, DDIM inversion, as an injective function, will map $x$ and $y$ to distinctive vectors in the latent space. Therefore, the generator can map these distinctive latent vectors back to the original signals $x$ and $y$, respectively, given a sufficiently flexible generator and a well-designed loss function. However, this lossless compression is only achievable when two different data remain distinguishable after downsampling. Hence, the downsampling factor should be carefully selected to ensure that the downsampled images are distinguishable. At resolutions like 64x64 (or even 32x32), natural images remain sufficiently distinguishable.

We express our gratitude to the reviewer for helpful reviews. Below, we faithfully answer the raised concerns, but we would be happy to provide additional experiments upon reviewer's request in our final revision.

Weakness 1. Can we integrate PaGoDA with SD?

Ans.

Yes, we can integrate PaGoDA in the latent space of SD. If $d$ (say 512) is the latent resolution of SD (data resolution could be up to 2K or 4K), we can train a DM on $d/2^n$ (say 64) dimension (which is much cheaper than training DM on full 512 latent dimension) and then distill/upsample with PaGoDA up to $d$. Although we focus on the core algorithm with pixel DM to pursue a fundamental understanding of core components, we are interested to integrate PaGoDA with SD in future to examine the degree of compression this integration can offer.

Weakness 2. Discuss the training costs compared to sampling-free alternatives.

Ans.

[Data Collection is Optional] While we utilized collected data-latent pairs mainly to save budget for model building, once training configurations are set, training data pairs can be obtained online during training.

[Minimal Overhead] In T2I, convergence is achieved around 20k iterations, requiring about 1 day with 8xH100, including latent calculation. This overhead is minimal compared to the total DM pretraining cost.

[Safer Training] We further address that sampling-based methods are safety-secure. During training, a classifier can filter out problematic samples related to privacy, NSFW, and copyright. Moreover, maintaining clear records of training data aids in post-mortem analysis.

Weakness 3. Adoption of KD objective in T2I is contradictory to reconstruction loss's motivation.

Ans.

[KD for High CFG] Figure 4 highlights that the latents from DDIM inversion are out-of-prior at high CFG scales. Therefore, a model trained with reconstruction loss performs poorly at these scales. To resolve this problem, we impose the KD loss. However, using KD-only limits student upper bounded by teacher.

[Effect of Recon+KD] When we add reconstruction loss, the FID-CLIP curve is shifted right (higher CLIP) and below (better FID) with the optimal CFG (w.r.t FID) falls from 2.5 to 1.15. We also report Figure A of the attached PDF of human evaluation, resulting in better image quality and prompt alignment.

[Eliminate KD] We are interested in eliminating KD objective by substituting CFG with CFG++ [A], a recent method with improved DDIM and DDIM inversion on guidance scale less than 1. On these scales, DDIM latent falls into in-distribution that facilitates model training without KD. We originally planned to explore this as future works but would be happy to investigate on this if reviewer requests in our final revision.

[A] CFG++: Manifold-constrained Classifier Free Guidance for Diffusion Models

Weakness 4. Using CLIP regularization makes the CLIP score evaluation incorrect.

Ans.

Figure A in the attached PDF shows that CLIP regularization is advantageous for prompt alignment under human judgement. To minimize potential bias in evaluation, we utilized different CLIP models for training (ViT-L/14 trained on YFCC100M) and evaluation (ViT-g/14 trained on LAION-2B).

Weakness 5. Clarify the CFG details for inference.

Ans.

The CFG scale in Tab. 5 and 6 is 1.15. As shown in Figure 14, CFG in PaGoDA exhibits a similar trend to teacher diffusion.

Weakness 6. Does PaGoDA work without adversarial objective?

Ans.

[Existence of Prior Holes] PaGoDA requires a GAN loss. Since PaGoDA's encoder is a deterministic mapping, it avoids posterior collapse but suffers from prior holes. These holes are unseen during training with recon-only loss, causing decoder to struggle in creating high-quality samples in those areas. Therefore, we need GAN loss to cover the entire prior manifold during training.

[Uniform Encoding] DDIM inversion encodes data uniformly into the prior manifold, as illustrated in Figure B of the attached PDF. This uniformity constrains generator, preventing mode collapse by densely regulating its output within the prior space through reconstruction loss.

[Well-Grounded GAN] Moreover, Theorem 3.1 proves that the optimal generator of a GAN-augmented distillation model matches data distribution, unlike previous works where optimal generator is heavily influenced by pretrained DM. Theorem 3.2 confirms the stability of combining GAN with reconstruction loss.

Question 7. Can we deactivate reconstruction/distillation losses for SR stages?

Ans.

Deactivating recon/distill losses causes model to solely depend on GAN. As seen in Figure 6, relying solely on GAN for SR results in object location shifted across resolution. This unwanted shift hinders the ability to control samples in base resolution. By activating reconstruction loss, the object's position remains consistent in resolution changes.

Question 8. Can we use a pretrained DM as a discriminator feature extractor?

Ans.

Yes, but note that feature extractors behave differently. According to [6], discriminative features prioritize local details over global shape, while generative features do the opposite. Since reconstruction loss strongly penalizes errors in shaping the global context, we used features from discriminative models to prioritize capturing local details with GAN loss.

Question 9. Could the ImageNet performance a result of overfitting?

Ans.

Table A of the attached PDF shows the Frechet Distance on DINOv2 feature space, comparing 50k samples with 50k val data. The degree of overfitting (FD val - FD train) in PaGoDA is comparable to baselines, implying PaGoDA is as robust as baselines in overfitting.

Weakness 1. To address the concerns about fair evaluation, we provide additional results below.

W1-1. GAN's discriminator pretrained on ImageNet biases the FID metric.

Ans.

[Frechet Distance (FD)] We agree with the reviewer that the use of ImageNet pretrained discriminator could bias the model towards achieving a good FID. Following EDM2 practice, we evaluate FD using DINOv2 features. Tables in the attached PDF imply that FID and FD trends are consistent in ImageNet 64x64, but not for high-resolution, which may indicate FID bias. Despite this, PaGoDA outperforms its teacher EDM at both resolutions on FD. We will update the manuscript to include a full set of results and discussions. Please also refer to R2-W9 and R3-W3.

W1-2. CLIP regularizer may also bias the evaluation.

Ans.

[Human Evaluation] We conduct human evaluation to fairly measure the effect of CLIP loss at Figure A of the attached PDF. According to the figure, the adoption of CLIP regularization yields positive influence on prompt alignment. Also, to minimize the undesirable bias in FID/CLIP scores, we utilized different CLIP models for training (ViT-L/14) and evaluation (ViT-g/14). These models were trained on different datasets: ViT-L/14 with YFCC100M and ViT-g/14 with LAION-2B.

Weakness 2. To address the novelty concerns, we clarify our contributions below.

W2-1. The combination of a regression loss and a GAN loss has been explored for T2I generation in several studies.

Ans.

[DDIM vs. DDIM Inversion] Although the reviewer classifies both reconstruction loss (from DDIM inversion) and distillation loss (from DDIM) as regression loss, they function differently. Previous distillation models [3, 4] adopt DDIM for distillation, starting from the prior latent and obtaining synthetic data, which may harm model performance if the synthetic data is of low-quality. In contrast, DDIM inversion in PaGoDA, starting from the real data and obtaining its latent, can benefit from the high-quality of real data.

[Theoretically Grounded GAN Integration] Additionally, we show that existing methods combining GAN loss and DDIM-distillation loss do not optimally learn the data distribution, as noted in Theorem B.4 (extended Theorem 3.1) in the appendix. In contrast, Theorem 3.1 demonstrates that PaGoDA's objective ensures optimal data distribution—making it the first diffusion distillation approach to guarantee this theoretically. This advantage stems from using real input data $x$ to construct $(x,z)$ pairs for distillation. Empirically, Figure 5-(d) shows that PaGoDA's loss (recon+GAN) maintains performance, while KD+GAN loss degrades as the teacher model worsens (imperfect teacher model).

[GAN Stability] Moreover, previous works have not adequately addressed GAN stability concerns. Our Theorem 3.2 provides the first guarantee that GAN can be stable if combined with DDIM inversion. Addressing these questions on optimality and stability offers new insights into the distillation community.

W2-2. The use of distillation loss and CLIP loss are common practices [5, 6].

Ans.

Please note that we are not claiming CLIP regularization as our contribution. Instead, we advocate using the CLIP regularizer as an option to achieve better text alignment.

W2-3. Proposed components explored in literature reduce the novelty of the paper.

Ans.

[DDIM Inversion + Progressive Growing = Upsampling] We respectfully request the reviewer to re-evaluate our contribution. We are not complicating matters by adding well-established losses. Instead, we argue that DDIM inversion not only guarantees learning the data distribution (see Theorem 3.1) but, when combined with our progressively growing technique, offers an alternative to the upsampling methods used by Stable Diffusion and Cascaded Diffusion Models.

Weakness 3. We provide explanations to address concerns about methodological complexity.

W3-1. Using the standard GAN loss is simpler than the proposed classifier-free GAN, which complicates the objective.

Ans.

[Upsampling] We would like to address the reviewer's concern by connecting GAN with upsampling. Suppose $d$ is the base resolution where DM has been trained. To double the resolution, the GAN's real/fake data should also be doubled. Otherwise, the GAN cannot generate $2d\times 2d$ images with full details.

[Problem of Standard GAN] A problem arises when using standard approach of putting $\omega$-CFG teacher sample as GAN's real input. Since the teacher sample is of $d\times d$ dimension, without learning independent upsampling models like Cascaded Diffusion Models, there is no efficient way to upsample the teacher sample to $2d\times 2d$.

[Use Real-world Data] Our proposal is to use high-resolution real-world data. We put real-world data (downsized to $2d$ resolution) into GAN's real part instead of using teacher samples. We compare the standard approaches, and our proposal as follows:

• Standard way: generate $\omega$-CFG teacher sample and feed it to GAN's real part.
  • (-) Student resolution not extendable
  • (-) Reach to teacher quality at optimum
• Our proposal: retrieve real data, downsize it, predict $\omega$ of it, and put it into GAN's real part.
  • (+) Student resolution extendable
  • (+) Reach to data quality at optimum

[Use Released Checkpoint] We recognize that our approach may involve higher initial costs due to the need to train the classifier. However, once the classifier is publicly available, the training cost will be similar to the standard approach. We kindly ask the reviewer to focus on the qualitative benefits of our proposal.

[Concurrent Works] The literature [2-6], released publicly within 2-3 months before the NeurIPS submission, should generally be considered concurrent work to ours. We could add a section in our revision, but we think it is unfair to evaluate our method based on them.

Thank you to the authors for their response. I have increased my rating to 5 and am willing to accept the paper if other reviewers strongly support it. However, I still consider it borderline due to several issues.

1. The new experiments using dino metrics demonstrate that the results are not completely biased, which is appreciated. However, the method performs worse than EDM2 and possibly other untested methods. For the revision, it is crucial to correct numerous inaccurate claims regarding the state-of-the-art performance, initially attributed to a biased FID evaluation. I recommend conducting additional experiments using unbiased GAN discriminators, such as non-initialized or diffusion GANs.

2. I acknowledge the distinction between DDIM and DDIM inversion for reconstruction training. However, current method also needs to employ the DDIM distillation loss for text to image. Additionally, the improvement between these two may not be significant once the biased GAN issue is addressed. I suggest rerunning the experiment shown in Figure 5(c) for the revision.

3. I appreciate the clarifications on the benefits of certain components in the upsampling process. However, it remains unclear how much more advantageous the classifier-free GAN is in a broader setting compared to the simpler, well-tested GAN loss with generated image as real data. Upsampling may not be necessary in real world setting. For generation up to 1K resolution, LDM is sufficient; for higher resolutions, training a separate, smaller super-resolution network is often more straightforward.

4. Concerning related works, most references are old enough according to NeurIPS policy. Additionally, it is important to discuss any literature that has influenced our understanding. The papers I mentioned, along with others, are closely related and warrant a more thorough discussion.

We deeply appreciate the reviewer for valuable and insightful feedback. We will revise the paper thoroughly reflecting all discussions. Below we further address the raised concerns.

Questions 1.

Ans. We would like to express our deepest gratitude to the reviewer for pointing out the bias in the FID. We fully agree with the reviewer's opinion on this matter and plan to conduct additional experiments using the unbiased GAN discriminator suggested by the reviewer, reflecting this in the final revision.

Question 2.

Ans. To further assess the effectiveness of the reconstruction (DDIM inversion) loss, we present our findings in Figure A. PaGoDA with reconstruction loss shows stronger performance in prompt alignment compared to PaGoDA without it, highlighting the benefits of incorporating reconstruction loss. Nevertheless, we plan to rerun the experiments presented in Figure 5-(c) for the revision, as suggested by the reviewer.

Question 3.

Ans. We agree with the reviewer. In a more practical scenario, as the reviewer said, it could be easier to generate higher-resolution samples by first generating 1K-resolution with LDM, and upscaling it into higher-resolution with a separate module. Although, we believe that PaGoDA could work effectively in the following cases.

• [Inference Speed] Due to the upsampling capability, PaGoDA can replace LDM for 1K generation with halved cost compared to the 1-step LCM, as illustrated in Figure 1. In the LDM framework, regardless of the number of steps taken to synthesize the latent representation, the generated latent must be decoded back to the pixel space. This additional decoder evaluation incurs computational costs nearly equivalent to those of U-Net evaluation, accounting for about 50% of the total computation in 1-step LCM generation. This inherent and fundamental limitation of the LDM framework is completely overcome in PaGoDA, which directly trains on the pixel space, enabling sampling at half the cost compared to the 1-step LCM.
• [Latent PaGoDA] If we want to keep the LDM framework for 1K generation, we can apply PaGoDA in the latent space of LDM to lighten the training load. Given that diffusion training accounts for most of the computational cost, PaGoDA's resource demands are nearly the same as those for diffusion training at resolution lower than the full latent dimension. In contrast, LDM requires training the diffusion model at the full latent resolution, which is considerably more costly. Therefore, PaGoDA offers a substantially reduced training budget compared to conventional LDM.

Question 4.

Ans. We agree with the reviewer and we will discuss related literature thoroughly in our paper revision.

[Final Comments from Authors] We are planning experiments with the aim of thoroughly addressing all the reviewer's concerns by diligently fulfilling the requests. Given these efforts in mind, we respectfully ask the reviewer if our paper still remains borderline.