You Only Sample Once: Taming One-Step Text-to-Image Synthesis by Self-Cooperative Diffusion GANs

Thank you for your time and effort in reviewing our paper. We hereby address the concerns below.

As mentioned in the paper, "It is hard to extend GANs on large-scale datasets due to training challenges," yet there are existing works that successfully train GANs with large-scale datasets (e.g., GigaGAN with 2.7 billion training image-prompt pairs, Diffusion2GAN with 12 million training image-prompt pairs). It is important to compare these methods.

We clarify that we aim to emphasize that extending pure GANs on large-scale datasets is challenging (instead of impossible), though there are existing works that successfully train GANs with large-scale datasets. In particular, despite its complicated design, GigaGAN takes 4,783 A100 days for training, which is too expensive to train. On the other hand, Diffusion2GAN is a diffusion-GAN hybrid model in a similar spirit of our YOSO. However, Diffusion2GAN requires a massive noise-image paired dataset, consuming a high cost of 15 A100 days not to mention their training cost, while our training cost via fine-tuning SD is less than 10 A800 days.

Why use the bottleneck layer of the UNet to compute the latent perceptual loss? Some works (e.g., DIFT, PnP, FasterDiffusion) suggest that encoder features are more important than both the encoder and bottleneck features.

Our empirical findings show that using bottleneck features works better than using encoder features in our context. Specifically, when using encoder features, the performance of YOSO-SD-LoRA would drop from 28.33 HPS to 27.95. This may be due to the perceptual loss being more dependent on features from "deeper layers", which differs from performing other downstream image tasks.

Regarding zero-shot FID on COCO dataset.

Previous work [a,b] suggests that zero-shot COCO FID may not be a good metric for evaluating modern text-to-image models. Pick-a-Pic [a] found that zero-shot COCO FID can even be negatively correlated with human preferences in certain cases. SDXL [b], which is widely known to be a more powerful generative model than SD 1.5, has a worse zero-shot COCO FID score than SD 1.5 (See Appendix F in [b]). In this work, we adopted multiple machine metrics that correlate with human preferences, including Image Reward, HPS, and AeS. These metrics actually correlate better with human preferences, as detailed in their respective papers [c,d], while our work shows the state-of-the-art performance of these metrics.

Still, we are glad to provide a comparison regarding zero-shot COCO FID as follows.

Results show that our model demonstrates competitive performance in both FID and CLIP scores when compared to modern text-to-image models and other diffusion distillation methods. This achievement is built upon our approach's superior performance in machine metrics that better represent human preferences.

It’s better to provide experiments comparing with exiting SOTA one-step generative methods (e.g., SwiftBrush) to prove that YOSO enhances the inference efficiency and high-quality generation.

Following your suggestion, we compared SwiftBursh in terms of zero-shot FID and CLip Score in the aforementioned table. The results show that YOSO is better in both metrics. Besides, we have compared YOSO to DMD [e] in our original paper, while DMD is considered as a stronger baseline compared to SwiftBrush. In particular, both DMD and SwiftBrush are developed based on variational score distillation [f], while DMD has an additional ODE regression loss for enhancing performance.

Thank you for your continued feedback. We appreciate the opportunity to clarify and address your concerns.

Why Use the Bottleneck Layer for Latent Perceptual Loss?

First, it is important to highlight that our contribution is proposing the latent perceptual loss (LPL) computed using features from a pre-trained latent diffusion model (DM), which contrasts with the traditional MSE loss. Our choice of using the bottleneck layer for computing LPL is motivated by its ability to provide a deeper and more compact feature representation. This compactness allows the perceptual loss to capture higher-level, abstract features, leading to improved perceptual quality in the generated images.

While the decoder layers are indeed deeper in terms of network depth, their outputs are closer to the original latent space. This proximity may cause the decoder features to focus more on reconstructing low-level details rather than capturing the high-level perceptual aspects, which might not be as beneficial for perceptual loss purposes.

Ablation Studies on Different Layers.

It is a good suggestion. We conducted additional ablation experiments using features from the encoder, bottleneck, and decoder layers individually, as well as combining all layers for computing the LPL. Due to the limited time during the rebuttal period, we based our experiments on the LCM method for efficiency, using LoRA for fine-tuning SD 1.5 with 3k training iterations.

The quantitative results are summarized below:

We also include visual comparisons among variants in Figure 11 located in the Appendix.

From both the quantitative results and qualitative comparisons, we observe:

• Using bottleneck features for computing LPL achieves the best performance, outperforming other variants in terms of HPS and CLIP Scores.
• LPL consistently outperforms MSE, regardless of which feature layer is used, demonstrating the effectiveness of the latent perceptual loss as a whole.
• While combining all layers (Encoder + Bottleneck + Decoder) does improve over using individual encoder or decoder layers, it does not surpass the performance of using the bottleneck layer alone. This suggests that the bottleneck layer captures the most relevant features for the perceptual loss in our context.

Regarding Evaluation Metrics. We would like to reiterate that zero-shot FID is not a reliable metric for evaluating text-to-image models, as discussed in prior works [a, b]. CLIP Score is a more appropriate metric in this context, and we have provided the additional CLIP Scores as you recommended.

[a] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, NeurIPS 2023.

[b] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, ICLR 2024.

Thank you for your valuable feedback. We are glad to provide further clarification.

I have a different perspective on the view that the output of decoder layers is closer to the original latent space. In SD, the decoder consists of multiple layers at different scales. Only the output of the final layer is closer to the original latent space, while the outputs of the other layers in the decoder are similar to the bottleneck's output and even contain richer information (see Figure 2 in [A] and Figure 3 in [B]). It would be more appropriate to use the outputs of different-scale layers in the decoder to compute the perceptual loss. Which scale layer's output from the decoder is used in the new table? The output of the final layer in the decoder is closer to the latent code and is not suitable for computing the perceptual loss.

Clarification on Our View of Outputs of Decoder Layers We clarify that our statement that "the outputs of decoder layers are closer to the original latent space" is made in comparison to the encoder layers or the bottleneck layer. Intuitively, as we progress deeper into the decoder, the feature representations tend to more closely approximate the original latent space. For instance, as observed in Figure 3 of [B], the features from decoder layer 4 are significantly closer to the original latent space compared to those from layer 1. Similarly, the features from layer 7 are closer to the original latent space than those from layer 4.

Decoder Layers Used in Our Experiments We compute the latent perceptual loss using all decoder layers except the last one.

We agree that decoder layers may contain richer details, but too many details remaining might make the network focus more on reconstructing low-level details when computing the perceptual loss, which could be detrimental to the calculation of perceptual loss.

To investigate this further, we conduct additional experiments in using different-scale decoder layers for computing the LPL, similar to the setting in [B]. The results are presented in the tables below.

It can be observed that using decoder layers=1 clearly performs the best among the different decoder layer configurations. This is different from the observation in [B], where they found layers=4-11 performs best in their context.

This is because [B] aims to generate images that are similar but not identical to the reference image by replacing the outputs/features of certain layers in the diffusion network. This direct replacement operation can only leverage the outputs/features themselves, making it necessary to preserve as much detail as possible (without leaking appearance) to retain the semantics of the reference image.

In our case, the LPL optimizes $||f(x)-f(\text{target})||_2^2$, where the student network is trained via backpropagation through $f(x)$. This allows it to not only leverage the features themselves but also utilize the gradient information of the perceptual network $f$ (i.e., the pre-trained diffusion). As a result, the layer selection that is suitable for [B] might lead the perceptual loss to overly focus on low-level details in our case.

More importantly, regardless of which layers are used to compute LPL, it is significantly better than MSE. This highlights the effectiveness of our proposed latent perceptual loss.

Thank you for your time and effort in reviewing our paper. We very much appreciate your acknowledgment of our proposed self-cooperative learning and the efficiency of our YOSO. We hereby address the concerns below.

Clarity of Key Contribution: This paper lacks clarity on whether YOSO is a new model architecture that can be trained from scratch or a step distillation methodology that should be applied on pre-trained models. Further elaboration, possibly with additional diagrams, would make the paper more accessible and clear.

As clearly stated in the abstract, YOSO can support both training from scratch and finetuning pre-trained diffusion models, since the loss for training YOSO does not depend on a pre-trained diffusion model. We have included the algorithm description of training YOSO from scratch in Algorithm 1 which is located in Appendix D in our original manuscript.

Experimental Scope and Robustness: This paper introduces YOSO both as a new model and a novel distillation techniques for pre-trained models. However, the former part was only evaluated on the CIFAR-10 dataset, additional benchmarks are necessary to substantiate its effectiveness across more diverse and complex datasets.

To further demonstrate that YOSO can serve as a new model trained from scratch, we conducted additional experiments on ImageNet-64 that adopt EDM architecture on both generator and discriminator for training YOSO from scratch. The results are shown in the following table:

Table A: Results obtained by training from scratch

It can be seen that YOSO has clearly better performance than CT and iCT, and achieves comparable performance to state-of-the-art diffusion models and GANs.

Novelty in Methodology: Besides self-cooperative learning, the adoption of decoupled scheduling, perceptual and consistency loss isn't novel to the field, and while the rationale is sound, these sections could benefit from more detailed comparison with previous works and how it is different from them.

Our proposed decoupled scheduler integrates the GAN loss and diffusion distillation loss (consistency loss) in a simple and effective way, which ensures that we just train one stage. In contrast, although some works [a, b] have combined GAN loss with diffusion distillation loss, without our decoupled scheduler, they have to train the model in multiple stages with the scheduler coupled between GAN loss and distillation loss. Moreover, the proposed Latent Perceptual Loss provides a method to efficiently compute perceptual loss in latent space using pre-trained diffusion models. In contrast, previous work [c] requires decoding latent through a VAE decoder and then computing LPIPS loss in data space, which is computationally expensive.

[a] SDXL-Lightning: Progressive Adversarial Diffusion Distillation, arXiv.

[b] Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis, NeurIPS 2024.

[c] InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation, ICLR 2024.

Thank you for your time and effort in reviewing our paper. We hereby address the concerns below.

There is a lack of important references in the Background, such as [1,2,3], which are all about combining Diffusion models and GANs. The differences between the proposed method and these methods [1,2,3] need to be emphasized and discussed.

Thanks for mentioning these excellent papers. We have discussed UFOGen [3] in our original manuscript, and we provide a discussion of [1,2] as follows. Similar to UFOGen [3], the method proposed in [1] injects noise into samples to stabilize adversarial training. This leads to subpar one-step generation learning efficiency. On the other hand, diffusion2GAN [2] relies on using diffusion models to construct numerous noise-image pairs (3M pairs in finetuning SD) for organizing regression loss to stabilize training, which is computationally expensive. Specifically, constructing 3M noise-image pairs using SD 1.5 requires 15 A100 days according to the data in InstaFlow, while our computational cost for fine-tuning SD using LoRA is less than 10 A800 days. This highlights the efficiency of our proposed method.

In the experimental part, it is also necessary to compare with these methods [1,2,3].

In the original paper, we compared UFOgen [3] on CIFAR-10, with results shown in Table 3 of the paper. Below we provide an experimental comparison with [1, 2] on CIFAR-10:

It can be seen that our approach achieves a significantly lower FID score of 1.81, representing a 43% improvement over the next best method, Diffusion2GAN [2] at 3.16.

There is no comparison with the current SOTA methods in terms of model complexity (such as the number of network parameters and training inference time).

In comparison with diffusion distillation methods, we consistently maintain the same backbone for the student model, ensuring that all methods have identical parameter counts and single-step inference costs. Moreover, our proposed YOSO just requires less than 10 A800 GPU days to fine-tune SD 1.5, while InstaFlow requires 199 A100 GPU days to fine-tune SD 1.5 and most other works (e.g., SD-Turbo [a], PeRFlow [b], and Hyper-SD [c]) do not report their training cost.

[a] Adversarial Diffusion Distillation, ECCV 2024.

[b] PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Accelerator, NeurIPS 2024.

[c] Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis, NeurIPS 2024.

In the TEXT-TO-IMAGE GENERATION experiment, there is no comparison with the GAN-based methods, such as [4]. It is also worth discussing whether the proposed method is better or more efficient than the GAN-based methods.

Our approach has a significant advantage in training costs compared to pure GANs, since we can efficiently fine-tune the pre-trained diffusion models. Specifically, while GigaGAN [4] requires 4,783 A100-days for training, we need less than 10 A800-days to fine-tune a pre-trained diffusion model (SD-v1.5 and PixArt-$\alpha$).

Regarding model complexity, the author should compare it like Table 1 and Table 2 in the UFOGen paper.

Please refer to Table A above. It can be seen that YOSO is one of the most efficient models.

A fair comparison should be conducted with GigaGAN under the same experimental conditions.

We provide a comparison of zero-shot COCO FID with GigaGAN. Please refer to Table A above.

References.

[a] Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation, NeurIPS 2023.

[b] Semi-Implicit Denoising Diffusion Models (SIDDMs), NeurIPS 2023.

[c] Maximum Likelihood Training of Score-Based Diffusion Models, NeurIPS 2021.

[d] Improved Techniques for Training Consistency Models, ICLR 2024.

[e] One-step Diffusion with Distribution Matching Distillation, CVPR 2024.

[f] SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation, CVPR 2024.

[g] InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation, ICLR 2024.

[h] Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis, NeurIPS 2024.

[i] Adversarial Diffusion Distillation, ECCV 2024.

[j] SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, ICLR 2024.

[k] Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis, arXiv.

[l] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation, NeurIPS 2023.

Thank you for your time and effort in reviewing our paper. We very much appreciate your insightful comments and your recognition of our work. We hereby address the concerns below.

Lack of the ablation study on training techniques. The paper introduces training techniques like Informative Prior Initialization (IPI) and v-prediction, which are valuable training methods that could also benefit other models. However, the lack of an ablation study on these components makes it difficult to assess their specific impact and to directly compare YOSO's performance with other models that could potentially leverage these techniques.

Thank you for acknowledging our technical contributions. We would like to emphasize that we have conducted comprehensive ablation studies on most of our proposed techniques (including IPI, Latent Perceptual Loss, Decoupled Scheduler, and Annealing Strategy) on text-to-image generation, with results presented in Table 4 of our paper. Here we conducted additional ablation experiments on quick adapt to v-prediction using PixArt-$\alpha$. We evaluate the variants in training PixArt-$\alpha$ on 5k training iterations and 1-step generation. The results are shown below:

It can be seen that using eps-prediction alone for one-step generation yields inferior performance, while naive v-prediction (directly using v-prediction in training) achieves decent results. However, quick adaptation to v-prediction demonstrates significantly better performance, validating the effectiveness of our proposed quick adaptation to v-prediction.

Hyperparameter sensitivity. The timestep difference between the teacher and student latents, a key factor in the Decoupled Scheduler, is a hyperparameter that directly impacts training stability and eventual performance. Does it need to be tuned for different datasets? Could you show how robust the method is to this hyperparameter?

Good comment. In all experiments in our paper, we consistently used 250 as the timestep difference between the teacher and student latents in our cooperative adversarial loss. This demonstrates that the hyperparameter is not required to be tuned among different settings. However, we are glad to provide experiments with varying timestep differences. Specifically, we used Lora finetuning SD-v1.5 with 32 batch size and 3000 iterations (note that we used significantly fewer computational resources compared to the YOSO-SD-lora shown in the paper, resulting in lower performance, but this does not affect the relative performance). We report the results below:

The results show that YOSO can achieve consistently good performance by varying the timestep difference.

In Eq (3): why did you use different lambda for different t?

This is because samples with higher noise have a lower signal-to-noise ratio, making it more difficult to reconstruct samples from them, resulting in higher variance in learning. Therefore, we need a weight that can specify the learning importance of different timesteps. It's worth noting that this approach is actually widely used in diffusion [a] and diffusion distillation [b,c] papers.

[a] Elucidating the Design Space of Diffusion-Based Generative Models, NeurIPS 2022.

[b] Consistency Models, ICML 2023.

[c] Improved Techniques for Training Consistency Models, ICLR 2024.

Thank you for your time and effort in reviewing our paper. We very much appreciate your acknowledgement of our proposed training strategies and the advantages over qualitative metrics. We hereby address the concerns below.

The presentation of the visualization results in this paper is somewhat insufficient, including a comparison with YOSO-PixArt-alpha and other Few-Step models, such as PixArt-delta, and it might be worth attempting to compare with the original models as well.

Thanks for your advice. We have included the visualization comparison to PixArt-$\alpha$ (original teacher model) and PixArt-$\delta$ on 1024 resolution in Figure 9 located in the appendix. As can be observed, our method produces significantly better image quality compared to PixArt-$\delta$ and achieves comparable results to the multi-step teacher model.

The abbreviation "NEF" is not explained.

The "NFE" denotes Number of Function Evaluations. We have explained this abbreviation in our revision.

Should the steps for Hyper-SD-LoRA and YOSO-LoRA in Table 2 be equal to 1?

Not really. Table 2 evaluates the performance of Hyper-SD-LoRA and YOSO-LoRA in both 1-step and 4-step generation.

Figure 5 does not specify the comparative configuration for model inference—does it keep the prompt unchanged while changing the same seed?

Yes. We keep the prompt unchanged while changing the random seed. The random seed is the same for each image among models.

Why were only SD-LoRA and PixArt-Full experiments conducted in the experimental setup, without including SD-Full and PixArt-LoRA related experiments?

In short, this is for fair comparison following the same experimental setup of existing work. Specifically, we found that many works [a,b] used LoRA fine-tuning on SD 1.5. For the sake of fair comparison, we also only applied LoRA fine-tuning for YOSO-SD. As for Pixart, previous works [c] used full fine-tuning, so we also adopted full fine-tuning to achieve the best possible results and ensure the fair comparison.

[a] Trajectory Consistency Distillation: Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping

[b] Hyper-SD: Trajectory Segmented Consistency Model for Efficient Image Synthesis, NeurIPS 2024.

[c] PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation, ECCV 2024.

I am curious about how the zero-shot one-step 1024 resolution is effective. As far as I know, 1024 has a different ar_size compared to 512, and the posemb scale is also different. Wouldn't these factors significantly impact the weight distribution of the 1024 model?

This is because the weights of the 1024 model are inherited from the 512 model, and there is a high similarity between their weights. We only merged the compatible model weights. In particular, the overall cosine similarity between the weight of 512 model and the weight of 1024 model is 0.9902. Based on this zero-shot adaptation capability to the 1024 model and YOSO's rapid convergence (requiring only around 20k iterations), we hypothesize that the weights learned by YOSO are primarily responsible for generation acceleration which activates some inherent capability within the pre-trained diffusion model. The reason why YOSO-SD-LoRA has some capability to adapt to ControlNet might also be similar. This might also explain why YOSO-SD-LoRA has some compatibility with ControlNet and different unseen customized models finetuned from SD.

Without retraining, will increasing the sampling steps, similar to LCM, improve the results? If further enhancements are desired for better generated image quality, what would be the subsequent optimization directions based on the proposed approach?

Yes. The YOSO can be used in different sampling steps. We have shown the performance of YOSO in both 1-step and 4-step generation in Table 2. The 4-step generation has a better HPS of 30.50 compared to the 1-step generation which has an HPS of 28.33. This indicates that YOSO can benefit from more sampling steps similar to LCM. To further improve image quality, we may need to develop crafted objectives for few-step generation, or combine with other advanced techniques, such as distribution matching via score distillation [d, e] and human feedback learning [f].

[d] ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation, NeurIPS 2023.

[e] Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models, NeurIPS 2023.

[f] ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation, NeurIPS 2023.