Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation

We thank the reviewer for recognizing the state-of-the-art performance of the proposed SiD-LSG.

Novelty of LSG: We would like to emphasize that while CFG has been utilized previously, its application to the training of the fake score network represents a novel aspect of our Long-Short Guidance (LSG) strategy. This combination of SiD and LSG has proven effective in achieving a better compromise between low FID and high CLIP scores, as further illustrated in the newly added Table 3 and Figure 9 in the Appendix.

Training Efficiency: It is essential to clarify that generating fake images and incorporating CFG for distillation are standard practices among existing distillation methods. As depicted in Figures 2-5, SiD-LSG swiftly generates photo-realistic images and continues to lower FID and enhance CLIP scores at an approximately exponential rate before signs of plateauing appear. A key advantage of SiD-LSG is its seemingly consistent improvement as computational resources increase—there is no observed risk of FID (CLIP) scores worsening after certain iterations. However, due to the exponential rate of improvement (approximately log-log linear in FID (CLIP) vs. iterations) observed in our experiments, the rate of progress naturally diminishes over time in linear terms.

Role of CFG: As illustrated in Figure 3 (Right) and the newly added Table 3 in the Appendix, our model functions without CFG but yields unsatisfactory CLIP scores, indicating suboptimal alignment of the generated images with the provided text prompts. This supports prior observations on the crucial role of CFG in improving text-image alignment in established text-image diffusion models, such as SD1.5 and SD2.1-base. Given that SiD-LSG distills from these models, its dependence on CFG is anticipated.

CFG in FLUX: Regarding the question about FLUX, it is essential to note that FLUX is not yet fully open-access, and no detailed technical report is currently available that elaborates on its design, training, and inference details. Our response is based on publicly available information.

Based on available resources, FLUX still relies on CFG to enhance its text-image alignments:

• FLUX.1 [pro] allows a guidance scale between 2 and 5, which "Controls the balance between adherence to the text prompt and image quality/diversity. Higher values make the output more closely match the prompt but may reduce overall image quality. Lower values allow for more creative freedom but might produce results less relevant to the prompt," according to https://replicate.com/black-forest-labs/flux-pro.
• FLUX.1 [dev] allows a guidance scale between 0 and 10, as stated on https://replicate.com/black-forest-labs/flux-dev. FLUX.1 [dev], described as an open-weight, guidance-distilled model for non-commercial applications on https://blackforestlabs.ai/, is directly distilled from FLUX.1 [pro] and offers similar quality and prompt adherence capabilities while being more efficient than a standard model of the same size.
• FLUX.1 [schnell] is noted as a guidance and step-distilled variant on https://github.com/black-forest-labs/flux.

It would be intriguing to explore whether SiD-LSG could be applied to distill FLUX [pro], [dev], and [schnell]. Theoretically, SiD-LSG is well-suited for this task as it does not require access to the training data used for FLUX. Practically, we must consider the licensing implications of FLUX and address technical challenges such as adapting from the Distributed Data Parallel (DDP) framework, which SiD-LSG currently uses, to alternative distributed training frameworks that allow for model splitting across GPUs. We are keen to pursue this exploration in our future work, contingent upon a clear understanding of licensing implications and the availability of computing resources.

We appreciate the reviewer's insightful summary and detailed discussion on both the long and short CFG strategies. We address the raised points as follows:

Point 1: A prevailing challenge in current text-to-image diffusion models and their distilled variants lies in optimizing distribution matching—characterized by low FID and robust generation diversity—and aligning generated images with provided text prompts, which is reflected in high CLIP scores. In existing literature, it is not uncommon to emphasize the FID for models trained or inferred under a low CFG scale, while using a model under a high CFG to report CLIP and visualize generations. This practice, which we explicitly avoided, often obscures the true performance trade-offs.

In our research, we consistently report both FID and CLIP for every model we distill, directly addressing this contradiction. We introduce Long-Short Guidance (LSG) as a method to mitigate, although not entirely resolve, this issue. Our results demonstrate that LSG achieves a more balanced performance between low FID and high CLIP compared to both long and short CFG strategies.

The underlying causes of this contradiction likely differ among model types:

• Teacher Diffusion Models (e.g., SD1.5 and SD2.1-base): The high-guidance scale during inference forces these models to concentrate on high-density regions of the data distribution conditioned on text prompts, achieving lower FIDs but at the expense of reduced diversity and poorer distribution matching in sparser regions.
• Data-Free Distillation Methods: Typically using long CFG strategies, these models emulate the behavior of the CFG-enhanced teacher, leading to similar challenges.
• Adversarial Boosted Distillation Methods (e.g., ADD by Sauer et al., 2023b): These methods typically perform clearly worse without their adversarial components. They are prone to adversarial loss inducing mode collapse or seeking behaviors, which can significantly compromise diversity and negatively impact FID.

Point 2: We thank the reviewer for their valuable suggestions. To more effectively demonstrate the effectiveness of our proposed LSG, we have incorporated Table 3 and Figure 9 in the Appendix. Drawing an analogy between CLIP vs. FID and the True Positive Rate vs. False Positive Rate in a Receiver Operating Characteristic (ROC) curve, Figure 9 illustrates that LSG achieves the largest Area Under the CLIP-FID Curve. This underscores its superiority over both the Long and Short CFG strategies.

LSG applies CFGs to the evaluation of the teacher and to both the training and evaluation of the fake score network, guiding the learning of the student generator. As CFGs are not utilized in reverse sampling but only in guiding the learning of the fake score and the single-step generators, they provide opportunities to better preserve the matching of the data distribution.

Thank you for the author’s response, which addressed many of my concerns. I’ve decided to keep the score.

We appreciate the reviewer for providing a clear list of the contributions of this paper and thank you for your support.

We would like to highlight a novel aspect of this work: the introduction of long and short CFG strategies, which deviate from the traditional long strategy typically employed in this field. To illustrate this, Table 3 and Figure 9 newly added into the Appendix demonstrate that the proposed Long-Short Guidance (LSG) achieves a superior balance in maximizing CLIP scores and minimizing FID, as evidenced by a larger area under the CLIP-FID curve compared to other strategies.

Our focus was on the well-benchmarked SD1.5 and SD2.1-base, and we hope that our open-source code and model checkpoints will enable others to extend our method to both existing and future diffusion/flow-based generative models. We plan to further enhance SiD-LSG, with anticipated demonstrations of these improvements not only on SD1.5 and SD2.1-base but also on more advanced models such as Stable Diffusion XL/3.5 and other open-source diffusion/flow-matching models based on Unet/DiT architectures.

The current approach is also applicable to v-prediction. Using the Diffuser library to transition from epsilon prediction to v-prediction, the only two modifications involve changing the noise schedule from DDPM to DDIM and adjusting the loss for the fake score network as follows:

#Sudo code:
images = G_theta(z, contexts).detach()
noise_fake = sid_sd_denoise(unet=fake_score_ddp, images=images, noise=noise, contexts=contexts, timesteps=timesteps,
                            noise_scheduler=noise_scheduler,
                            text_encoder=text_encoder, tokenizer=tokenizer, 
                            resolution=resolution, dtype=dtype, predict_x0=False, guidance_scale=cfg_train_fake)
if noise_scheduler.config.prediction_type == "v_prediction":
    target = noise_scheduler.get_velocity(images, noise, timesteps)
    loss = (noise_fake - target) ** 2
    snr = compute_snr(noise_scheduler, timesteps)
    loss = loss * snr / (snr + 1)
else:
    loss = (noise_fake - noise) ** 2

loss = loss.sum().mul(loss_scaling / batch_gpu_total)

Under this adaptation, the SiD-LSG algorithm can then be directly applied to distill Stable-Diffusion-2-1 that uses v-prediction.

Thank you for your constructive feedback. Before addressing each of your concerns in detail, we’d like to clarify a key misunderstanding regarding SwiftBrush, which we had initially found unclear ourselves. Specifically, we sought clarification from the authors on their statement: "During training, a guidance scale of 4.5 is used for both teachers."

The authors confirmed that “Classifier-Free Guidance (CFG) was not applied to the LoRA teacher in Equation 6. We directly use the output from the LoRA teacher and subtract it with random Gaussian noise ε.”

In other words, CFG in SwifthBrush is used during the inference phase of the fake score network but is not applied during training (the "training" in SwiftBrush refers to the training of the generator). Thus, we stand by our claim that we are the first to incorporate CFG into the training of the fake score network.

Thank you for reviewing our response and posing additional questions, which we are pleased to address.

1. It is important to highlight that there is no equivalent of $\kappa_3$ in the Diff-Instruct/DMD/SwiftBrush loss frameworks.

   At the beginning of our study, we tested various combinations of $\kappa_{1,2,3,4}$ and quickly eliminated those that converged too slowly or diverged. Specifically, the combination $\kappa_1=1$, $\kappa_2=\kappa_3>1$, and $\kappa_4 \in${$\kappa_2,1$} was discarded early due to poor performance, as depicted in Figure 11 newly added into the Appendix.

2. We will further clarify the distinctions between long, short, and long-short guidance strategies in our next revision.

3. We emphasize that SwiftBrushv2 is not entirely a data-free method, as it requires more than just access to the teacher model and user-provided text prompts. Unlike SwiftBrush, which is data-free but yields a higher FID, SwiftBrushv2 substantially improves upon its predecessor. However, it achieves this through extensive engineering efforts, including initialization from SD-Turbo—a model trained with adversarial diffusion distillation that requires real data access, thereby disqualifying it as data-free.

• SwiftBrushv2 employs several enhancements to lower its FID from above 15 to 8.77, and then to 8.14. These modifications could also benefit SiD-LSG, which had already achieved an FID of 8.15 without them:

  • Leveraging pretrained weights from SD-Turbo to initialize the student network within SwiftBrush’s training framework.
  • Augmenting its prompts with data not only from the 1.5M Journey DB but also from 2M entries from LAION.
  • Introducing a clamped CLIP loss to directly boost the CLIP score. Although this undoubtedly increases the CLIP score, it remains uncertain whether it genuinely enhances text-image alignment as significantly as the improved score suggests.
  • Applying post-training integration of weights from two differently trained models, which involves training the models separately and merging their weights afterward.
  • Incorporating DMD-like image regularization with a human-feedback dataset of 200K images from LAION-Aesthetic-6.25+ to further decrease the FID from 8.77 to 8.14, making SwiftBrushv2 categorically not data-free.

• We observe that SiD-LSG's FID of 8.15 not only outperforms DMD but also DMD2, which has an FID of 8.34. DMD2 introduces several techniques to enhance DMD, these could potentially benefit SiD-LSG as well.

• In summary, SwiftBrushv2 and DMD2, which we regard as concurrent works, explore enhancements to their base models derived from Diff-Instruct/variational score distillation by incorporating various additional techniques. In contrast, SiD-LSG concentrates on elevating the base data-free SiD model, derived from a mode-based Fisher divergence, to a strong contender in the field of text-to-image generation, without relying on these extra efforts. Consequently, SiD-LSG and the approaches used in SwiftBrushv2 and DMD2 are naturally complementary to each other.

4. Regarding precision and computational efficiency, although we have achieved state-of-the-art results under FP32 with acceptable computing demand, our experiments with FP16 showed faster convergence but typically resulted in FID scores more than 2 points worse than those achieved with FP32. Our current hypothesis is that adjusting the loss scaling for both the generator and the fake score network might help prevent numerical under/overflow, commonly encountered in FP16 environments. Presently, we have not applied loss scaling. Alternatively, employing PyTorch autocast and GradScaler for mixed precision could potentially enhance performance. We consider these adjustments less critical at this stage but plan to explore them in future research. Additionally, we observe that using FP16 for inference, as is standard in SiD-LSG when reporting FID and CLIP scores, does not adversely affect performance.

Thank you the authors for your response. I see better the difference between your long and short CFG mechanism and the one used in SwiftBrush.

1. So in SwiftBrush, $\kappa_1 = 1$ and $\kappa_2 = \kappa_3 > 1$, while in your method $\kappa_1 = \kappa_2 = \kappa_3 > 1$. I am not sure if the difference has a significant effect. Can you provide the results of your system when using the same cfg mechanism used in SwiftBrush with $\kappa_1 = 1$, $\kappa_2 = \kappa_3 > 1$, $\kappa_4$ is either 1 or equals to $\kappa_2$, which is not provided in your paper?

2. The further discussion text for the long/short/long and short strategies should be moved to the main text. Also, I still find the definitions of these strategies quite forced.

3. SwiftBrushv2 has a data-free version with FID 8.77. Its best version, with FID 8.14, only employs a small set of 200k images for regularization, which DMD did. DMD2 starts with a data-free pipeline (before adding the GAN loss), though its data-free version seems to have bad quality.

4. I see that you can greatly improve the training efficiency with FP16. However, according to Fig. 10 in the Appendix, your FP16 versions have worse FID scores than the FP32 ones. I assume the numbers reported in Table 1 are with FP32. Could you provide the key results with FP16 to see how much performance drops when employing that efficient training?