Thank you for thoroughly reviewing our work and for your insightful and helpful suggestions for improving our paper. We provide our response in the following. We have made every effort to address all your concerns. Please don’t hesitate to let us know if your concerns/questions are still not clearly resolved. We are open to active discussion and will do our best to address your concerns during the discussion period.

W1 & Q1. lacking important details & single precise definition of how the proposed approach actually performs

→ We apologize for any ambiguity regarding how the proposed distillation method performs. The main reason we explained the distillation part in almost two pages is that we needed to address two main aspects simultaneously: 1) pruning the SDXL-Base model and building efficient KOALA U-Nets, and 2) exploring the effective feature distillation strategy. Furthermore, our key finding for the distillation is that self-attention is the most crucial part. To provide sufficient evidence, we conducted in-depth qualitative (refer to Fig. 3) and quantitative (refer to Tab. 3 and Tab. 10) analyses, which took up considerable space in the manuscript.

In addition, as you suggested, we introduce a brief summary of self-attention-based distillation:

• For every transformer block composed of transformer layers at each stage, we allow the student model to mimic the self-attention features from the teacher model (refer to Fig. 2).
• At the highest feature level (e.g., DW-1 & UP-3), since there are no transformer blocks, we compare only the last output features at that stage (refer to Fig. 2 & Tab. 10d).

Since we don’t have enough space to address all of our findings (e.g., four findings) for the effective distillation strategy, we kept only two points in Sec. 3.1.2 and moved the other findings to Appendix B, along with Table 10. As a result, it may cause some details to be missing.

Let us describe more details of four findings for our distillation strategy:

1. Among different feature types for distillation, self-attention shows the best performance (Tab. 3a) and represents more discriminative features (Fig. 3 & Fig. 10 in Appendix).
2. Self-attention at the decoder (UP-1 to UP-3) has a larger impact on the quality of generated images than the encoder (DW-1 to DW-3). Therefore, we keep more transformer blocks in the decoder part when pruning SDXL-Base’s U-Net (Tab. 3b).
3. When comparing self-attention features, the features of the early layers in the transformer block, which is composed of multiple layers, are more significant for distillation (Tab. 10c). This is supported by the feature cosine similarity analysis, which shows that early layers exhibit more distinct feature discrimination (Sec. B3 & Fig. 11).
4. The combination of the last output features (LF) at the highest feature level and the self-attention features at the other stages shows the best results (Tab. 10d).

W2. the entire discussion on the U-Net architecture could be a single table.

→ Thank you for your suggestion. As you pointed out, we will consolidate the U-Net section into a single, more concise and clear table to make it more approachable and easier to understand.

We deeply appreciate your valuable feedback and the thoughtful reconsideration of our paper's score. We will make sure to carefully reflect on your suggestions and update the paper to improve its organization and conciseness. Once again, thank you very much for the time and effort you dedicated to thoroughly reading and reviewing our paper.

Thank you for thoroughly reviewing our work and for your insightful and helpful suggestions for improving our paper. We provide our response in the following. We have made every effort to address all your concerns. Please don’t hesitate to let us know if your concerns/questions are still not clearly resolved. We are open to active discussion and will do our best to address your concerns during the discussion period.

W1. focused on specifically to SDXL & would be appled to transformer-based models

→ Please refer to G1 section in the general response.

Q1. Confusion Between the Terms Block and Layer

→ The SDXL-Base architecture consists of convolutional residual blocks and transformer blocks, each of which is composed of multiple transformer layers. A potential point of confusion is the distinction between transformer blocks and transformer layers. Our intention is to use ‘block’ as a higher-level concept, referring to a group of ‘layers.’ Therefore, we will reiterate this definition in the manuscript to avoid any misunderstanding. Thank you for your comment. ”

Q2. SA+LF combination

→ We apologize for the confusion. The final feature combination for the distillation training is self-attention (SA) from all transformer layers plus the last feature (LF) from the highest feature level (DW1+UP3), as shown in Fig. 2 and Tab. 10d. Using only SA features already shows superior performance compared to using only LF in BK-SDM. However, we have found that adding the last features at DW-1 and UP-3, where there are no self-attention layers, further boosts performance. To explain the cause of this confusion, due to space constraints in the manuscript, we moved the optimal feature combination to the appendix along with Tab. 10(d) and missed mentioning SA+LF in the main text. We will clarify this and update the manuscript. We appreciate your comment once again.

Q3. "KOALA" acronym

→ Thank you for your kind advice. As you noted, the explanation of the "KOALA" acronym is either implicitly included in the figure captions or found later in the paper (lines 237-238). We will refine and move the explanation to the introduction section in the final version.

Q4. Tab.4

→ We intended to use the phrase “in the second row of the table” to indicate that the second data source is demonstrated in the second row of Table 4. However, as the reviewer noted, the current demonstrations might cause confusion for readers. Therefore, we will revise the notation (to ensure consistency of data source notation both in the table and the document) and clarify the phrasing in the final version.

We sincerely thank you for your thoughtful review and for considering the impact of Table 8 in your comment. If possible, we would greatly appreciate it if you could provide further details or explain which specific aspects of Table 8 failed to persuade you. We genuinely hope you might give us the chance to address any concerns. Once again, we are truly grateful for the time and effort you have dedicated to thoroughly reviewing our paper.

As we near the end of the discussion phase, we would like to once again express our sincere gratitude for the time and effort you dedicated to thoroughly reviewing our paper. Your detailed suggestions, such as clarifying notations and refining the formatting of experimental results, are greatly appreciated. We will carefully revise our manuscript to reflect your valuable feedback. Once again, thank you very much for your thoughtful and comprehensive review.

Thank you for thoroughly reviewing our work and for your insightful and helpful suggestions for improving our paper. We provide our response in the following. We have made every effort to address all your concerns. Please don’t hesitate to let us know if your concerns/questions are still not clearly resolved. We are open to active discussion and will do our best to address your concerns during the discussion period.

W1. extent of degradation in text rendering & quantitative comparison

→ Please refer to G2 section in the general response

W2. Comparison with training-free methods

→ Thank you for your valuable insight. To the best of our knowledge, very recent training-free acceleration methods for diffusion models leverage the redundancy of features over the denoising steps. These methods (e.g., DeepCache [1], Cache Me If You Can [2]) cache and retrieve features of previous steps for efficient inference of the current step. In this rebuttal, we compare our KOALA model with DeepCache, which has an official implementation available. Using their code, we measured the inference cost, memory footprint, latency, and performance in terms of HPSv2 and Compbench scores. The results are summarized in the table below. Compared to the original SDXL-Base, DeepCache achieves faster inference speed at the cost of performance and increased memory usage. We speculate that the increased memory usage of DeepCache is due to the caching and retrieval process of features, which requires additional memory. In contrast, when compressing the same SDXL-Base model, our KOALA model reduces both memory usage and latency while achieving superior performance compared to DeepCache. Additionally, due to the step-distilled teacher and longer training, our KOALA model is capable of generating more faithful images with fewer diffusion steps, resulting in lower latency. Furthermore, the training-free method is orthogonal to our approach, we expect that combining the training-free method with ours could exhibit a synergy effect, suggesting a promising future direction.

*Note: KOALA-SDXL-Base and KOALA-Lightning denote models trained with the SDXL-Base teacher (1st row of Tab. 5) for 100K iterations and with the SDXL-Lightning teacher (11th row of Tab. 6) for 500K iterations, respectively.

W3. Comparison with step-distillation methods

→ We would like to clarify that we have already compared the step-distillation methods, such as SDXL-Turbo and SDXL-Lightning, in Sec. 4.2 and Tab. 6 of our manuscript. Below, we re-summarize a comparison with step-distillation methods, including the more recent work, PCM [3], in the following table. We observe that, compared to SDXL-Lightning and PCM at a $1024^2$ resolution, our KOALA-Lightning-700M demonstrates competitive performance while achieving lower latency and memory usage with a model size that is $3\times$ smaller.

*Table note: SDXL-Turbo generates images with $512^2$ resolution while the other models use $1024^2$ resolution.

→ Furthermore, we have verified the synergy effect between the step-distillation method (e.g., PCM) and our KOALA backbones during this rebuttal period. We performed the step-distillation training using PCM with our KOALA backbones and obtained results as shown in the table below. Due to their efficiency, our KOALA backbones enable PCM to further boost speed-up with only a slight performance drop compared to PCM with the SDXL-Base backbone. Additionally, we presented qualitative comparisons of our PCM-KOALA models with PCM-SDXL-Base in Fig. 1 of the attached general_response.pdf file (Please see the PDF file).

*Table note: SDXL-Turbo and SDXL-Lightning do not provide training code, while PCM releases the official training code, allowing us to train PCM with our KOALA backbones.

Q1. a better teacher model should have greater potential to distill a superior student model.

→ We would like to clarify that in the main state-of-the-art comparison table (e.g., Tab. 6) of our manuscript, SDXL-Lightning shows better or comparable performance to SDXL-Base on HPSv2 and CompBench, respectively. Thus, SDXL-Lightning could be considered a better teacher model. Your statement that “a better teacher model should have greater potential to distill a superior student model” aligns with our findings. Indeed, we observed that the KOALA models distilled from SDXL-Lightning achieve better performance than those distilled from SDXL-Base as shown in Tab 3.

[1] Ma et al. DeepCache: Accelerating Diffusion Models for Free. CVPR 2024.
[2] Wimbauer et al. Cache Me if You Can: Accelerating Diffusion Models through Block Caching. CVPR 2024.
[3] Wang et al. Phased Consistency Model. Arxiv (2405.18407) 2024.

Thank you for thoroughly reviewing our work and for your insightful and helpful suggestions for improving our paper. We provide our response in the following. We have made every effort to address all your concerns. Please don’t hesitate to let us know if your concerns/questions are still not clearly resolved. We are open to active discussion and will do our best to address your concerns during the discussion period.

W1-1. specific to the SDXL U-Net and heuristics, not a generalizable approach

→ Please refer to G1 section in the general response.

W1-2. novelty and contributions to the academic community

→ Novelty: We believe that our self-attention-based distillation approach for text-to-image model compression is novel, supported by two key points.

• To the best of our knowledge, our approach is the first to identify that self-attention is the most crucial element for feature distillation in T2I model compression.
• The self-attention-based distillation approach can be generally applied across various architectural designs of diffusion models (e.g., U-Net-based and transformer-based) that consist of transformer blocks with self-attention layers.

→ Contribution: We believe that we have made two contributions to the academic community from the perspective of practical impact.

• An efficient T2I Backbone for Consumer-Grade GPUs:

  • Many downstream tasks, such as personalized image generation [1] and image-editing [2], leverage SDXL-Base as a de facto backbone due to its open-source model and superior performance. However, the SDXL model barely works on GPUs with more than 11GB of VRAM due to its substantial model cost, and its slower latency may cause delays in the outcomes of new research using it. To overcome this limitation, our efficient KOALA T2I backbones can serve as a cost-effective alternative, facilitating further downstream research, especially for those working in limited-resource environments. Furthermore, our efficient KOALA models can assist practitioners and designers by enabling rapid ideation and supporting their creativity with low latency in a consumer-grade GPU environment.

• An effective recipe of distillation for T2I model compression:

  • We presented three key lessons in our paper: i) self-attention-based distillation strategy, ii) data considerations, and iii) the influence of the teacher model. Additionally, through an in-depth analysis of each lesson, we offer general insights into popular U-Net-based diffusion models, such as the role and computational burden of each layer. We strongly believe that the lessons and insights we provide will benefit researchers and developers striving to make T2I models more efficient.
  • In the large language models (LLMs) field, larger models continue to emerge rapidly, and there have been many efforts to compress these models. Similarly, new and larger T2I models have also been developed. However, unlike in the LLM field, there have been relatively few attempts to compress T2I models due to factors such as the high cost of training (i.e., healing) and data scarcity (i.e., high-resolution and expensive copyright). In this context, we believe that our lessons can be particularly helpful for the community, especially for those working with resource constraints.
  • It is important to note that research providing crucial recipes for effective training, alongside proposing entirely novel methods, has historically played a significant role in revolutionizing their respective fields and boosting overall performance. For instance, Improved GANs [3] for GANs, MoCo-v3 [4] for self-supervised learning, DeiT [5] for transformers, and ConvNext [6] for CNNs have all made substantial impacts. Similarly, we hope our work brings valuable insights and benefits to the research community.

W2. Discussion on failure cases

→ Please refer to G2 section in the general response.

[1] Ruiz et al. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. ICLR 2023
[2] Zhang et al. Adding Conditional Control to Text-to-Image Diffusion Models. ICCV 2023
[3] Salimans et al. Improved Techniques for Training GANs, NeurIPS 2016
[4] Chen et al. An Empirical Study of Training Self-Supervised Vision Transformers. ICCV 2021
[5] Touvron et al. Training data-efficient image transformers & distillation through attention. ICML 2021
[6] Xie et al. A ConvNet for the 2020s. CVPR 2022

We are sincerely grateful for the time and effort you dedicated to thoroughly reading and reviewing our paper. Your feedback, including the emphasis on novelty and contribution, the generality of the methodology, and the discussion on failure cases, will be carefully reflected upon as we update the paper to enhance its completeness. Once again, thank you very much for your thoughtful and thorough review.