Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation

We sincerely appreciate the reviewer's insightful comments and encouraging feedback regarding our motivation for pruning text encoders, our embedding-space-based similarity approach, the justification for using MSE over cosine similarity, and the clarity and validity of our experimental design and evaluations.

---

Question 1. Clarification on pruned model's unconditional score effectiveness and impact on FID when guiding original conditional scores.

Response: We believed that it is worth reporting and discussing (at least in the Discussion section) this interesting phenomenon of decreasing FIDs since this was also observed in many different contexts such as fine-tuning for concept erasing [1–3], distillation [4–5], quantization [6], or compression [7–8], despite an intuitive expectation of potential performance degradation. Furthermore, as you suggested, our experimental results, summarized in Table 1, indicate that guiding conditional embeddings from the Skrr-compressed model using unconditional embeddings from the dense model leads to degraded FID performance compared to employing both conditional and unconditional embeddings entirely from the Skrr-compressed model. A deeper exploration of this intriguing phenomenon and its underlying mechanisms would constitute compelling future work.

Table 1. FID and CLIP score with combinations of conditional / unconditional scores from dense / compressed models in PixArt-$\Sigma$.

[1] Lu, et al. Mace: Mass concept erasure in diffusion models. CVPR (2024).
[2] Zhang, et al. Forget-me-not: Learning to forget in text-to-image diffusion models. CVPR (2024).
[3] Lee, et al. Concept pinpoint eraser for text-to-image diffusion models via residual attention gate. ICLR (2025).
[4] Zhao, et al. MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices. In ECCV (2024).
[5] Feng, et al. Relational diffusion distillation for efficient image generation. ACM MM (2025).
[6] Li, et al. SVDQuant: Absorbing Outliers by Low-Rank Component for 4-Bit Diffusion Models. ICLR (2025).
[7] Yuan, et al. Ditfastattn: Attention compression for diffusion transformer models. NeruIPS (2024).
[8] Chen, et al. $\Delta$-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers. arXiv (2024).

---

Question 2. Typo error.

Response: Your understanding is correct, and the point you mentioned is indeed a typo. Thank you for pointing this out. The values of $\text{Metric}_1$ for (b) and $c$ in Figure 4 should be swapped.

---

Question 3. Experiments on extreme sparsity.

Response: Thank you for proposing this insightful experiment. We conducted evaluations under high sparsity levels beyond 50%, and these results are presented in Table 2. While other pruning methods produce images with severely compromised fidelity under such extreme sparsity, Skrr, despite exhibiting some degree of performance degradation, continues to generate images with relatively high fidelity that remain reasonably aligned with the textual descriptions. We will include these detailed experimental results in the supplementary materials of our revised manuscript.
Table 2. Quantitative results on sparsity over 50% with PixArt-$\Sigma$.

---

Question 4. FLOPs reduction in Skrr.

Response: Skrr not only reduces memory and parameters but also lowers FLOPs compared to the dense model, as shown in Table 2 of the main manuscript. While the Re-use mechanism slightly increase FLOPs relative to methods using pruning alone, the text encoder constitutes a minor fraction of the total T2I pipeline. As a result, Skrr reduces overall FLOPs by 0.04% compared to the dense model, with at most a 0.17% increase over other pruning approaches.

We sincerely appreciate the reviewer's insightful comments and encouraging feedback regarding the clarity of our proposed two-stage pruning method (Skip and Re-use), as well as our comprehensive evaluation using multiple widely recognized metrics and state-of-the-art diffusion models.

---

Question 1. Computational cost of beam search-based algorithm.

Response: Thank you for the valuable suggestion. As you correctly pointed out, beam search introduces additional computational overhead, and we have clearly articulated its time complexity with respect to the number of transformer blocks $L$ and beam size $k$ in Table 1. However, since $k$ is set to a relatively small value (specifically, $k=3$ in Skrr) compared to $L$ (with $L=48$ for T5-XXL), the additional computational overhead remains minimal. Furthermore, it is crucial to emphasize that this overhead occurs only once during the pruning stage when blocks are selected and does not recur during the actual image synthesis phase. Thus, the computational cost from beam search has negligible practical impact on the applicability and efficiency of Skrr. We will include a detailed analysis addressing this aspect explicitly in our revised manuscript.

Table 1. Time complexity of each block-wise pruning method for calibration. For Skrr, $k=3$.

---

Question 2. Clarification on statistical significance and stability of FID improvements observed under text encoder pruning.

Response: We emphasize that our evaluation of FID scores was conducted using the MS-COCO dataset with 30,000 prompts, a sufficiently large and widely used benchmark to ensure robustness and statistical reliability. The use of such an extensive dataset substantially reduces the likelihood that the observed improvements in FID scores at high sparsity are due to randomness or evaluation bias. Additionally, it should be noted that the calibration subset derived from CC12M and the MS-COCO dataset employed for FID evaluations are entirely disjoint, further diminishing concerns regarding dataset-induced bias.

We believed that it is worth reporting and discussing (at least in the Discussion section) this interesting phenomenon of decreasing FIDs since this was also observed in many different contexts such as fine-tuning for concept erasing [1–3], distillation [4–5], quantization [6], or compression [7–8], despite an intuitive expectation of potential performance degradation. A deeper exploration of this intriguing phenomenon and its underlying mechanisms would constitute compelling future work.

[1] Lu, et al. Mace: Mass concept erasure in diffusion models. CVPR (2024).
[2] Zhang, et al. Forget-me-not: Learning to forget in text-to-image diffusion models. CVPR (2024).
[3] Lee, et al. Concept pinpoint eraser for text-to-image diffusion models via residual attention gate. ICLR (2025).
[4] Zhao, et al. MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices. In ECCV (2024).
[5] Feng, et al. Relational diffusion distillation for efficient image generation. ACM MM (2025).
[6] Li, et al. SVDQuant: Absorbing Outliers by Low-Rank Component for 4-Bit Diffusion Models. ICLR (2025).
[7] Yuan, et al. Ditfastattn: Attention compression for diffusion transformer models. NeruIPS (2024).
[8] Chen, et al. $\Delta$-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers. arXiv (2024).

We sincerely appreciate the reviewer's insightful comments and encouraging feedback.

---

Question 1. Novelty claiming on task.

Response: We acknowledge the reviewer's comment regarding the overstatement of novelty. We appreciate this important point and will carefully revise our manuscript to more accurately reflect the nature of our contribution. Specifically, we will clarify that our primary novelty lies in effectively addressing the particular challenges involved in pruning text encoders within text-to-image (T2I) diffusion models, rather than implying broader methodological novelty.

---

Question 2. Evaluation on additional benchmark.

Response: GenEval was selected in this work since it is a widely adopted benchmark among prior T2I works, thus can provide direct comparisons with our proposed method. To relieve your concern for additional evaluation, we now evaluate our Skrr on two more benchmarks (Tables 1 and 2 below), T2I-CompBench [1] (generating 4 images per prompt, thus 3,600 images total) and DSG-1K [2] (generating 4 images per prompt, thus 4,240 images total), in BLIP-VQA score and DSG score, respectively. These new results are consistent with our prior evaluation, demonstrating Skrr’s robustness across diverse and rigorous benchmarks.
Table 1. Quantitative results on BLIP-VQA score in T2I-CompBench dataset with PixArt-$\Sigma$.

Table 2. Quantitative results on DSG score in DSG-1k dataset with PixArt-$\Sigma$.

[1] Huang, et al. T2I-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. NeurIPS (2023).
[2] Cho, et al. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. ICLR (2024).

---

Question 3. About Lipschitz continuity assumption.

Response: The assumption of Lipschitz continuity is widely used in deriving convergence rates and error bounds. Prior work has empirically shown that reasonably trained large transformer models satisfy this condition [3], and theoretical analyses have confirmed its validity in diffusion models [4]. Thus, our use of this assumption is well-justified, supporting the relevance of our theoretical framework to real-world T2I models.
[3] Khromov & Singh. Some Fundamental Aspects about Lipschitz Continuity of Neural Networks. ICLR (2024).
[4] Liang, et al. Unraveling the smoothness properties of diffusion models: A gaussian mixture perspective. arXiv (2024).

---

Question 4. Evaluation on FLUX.

Response: We provide extensive additional qualitative comparisons for stronger T2I models, such as SD3 and FLUX, in the appendix. Nevertheless, to directly address your concern regarding generalizability, we have also conducted quantitative evaluations on the FLUX.1-schnell model, and these results are presented in Table 3 below. The outcomes clearly demonstrate that our method preserves the performance of dense models in FLUX compares to other baselines, reinforcing our claim regarding the broad applicability and generalizability of our pruning strategy across multiple strong T2I diffusion models.

Table 3. Quantitative results on FLUX.1-schnell.

---

Question 5. Relationship between Figure 3 and T2I models.

Response: Figure 3 presents experimental results specifically obtained from the T5-XXL encoder, which is widely adopted in T2I models. Note that T5-XXL is independently trained from T2I models, so it is reasonable to investigate this component separately while still having a strong relevance to the T2I context. To avoid potential confusion, we will revise the text to explicitly highlight the connection to T2I.

We sincerely appreciate the reviewer's insightful comments and encouraging feedback regarding our method's superior performance over existing blockwise pruning techniques, theoretical support, and extensive experiments across diverse metrics.

---

Question 1. Evaluation on additional benchmark.

Response: GenEval was selected in this work since it is a widely adopted benchmark among prior T2I works, thus can provide direct comparisons with our proposed method. To relieve your concern for additional evaluation, we now evaluate our Skrr on two more benchmarks (Tables 1 and 2 below), T2I-CompBench [1] (generating 4 images per prompt, thus 3,600 images total) and DSG-1K [2] (generating 4 images per prompt, thus 4,240 images total), in BLIP-VQA score and DSG score, respectively. These new results are consistent with our prior evaluation, demonstrating Skrr’s robustness across diverse and rigorous benchmarks.

Table 1. Quantitative results on BLIP-VQA score in T2I-CompBench dataset with PixArt-$\Sigma$.

Table 2. Quantitative results on DSG score in DSG-1k dataset with PixArt-$\Sigma$.

[1] Huang, et al. T2I-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. NeurIPS (2023).
[2] Cho, et al. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. ICLR (2024).

---

Question 2. Clarification on Skrr's practical applicability considering performance relative to dense models.

Response: As you may know, true practicality does not only needs excellent performance, but also requires efficient memory usage and we believe that our proposed Skrr can provide a way to controling this trade-off while minimizing performance degradation. As shown in all quantitative results, including Table 1 and Table 2 in question 1, Skrr consistently achieves better performance than all baseline pruning techniques, even at high sparsity. Furthermore, qualitative examples in both the main text (Fig. 5) and appendix (Fig. A10-A16) clearly show that Skrr maintains visual fidelity and semantic alignment with the dense model. These results underscore Skrr’s strong practical applicability despite compression. Additionally, Skrr is entirely training-free, which enhances its practicality and distinguishes it as a meaningful contribution to the field.

---

Question 3. Practical utility considering memory efficiency versus performance trade-off in text encoder pruning.

Response: We would like to clarify that our method is training-free, thus there is no claim regarding training; instead, our approach is explicitly designed for inference scenarios. In practical situations where it is challenging to anticipate incoming text prompts, offline feature extraction becomes impractical. Consequently, reducing the size of the text encoder is essential for effective model deployment. Reviewer v9A7 also acknowledged this motivation, explicitly stating, "I agree with this claim as the size of the text encoder significantly affects the memory and size of the checkpoint in storage." Concerns regarding performance drops associated with memory-efficient designs have already been addressed above.

---

Question 4. Clarification on the representativeness and generalizability of the CC12M calibration dataset for text-to-image tasks.

Response: We would like to clarify that the CC12M dataset is widely recognized and extensively utilized for training and evaluating various T2I models, including Stable Diffusion 3. To address your concern about potential overfitting to our calibration dataset, we emphasize two key points: First, all evaluation metrics employed in our experiments—FID, CLIP, DreamSim, GenEval, T2I-CompBench, and DSG score—were computed on datasets entirely distinct and mutually exclusive from the calibration set. Consistent performance across these diverse benchmarks strongly suggests minimal risk of overfitting. Second, since all baseline methods evaluated in our experiments utilized the same calibration set for their respective pruning strategies, the fairness and comparability of our results remain intact.