Accelerate High-Quality Diffusion Models with Inner Loop Feedback

1. My main concern is about writing, there are a lot to be improved, I just list some examples here:

• Avoid using verbal expressions and make the sentences concise. For example, “we already know that” (L96), “one does not need to store an entire additional set of models weights” (L92), “This is clearly not optimal” (L107), “feed its output features to the feedback” (module?) (L68), “for this caching” (L215) etc.
• Make the long sentences logically fluent. E.g., “Different diffusion models have different learned weight values, and training quickly, without overfitting, is quite challenging.” The first part of the sentence does not seem to be related to the second part.
• Always give a statement followed by an explanation. E.g., “One cannot train this feedback mechanism with basic random initialization; the magnitude of the feedback will be too large, and the training will diverge.” Why is the magnitude too large? If it’s experimental observation you should refer to the experiment section for why it fails and how to fix it.
• Be specific and always support with data. E.g., “ILF is sometimes even better than the 20 step baseline”.

2. The motivation in L127 mentioned “redundancy” but not explained where it comes from. Is the motivation trying to improve caching since it has drawbacks (L107).

3. In the abstract “While many efforts to optimize diffusion models focus on achieving acceptable image quality in extremely few steps (1-4 steps), our emphasis is on matching best case results (typically achieved in 20 steps) while significantly reducing runtime.” Could you explain how “acceptable image quality” differs from “matching best case result”? Or is the emphasis that you are not using extremely few steps?

4. Is it possible to reduce the steps further? The design seems to be flexible to adjust the caching/skipping.

5. In fig. 8, why does image reward drop after a certain time for different feedback schedules?

Table 1 shows that the performance improvement of ILF over the baseline method with steps=2 is not always significant.

In the Fast Approximate Distillation section, presenting the final noise expression with a formula would improve readability.

• The method is examined on only transformer based networks, which limits the evaluation of the approach.
• The method is compared to caching that uses UNet architecture, making the comparison less effective.
• Line 264 there is a typo, please rephrase.
• The approach fine-tunes the network to predict score different than the one used in training, what is the advantage of that? also, how does it compare to fine-tuning the network itself without the feedback using this setup?
• The method is compared only to DPM-Solver++, and most methods benchmark ImageNet-64x64, which is not reported. Alternatively, you may report MS COCO-30k to align your evaluation with the literature.
• The method can be viewed as consistency model or even a self-distillation based approach, I was expecting to see comparison to SOTA methods from these classes.

• Comparison to Step Distillation Approaches: ILF introduces a self-distillation approach to accelerate the inference of diffusion models. Since this concept has been explored in previous works [1,2,3], a comprehensive comparison with these methods is essential. However, ILF is primarily compared to a caching-based approach that does not involve step-skipping training. The only comparison to another distillation method is a brief qualitative ablation (Fig. 11) with just three samples. This raises a few concerns: (i) The experiment is poorly documented, and it is unclear which specific method was used. Although additional details are mentioned as being in the appendix, I could not find them. Could the authors please refer me to where these details are? (ii) As noted in lines 469-472, the results are reported to be similar to ILF. This highlights the need for a more thorough comparison, including a numerical evaluation of performance and inference runtime of ILF against other step distillation methods. In summary, I believe including some of these approaches ([1,2,3]) as baselines in the main results of the paper is essential.

• Results of Less Denoising Steps Baseline: When observing Tab. 1, one can notice that ILF offers a relatively minor improvement over a simple baseline which simply uses less denoising steps. This baseline performs similarly in both prompt-aware metrics and FID but does not require any additional data or model training, whereas ILF reportedly requires 100 H100 GPU hours (as mentioned in L343 and L480) making ILF a less favorable option. Furthermore, this baseline is excluded from the qualitative comparisons in Fig. 7. Could the authors please add these additional qualitative results to Fig. 7?

• Inconsistency of Experimental Section: The experimental section presented several variations of the proposed ILF. For instance, the main quantitative results (Tab. 1) use a different acceleration rate than the main qualitative comparisons (Fig. 7), and yet another rate is used for the ablation in Table 3. While I understand that these are all different choices for hyper-parameters, there should be some consistency, particularly between the main qualitative and quantitative results. Otherwise, it becomes very difficult to interpret the results presented.

• Quality of Writing: Some parts of the paper are poorly written, and hard to follow. The main issue for me is in the introduction section, which should be revised. Additionally, lines 350-360 and 469-472, both describing experimental settings, are specifically unclear.

[1] Liu, Xingchao, et al. "Instaflow: One step is enough for high-quality diffusion-based text-to-image generation." The Twelfth International Conference on Learning Representations. 2023.

[2] Sauer, Axel, et al. "Adversarial diffusion distillation." European Conference on Computer Vision. Springer, Cham, 2025.

[3] Meng, Chenlin, et al. "On distillation of guided diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.