Diffusion-based Extreme Image Compression with Compressed Feature Initialization

Thank you for your time and constructive comments. We have revised the manuscript based on your comments and address the weaknesses and questions raised in your review below:

Response to Weakness 1:

While recent works have explored leveraging the robust generative capability of Stable Diffusion for better perceptual quality (e.g, DiffBIR [1] and DiffEIC [2]) or using residual diffusion for acceleration (e.g., ExposureDiffusion [3] and Resshift [4]), none have attempted to combine Stable Diffusion with residual diffusion. To the best of our knowledge, we are the first to successfully integrate Stable Diffusion into a residual diffusion framework.

Response to Weakness 2:

Thank you for raising this concern. We have carefully reviewed our experimental setup and found no issues with the implementation. The observed color deviations in Text+Sketch are expected, as this method reconstructs images solely from sketches and semantic text. This behavior is consistent with the results reported in its original paper. Additionally, the slight brightness bias observed in PerCo-20 is consistent with the results shown in the PerCo paper.

Response to Weakness 3:

Thank you for highlighting this concern. To address it, we have compared our baseline with DiffEIC. Note that the only difference between the two lies in the compression module. As shown in Fig. 6 and Table 2 (left) of the revised manuscript, the negligible differences in performance between DiffEIC and our baseline demonstrate that the choice of entropy model has minimal impact on overall performance. This confirms that the improvements in our method are primarily attributed to the proposed Relay Residual Diffusion (RRD) and Fixed-Step Fine-Tuning (FSFT) strategies, as further validated by the ablation study presented in Fig. 6 and Table 2 (left).

Table 2(left): The impact of RRD and FSFT on performance. Performance is represented by BD-rate (%), using DiffEIC as the anchor. Distortion metrics include PSNR, MS-SSIM, and SSIM. Perceptual metrics include DISTS, FID, KID, NIQE, and LPIPS. DS denotes the number of denoising steps. 2/5 denotes that we use 2 denoising steps for the two models with larger bpp and 5 steps for the remaining models. | Methods | DS | Distortion | Perception | Average | |----------------|-------|------------:|------------:|---------:| | Baseline | 50 | 7.4 | -1.8 | 2.8 | | +RRD | 2/5 | -31.0 | 12.7 | -9.1 | | +RRD+FSFT | 2/5 | -42.2 | -36.6 | -39.4 |

Response to Weakness 4:

Thank you for this valuable comment. Following your suggestion, we have reselected DiffEIC [2] as the anchor and added detailed notes to the table captions, as shown in Table 2 and Table 3 of the revised manuscript.

Table 3: BD-rate (%) for different methods on the CLIC2020 dataset with DiffEIC as the anchor. For distortion-oriented methods (i.e., BPG, VVC, and ELIC), we omit their perceptual metrics. The best results are highlighted in bold. | Methods | | | Perception | | | | Distortion | | Average | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | | DISTS | FID | KID | NIQE | LPIPS | PSNR | MS-SSIM | SSIM | | | BPG | - | - | - | - | - | -66.2 | -32.8 | -40.3 | - | | VVC | - | - | - | - | - | -77.8 | -51.3 | -58.6 | - | | ELIC | - | - | - | - | - | -82.7 | -54.6 | 66.7 | - | | HiFiC | 201.8 | 248.2 | 372.6 | -28.7 | 63.4 | -29.1 | 2.7 | 14.7 | 105.7 | | VQIR | 71.8 | 183.9 | 156.7 | 32.4 | 51.3 | 16.4 | 43.9 | 57.8 | 76.8 | | PerCo | 66.1 | 67.6 | 65.1 | 5.2 | 67.7 | 33.9 | 69.2 | 77.7 | 56.6 | | MS-ILLM | 28.5 | 40.9 | 34.6 | -85.4| -44.7| -75.4 | -44.7 | -38.5 | -21.5 | | RDEIC (Ours)| -17.9 | -18.3| -22.1| -83.7| -40.8| -61.3 | -32.7| -32.7| -38.7 |

Reference

[1] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu, Ben Fei, Bo Dai, Wanli Ouyang, Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior. arXiv preprint arXiv:2308.15070, 2023.

[2] Zhiyuan Li, Yanhui Zhou, Hao Wei, Chenyang Ge, and Jingwen Jiang. Towards extreme imagecompression with latent feature guidance and diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology, 2024.

[3] Yufei Wang, Yi Yu, Wenhan Yang, Lanqing Guo, Lap-Pui Chau, Alex C Kot and Bihan Wen. Exposurediffusion: Learning to expose for low-light image enhancement. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.

[4] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

Dear Reviewer X8Q6,

We hope you have had the opportunity to review our responses and clarifications. As the discussion period is nearing its conclusion, we would greatly appreciate it if you could confirm whether our updates have adequately addressed your concerns.

Thank you for your time and consideration.

Best regards,

The Authors

Thank you for your time and constructive comments. We have revised the manuscript based on your comments and address the weaknesses and questions raised in your review below:

Response to Weakness 1 & Question 1:

Following your suggestion, we have included additional ablation experiments on computational complexity. The proposed relay residual diffusion (RRD) framework enables image reconstruction with 5 or even 2 denoising steps, significantly improving the computational efficiency during decoding. As shown in Table 2 (right) of the revised manuscript, incorporating RRD reduces the denoising time by a factor of 10$\times$ to 25$\times$ compared to the baseline:

| Methods | DS | Denoising Time | Speedup | |---------------|:------:|:----------------------:|---------:| | Baseline | 50 | 4.349 ± 0.013 | 1× | | +RRD | 5 | 0.434 ± 0.002 | 10× | |+RRD | 2 | 0.173 ± 0.001 | 25× |

Additionally, the fixed-step fine-tuning strategy is purely a fine-tuning strategy and does not introduce any additional computational overhead during decoding. We have incorporated this discussion into the Ablations section.

Response to Weakness 2 & Question 2:

In the revised manuscript, we have expanded our comparative analysis by including additional baseline methods, namely the traditional compression standard BPG [1] and the VAE-based compression method ELIC [2]. These additions provide a broader context for evaluating the performance of our approach relative to both traditional and modern learning-based compression techniques.

Response to Weakness 3 & Question 3:

Thank you for raising this concern. To assess the robustness and generalization ability of RDEIC, we have conducted additional experiments on the larger MS-COCO 30k dataset, which comprises 30,000 images spanning a diverse range of categories and content types. This dataset was constructed by selecting the same images from the COCO2017 training set [3] as utilized in PerCo [4].

As shown in Fig. 9 of the revised manuscript, RDEIC maintains consistent performance across this expanded dataset, demonstrating its ability to generalize effectively to unseen data, even in scenarios with more diverse and challenging content. Additionally, visualized examples of reconstructed images are provided in Fig. 16 of the revised manuscript to further illustrate the robustness of our approach.

We have included this discussion in Appendix C of the revised manuscript.

Reference

[1] Fabrice Bellard. Bpg image format. 2014. URL https://bellard.org/bpg/

[2] Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.5718–5727, 2022.

[3] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209–1218, 2018.

[4] Marlene Careil, Matthew J. Muckley, Jakob Verbeek, and Stéphane Lathuilière. Towards image compression with perfect realism at ultra-low bitrates. In The Twelfth International Conference on Learning Representations, 2024.

Dear Reviewer PdRA,

Thank you very much for your positive feedback on our work! We hope you have had the chance to review our responses and clarifications. As the discussion period is nearing its conclusion, we would greatly appreciate it if you could confirm whether our updates have fully addressed your concerns.

Thank you again for your time and thoughtful review.

Best regards,

The Authors

Response to Question 1:

Diffusion-based extreme image compression methods are known for their exceptional performance in perceptual quality but often struggle to achieve high fidelity. As shown in Fig. 4 of the revised manuscript, diffusion-based approaches (solid lines) generally outperform other methods (dashed lines) in perceptual quality while exhibiting lower scores on fidelity metrics such as PSNR, MS-SSIM, and SSIM.

Within this context, the proposed RDEIC achieves notable fidelity improvements compared to existing diffusion-based methods, such as DiffEIC [1] and PerCO [2]. These improvements align with our stated motivation to address the fidelity limitations of diffusion-based extreme image compression methods. While the fidelity of our RDEIC may not yet surpass that of traditional or other learning-based methods, it remains a meaningful step forward in improving fidelity of diffusion-based approaches.

Response to Question 2:

First, in image compression, it is common practice not to specify which model’s results are used when presenting comparative experiments, as each model corresponds to a different compression ratio (measured in bpp ). By default, results from all models are included to provide a comprehensive comparison. For instance, each point on the performance curves in Fig. 4 represents a model trained with a specific $\lambda_r$ value. Additionally, in Table 2 and Table 3 of the revised manuscript, the BD-rate (%) is calculated based on the performance of all models.

Second, in all experiments, we set the guidance scale $\lambda_s=1$ by default unless otherwise specified. This clarification has been included in the revised manuscript.

Response to Question 3:

As stated in the clarification, 2/5 indicates that we use 2 denoising steps for the two models corresponding to larger bpp values and 5 steps for the remaining three models. The performance points from these five models collectively form the performance curve, serving as the anchor for comparison. In this table, distortion metrics include PSNR, MS-SSIM, and SSIM, while perceptual metrics include DISTS, FID, KID, NIQE, and LPIPS.

Follow your suggestion, we have added detailed notes to the table caption to clarify this information and included the performance curves in Fig. 6 of the revised manuscript to provide a clearer demonstration of the results.

Table 2: The impact of RRD and FSFT on performance (left) and speed (right). Performance is represented by BD-rate (%), using DiffEIC-50 as the anchor. Distortion metrics include PSNR, MS-SSIM, and SSIM. Perceptual metrics include DISTS, FID, KID, NIQE, and LPIPS. DS denotes the number of denoising steps. 2/5 denotes that we use 2 denoising steps for the two models with larger bpp and 5 steps for the remaining models. FSFT is a fine-tuning strategy that does not affect speed. | Methods | DS | Distortion | Perception | Average | | Methods | DS | Denoising Time | Speedup | |---------------|:------:|------------:|------------:|---------:|----------|---------------|:------:|:----------------------:|---------:| | Baseline | 50 | 7.4 | -1.8 | 2.8 | | Baseline | 50 | 4.349 ± 0.013 | 1× | | +RRD | 2/5 | -31.0 | 12.7 | -9.1 | | +RRD | 5 | 0.434 ± 0.002 | 10× | | +RRD+FSFT | 2/5 | -42.2 | -36.6 | -39.4 | |+RRD | 2 | 0.173 ± 0.001 | 25× |

Response to Question 4:

Thanks for pointing out this issue. For the two models corresponding to larger bpp values, we use 2 denoising steps, while for the remaining three models, we use 5 denoising steps. As shown in Fig. 4 of the revised manuscript, the performance points of these five models with different denoising steps collectively form the performance curve, serving as the anchor for calculating the BD-rate. To avoid unnecessary confusion, we have removed this column from Table 1 of the revised manuscript.

Reference

[1] Zhiyuan Li, Yanhui Zhou, Hao Wei, Chenyang Ge, and Jingwen Jiang. Towards extreme imagecompression with latent feature guidance and diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology, 2024.

[2] Marlene Careil, Matthew J. Muckley, Jakob Verbeek, and Stéphane Lathuilière. Towards image compression with perfect realism at ultra-low bitrates. In The Twelfth International Conference on Learning Representations, 2024.

Thank you for your time and constructive comments. We have revised the manuscript based on your comments and address the weaknesses and questions raised in your review below:

Response to Weakness 1:

Thank you for pointing out these relevant references. SeeSR [1] and PASD [2] embed the LR latent into the initial random noise at the terminal diffusion timestep N (1000) during inference, but still require numerous denoising steps for reconstruction (e.g., 50 in SeeSR and 20 in PASD). Resshift [3] constructs a Markov chain that transfers between degraded and target features by shifting the residual between them, substantially improving transition efficiency. However, ResShift's redesigned diffusion equation and noise schedule prevent it from leveraging the robust generative capability of pre-trained Stable Diffusion.

In contrast, our RDEIC directly derives a novel residual diffusion equation from Stable Diffusion’s original diffusion equation, enabling seamless integration of per-trained Stable Diffusion to leverage its robust generative capability. To the best of our knowledge, this is the first successful integration of Stable Diffusion into a residual diffusion framework.

We have incorporated this discussion into the Related Work section to clarify the novelty of our approach.

Response to Weakness 2:

Thanks for your insightful comment. Residual diffusion allows us to construct the starting point using a smaller timestep $N$ (300) instead of the terminal diffusion timestep $T$ (1000). As shown in Fig. 2(b), the resulting $\boldsymbol{z}_N$ retains most of the information from the compressed features $\boldsymbol{z}_c$, providing a strong foundation for detail generation. Additionally, starting from $N = 300$ naturally avoids the randomness and error accumulation associated with sampling from $n = 1000$ to $n = 300$.

For Stable Diffusion, we leverage its robust generative capability to achieve high perceptual reconstruction at extremely low bitrates. It is important to clarify that Stable Diffusion is not used to resolve the randomness caused by pure noise or improve fidelity but rather to enhance the perceptual quality of the reconstruction.

We have rephrased this content in the third paragraph of Sec. 1 to improve clarity and presentation.

Response to Weakness 3:

1. For Stable Diffusion, we also start from $\boldsymbol{z}_N$ rather than pure noise. Since $\boldsymbol{z}_N$ retains most of the information from the compressed feature and no additional control conditions (e.g., text) are applied, the direct output of Stable Diffusion is low-frequency images lacking high-frequency details.

2. Our controllable detail generation method (Eq. (11)) aligns in form with classifier-free guidance (CFG). At each denoising step, the predicted noise is decomposed into two components, and the reconstruction is controlled by adjusting the guidance scale $\lambda_s$. In this framework, Stable Diffusion corresponds to the case where $\lambda_s = 0$.

Reference

[1] Rongyuan Wu, Tao Yang, Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang. Seesr: Towards semantics-aware real-world image super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 25456–25467, 2024.

[2] Tao Yang, Rongyuan Wu, Peiran Ren, Xuansong Xie, and Lei Zhang. Pixel-aware stable diffusion for realistic image super-resolution and personalized stylization. arXiv preprint arXiv:2308.14469, 2023.

[3] Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by residual shifting. In Thirty-seventh Conference on Neural Information Processing Systems, 2023

Dear Reviewer Ejkn,

We hope you have had the opportunity to review our responses and clarifications. We would be grateful if you could confirm whether our updates have fully addressed your concerns. Should you have any further comments or questions, we would be more than happy to address them at your convenience.

Thank you once again for your valuable time and thoughtful feedback. We genuinely appreciate your efforts in reviewing our work.

Best regards,

The Authors

Thank you for your timely response and thoughtful feedback. We are glad to have clarified some of your concerns and would like to address the remaining points in detail.

1. Regarding Starting Point

First, after reviewing their papers and codebases, we found that the initial points in both methods are not adjustable as you mentioned. Specifically, in CCSR, the initial point is set to pure noise, as defined in its code:

x_T = torch.randn(shape, device=model.device, dtype=torch.float32).

AddNet, on the other hand, primarily focuses on controlling conditions during the denoising phase. Its initial point is similar to PASD and SeeSR, as reflected in its code:

parser.add_argument("--start_point", type=str, choices=['lr', 'noise'], default='lr') # LR Embedding Strategy, choose 'lr latent + 999 steps noise' as diffusion start point.

Second, these previous methods only predict noise and do not consider the residual between degraded and target features. In contrast, our approach directly uses compressed features with slight noise as the starting point during training, enabling the network to simultaneously remove both noise and residual. This ensures consistency between the training and testing phases, improving both efficiency and reconstruction quality. We believe this makes our work novel compared to these prior methods.

2. Regarding similarities to ResShift

We acknowledge that our work shares a conceptual similarity with ResShift in utilizing residual modeling. However, we extend this idea by incorporating the powerful generative capability of pre-trained text-to-image diffusion models like Stable Diffusion into the residual diffusion framework—an area that has not been explored in previous works.

It is important to note that Stable Diffusion is trained for noise prediction and cannot directly handle residuals. To leverage Stable Diffusion's robust generative capability while enabling the network to process residuals, we designed a new diffusion equation that satisfies the following conditions:

• The diffusion equation retains the same structure as Stable Diffusion's equation, allowing seamless integration and utilization of its generative capability.

• It can progressively add residuals in a manner analogous to noise addition.

• $\boldsymbol{z}_N = \sqrt{\bar{\alpha}_N} \boldsymbol{z}_c + \sqrt{1-\bar{\alpha}_N}\epsilon_N$, where $\boldsymbol{z}_c$ is the compressed features.

Specifically, we derived the following equation from Stable Diffusion’s diffusion framework:

• $\boldsymbol{z}_{n} = \sqrt{\bar{\alpha}_n} (\boldsymbol{z}_0 + \lambda \frac{\sqrt{1-\bar{\alpha}_n}}{\sqrt{\bar{\alpha}_n}} \boldsymbol{e}) + \sqrt{1-\bar{\alpha}_n} \epsilon_n = \sqrt{\bar{\alpha}_n} \boldsymbol{z}_0 + \sqrt{1-\bar{\alpha}_n} (\lambda \boldsymbol{e} + \epsilon_n)$, where $\lambda=\frac{\sqrt{\bar{\alpha}_N}}{\sqrt{1-\bar{\alpha}_N}}$.

This innovation allows the network to combine residual modeling with the robust generative capabilities of Stable Diffusion. Therefore, summarizing our work as “just transferring residual modeling to Stable Diffusion” does not fully capture the novelty and depth of our contributions.

Furthermore, while our implementation leverages Stable Diffusion, the underlying methodology and derivation are general and can be readily extended to other text-to-image diffusion models.

Our Perspective on Innovation

Finally, we believe that discussing innovation without considering the specific task is inappropriate. In this work, our goal is to introduce residual diffusion to overcome the efficiency and fidelity limitations of existing diffusion-based extreme image compression methods. During this process, we addressed several critical challenges, including:

• Integrating residual diffusion with pre-trained text-to-image diffusion models.

• Achieving end-to-end training of the compression module within a residual diffusion framework.

• Resolving inconsistencies between timestep-independent training and inference.

• Enabling diverse reconstruction outputs without compromising efficiency.

Experimental results demonstrate that our method achieves significant improvements in both reconstruction performance and computational efficiency. We believe that, in the field of image compression, our work is novel and makes meaningful contributions.

We hope this response addresses your concerns and clarifies the novelty and contributions of our work. Thank you again for your valuable feedback.

Thank you for your time and constructive comments. We have revised the manuscript based on your comments and address the weaknesses and questions raised in your review below:

Response to Weakness 1:

As stated in the clarification, our focus is on improving the diffusion process and training strategy rather than proposing a novel pipeline. In summary, our primary innovation are as follows:

• Relay Residual Diffusion : We propose a relay residual diffusion that effectively combines the efficiency of residual diffusion with the powerful generation capability of Stable Diffusion. **To the best of our knowledge, we are the first to successfully integrate Stable Diffusion into a residual diffusion framework. **

• Fixed-Step Fine-Tuning: We design a fixed-step fine-tuning strategy that eliminates the discrepancy between training and inference, significantly enhancing reconstruction performance.

• Controllable Detail Generation: We introduce a controllable detail generation method that enables a trade-off between smoothness and sharpness, allowing users to adjust the reconstruction results according to their preferences.

Given these contributions, we believe it is inappropriate to assess the innovation of this work solely based on the pipeline structure. Regarding comparisons with GLC, we regret that we could not include results due to the unavailability of GLC’s official code. However, our comparisons with PerCo, DiffEIC, and other methods are sufficient to demonstrate the superiority of our proposed RDEIC.

Response to Weakness 2:

Thanks for your comment. RDD is not mentioned in this paper; did you mean RRD?

Our RDEIC uses the same generative prior (Stable Diffusion) as DiffEIC. RRD is not a ``better generative model'' but rather the novel relay residual diffusion framework proposed in this paper, which combines the efficiency of residual diffusion with the robust generative capability of Stable Diffusion. Referring to Table 2(left) of the revised manuscript, applying the proposed RRD to DiffEIC would indeed improve its performance, as the difference between DiffEIC and our Baseline lies only in the compression module.

Table 2(left): The impact of RRD and FSFT on performance. Performance is represented by BD-rate (%), using DiffEIC as the anchor. Distortion metrics include PSNR, MS-SSIM, and SSIM. Perceptual metrics include DISTS, FID, KID, NIQE, and LPIPS. DS denotes the number of denoising steps. 2/5 denotes that we use 2 denoising steps for the two models with larger bpp and 5 steps for the remaining models. | Methods | DS | Distortion | Perception | Average | |----------------|-------|------------:|------------:|---------:| | Baseline | 50 | 7.4 | -1.8 | 2.8 | | +RRD | 2/5 | -31.0 | 12.7 | -9.1 | | +RRD+FSFT | 2/5 | -42.2 | -36.6 | -39.4 |

Response to Weakness 3:

Thank you for you insightful comment. In the revised manuscript, we have selected more appropriate visualization results in Fig. 1 and Fig. 5 to better illustrate the advantages of RDEIC.

Response to Weakness 4:

In the revised manuscript, we include the reconstruction performance of the SD autoencoder in Fig. 4 (indicated by the black horizontal line), which represents the theoretical upper limit of RDEIC’s performance. For bpp $>$ 0.06, the SD autoencoder performs worse than MS-ILLM in terms of NIQE, which explains why RDEIC also underperforms MS-ILLM on this metric.

Additionally, as NIQE is a no-reference metric, a lower NIQE score does not always indicate better reconstruction quality in the context of extreme image compression. For instance, Text+Sketch achieves the best NIQE score but produces reconstructions that significantly deviate from the original image.