DiffPC: Diffusion-based High Perceptual Fidelity Image Compression with Semantic Refinement

Thank you for your patient reading and feedback! Your recognition of the comprehensiveness of our experimental work and the novelty of our methods is truly motivating for us. We will strive to provide additional explanations of our approach to address any concerns you may have.

W1-2:Unclear definition, wrong format

R1: Thank you for pointing that out. Upon review, we have identified the issues you mentioned in the manuscript. We have made corrections according to standard definitions to enhance readability. Regarding the definitions of $c$ and $s$, as they have already been explained in Sec 2 and Sec 3.1 for consistency throughout the document, we have decided to maintain these identifiers unchanged. We appreciate your understanding

W3:Similar works that fine-tune pre-trained diffusion models for low-quality image enhancement include.

R2: Thank you for your feedback! The tasks investigated in the works referenced by [1] [2] [3] all focus on image inverse problems, which differ from the compression task addressed in our study. The commonality lies in the utilization of pre-trained diffusion models as priors.

Specifically, image inverse problems aim to recover the original image by fitting a degradation process under known conditions, whereas image compression tasks involve exploring more compact representations under known prior types to achieve an optimal rate-distortion trade-off, which entails complex bit allocation issues.

Furthermore, our proposed framework incorporates the IC-Control Net and the hybrid semantic refinement module, both of which are innovative structures. These unique components differentiate our work from those referenced articles, as they do not feature similar or identical structures.

Q1:In Line 45, are realism and perceptual fidelity truly equivalent concepts, or could their distinctions be further explored?

R3: Thank you for your feedback! We have referenced several previous studies [5] [6] and maintained this terminology. Actually, realism does not have a precise definition; we use the term "realism" to refer to images that achieve higher human ratings. Additionally, perceptual metrics such as LPIPS and DISTS significantly enhance human perceptual evaluations and serve as crucial tools for quantitatively assessing photorealism. Overall, we believe that equating realism with perceptual quality does not lead to confusion or unfairness in experimental results.

Q2:The experimental section demonstrates that PSNR and SSIM metrics for generative models are not optimal. Could some insightful explanations be provided for why this is the case?

R4: Thank you for your feedback! In Section 1, we mentioned: [4] highlighted the triple trade-off between distortion (PSNR), perceptual realism (perceptual metrics), and bit rate (bpp) at extremely low bitrates. [5] pointed out that at ultra-low bitrates, the significance of distortion (PSNR) as an evaluation metric diminishes, with higher PSNR values not necessarily indicating better image quality [6]. In our experiments in Section 4.2, we demonstrated the best perceptual quality and outperformed existing diffusion baselines regarding distortion metrics like PSNR.

In conclusion, our proposed approach achieves the optimal triple trade-off between perceptual realism, distortion, and bit rate.

Dear Reviewr suRn

Thank you once again for dedicating your valuable time to reviewing our paper and providing constructive comments! We humbly request your brief attention to our responses addressing your concerns. Any feedback you can provide would be immensely appreciated. Moreover, if your inquiries have been adequately addressed, we would be thankful if you could kindly consider adjusting your rating accordingly. Additionally, we remain at your disposal to address any further questions before the conclusion of the rebuttal period.

Sincerely,

The Authors

We highly value your initial feedback and understand that you may have a busy schedule. However, we kindly urge you to take a moment to review our responses to your concerns.

In the appendix A.7.3, we provide additional visual results on bit allocation, along with supplementary ablation studies, illustrating the contribution of our proposed multi-scale compressor to optimizing rate allocation. Our approach utilizes a pre-embedding module to decouple image semantics from quantization codewords, further enhancing the global stability of the diffusion prior by incorporating textual semantics. To the best of our knowledge, this has not been attempted in previous works leveraging diffusion priors.

In conclusion, we humbly request your response once again, as we are eager to engage in further discussion with you.

Dear Reviewer suRn,

We notice that all other reviewers have posted their post-rebuttal comments to our response but we still have not received any further information from you. We greatly appreciate your initial comments. We fully understand that you may be extremely busy at this time. However, we kindly request that you take a moment to review our responses addressing your concerns. Due to the passing of the deadline for manuscript revisions, we may not be able to supplement a substantial amount of experiments and visual results to address your concerns, for which we express deep regret. You may peruse our discussions with other reviewers, as this might aid in bolstering your confidence in our work.

In conclusion, we eagerly anticipate your response and are more than willing to engage in further discourse with you.

Best Regards,

Paper 409 Authors

Thank you for your response! We have noted your skepticism regarding the novelty of our work. Allow us to reiterate our contributions and clarify the significant distinctions between our research and the works you referenced [1] [2] [3].

We have introduced a novel compression framework featuring a multi-scale compressor that adjusts rate allocation based on the prior of a pre-trained T-I (Text to Image) LDM. This approach effectively reduces the bit rate while preserving high-frequency details, enabling more realistic reconstructions at the decoding stage. Further, our innovative use of a pre-embedding module decouples high-level semantics from image prompts, and combined with textual prompts, it generates a hybrid semantics, fully leveraging the generative potential of the T-I conditional latent diffusion model.

Compared to works [1], [2], and [3], our research not only differs in the tasks undertaken but also exhibits significant distinctions in the technical paths and details involved in utilizing diffusion models:

• [1] focuses on enhancing medical images through a diffusion model that employs the DDPM architecture in the image domain. This work addresses a super-resolution task requiring a known degradation matrix $H$. Furthermore, it does not leverage the prior knowledge of pre-trained models, opting for full training on the target dataset. Additionally, this study does not utilize any conditional prompts.
• [2] explores low-light image enhancement using the DDPM model, proposing a novel analytical sampling process and training objectives based on a known exposure process. Since our work does not stem from such tasks and lacks known degradation priors to underpin the training of the diffusion model, our training objectives, sampling processes, and the diffusion model backbone employed are markedly different.
• [3] involves image enhancement using a pre-trained LDM model, which first learns the degradation process through a restoration network and subsequently samples from the pre-trained LDM's prior to further enhancing the restored images. In contrast, our work targets compression tasks, where the degradation process is complex and cannot be simulated through a simple network as in super-resolution tasks. Moreover, [3] does not incorporate any high-level semantic prompts, which we believe inevitably leads to suboptimal results.

In summary, there are clear differences in applying diffusion frameworks across various tasks. We believe it is unreasonable to diminish the novelty of our work solely due to the shared use of a diffusion model backbone; however, we deeply respect your viewpoints and comments. We would be most grateful if you are willing to engage in further discussion.

[1] Ma, J., Zhu, Y., You, C., & Wang, B. (2023) in "Pre-trained Diffusion Models for Plug-and-Play Medical Image Enhancement" (MICCAI).

[2] Wang, Y., Yu, Y., Yang, W., Guo, L., Chau, L. P., Kot, A. C., & Wen, B. (2023) in "ExposureDiffusion: Learning to Expose for Low-Light Image Enhancement" (ICCV).

[3] Lin, X., He, J., Chen, Z., Lyu, Z., Dai, B., Yu, F., ... & Dong, C. (2023) in "DiffBIR: Towards Blind Image Restoration with Generative Diffusion Prior".

We greatly appreciate your constructive suggestions and recognition of the innovativeness of our work! Your acknowledgment of the significance and comprehensiveness of our experiments is truly inspiring! Your feedback contributes to the refinement of our endeavors, and we will exert utmost diligence in addressing your inquiries and dispelling any uncertainties.

W1: The authors do not describe some representative perceptual image compression methods corresponding to their work in the related work.

R1: Thank you for your suggestions. We have thoroughly considered these relevant works and incorporated them into our related work section.

W2,3,4,5,7: typo error & issue of article clarity

R2: Thank you for your careful reading and feedback! We apologize for any reading obstacles caused by typo errors. Following your advice, we have included citations to relevant works in the revised version, corrected the corresponding descriptions. This includes changes to Fig. 2 and Fig. 9, as well as adjusting all references from TAC to TAD.

W6:Please compare your method with VQGAN-based compression (DCC2024), TACO (ICML2024), and DiffEIC (IEEE TCSVT2024) and add them to Figures 4 and 5. VQGAN-based IC performs extreme image compression using codebook prior. TACO uses the text prior and DiffEIC uses the diffusion prior, which are very relevant to your work.

R3: Your suggestions are quite meaningful and assist us in further refining our experiments. Following your advice, we have incorporated comparisons with these latest baselines in Figures 4 and 5. Here, we present some of the results:

Table 1. Tests performed on CLIC20. The metric is BD-rate（%）, based on DIffPC. A higher BD-rate indicates a larger gap with DiffPC.

We conducted our experiments using the official provided code and checkpoints. Observations on the CLIC20 dataset reveal that our proposed DiffPC consistently outperforms the baseline schemes in perceptual metrics. While DiffPC shows slightly lower performance in pixel-level distortion metrics (PSNR) compared to TACO, it exhibits significant advantages in statistical fidelity and perceptual fidelity.

We have plotted the test results of [3] in Figures 4 and 5. Due to the relatively aggressive fine-tuning approach employed by the VQGAN based [3], the available bitrate range is limited, making it challenging to select sufficient data points for reliable BD-rate computation. Furthermore, we have noticed that if the bitrate approaches or exceeds the maximum bitrate range accommodated by [3], the performance of VQGAN may even deteriorate as the bitrate increases, indicating concerns regarding its bitrate generalization performance. In contrast, DiffPC demonstrates stronger bitrate generalization capabilities and achieves comparable ultra-low bitrate performance to VQ-GAN based on less training data (0.08Million vs. 1.2Million).

[1]Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior. IEEE TCSVT 2024.

[2]Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity. ICML 2024

[3]Extreme Image Compression using fine-tuned VQGANs. Data Compression Conference 2024

Dear Reviewr 1ruM

Thank you once again for dedicating your valuable time to reviewing our paper and providing constructive comments! As the end of the discussion period approaches, we kindly ask if our responses have satisfactorily addressed your concerns. Your feedback would be greatly appreciated, and we would be delighted to engage in further discussions if needed.

Sincerely,

The Authors

I think you need to go over your manuscript again and make sure there are no mistakes. I have improved the score and wish you good luck.

Q11: What impact do multi-scale features have on performance? In Eq. (12), the loss is calculated based on the difference between $z_0$ and $c$ , where multi-scale features do not appear to be necessary. Additionally, the authors have not demonstrated the role or necessity of multi-scale features in the ablation study. Clarifying these points would strengthen the paper’s argument for including multi-scale features.

R4: We sincerely appreciate your constructive suggestions. To address your concerns, we will first present quantitative analyses through ablation experiments on the multi-scale feature structure:

Table 2. The ablation experiments were conducted on the DIV2K dataset, utilizing the BD-rate (%) metric with DiffPC as the baseline.

It is evident that the removal of the Multi-feature component diminishes the model's performance. While this reduction may lead to a decrease in the model's parameter count, we deem it worthwhile to introduce a modest number of parameters to achieve a stable enhancement in performance. Furthermore, we have integrated the ablation experiments into Figure 8 in the main text and appended an analysis of this module.

Subsequently, we present our perspective on this multi-scale structure.

• Primarily, the utilization of multi-scale features, as demonstrated in various other works [1] [2], has been validated for its role in broadening the receptive field and preserving high-frequency information. Our intent is to augment the perceptual range of the encoding end towards the original image by amalgamating multi-scale features.
• Furthermore, the original design intent of Stable diffusion for image generation directly results in the fabrication of high-frequency details in the generated images, rather than preserving them from the original image. These high-frequency signals are discarded during the encoding process by $\mathcal{E}(\cdot)$, whereas our designed multi-scale structure aims to fully recover these high-frequency details.
• Your insightful observation regarding "the loss is calculated based on the difference between $z_0$ and $c$, where multi-scale features do not appear to be necessary" is astute. Indeed, in high-bitrate scenarios, multi-scale structures may not be essential as these additional high-frequency signals are likely to be discarded again due to the constraints of the optimization loss. However, it is imperative to note that our rate-distortion loss is just a part of our training optimization objective, and these extra high-frequency signals are beneficial for optimizing the likelihood bound $\mathcal{L}_{CSD}$ of the denoising model. Particularly in low-bitrate scenarios, where the distortion constraints are relatively weak, details originally contained in the signal $c$ are significantly discarded. Through the multi-scale structure, we aim to complement these high-frequency details, prompting the compressor to focus more on textures rather than low-frequency semantics. This is crucial as the low-frequency semantics are compensated for in subsequent stages through diffusion priors and semantic enhancements in the second phase.

[1] Li, Feng, et al. "Lite detr: An interleaved multi-scale encoder for efficient detr." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[2] Chang, Jia-Ren, and Yong-Sheng Chen. "Pyramid stereo matching network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

Q12: In Eq. (13), how is the initial value of $w$, specifically $\sigma^2_{z_0}$, determined in practice? Additionally, how do you ensure that the trainable hyperparameter $w$ correctly learns to achieve precise bit allocation?

R7: Thank you for your comments. In fact, $\sigma^2_{z_0}$ is modeled during encoding in the analytical encoder $\mathcal{E}(\cdot)$ of Stable diffusion. $\mathcal{E}(\cdot)$ outputs both the mean and variance of the variable $z_0$, yet during training and sampling, typically only the mean is used as an estimate for the variable $z_0$. Hence, acquiring $\sigma^2_{z_0}$ is straightforward. Furthermore, we initialize the parameter $w$ with $\sigma^2_{z_0}$. During training, $w$ is updated; in the initial stages, the model focuses on rate-distortion optimization, resulting in minimal fluctuations in the value of $w$. As training progresses, slight updates are made to $w$ to align with the optimization objective of the diffusion loss $\mathcal{L}_{CSD}$. Overall, throughout the training process, $w$ consistently approximates the most accurate bit allocation, requiring no additional adjustments.

We greatly appreciate your constructive suggestions and recognition of the innovation in our work! Your acknowledgment of the innovation and comprehensiveness of our experiments is truly inspiring. The suggestion you made regarding additional ablation experiments has sparked our enthusiasm. We will exert our utmost efforts to address your inquiries and alleviate any concerns you may have.

W1: The paper lacks a thorough comparison and analysis of computational complexity. As diffusion models inherently introduce high decoding complexity, it is essential to evaluate and compare this aspect against previous SOTA methods.

R1: Thank you for your constructive suggestions! To alleviate your concerns, we have included comparative experiments on model complexity with state-of-the-art solutions in the "Joint Response about Model Complexity." These experiments have also been added to Appendix Sec A.5. For a more detailed discussion, you can refer to our revised version.

W2: Other existing image compression methods [1, 2] could be discussed and compared.

R2: Thank you very much for your suggestions; they are invaluable in helping us enhance our experiments further. Following your advice, we have incorporated comparisons with these latest baselines. Here, we present some of the results:

Table 1. Tests performed on CLIC20. The metric is BD-rate（%）, based on DIffPC. A higher BD-rate indicates a larger gap with DiffPC.

We conducted our experiments using the official provided code and checkpoints. It can be observed that on the CLIC20 dataset, our proposed DiffPC consistently outperforms the baseline solutions in perceptual metrics. While DiffPC may lag slightly behind TACO in pixel-level distortion metrics (PSNR), it exhibits significant advantages in statistical fidelity and perceptual fidelity. Furthermore, we have included the above experiments in the revised experimental section, where you can find more results in Sec 4.2.

[1] Zhiyuan Li, Yanhui Zhou, Hao Wei, Chenyang Ge, and Jingwen Jiang. Towards extreme image compression with latent feature guidance and diffusion prior. IEEE Transactions on Circuits and Systems for Video Technology, 2024.

[2] Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, and Jaeho Lee. Neural image compression with text-guided encoding for both pixel-level and perceptual fidelity. In International Conference on Machine Learning, 2024

Q1-9: Typo error & writing clarity.

R3: Once again, we express our gratitude for taking the time to review our paper. In the revised version, we have rectified all the errors you mentioned and have clarified some ambiguous statements and images. Next, we will provide specific responses to some of the issues raised.

Q4: ...Removing the TAC ... , should TAC actually be TAD ?

R: This is a typo error; we have now corrected all instances of "TAC" in the images and sections to "TAD."

Q6: Is Eq. (3) accurate? $\mathcal{D}$ is the predicting noise rather than the mean

R: Thank you for pointing that out. We have revised the wording and replaced the denoising network $\mathcal{D}(\cdot)$ with the neural network $\mathcal{M}(\cdot)$, providing a more precise explanation.

Q7: In Eq. (13), what does $\eta$ denote in $\mathbf{TAD}_{\eta}$?

R: Here, $\eta$ represents the parameters of the network $\mathbf{TAD}{\eta}$, emphasizing that $\mathbf{TAD}{\eta}$ is a neural network with trainable parameters.

Q10: The authors mention that the importance-weighted loss assigns more bits to texture details, enabling a more precise reconstruction of these features. Could the authors provide a visualization result of the bit allocation to support this conclusion?

R3: Thank you very much for your insightful suggestions! In the revised version of the paper, we have provided visualizations of bit allocation in the Sec A.7.3, confirming the effectiveness of the importance-weighted loss proposed. Compared to not using $\mathcal{L}_{imp}$ , DiffPC allocates fewer bits in flat regions (such as the sky) and concentrates bits more in areas with complex textures.

Dear Reviewer GPEP

We deeply value your initial feedback and understand that you may have a busy schedule. However, we kindly request that you take a moment to review our responses to your concerns. Any feedback you can provide would be greatly appreciated. We are also available to address any additional questions before the rebuttal period concludes.

Sincerely,

The Authors

Thank you for your response! We are grateful for your recognition of the originality of our working methods and appreciate your efforts to enhance our work!

Q2: I would like to understand the performance comparison between directly using VAE compression and DiffPC.

R3: Thank you for your inquiry. We are uncertain if we have fully grasped your query; we present two interpretations:

1. What distinguishes the efficacy of our proposal from other VAE-based compression schemes?
2. How would the performance of DiffPC alter if the foundational denoising network were removed?

• Regarding the first interpretation, the baseline model we compare, ELIC, is a neural compressor entirely structured upon VAE principles. Through experimentation, the significant performance gap between ELIC and DiffPC becomes readily apparent.
• For the second interpretation, we have conducted ablation experiments to assist in clarifying any uncertainties you may have.

Table 1. Remove noise network performance comparison. Tested on KodaK24 with a BD-rate (%) based on DiffPC (Ours)。

It can be observed that the removal of the denoising network would have a catastrophic impact on overall performance, as the diffusion model prior we leverage stems directly from the denoising network. In low bit-rate scenarios, utilizing the prior from a pre-trained model can significantly enhance the quality of reconstruction [4]. If we have misconstrued your inquiry, please inform us, and we will promptly address your concerns. If you believe that additional experiments would be beneficial to address this issue, please let us know, and we will promptly provide further clarification for you.

We trust that our explanations and experimental data can alleviate your apprehensions. We believe that compression schemes based on diffusion models hold great promise, whether at the image level or in compression tasks geared towards downstream applications, as diffusion models can offer robust priors as a powerful aid.

[4] Careil, Marlene, et al. "Towards image compression with perfect realism at ultra-low bitrates." The Twelfth International Conference on Learning Representations. 2024

Thank you for your constructive feedback! We have observed that your concerns do not lie in the technical details of our approach but rather in the skepticism towards the use of pre-trained diffusion models in the technical roadmap. We would be more than willing to offer qualitative and quantitative experiments and explanations to help you better understand our framework.

W1: While pre-trained T2I diffusion models can enhance the quality of compressed images, their high computational cost makes them an unacceptable solution for image compression.

R1: We fully understand your concerns; indeed, multiple sampling does introduce significant decoding latency. We sincerely apologize for not analyzing the decoding complexity in the initial draft of the article. You can refer to the additional experiments included in our "Joint Response about Model Complexity" where we elucidate that the complexity of our approach is not "unacceptable." You can observe that our proposed solution maintains a comparable encoding latency on the encoding side and can reduce decoding latency by decreasing the sampling steps. Even with a sampling step of 50, we do not consider the decoding speed of DiffPC to be unacceptable. This is because the decoding latency of our approach is on par with Cheng20 and superior to the diffusion-based baseline Perco.

In conclusion, we deem our diffusion-based framework to be profoundly significant, particularly in addressing the challenge of cold data storage. As elucidated in our manuscript, numerous enterprises house vast repositories of cold data—information that sees infrequent access but necessitates algorithms with high compression rates to mitigate storage expenditures. Put differently, in this scenario, prioritizing high compression rates and fidelity over real-time decoding proves to be more imperative.

Q1：Although latent diffusion can enhance semantic details, it can also compromise some structural details, such as small faces and small text, which are not demonstrated in the paper.

R2: Thank you very much for your inquiry. We understand your concerns and apologize for not addressing this issue adequately in the main text. In response to your query, we will provide qualitative experiments to elucidate. In the revised version's appendix, we present extra visual results regarding the "small faces" and "small text" you mentioned in Sec A.6.

Next, we will address your question in the context of "low bit rates" mentioned in the paper.

• Firstly, concerning the classification of "text", we categorize it into large, medium, and small classes. In the realm of small text, even the state-of-the-art neural compressors like ELIC, based on traditional rate-distortion optimization, fail to reconstruct legible text. Similarly, Gan-based methods [2,3] also fall short. With the premise of lost textual semantics during decoding, our proposed DiffPC decoding yields images with sharper edges and enhanced perceptual quality.
• In the classification of 'medium text', all generative compression schemes exhibit some degree of text distortion. However, notably, GAN-based approaches [1,2] display more pronounced artifacts and structural deficiencies. In contrast, DiffPC, while maximizing the preservation of textual semantics, maintains the details and textures of the remaining parts. In the category of 'large text', all methods can successfully restore high-fidelity textual information.

Similarly, we categorize faces into large, medium, and small classes, as depicted in the Fig.14.

• In the large and medium categories, DiffPC maintains remarkable structural consistency without distorted details. Conversely, Gan-based approaches exhibit structural distortions, while ELIC shows severe blurring, making facial recognition challenging.
• In the small face category, both Gan-based methods and ELIC demonstrate significant distortion and structural disarray. In contrast, DiffPC, at the expense of a certain level of semantic consistency, enhances overall structural coherence and texture details.

Past research indicates a triad trade-off of distortion, realism, and bit rate in compression schemes under low bit-rate scenarios [3]. For the small text and small face scenarios you mentioned, existing compression approaches fall short of providing perfect reconstruction. However, we believe that our proposed DiffPC strikes the best balance between distortion and realism in these contexts.

[1]Mentzer, Fabian, et al. "High-fidelity generative image compression." Advances in Neural Information Processing Systems 33 (2020): 11913-11924

[2]Muckley, Matthew J., et al. "Improving statistical fidelity for neural image compression with implicit local likelihood models." International Conference on Machine Learning. PMLR, 2023.

[3]Yochai Blau and Tomer Michaeli. Rethinking lossy compression: The rate-distortion-perception

Dear Reviewer HiqS

Thank you once again for dedicating your valuable time to reviewing our paper and providing constructive comments! As the end of the discussion period approaches, we kindly ask if our responses have satisfactorily addressed your concerns. If you have any further inquiries regarding our utilization of the diffusion model approach, or if you feel that your feedback has been misconstrued by us, please do not hesitate to inform us. We are more than willing to engage in timely discussions with you.Furthermore, if your questions have been addressed, we would be grateful if you could consider updating your rating accordingly.

Sincerely,

The Authors

We deeply value your initial feedback and understand that you may have a busy schedule. However, we kindly request that you take a moment to review our responses to your concerns.

Utilizing the prior knowledge of diffusion models for image compression holds significant promise, and some works [1] [2] [3] in this domain have garnered widespread recognition. We present additional visual results in the appendix, where you can clearly observe that our approach maintains exceptional fidelity even at ultra-low bit rates. Furthermore, the complexity of diffusion models is not sensitive to image resolution, which allows our method to demonstrate superior encoding and decoding speeds compared to sequence autoregressive entropy models when handling high-resolution images.

In conclusion, we kindly request your response once more, as we are eager to engage in further discussion with you.

[1] Towards Extreme Image Compression with Latent Feature Guidance and Diffusion Prior. IEEE TCSVT 2024.

[2] Careil, Marlene, et al. "Towards image compression with perfect realism at ultra-low bitrates." The Twelfth International Conference on Learning Representations. 2024

[3] Yang, Ruihan, and Stephan Mandt. "Lossy image compression with conditional diffusion models." Advances in Neural Information Processing Systems 36 (2024).