Towards image compression with perfect realism at ultra-low bitrates

We thank the reviewer for their time and constructive comments and address the questions below:

Though the perceptual quality is SOTA, the decoding complexity is not compared. It is better to show the decoding complexity since diffusion models tend to be slower. Comparing the decoding complexity (i.e. latency and flops) with previous perceptual image compresion methods can provide more helpful information to the community.

As stated in the joint answer, we performed tests for encode and decode speed and added a table with the results to the Appendix.

Missing citations. Following Hoogeboom et al 2023, previous SOTA PO-ELIC should be compared or at least discussed if it is hard to make direct comparison: He, Dailan, et al. "PO-ELIC: Perception-Oriented Efficient Learned Image Coding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Thank you for pointing out to this relevant reference. In the related work section of the revised manuscript, we add a discussion on PO-ELIC which builds upon ELIC by adding a mixture of losses including adversarial ones to achieve better perceptual quality. They evaluate at bigger bitrates than us (their minimum bitrate is 0.075), and they do not provide code so it is difficult for us to include it in our benchmark. We already compare to three other adversarial-based codecs: HifiC as well as MS-ILLM and Multi-Realism which are more recent than PO-ELIC.

Other Suggestions. The idea of conditioning the generative codec on image representation along with an extra descriptor (which is losslessly coded) is very interesting. I noticed that the following work provides a theoretical perspective on this conditional coding setting and also explains that the perceptual quality measured by FID is not affected by bitrate, please consider discussing this to provide more information to readers. Xu, Tongda, et al. "Conditional Perceptual Quality Preserving Image Compression." arXiv preprint arXiv:2308.08154 (2023).

Thank you for providing this relevant reference. This work introduces the notion of conditional perceptual quality, by extending perceptual quality to be conditioned on a specific side information. Given that we condition our generative codec on a losslessly encoded text description with the objective of achieving better preservation of text and local semantic information (measured by CLIP score/mIoU) and optimal image quality (FID), our work is very related to the idea of designing an optimal conditional perceptual quality preserving codec. We include this discussion in the related work.

We thank the reviewer for their time and constructive comments and address the questions below:

I am not really sure what is the point of evaluating LPIPS in Fig.3. Obviously MS-ILLM and HiFiC are trained with LPIPS, and this brings unfair advantage over the proposed approach. And as the authors are optimizing their approach towards divergence based perceptual quality [Blau 2018], using FID and KID should be enough. Reporting MS-SSIM and mIoU is also weird.

We agree with the reviewer that our goal is to design a perceptual codec and that FID/KID should be sufficient to evaluate the perceptual quality of our model. As we wanted to compare in the setting of RDP theory [Blau 2019], we included an evaluation on image similarity metrics LPIPS and MS-SSIM because they are widely used in the compression litterature. As rightly pointed out, LPIPS and MS-SSIM are not very meaningful in the low bitrate regime (see Fig. 14 in Appendix), so we included mIoU and CLIP score as they offer alternative ways of assessing distribution matching (mIoU having spatial localization and CLIP using a different image recognition mechanism). In the revised manuscript we further clarify the selection of metrics for evaluation.

The authors think achieveing a low MS-SSIM, PSNR and LPIPS is a weakness, which I can not agree. By RDP theory [Blau 2019], it is absolutely normal for perceptual codec being outperformed on image-wise distortion metrics.

We agree with the reviewer. We include this discussion in 'Limitations' paragraph of the paper.

The authors are enouraged to present results in their original aspect ratio. Currently some results in Fig. 1 looks weird as they are squeezed horizontally.

Thanks for the valuable remark. We update all figures in the paper with their original aspect ratio.

In MS-ILLM, the VQ-VAE is forzen once after pre-trained. While in this paper the VQ-VAE participate the end-to-end training. Then a natural question to ask is whether include the VQ-VAE of MS-ILLM in training improves the result?

In our model, we use an autoencoder that maps RGB images to the latent space of the diffusion model. This autoencoder is pretrained and frozen when training our models and we train a small hyper-encoder in the latent space for vector quantization (see paragraph 'Implementation details' in Section 4.1). In MS-ILLM, the VQ-VAE is only used as a continuous-to-discrete mapping for the purpose of training an adversarial loss. Our model does not use an adversarial loss, so our assessment is the two are not comparable. If we misunderstood the reviewer’s intent, we could reconsider a rephrasing of this request.

A recent preprint [Conditional Perceptual Quality Preserving Image Compression] also justifies the adoptation of side information in image compression. The authors can probably connect their work to rate-distortion-perception trade-off with the approach similar the in preprint.

Thank you for providing this relevant reference. This work introduces the notion of conditional perceptual quality, by extending perceptual quality to be conditioned on a specific side information. Given that we condition our generative codec on a losslessly encoded text description with the objective of achieving better preservation of text and local semantic information (measured by CLIP score/mIoU) and optimal image quality (FID), our work is very related to the idea of designing an optimal conditional perceptual quality preserving codec. We include this discussion in the related work.

We thank the reviewer for their time and constructive comments and address the questions below:

The training and inference complexity of such diffusion-based methods is usually large. The authors should provide some preliminary results like running time or memory consumption.

As stated in the joint answer, we performed tests for encode and decode speed and added a table with the results to the Appendix.

Another existing diffusion-based image compression framework [1] combining text and image information can be discussed. [1] Pan, Zhihong, Xin Zhou, and Hao Tian. "Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models." arXiv preprint arXiv:2211.07793 (2022).

Thanks for pointing out this relevant reference. In [1], they optimize a textual embedding on top of a pretrained text-to-image diffusion model. They also design a compression guidance method used at each denoising step during inference to better reconstruct original images. Their lowest compression rate is 0.042 bits per pixel which is lower than most of the compression baselines, but still much higher than 10x our minimum bitrate of 0.0031 bits per pixel. We include this paper in the related work.

A minor question: on page 4, in “Local spatial encoding” paragraph line 6, “Third, we proceed to quantization of $H_s$ to obtain $z_g$ via vector quantization”. From my understanding, $z_g$ should be $z_l$ ? (ps, a typo here: “quatization” to “quantization” )

Thanks a lot for pointing out the two typos, we correct them in the revised manuscript.

We thank the reviewer for their time and constructive comments and address the questions below:

The proposed method only optimizes the distortion/perceptual term but drops the rate term. It is not justified why the rate term can be dropped. What impact will have if include the rate term into optimization?

In our approach, the rate is controlled through two hyper-parameter settings: the spatial resolution of the hyper-latents, and the vocabulary size. Therefore, the rate can be considered fixed when training our models, and therefore dropped from the objective function. To clarify this, we rephrase the first paragraph of section 3.1.

One limitation of diffusion model is the inference speed. This paper do not provide the running time of the proposed method, especifically the decoding time. A fair comparsion of running time (both encoding and decoding ) with the competing methods (both learned and traditional) should be provided and discussed.

As stated in the joint answer, we performed tests for encode and decode speed and added a table with the results to the Appendix.