Lossy Compression with Pretrained Diffusion Models

Thank you for your review.

In response to the raised weaknesses:

The method's performance is constrained by the fidelity limits of the Stable Diffusion models' variational autoencoders, affecting compression quality at higher bitrates.

This is indeed a limitation of our method. However, note that this is much less of a limitation for our newly added results using Flux-dev, which has a much higher fidelity autoencoder.

The reverse-channel coding process, while optimized, still presents computational challenges, particularly for very large or very small DKL values.

As explained in Section 3, our optimization process allows us to mostly circumvent the problem of $D_{KL}$ values which are impractically large or small. $D_{KL}$ is too small, we take larger steps. If $D_{KL}$ is too large, we break up the distribution into manageably-sized chunks. This is our main innovation on the DiffC algorithm that makes reverse-channel coding practical.

3. The paper does not extensively explore the combination of DiffC with other conditional diffusion approaches, which could potentially enhance performance further.
4. The reliance on Stable Diffusion's image size limitations may affect the generalizability of the method to images outside the training distribution.

Weaknesses 3 & 4 are important points, but we regard them as outside the scope of the current work. These are improvements we plan to explore in future work.

Questions:

How does the performance of your method vary with image sizes that deviate from the training distribution...?

To answer your question, see the new Appendix A.7, we have added R-D curves for DiffC on OOD image sizes. While it is outside the scope of our current work, we think that DiffC could be combined with approaches like https://elasticdiffusion.github.io/ to compress images at a more flexible range of sizes and aspect ratios.

Can you provide a more detailed comparison between your method and other state-of-the-art compression techniques...?

As requested, we have added comparisons to SoTA compression methods: HiFiC, PerCo, MS-ILLM, and DiffEIC. As promised, our method performs competitively on rate/distortion/perception curves against these other compression methods.

In the leftmost plot of Figure 3, could the authors explain why the performance of the 163-step is worse than that of the 54-step?

This is a good observation, and I wish we could explain it, but currently we cannot. Note thematically similar results in the Appendix A.7, where using lower-than-actual $D_{KL}$ values during compression can actually improve the PSNR rate-distortion curve. According to our assumptions in Section 3.2, the 163 step schedule should achieve the best rate-distortion curve by any reasonable distortion metric. But this conclusion depends on the assumption that our reverse-channel coding scheme will sample perfectly from the forward process. We strongly suspect that these unexpected results are downstream of some approximation error in our reverse-channel coding scheme, and some unexplored tradeoffs where it can actually be more efficient to save bits by only approximately instead of exactly sampling the forward process. We consider this an important topic in our planned future work.

Thank you for your review and suggestions. Responses to your comments below:

The paper lacks sufficient innovation and resembles more of an engineering improvement on existing methods.

To clarify the significance of our contribution, see both our top-level reply to all reviewers, as well as the newly added blue text in our introduction.

The diffusion model requires multiple inference steps for both encoding and decoding, resulting in significant computational overhead.

This is true, and is a significant limitation of our method. While mitigating this is outside the scope of our current work, we consider it a significant opportunity for future research. It's worth noting that other diffusion-based generative compression methods also require multiple denoising steps. The newly added Table 1 shows that DiffEIC and PerCo have comparable decoding times to DiffC.

Although the authors claim that their method is competitive with other state-of-the-art compression methods, no comparisons are provided in the paper.

In the latest revision of the paper, we have added comparisons to HiFiC, PerCo, MS-ILLM, and DiffEIC. We have also added a table with the # of model parameters and average encoding/decoding times for each method. We are pleased to report that DiffC performs competitively.

On page 2, line 100, the description of CDC appears to be inaccurate...

We apologize for the mistake and have changed the wording accordingly.

In Section 4.2, why is there no R-D curve... for Stable Diffusion XL Base and Refiner?

We have added an RD curve for the no-refiner version of SDXL to Figure 2. Note that it is very close in performance to the version with the refiner, but with slightly lower CLIP and quality scores.

In Section 3.5... Does this mean that the D_KL values used in practice are manually set? Could the authors provide empirical results...? Could RD trade-offs be achieved by adjusting the values?

Good questions! Yes, the $D_{KL}$ values used in practice are manually set. As requested, we have added a new Appendix A.7 with further results on this topic. The figure in appendix A.7 shows the impact of increasing or decreasing the $D_{KL}$ values. RD-tradeoffs could definitely be achieved by tuning these parameters, as our figure in the appendix shows. Unfortunately, we do not yet have a good understanding of the tradeoffs involved here. We consider further analysis of this topic to be an important part of our planned future work.

Thank you for your review and suggestions.

The significance of this work is profound because it addresses the remaining problems in the DiffC algorithm, such as inference time and reverse-channel coding.

Thank you for the kind words; we are also excited about the future of the DiffC algorithm. We hope our work helps advance the research on this algorithm from theory to practice.

(1)The manuscript looks more like a technology report than an academic paper.

We have made many significant changes to the new rebuttal version of the paper, which we hope address your concerns here. See our general comment for a list of changes. New content is highlighted in blue.

(2)The authors do not provide quantitative comparisons with state-of-the-art extreme image compression methods (VQ-based methods, diffusion-based methods) and show the advantages of the proposed method.

As requested, we have added comparisons to SoTA compression methods: HiFiC, PerCo, MS-ILLM, and DiffEIC.

(3) Although the main contribution of the paper is the implementation of the DiffC algorithm, the author should provide the model complexity of the proposed method (e.g., network parameters, FLOPs, encoding/decoding time, etc.).

We have also added Table 1, which reports the parameter counts and average encoding/decoding times for each compression method we evaluated.

In Section 4.1, the prompts do not support image compression or reconstruction. This result contradicts many previous prompt-guided image compression methods. The author should add more analysis and explain why they do not work.

We were also surprised by this result. Since the last revision of the paper, we have also experimented with BLIP-generated prompts and arrived at the same conclusion. We have also added rate-distortion curves for SDXL, and Flux using BLIP prompts. While there is no clear advantage to using prompts with either SD 1.5 or 2.1, we note that both SDXL and Flux show significantly improved quality scores at low bitrates when prompts are employed (See Appendix Figure 11 on page 20).

As to your question about why prompt conditioning is less effective for our method compared to other prompt-guided image compression methods, this is a challenging question to answer. DiffC is a different compression method with different characteristics. For now we have no concrete answer to your question; it is something we would like to understand better in our future work.

Thank you for your review and suggestions.

I failed to find the comparisons of this idea with other SOTA image compression methods. Could you present the comparisons of this idea with other SOTA image compression methods?

As requested, we have added comparisons to SoTA compression methods: HiFiC, PerCo, MS-ILLM, and DiffEIC. As promised, our method performs competitively on rate/distortion/perception curves against these other compression methods.

First, I don't think the main contribution is significant. The major idea had already been proposed several years ago.

We have also added some discussion in the introduction emphasizing the significance of our contribution. Also see our top-level reply to all reviewers for a defense of our paper's significance.

Please refer to the latest revision of the paper. New content is highlighted in blue.