Zero-Shot Image Compression with Diffusion-Based Posterior Sampling

• Variance Minimization: The reviewer makes an interesting point of directly minimizing the entropy. Yet, finding the best measurement at each step for lowering the entropy of an unknown distribution is not trivial. By defining the uncertainty as the MSE attained by the linear minimum-MSE (MMSE) predictor of x based on the measurements y, the uncertainty directly correlates to the posterior distribution’s covariance. Thus, a closed-form solution for the optimal linear measurement exists. This closed-form solution indirectly reduces the entropy. For example, if the posterior distribution is a Gaussian, the optimal linear measurement for lowering L2 error is also the optimal in reducing the entropy. This is because a Gaussian distribution remains a Gaussian when conditioned on a linear measurement, and its entropy is proportional to the log determinant of the covariance matrix. We will add this discussion in the paper.
• Non-Linear Measurements: while diffusion-based non-linear solvers (such as PiGDM) exist, they have yet to be employed for an active acquisition scheme (such as AdaSense). This is probably because finding the optimal non-linear measurements even for L2 error is non-trivial, unlike the linear case. Nevertheless, we demonstrate that even when limited to linear transforms, by constructing an image-specific transform we are able to formulate a highly effective compression scheme.
• Related Work: We will refer to these works in a revision (we already refer to [1]). Yet, we note that [1] and [2] both use networks that are trained specifically for the task of image compression, unlike our method, which is entirely based on off-the-shelf pre-trained generative networks. Additionally, our compression scheme is independent from neural compression methods, unlike the cited works which integrate existing neural compression methods with advances in conditional image generation, thus providing a novel take on the compression task.

[1] Careil, Marlene, et al. "Towards image compression with perfect realism at ultra-low bitrates." The Twelfth International Conference on Learning Representations. 2023.

[2] Xu, Tongda, et al. "Idempotence and perceptual image compression." arXiv preprint arXiv:2401.08920 (2024).

• Computational Costs: As mentioned, PSC requires many NFEs to generate the multitudes of samples required at each step. More specifically, we used an order of 10,000 NFEs for compressing each image (depending on the required rate of course, as our method is progressive). While this computational cost is much higher than JPEG, so is the image quality while also maintaining much lower distortion for the same rates. We stress that this is the first introduction of such a posterior-sampler based compression method, and we expect that major speed-ups could be made in the near future, with the improvements of diffusion models and posterior samplers (such as flow-based methods) in general, and with specific algorithmic ideas that fit the architecture of the proposed algorithm.
• Pre-trained Diffusion Model: The diffusion model is central to the proposed compression algorithm: It is used for the active acquisition phase of the compression as well as for the final restoration step. This model provides the image prior for the compression algorithm by guiding the measurement process. At each step, a multitude of posterior samples are generated with the pretrained diffusion using a posterior sampler, enabling the selection of the next measurement. For these reasons, we believe the emphasis on the zero-shot diffusion to be appropriate.
• Experimental Section: Could the reviewer please highlight what experiments would benefit the paper? We will gladly add more experiments.
• Figures: We will make an effort to expand on the figures in the text.
  • Fig. 2: The line represents the different choices along the Rate-Distortion curve which may be chosen by a user compressing with our algorithm. As described in the paper, the flexibility to work at any desired rate is an advantage of our compression method, while also existing in several classic approaches such as JPEG.
  • Fig. 4: We will modify the diagram to add detail and improve its quality.
  • Fig. 5: The results exhibit lower quality on the right due to the much lower compression rate, as indicated by the BPP (bits per pixel) values on the images.
  • Fig. 6: The results on the right lose detail due to the lack of a text-prompt in generation, causing less specificity in the images generated for the active acquisition phase and thus causing image quality to drop.

[1] Song, Yang, et al. "Score-based generative modeling through stochastic differential equations." arXiv preprint arXiv:2011.13456 (2020).

[2] Kingma, Diederik, et al. "Variational diffusion models." Advances in neural information processing systems 34 (2021): 21696-21707.

• High Computational Cost: As mentioned, PSC requires many NFEs to generate the multitudes of samples required at each step. More specifically, we used an order of 10,000 NFEs for compressing each image (depending on the required rate of course, as our method is progressive). We stress that this is the first introduction of such a posterior-sampler based compression method, and we expect that major speed-ups could be made in the near future, with the improvements of diffusion models and posterior samplers (such as flow-based methods) in general, and with specific algorithmic ideas that fit the architecture of the proposed algorithm.
• ICLR Standards: Our paper presents an original formulation for using existing diffusion models for image compression; We stress that there are no similar posterior sampling-based compression methods existing in the literature. The introduced method is SOTA - it performs competitively to HiFiC [1], a leading training-based neural compression method. Furthermore, the proposed scheme is very flexible, offering a progressive compression that other learning-based methods do not serve. True, the proposed algorithm is of high complexity, as described above, and we believe that future work will seek and find ways to speed it up substantially. In summary, we believe our work meets the standards of ICLR and beyond.
• Theoretical Framework: We will revise the paper by better explaining the uncertainty reduction framework that justifies our method.
• Complicated Exposition: We thank the reviewer for this suggestion, and we will simplify our exposition, specifically of AdaSense.
• Related Research: We will refer to these works in our revision, thank you for bringing these works to our attention. The cited works are neural compression methods, and are trained directly for image compression, unlike our zero-shot method. We note that while [2] is flexible in terms of compression rate, similar to our method, it also requires hundreds or thousands of computationally costly NFEs to achieve its flexibility.

[1] Mentzer, Fabian, et al. "High-fidelity generative image compression." Advances in Neural Information Processing Systems 33 (2020): 11913-11924.

[2] Gao, Chenjian, et al. "Flexible neural image compression via code editing." Advances in Neural Information Processing Systems 35 (2022): 12184-12196.

• Writing Quality: We will make a sincere effort to make the contributions clearer.
• Required NFEs: We use 25 NFEs to generate samples with DDRM (as specified in the supplementary material). As for the overall complexity, we discuss it hereafter. We will make these details clear in a revised version.
• Reducing Uncertainty: We define the uncertainty as the MSE attained by the linear minimum-MSE (MMSE) predictor of x based on the measurements y. With this definition, the uncertainty directly correlates to the posterior distribution’s covariance, and thus a closed-form solution for the optimal linear measurement exists. By acquiring additional measurements, or in our case increasing the size of the compressed representation, we can decrease this measurement of uncertainty in the final restored image.
• Novelty of Method: Indeed, neither the posterior samplers used or active acquisition methods are contributions of our work. Nevertheless, we believe that the use of these pre-existing tools (which also include the pretrained diffusion models in our case) for new domains is both useful and original. We note that we are not aware of any other work employing posterior sampling of any kind, regardless of the methods, for image compression.
• High Computational Cost: As mentioned, PSC requires many NFEs to generate the multitudes of samples required at each step. More specifically, we used an order of 10,000 NFEs for compressing each image (depending on the required rate of course, as our method is progressive). We stress that this is the first introduction of such a posterior-sampler based compression method, and we expect that major speed-ups could be made in the near future, with the improvements of diffusion models and posterior samplers (such as flow-based methods) in general, and with specific algorithmic ideas that fit the architecture of the proposed algorithm.
• Non-Linear Measurements: while diffusion-based non-linear solvers exist, they have yet to be employed for an active acquisition scheme (such as AdaSense). This is probably because finding the optimal non-linear measurements even for L2 error is non-trivial, unlike the linear case. While developing such an active acquisition scheme would be of high interest, we believe it to be out of the scope of our work, which attempts to use the existing active acquisition tools for image compression.
• PiGDM: We find that restoring the compressed representation with PiGDM yields images with better perceptual quality. We highlight that this does not necessarily make them better, as their distortion is greater than the restorations produced by DDRM. This tradeoff is expected from different restoration algorithms according to the Perception Distortion Tradeoff [1], and we leave it to the user to choose what restoration algorithm to use to restore the compressed representation. Also, we note that PiGDM was not used during the active acquisition phase of PSC, but only during the final restoration.
• Neural Compression: We have compared PSC to one notable neural compression method, HiFiC [2], and we reach slightly better performance. Yet, as a zero-shot method that has never seen the task of image compression in training, we do not expect our method to be SOTA compared to all training-based neural compression methods. We should mention that in our experiments, we ran our method in its simplest form, without additional improvements, such as better entropy coding, better optimization of the projection directions, etc..
• More Efficient Models: To the best of our knowledge, methods such as consistency models (CM) are not designed for posterior sampling of the form we require in the proposed compression algorithm. Adjusting CM to accommodate this task will need many NFE’s as it will necessarily rely on an iterative evaluation of the network. The exploration of such faster sampling methods is an interesting future work.

[1] Blau, Yochai, and Tomer Michaeli. "The perception-distortion tradeoff." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[2] Mentzer, Fabian, et al. "High-fidelity generative image compression." Advances in Neural Information Processing Systems 33 (2020): 11913-11924.