Essential Reference

We thank the reviewer for highlighting the insightful work by Huan Liu et al. (ICLR 2022). It is compelling to address tasks such as compressed image restoration by formalizing them as a distribution shift (via optimal transport) with an informational bottleneck. We will include a discussion of this important reference in the related work section of our revised manuscript.

Codebook Size Adjustment in Different Timesteps

Dynamically adjusting the codebook size across timesteps may indeed offer potential benefits. A positive indication of this can be seen in Appendix B.3, where using different (fixed, not dynamically adjusted) codebook sizes for different timesteps leads to improved performance. However, identifying the “optimal” codebook size for each step can be challenging, as modifying the codebook size at some timestep can affect the outcomes of all subsequent timesteps. We will include this in the discussion as an interesting option for future research.

DDCM Bitrate and File Size Clarification

Please note that the reported bitrate is precisely that of the compressed file size, as in traditional compression methods. Indeed, the mentioned logarithmic relationship $T\cdot log_{2}(K)$ between the codebook size $K$ and the number of sampling steps $T$ directly calculates the file size (in bits). This holds since each DDCM-generated image corresponds to a sequence of $T$ integers, each in $[0,K-1]$, that represent the indices of the codebook entries chosen during the generation (as depicted in Fig. 2). This sequence is then represented in a binary format and saved as a file. E.g., for $T=3,K=4$, the indices sequence [1,3,0] translates to the bitstream “011100”. We compare our method against previous compression approaches (including traditional ones) according to their file size (using BPP).

Experimental Setup and Results Clarification

We apologize for any confusion about our experimental setup and results. Our paper contains multiple different experimental settings (starting at lines 157(R), 188(R), 309(R), and 368(R)), involving image generation, compression, and restoration, following common practices in each of these fields. Let us clarify.

Section 4 in the paper

This section shows that DDCM maintains competitive generation capabilities relative to DDPM, even with small codebooks (lines 158(R)). Specifically, we show that DDCM achieves comparable performance to DDPM as measured by standard generative metrics like FID (Fig. 3, main text), KID (Fig. 8, App. A), and through visual inspection (Figs. 9 & 10, App. A) across two distinct datasets: ImageNet and MS-COCO. This experiment is intended to validate our hypothesis regarding the redundancy of the continuous Gaussian representation space utilized in traditional DDMs (lines 016(R) & 182(L)).

Section 5 in the paper

This section introduces our perceptual image compression method, based on DDCM. We compare our method to prior approaches using standard rate-perception-distortion evaluations [R2], assessing perceptual quality (FID) and distortion (LPIPS, PSNR) across multiple bitrates (BPP). These quantitative comparisons are provided in Fig. 5, complemented by qualitative assessments in Fig. 4, following common evaluation practices. Both our quantitative and qualitative results demonstrate the superiority of our proposed method over existing techniques (lines 243-258(L)). In Appendix B, we also include additional qualitative results, as well as additional quantitative results for higher-resolution images (Figs. 11-16).

Section 6 in the paper

Here, we propose extending our compression framework to handle more general compressed conditional generation tasks, such as compressed image restoration. Our experiments in the main text cover two tasks: zero-shot posterior sampling (Sec. 6.1) and blind face image restoration (Sec. 6.2). We compare our method with previous state-of-the-art approaches, assessing both distortion (PSNR) and perceptual quality (FID), thus adhering to established evaluation protocols in image restoration [R1]. The qualitative and quantitative results, presented in Figs. 6 & 7, show that our method achieves superior perceptual quality compared to previous methods (lines 353-357(L) & 421-428(L)). Apps. C.3 & C.4 provide additional results (Figs. 17-22).

We hope this clearly addresses the concerns raised regarding the clarity and comprehensiveness of our experimental setup and results.

Superior Perceptual Compression Performance

Kindly note that our image codec is designed for perceptual image compression, meaning that we aim to achieve best output perceptual quality. Due to the rate-perception-distortion tradeoff [R2], this is expected to compromise the distortion (e.g., PSNR, LPIPS) of our method. Indeed, since our method achieves the lowest (best) FID (perceptual quality) for almost all bitrates and datasets (Figure 5), the rate-perception-distortion tradeoff explains why our method does not always achieve the best distortion. However, note that we do achieve better distortion compared to previous perceptual image compression methods such as PerCo (SD), while also achieving better perceptual quality.

Generation Evaluation Using No-Reference Metrics

We kindly note that it is not common practice to evaluate image generation methods with no-reference quality measures (e.g., NIQE [R3]), since the goal in image generation is to sample from a data distribution. No-reference quality measures only assess the quality of the generated images and not their diversity, whereas metrics such as FID and KID (which we report in the paper) assess quality and diversity simultaneously.

References

[R1] Yochai Blau & Tomer Michaeli. The perception-distortion tradeoff. CVPR 2018.

[R2] Yochai Blau & Tomer Michaeli. Rethinking lossy compression: The rate-distortion-perception tradeoff. ICML 2019.

[R3] Anish Mittal et al. Making a ‘completely blind’ image quality analyzer. ACCSC 2012.

Fig. 3 FID

Yes, the FID scale in Fig. 3-L is $10^1$. The FID of DDPM is 9.21 (black dashed line). Indeed, some prior works with the same DDM reported lower FID on ImageNet 256x256, when using 50k generated samples [R2]. However, we use only 10k generated samples (see L171-L) to reduce computation time, similarly to [R3,R4]. Decreasing the number of samples is known to increase the FID [R1]. To validate this, we computed the FID between the same 50k reference images and two random subsets from the ImageNet 256x256 training set, consisting of 10k and 50k images. These subsets obtained FID=4.934 and FID=1.968, respectively.

Note that this experiment only aims to demonstrate DDCM's competitiveness with DDPM, specifically for small codebooks. Indeed, the modest decrease in FID from 10.5 to 9.5 (when increasing $K$) is a strength of our method rather than a limitation: even for $K=2$, DDCM achieves FID$\approx$10.5 (competitive with DDPM). This supports our hypothesis (L016-R & L182-L) that the Gaussian representation space used by standard DDMs is redundant.

Fig. 5 & VAE Bound

The FID scores for the compression results in Fig. 5 are in log scale.

Regarding the “VAE bound,” we apologize for the misunderstanding, this term was not clarified in the paper. In some experiments we employ a latent DDM (specifically, SD 2.1), so we are compressing the latent VAE encoding of a given image, rather than the image itself. Since encoding-decoding an image using a VAE typically does not yield perfect image reconstruction, the distortion of any VAE-latent-space-based compression method (e.g., ours, PerCo, PSC) is bounded by that of the VAE encoder-decoder. The “VAE bound” in the figure corresponds to the distortion resulting from encoding and decoding the original images using only the VAE, without any additional compression. This bound is important to present since it allows to distinguish the distortion caused by the VAE from that caused by the compression scheme.

Thanks for these important points. We will clarify them in the revised paper and replace the term “VAE-bound” with “SD 2.1 Encoder-Decoder bound.”

Eq. (7) Efficiency

Please note that Eq. (7) does not involve any gradient computation, iterative optimization, or use of a neural network. It involves a straightforward matrix multiplication followed by picking the maximum out of K scalars. Empirically, on an L40S GPU with $K=8192$ and 256x256 images, this operation takes 0.357ms on average (over 100 trials). This is negligible compared to the DDM’s forward pass, which takes 57.2ms on average.

Tokenization Methods

We appreciate this suggestion. Viewing DDCM-based compression as an image tokenization method with random noises acting as the tokens is interesting. We will discuss this in our paper revision.

Numerical Results

Thanks for this valuable suggestion. We will add numerical comparisons to the appendix.

Applying to Flow Matching Models

Although flow-matching models typically generate samples by solving an ODE, several works (e.g., [R5,R6]) have shown they can also generate samples by solving an SDE, which involves adding noise at each generation step. Thus, we believe that such flow-matching SDEs may be discretized as well, similarly to DDCM.

Resolution Generalization

DDCM is easily generalizable to any resolution. Instead of pre-sampling the codebooks with a particular resolution, one can rely on a shared-seed random generator to dynamically generate codebooks for different resolutions. Specifically, one can store/transmit the resolution as part of the bitstream and, before compressing/decompressing an image, randomly sample the codebooks for the given resolution. Using the same seed for compression and decompression ensures that both would rely on the same codebooks.

If no shared-seed random generator is available, one can pre-sample the codebooks for a maximal allowed resolution, store/transmit the resolution of the image, and dynamically slice smaller codebooks from the pre-sampled ones.

Therefore, the generalizability of DDCM to different resolutions is determined only by that of the pre-trained DDM.

Learnable Codebooks

Learning the codebooks may indeed enhance DDCM’s performance (as noted in L433-R), particularly in compression. We investigate this in our future work.

References

[R1] Mikołaj Bińkowski et al. Demystifying MMD GANs. ICLR 2018.

[R2] Cheng Lu et al. DPM-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NeurIPS 2022.

[R3] Cheng Lu et al. DPM-Solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv:2211.01095.

[R4] Kaiwen Zheng et al. DPM-Solver-v3: Improved diffusion ODE solver with empirical model statistics. NeurIPS 2023.

[R5] Jeongsol Kim et al. FlowDPS: Flow-driven posterior sampling for inverse problems. arXiv:2503.08136.

[R6] Litu Rout et al. Semantic image inversion and editing using rectified stochastic differential equations. ICLR 2025.