Optimal Neural Compressors for the Rate-Distortion-Perception Tradeoff

We thank the reviewer for their helpful feedback and comments, and recognizing the work as bridging the gap between theory and practice. We address the comments in the following.

Responses to weaknesses:

W1. Extending the proposed framework to state-of-the-art high-resolution image compressors is an important direction of our current research. The approach described in the current work could be extended to NTC architectures that handle image data. State-of-the-art architectures for RDP would include HiFiC [1] and follow-up works such as ILLM [2], which are architecturally derived from [3]. Therefore, following [4], we can apply lattices product-wise across the channel dimension, without incurring any notable increase in computational complexity due to the use of dithering or lattice quantization. PD-LTC would involve adding a random dither vector at the decoder in the latent space. The advantage it provides would be an ability to enforce tighter perception constraints compared to deterministic codecs, which we showed for the Speech and Physics datasets. We have preliminary results that show an improvement of FID scores with PD-LTC using the integer lattice with the ILLM model for images: at 0.06 bpp, FID reduces from 5.2 to 4.7 while PSNR remains around 29.1, and at 0.12 bpp, FID reduces from 4.1 to to 3.8 while PSNR improves 30.5 to 30.7. The gain is expected to improve for more efficient lattices. Incorporating SD-LTC in state of the art image compression methods such as ILLM requires some adjustments and variations of the vanilla setting discussed in the present paper. This is because state-of-the-art methods such as [2, 3] use hyperprior entropy models where the latent $y$ is first mean-shifted before quantization and entropy-coding: $y$ is quantized as $Q(y-\mu)+\mu$ where $\mu$ is the mean output from the hyperprior model. We believe that understanding the interplay between $\mu$ and the random dither vector is important in analyzing the role of dithering in coordinating the encoder and decoder and investigating the benefit when random dithering is incorporated in such entropy models. As a result, we believe that a full set of results on image sources requires a more careful investigation of how random dithering should be integrated with the hyperprior models, and this deserves its own separate work.

Nonetheless, our current set of experiments on audio and scientific measurement data, which demonstrate performance benefits for moderate-dimensional data, already illustrate the practical benefits of our design for RDP-oriented compression of scientific and audio sources. Moreover, the theoretical results themselves bridge several sub-fields of neural compression and information theory, including dithered lattice quantization, lattice Gaussian coding, rate-distortion-perception theory, and nonlinear/lattice transform coding. We believe the proof techniques and theorems to be of potential independent interest to all these sub-fields, providing additional value to the paper’s contributions.

W2. The theoretical results provided are asymptotic, i.e., to achieve the fundamental limits, a very large dimensional lattice would need to be used. For practical uses, a finite-dimensional lattice must be chosen. The experimental results show that the predicted behaviors are still exhibited by finite-length lattice codes (such as $E_8$), which is enough to significantly bridge the gap to $R(D, P)$ and $R(D/2, \infty)$. Using higher-dimensional lattices would help bring the performance even closer to the fundamental limits. Regarding the low-rate regime in Fig. 4, as mentioned in lines 335-336, this is an artifact due to the limitations regarding the Monte-Carlo integration technique for LTC borrowed from Lei et al (2025); see the ablation studies of Lei et al (2025) for more details. Designing lattice-specific entropy models that can avoid this issue is important future research, and is out of the scope of the current work.

W3. For the real-world sources, we swept $\lambda_1$ (the distortion weight) and kept $\lambda_2$ (the perception weight) fixed to a positive value. This allows us to compare rate-distortion and rate-perception tradeoffs across methods; note that the two plots should be examined jointly together, and that the individual plots do not show the rate-perception (or rate-distortion) performance for a fixed distortion (or perception). We can see that the pairs of figures show how better lattice packing and more shared randomness improve the rate-distortion tradeoff for the perception constraints shown. In addition, (i) deterministic NTC/LTC cannot enforce the perception constraint at low rates compared to PD-NTC/LTC and SD-NTC/LTC, and (ii) SD-LTC achieves a superior rate-distortion tradeoff compared to PD-LTC while also achieving a lower perception. While not directly shown, the perception-distortion tradeoff depends on the rate regimes, and the way the compressors were trained. At large rates, the distortion being small makes a small perception easy to achieve; on Speech (PD-LTC), this would be observed if larger rates were shown. At near-zero rates, the objective is essentially just rate and perception, so the model would prioritize making the perception small. At moderate rates, there is more tension between the perception and distortion constraints, and the tradeoff regime depends on how $\lambda_1$ and $\lambda_2$ relate. If a larger $\lambda_2$ value was chosen for the moderate rate points (e.g., for PD-LTC/NTC), then their perception values would be lowered, at the cost of worse distortion. If $\lambda_1$, $\lambda_2$ were further chosen more particularly, a nicer-looking convex plot would be seen. However, this type of training and figure presentation that we do is consistent with prior RDP works such as [2], and as such the two plots should be examined jointly. We will include this discussion in a future revision of the paper.

W4. For both synthetic and real-world sources, the $s$ parameter is set as a learnable scalar parameter, constrained to be greater than 0.

W5. Regarding notation in Sec. 3.2, we agree it is a bit confusing, and will clarify the notation in a future revision of the paper.

Response to questions:

Q1. In QSD-LTC/NTC, $R_c$ is not included in the compression rate $R$. It is simply assumed that a back channel consisting of random bits at a rate of $R_c$ bits per latent dimension is available for use. This matches the setup and assumptions made by the theorems in Section 4, as well as prior work in information theory.

Q2. The proposed group-wise dithering is an interesting idea that may have potential in developing alternative efficient dithering methods when finite shared randomness is available. It seems that the proposed group-wise dithering would impose a hierarchical structure on how different dimensions of the dither vector are sampled. Thus, we believe it would implement some sort of nested dithering, which is in line with our proposed QSD-LTC method that is constructed with self-similar nested lattices. However, as mentioned in the last part of Remark B.1 in the paper, QSD-LTC ensures that the finite set of dither vectors chosen are uniformly distributed throughout the coarse lattice cell and thus maximally cover the cell, which is guaranteed by the properties of self-similar nested lattices. It is unclear whether the group-wise approach would select an equally good set of dither vectors. In addition, the example provided for the group-wise approach would still only generate $2^k$ total possible dither vectors of dimension $k+2$ if the "predefined rule" assigning seeds to groups is deterministic. Specifically, in the example above, let $s$ be a realization of the $2^k$ possible seeds (available at encoder/decoder), and $i \in \{0,1,2,3\}$ be the group index. If $f(s, i)$ is the rule that assigns a seed to group $i$, there are still only $2^k$ possible combined assignments $[f(s, 0), f(s, 1), f(s, 2), f(s, 3)]$ that generate dithers for the whole vector of dimension $k+2$, by enumerating $s$. If $f$ was random, then more than $2^k$ possibilities could be generated for the whole vector, but then the decoder would be unable to decode unless it had access to $f$’s randomness, which would necessitate additional shared randomness beyond the $k$ already shared. Thus it seems unlikely the group-wise approach would alleviate the challenges of a fixed $k$-bit random seed in high-dimensional cases. Overall, the group-wise approach could be promising, but it is unclear whether it provides nice properties on the dithers that QSD-LTC provides due to the self-similar nested lattices.

References

[1] Mentzer, F., Toderici, G. D., Tschannen, M., and Agustsson, E. High-fidelity generative image compression. NeurIPS, 2020.

[2] Muckley, M. J., El-Nouby, A., Ullrich, K., Jégou, H., and Verbeek, J. Improving statistical fidelity for neural image compression with implicit local likelihood models. ICML 2023.

[3] Ballé, J., Minnen, D., Singh, S., Hwang, S. J., and Johnston, N. Variational image compression with a scale hyperprior. ICLR 2018.

[4] E. Lei, H. Hassani, and S. Saeedi Bidokhti. Approaching rate-distortion limits in neural compression with lattice transform coding. 2024.

Thank you for your thorough response. Many of the reviewer’s concerns have been addressed through the rebuttal, and therefore, I am updating my rating to 5 accordingly.

We thank the reviewer for taking the time to provide valuable feedback and comments, and recognizing our work.

Extending the proposed framework to state-of-the-art high-resolution image compressors is an important direction of our current research. The approach described in the current work could be extended to NTC architectures that handle image data. State-of-the-art architectures for RDP would include HiFiC [1] and follow-up works such as ILLM [2], which are architecturally derived from [3]. Therefore, following [4], we can apply lattices product-wise across the channel dimension, without incurring any notable increase in computational complexity due to the use of dithering or lattice quantization. PD-LTC would involve adding a random dither vector at the decoder in the latent space. The advantage it provides would be an ability to enforce tighter perception constraints compared to deterministic codecs, which we showed for the Speech and Physics datasets. We have preliminary results that show an improvement of FID scores with PD-LTC using the integer lattice with the ILLM model for images: at 0.06 bpp, FID reduces from 5.2 to 4.7 while PSNR remains around 29.1, and at 0.12 bpp, FID reduces from 4.1 to to 3.8 while PSNR improves 30.5 to 30.7. The gain is expected to improve for more efficient lattices. Incorporating SD-LTC in state of the art image compression methods such as ILLM requires some adjustments and variations of the vanilla setting discussed in the present paper. This is because state-of-the-art methods such as [2, 3] use hyperprior entropy models where the latent $y$ is first mean-shifted before quantization and entropy-coding: $y$ is quantized as $Q(y-\mu)+\mu$ where $\mu$ is the mean output from the hyperprior model. We believe that understanding the interplay between $\mu$ and the random dither vector is important in analyzing the role of dithering in coordinating the encoder and decoder and investigating the benefit when random dithering is incorporated in such entropy models. As a result, we believe that a full set of results on image sources requires a more careful investigation of how random dithering should be integrated with the hyperprior models, and this deserves its own separate work.

Nonetheless, our current set of experiments on audio and scientific measurement data, which demonstrate performance benefits for moderate-dimensional data, already illustrate the practical benefits of our design for RDP-oriented compression of scientific and audio sources.

Regarding the types of lattice quantization methods, our theoretical results in Section 4 suggest that one needs a sphere-bound-achieving sequence of lattices to be optimal on the Gaussian source. In other words, on the iid Gaussian source, performance should be best when the normalized second moment (NSM) of the lattice is minimized, and therefore other lattice quantizers would be suboptimal (unless they coincide the NSM-optimal quantizer in each dimension). The lattices chosen, such as the moderate dimension ones used in the experiments, or the high-dimensional ones such as the polar lattices (mentioned in Sec 4), all have low complexity CVP and low NSM. For real-world sources, while our experiments in Section 5 suggest that lower NSM lattices provide better overall RDP performance, it is possible that even better RDP performance could be achieved using the methods in paper [a] which optimize the lattice directly while using the Babai rounding estimate to perform approximate CVP, which avoids the exponential lattice vector search, as there are currently no theoretical guarantees on more general sources. Combining the ideas in [a] with our proposed methods could be an interesting area for future research.

References

[1] Mentzer, F., Toderici, G. D., Tschannen, M., and Agustsson, E. High-fidelity generative image compression. NeurIPS, 2020.

[2] Muckley, M. J., El-Nouby, A., Ullrich, K., Jégou, H., and Verbeek, J. Improving statistical fidelity for neural image compression with implicit local likelihood models. ICML 2023.

[3] Ballé, J., Minnen, D., Singh, S., Hwang, S. J., and Johnston, N. Variational image compression with a scale hyperprior. ICLR 2018.

[4] E. Lei, H. Hassani, and S. Saeedi Bidokhti. Approaching rate-distortion limits in neural compression with lattice transform coding. 2024.

We thank the reviewer for recognizing the quality of the writing and theoretical contributions of our work.

We would like to first clarify several points of our work related to weakness 1 and question 1. On shared randomness, while it may be expected that more shared randomness should not hurt the performance, it is not immediately clear how shared randomness, infinite or finite, could be used to boost the performance in an efficient/optimal manner as promised by results from information theory. The extent of this performance gain is also unclear in practical settings (finite dimension $n$, real-world sources beyond Gaussian, etc). In the classical rate-distortion (RD) setting, no randomness is needed to achieve optimality. As a result, most neural codecs, whether oriented for RD or rate-distortion-perception (RDP), do not necessarily use shared randomness or randomness at all as a resource. Recent information-theoretic work (see Sec. 2) has shown that not only randomized reconstructions are needed to satisfy the perception constraint, but also infinite shared randomness is needed to achieve the RDP function. Any less shared randomness results in worse performance. However, these works do not provide constructive schemes that one may implement in practice. In contrast, our work provides a practical scheme that provably achieves the optimal tradeoffs for infinite and no shared randomness as derived by the information theory literature. The experimental results show that our proposed construction utilizes the shared randomness in an efficient manner as predicted by prior information theory work and the theory we established. These trends extend to compression of real-world sources in addition to Gaussians. Furthermore, the theory we provide is asymptotic (e.g., in the limit of large dimensions), but the experimental results show that the predicted behaviors are still exhibited by moderate-length lattice codes (such as $E_8$), which is enough to significantly bridge the gap to $R(D, P)$ and $R(D/2, \infty)$. Finally, while a theoretical proof of QSD-LTC (the finite randomness case) is left for future work (see below), the experimental results show that it can nearly achieve the performance of SD-LTC despite not having infinite shared randomness.

Regarding the benefits of LTC over NTC, Lei et al. (2025) consider the RD setting but do not address the RDP setting. As mentioned before, randomized reconstructions are necessary for RDP. It is therefore not immediately clear whether VQ-like coding (which is optimal for RD) is good for RDP; it is further unclear how VQ should be integrated with randomized codecs as VQ is deterministic. Our work reveals that VQ-like coding is indeed good for RDP (Appendix A), and then showed in Section 3 how lattices, which provide VQ-like regions, could be incorporated with shared or private randomness for neural compression via shared/private dithering. Thus, our experimental results confirm that under the RDP setting, SD/PD-LTC is indeed superior to SD/PD-NTC, is able to approach optimality (as predicted by Thms. 4.3, 4.5), and extends to general sources.

Q2. Regarding the MNIST figure, this is not unexpected. The behavior is the result of the RDP tradeoff being a triple tradeoff, the regimes where the perception constraint is active or not, and the way the compressors were trained. To generate the curves for the general sources, we swept $\lambda_1$ (the distortion weight) and kept $\lambda_2$ (the perception weight) fixed to a positive value. At large rates, the distortion being small makes a small perception easy to achieve. At near-zero rates, the objective is essentially just rate and perception, so the model would prioritize making the perception small. At moderate rates, there is more tension between the perception and distortion constraints, and the tradeoff regime depends on how $\lambda_1$ and $\lambda_2$ relate. If a larger $\lambda_2$ value was chosen for the moderate rate points (e.g., for PD-LTC/NTC), then their perception values would be lowered, at the cost of worse distortion. However, the current figure 5 still illustrates our main arguments: (i) deterministic NTC/LTC cannot enforce the perception constraint at low rates compared to PD-NTC/LTC and SD-NTC/LTC, and (ii) SD-LTC achieves a superior rate-distortion tradeoff compared to PD-LTC while also achieving a lower perception. If PD-LTC’s perception were lowered, it would only worsen its rate-distortion performance, but SD-LTC is already outperforming PD-LTC. If $\lambda_1$, $\lambda_2$ were further chosen more particularly, a nicer-looking convex plot would be seen. However, this type of training and figure presentation that we do is consistent with prior RDP works such as [1, 2], and as such the two plots should be examined jointly.

Q3. Regarding QSD-LTC, this is indeed more difficult. Similar to the proof of PD-LTC, QSD-LTC cannot make use of the additive channel equivalence of dithered lattice quantization. Therefore proving a fundamental limit for QSD-LTC would likely require a similar approach to PD-LTC based on lattice Gaussian analysis. However, unlike PD-LTC, it would require further understanding of the behavior of a discrete dither vector that is distributed over the nested lattices. To the best of our knowledge, this type of dithering has not been studied before in the context of lattice Gaussian coding. Therefore the QSD-LTC analysis deserves its own treatment based on new tools developed for dither vectors defined over nested lattices. We will update the paper with a remark reflecting this.

W2. The code was added to an anonymous link, however, it appears that an error occurred during the submission and this was not included in the supplementary material. While it could be added here, the new rebuttal policies disallow sharing of anonymous links, and we apologize for this inconvenience.

References

[1] Y. Blau and T. Michaeli. Rethinking lossy compression: The rate-distortion-perception tradeoff. ICML 2019.

[2] Muckley, M. J., El-Nouby, A., Ullrich, K., Jégou, H., and Verbeek, J. Improving statistical fidelity for neural image compression with implicit local likelihood models. ICML 2023.

We thank the reviewer for their helpful comments and feedback, and for recognizing the contributions of our work.

Extending the proposed framework to state-of-the-art high-resolution image compressors is an important direction of our current research. The approach described in the current work could be extended to NTC architectures that handle image data. State-of-the-art architectures for RDP would include HiFiC [1] and follow-up works such as ILLM [2], which are architecturally derived from [3]. Therefore, following [4], we can apply lattices product-wise across the channel dimension, without incurring any notable increase in computational complexity due to the use of dithering or lattice quantization. PD-LTC would involve adding a random dither vector at the decoder in the latent space. The advantage it provides would be an ability to enforce tighter perception constraints compared to deterministic codecs, which we showed for the Speech and Physics datasets. We have preliminary results that show an improvement of FID scores with PD-LTC using the integer lattice with the ILLM model for images: at 0.06 bpp, FID reduces from 5.2 to 4.7 while PSNR remains around 29.1, and at 0.12 bpp, FID reduces from 4.1 to to 3.8 while PSNR improves 30.5 to 30.7. The gain is expected to improve for more efficient lattices. Incorporating SD-LTC in state of the art image compression methods such as ILLM requires some adjustments and variations of the vanilla setting discussed in the present paper. This is because state-of-the-art methods such as [2, 3] use hyperprior entropy models where the latent $y$ is first mean-shifted before quantization and entropy-coding: $y$ is quantized as $Q(y-\mu)+\mu$ where $\mu$ is the mean output from the hyperprior model. We believe that understanding the interplay between $\mu$ and the random dither vector is important in analyzing the role of dithering in coordinating the encoder and decoder and investigating the benefit when random dithering is incorporated in such entropy models. As a result, we believe that a full set of results on image sources requires a more careful investigation of how random dithering should be integrated with the hyperprior models, and this deserves its own separate work.

Nonetheless, our current set of experiments on audio and scientific measurement data, which demonstrate performance benefits for moderate-dimensional data, already illustrate the practical benefits of our design for RDP-oriented compression of scientific and audio sources.

References

[1] Mentzer, F., Toderici, G. D., Tschannen, M., and Agustsson, E. High-fidelity generative image compression. NeurIPS, 2020.

[2] Muckley, M. J., El-Nouby, A., Ullrich, K., Jégou, H., and Verbeek, J. Improving statistical fidelity for neural image compression with implicit local likelihood models. ICML 2023.

[3] Ballé, J., Minnen, D., Singh, S., Hwang, S. J., and Johnston, N. Variational image compression with a scale hyperprior. ICLR 2018.

[4] E. Lei, H. Hassani, and S. Saeedi Bidokhti. Approaching rate-distortion limits in neural compression with lattice transform coding. 2024.