Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaptation

Weakness 1: Comparison with MS-ILLM

Response 1: We apologize that the comparison with MS-ILLM is missing. In the revised manuscript, we have compared with MS-ILLM [ref1] on both Kodak and DIV2K datasets, where the results are in Figure 3 and Figure 4 (revised manuscript, \textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}), respectively. One can see that, MS-ILLM is trained for specific R-D points, which achieve relatively higher quantitative performance in LPIPS and DISTS. However, even though, our method exhibits comparable performance in FID and KID, and better NIQE than MS-ILLM. Here, for simple clarity, we provide training steps (M), BD-rate (BD-LPIPS) results on the DIV2K dataset, and encoding/decoding time (s) on the Kodak dataset for quick comparison using VVC as an anchor, as follows:

Our method shows the fastest encoding/decoding time, which is $7\times$ faster than MS-ILLM in encoding, and $3\times$ faster than MS-ILLM in decoding. The comparisons demonstrate that our method can achieve a better trade-off between efficiency and effectiveness while preserving high flexibility to support variable bitrates in one model.

Weakness 2: Explanation on the obtaining for masks $m_1$, $m_2$, $m_3$

Response 2: In this work, considering that the masks are highly correlated with the target bitrate and corresponding granularity ratios, we would first clarify the details of the granularity ratio setting for specific bitrates. In our method, once the model is trained, we can obtain an index frequency table and assign the codes to each index during the entropy encoding process based on this table. By calculation, for a codebook with 1024 codes, the average code length of all the indices is $L=10.3875$. Supposing an input image with the size of $H\times W$, the amount of its assigned indices range is [$H/16\times W/16$, $H/4\times W/4$], which corresponds to full coarse-grained and fine-grained patch division. For each combination of granularity ratios ($r_1$, $r_2$, $r_3$), we can get its theoretical values of indices and masks by:

$\mathbf{Bpp_{Indices}}=\frac{\frac{H}{4}\times \frac{W}{4}\times r_1+\frac{H}{8}\times \frac{W}{8}\times r_2+\frac{H}{16}\times \frac{W}{16}\times r_3}{(H\times W)}\times L =\frac{(16r_1+4r_2+r_3)\times L}{256}$

$\mathbf{Bpp_{Mask}}=\frac{\frac{H}{4}\times \frac{W}{4}\times r_1+\frac{H}{8}\times \frac{W}{8}\times r_2}{H\times W} =\frac{4r_1+r_2}{256}$

where the total bpp is $(\mathbf{Bpp_{Indices}}+ \mathbf{Bpp_{Mask}}) =\frac{(16L+4)r_1+(4L+1)r_2+r_3}{256}, r_1\geq0, r_2\geq0, r_3\geq0, r_1+r_2+r_3=1$. Consequently, we can generate a query table for the theoretical bpps with ($r_1$, $r_2$, $r_3$). When receiving a user-given bpp, the model can automatically search the closest theoretical bpp in the query table and assign corresponding granularity ratios. In our investigations, we found that the error between the two values is less than 0.02, and the actual value is always slightly smaller than the theoretical value. Notably, our model allows users to modify the granularity ratios to mitigate such an error and produce better results. We also add the details in our Appendix A.3 (\textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}).

For the mask setting, with the given bpp and determined granularity ratios ($r_1$, $r_2$, $r_3$), our model automatically selects the patches with the top $r_3$ percent in order as the coarse ones. Through the encoder $E$, we can get a binary mask $m_3$ (0: not coarse, 1: coarse) and the corresponding elements in $z_3$ that are retained. Subsequently, $E$ removes the coarse-grained patches from the input $I$ based on $m_3$. We repeat the process on the remaining patches at medium granularity to generate $m_2$ and their corresponding elements in $z_2$. Finally, the rest patches are assigned to the fine-grained representation and $m_1$ can be calculated by 1-(m_2)$$\uparrow_2$$-(m_3)$$\uparrow_4.

Weakness 3: Typo errors

Response 3: Thanks for your careful reading. We have proofread the manuscript and revised the errors. The writing has been improved in the revised manuscript.

[Ref1] Matthew Muckley et al., Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models. ICML, 2023.

Weakness 4 & 5: Writing improvement in Section 3.2 and Section 3.3.

Response 4 & 5: Thanks for your suggestion, we have re-organized the description and explanation in Section 3.2 and Section 3.3, which are \textcolor[RGB]{147, 112, 219}{highlighted\ in \ purple} in the revised manuscript.

Questions 1 & 2: Clarity on the masks $m_1$, $m_2$, $m_3$ acquisition and setting of $r_1$, $r_2$, $r_3$

Response 1 & 2: In this work, considering that the masks are highly correlated with the target bitrate and corresponding granularity ratios, we would first clarify the details of the granularity ratio setting for specific bitrates. In our method, once the model is trained, we can obtain an index frequency table and assign the codes to each index during the entropy encoding process based on this table. By calculation, for a codebook with 1024 codes, the average code length of all the indices is $L=10.3875$. Supposing an input image with the size of $H\times W$, the amount of its assigned indices range is [$H/16\times W/16$, $H/4\times W/4$], which corresponds to full coarse-grained and fine-grained patch division. For each combination of granularity ratios ($r_1$, $r_2$, $r_3$), we can get its theoretical values of indices and masks by:

$\mathbf{Bpp_{Indices}}=\frac{\frac{H}{4}\times \frac{W}{4}\times r_1+\frac{H}{8}\times \frac{W}{8}\times r_2+\frac{H}{16}\times \frac{W}{16}\times r_3}{(H\times W)}\times L =\frac{(16r_1+4r_2+r_3)\times L}{256}$

$\mathbf{Bpp_{Mask}}=\frac{\frac{H}{4}\times \frac{W}{4}\times r_1+\frac{H}{8}\times \frac{W}{8}\times r_2}{H\times W} =\frac{4r_1+r_2}{256}$

where the total bpp is $(\mathbf{Bpp_{Indices}}+ \mathbf{Bpp_{Mask}}) =\frac{(16L+4)r_1+(4L+1)r_2+r_3}{256}, r_1\geq0, r_2\geq0, r_3\geq0, r_1+r_2+r_3=1$. Consequently, we can generate a query table for the theoretical bpps with ($r_1$, $r_2$, $r_3$). When receiving a user-given bpp, the model can automatically search the closest theoretical bpp in the query table and assign corresponding granularity ratios. In our investigations, we found that the error between the two values is less than 0.02, and the actual value is always slightly smaller than the theoretical value. Notably, our model allows users to modify the granularity ratios to mitigate such an error and produce better results. We also add the details in our Appendix A.3 (\textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}).

For the mask setting, with the given bpp and determined granularity ratios ($r_1$, $r_2$, $r_3$), our model automatically selects the patches with the top $r_3$ percent in order as the coarse ones. Through the encoder $E$, we can get a binary mask $m_3$ (0: not coarse, 1: coarse) and the corresponding elements in $z_3$ that are retained. Subsequently, $E$ removes the coarse-grained patches from $I$ based on $m_3$. We repeat the process on the remaining patches at medium granularity to generate $m_2$ and their corresponding elements in $z_2$. Finally, the rest patches are assigned to the fine-grained representation and $m_1$ can be calculated by 1-(m_2)$$\uparrow_2$$-(m_3)$$\uparrow_4.

** Weakness 1**: Comparing with HiFiC

** Response 1**: This is HiFiC trained separately to pursue their best performance on different specific R-D points. In this way, multiple fixed-rate models are necessitated to vary bitrates, leading to dramatic computational costs and inefficient deployment to cater to diverse bitrates and devices. In contrast, our method is customized to achieve flexible bitrate control within one unified model while preserving comparable perceptual quality and compression efficiency, which is the most important in this work. In Figure 5 (revised manuscript, \textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}), we further investigate the model efficiency on four terms: encoding time (s), decoding time (s), BD-rate (BD-LPIPS), and training steps (M). Here, for simple clarity, we provide the BD-LPIPS and the encoding/decoding time on the Kodak dataset in the following table for your reference:

For the inference speed, HiFiC suffers from critical time costs in both encoding and decoding. Our method achieves the fastest encoding/decoding time, which is $37\times$ faster than HiFiC in encoding and $67\times$ faster in decoding. Moreover, HiFiC as a single-point training method, requires independent training of $n$ models for $n$ R-D points. The proposed model requires only a single training session that enables compression across various bitrates, with the total training steps being substantially reduced to 0.6 million steps. By comparison, our method can achieve a promising balance among training costs, inference speed, and BD-rate saving.

Weakness 2: Comparison with MS-ILLM

Response 2: We apologize that the comparison with MS-ILLM is missing. In the revised manuscript, we have compared with MS-ILLM [ref1] on both DIV2K and Kodak datasets, where the results are in Figure 3 and Figure 4 (revised manuscript, \textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}), respectively. As we can see, MS-ILLM is trained for specific R-D points, which achieves relatively higher quantitative performance in LPIPS and DISTS. However, even though, our method exhibits comparable performance in FID and KID, and better NIQE than MS-ILLM. Here, for simple clarity, we provide training steps (M), BD-rate (BD-LPIPS) results on the DIV2K dataset, and encoding/decoding time (s) on the Kodak dataset for quick comparison using VVC as an anchor, as follows:

Our method shows the fastest encoding/decoding time, which is $7\times$ faster than MS-ILLM in encoding, and $3\times$ faster than MS-ILLM in decoding. The comparisons demonstrate that our method can achieve a better trade-off between efficiency and effectiveness while preserving high flexibility to support variable bitrates in one model.

Weakness 3: Tests on the CLIC 2020 dataset

Response 3: Thanks for your suggestion. We have provided the comparison on the CLIC2020 dataset, where the R-D performance and qualitative performance are shown in Figure 10 (Appendix, \textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}). Here, we provide the comparison in the following table for your reference:

It can be seen that Control-GIC achieves superior performance in most metrics over M&S and variable-rate method CTC. Compared to generative compression methods which are trained separately for multiple R-D points, our Control-GIC still maintains competitive performance, which validates that our method can achieve optimal trade-off between flexibility and effectiveness.

[Ref1] Matthew Muckley et al., Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models. ICML, 2023.

Dear Reviewer WUkA:

Hoping this message finds you well. Your comments and the provided insights are very valuable to our work. We will attach these analysis and additional experiments to our final version. As the rebuttal-discussion period is nearing its end, could you please review our response to see if it addresses your concerns? Your timely feedback will be extremely valuable to us. Could you read and let us know if there are more questions? We would be very grateful! Your decision is of utmost importance to us, and we earnestly hope that you will consider the significant contribution of this research to the field of image compression. Thank you very much!

Best regards,

All the authors

1. Considering performance, I believe it is acceptable for a variable bitrate solution to slightly underperform compared to a fixed bitrate solution. Similarly, if we aim for a lightweight approach, achieving faster encoding and decoding speeds with comparable performance is also acceptable. However, among the baselines compared in this paper, I did not observe such a case.

2. It seems that the paper does not specify how the query table is determined. Additionally, you mentioned using BFGS to determine $(r_1, r_2, r_3)$. Why was this method chosen? What is its principle? Can it guarantee an optimal result? The authors appear not to have provided an explanation.

Therefore, I maintain my score.

We much appreciate your timely feedback in your busy schedule. We are also very glad that you can provide such in-depth comments and analysis.

Comment 1: Model performance.

Response 1: In this work, our goal is to design a flexible, efficient, and effective generative compression method that accommodates a wide spectral range of bitrates in a unified model. Therefore, we compare with existing methods in 3 aspects:

1) Variable-rate methods: SCR and CTC. By the comparison in Figure 3, 4, and 5 (revised manuscript), our method achieves superior quantitative performance in most metrics and faster compression speed.

2) Generative methods: HiFiC, CDC, MS-ILLM, and MRIC. As you pointed out, their R-D performance is sometimes better than ours. But in perceptual quality LPIPS or non-reference quality assessment, our method actually achieves comparable performance while keeping faster speed in a variable-rate framework. Besides, in Figure 6 (revised manuscript), the results indicate that our method can reconstruct more preferable textures. Furthermore, based on the above response, it can be seen that even in a single R-D point pattern, our method is still comparable or superior with less encoding/decoding time.

3) Traditional CNN-based or non-deep learning-based codecs M&S, BPG, and VVC. These methods are prone to reconstruct compressed images based on the pixel-wise constraint (i.e. MSE), which shows obviously lower perceptual performance and worse subjective quality than ours as well as other generative compression methods.

Based on the above, we think that our method can achieve a better balance between flexibility, performance, and efficiency than existing methods. Therefore, we sincerely hope that you can reconsider the contributions and effects of this work.

Comment 2: Query table.

Response 2: The BFGS algorithm is widely recognized for its efficacy in addressing unconstrained optimization problems [Ref1-3]. Its core mechanism involves iteratively refining the search direction by approximating the inverse Hessian, which is the matrix of second derivatives of the objective function. Based on the findings in existing research [Ref1], BFGS has been demonstrated to benefit the convergence speed and model performance, which are essential for the efficient construction of our query table. Moreover, our model has demonstrated robustness across various granularity combinations. For clarity, here we provide some validation results on the Kodak dataset as follows:

We can observe that fine-tuning the ratio of the three granularities within a given combination has a minimal impact on image quality. The results indicate that varying the granularity ratios yields closely aligned bpp outcomes and further suggests that a certain target bpp can correspond to multiple solutions. Notably, there is no significant variation in the final image quality. This stability can be attributed to the model's comprehensive learning across all three granularities, which endows it with considerable robustness.

Based on these two reasons, we have selected the BFGS algorithm as our optimization method. Since the manuscript PDF revision has concluded, we guarantee that the pertinent algorithmic principles and processes will be detailed in the final version.

Last, we would like to thank you for your active discussion and valuable comments. We would be very grateful if you could kindly reconsider our contributions and explanations.

[Ref1] Nocedal, Jorge, and Stephen J. Wright, eds. Numerical optimization. New York, NY: Springer New York, 1999.

[Ref2] Gill, Philip E., Walter Murray, and Margaret H. Wright. Practical optimization. Society for Industrial and Applied Mathematics, 2019.

[Ref3] Bottou, Léon, Frank E. Curtis, and Jorge Nocedal. "Optimization methods for large-scale machine learning." SIAM review 60, no. 2 (2018): 223-311.

Dear Reviewer,

Thank you for your feedback. We noticed the lower score and would like to understand the reasons behind it. Could you please share any specific concerns or areas for improvement? We are committed to addressing them to enhance our work.

Best regards,

All the authors

Weakness 1: Compare with CDC in DISTS

Response 1: As we know, DISTS measures the structure and texture similarity between compressed and uncompressed images. CDC is implemented on the conditional diffusion model which leverages encoded latent variables containing sufficient structural and semantical information as the condition. Besides, a high bitrate indicates more accurate conditional information can be exploited to facilitate the reconstruction in CDC, resulting in better DISTS scores. We would like to explain that, in generative compression, the LPIPS, FID, or NIQE are also important metrics. As shown in Figure 3 and Figure 4 (revised manuscript, \textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}), our method achieves better performance.

This can be attributed to our dynamic granularity adjustment approach and the generative ability of VQGAN. Moreover, in literature, the methods including CDC trained separately to pursue their best performance on different specific R-D points often exhibit superiority over variable-rate compression methods. Our method is customized to achieve flexible bitrate control within one unified model while preserving comparable perceptual quality and compression efficiency. Even though, our method still performs comparably in most metrics across multiple datasets against CDC.

In addition, for the inference speed, CDC applies a lightweight diffusion variational autoencoder which benefits the encoding process but struggles with more decoding time due to its iterative reverse process. Our method achieves $1.5\times$ faster than CDC in encoding, and $45\times$ faster in decoding. The comparison on the Kodak dataset is as follows:

Weakness 4: Performance different patch size

Response 4: When the patch size is increased, then a single code needs to represent more image regions, and texture distortion occurs when local region details and textures are complex. Reducing the patch size can effectively solve the block artifact problem in the VQ method, and the reconstruction of local small faces in the image will be improved. In this work, we change the patch size based on the assignment of granularity ratios, where the results are shown in Figure 9 (Appendix A. 4, \textcolor[RGB]{0, 205, 205}{highlighted\ in\ cyan}). We use the results of CDC and MS-ILLM are used as references. The small face poses a critical challenge in image compression. Both CDC and MS-ILLM struggle to obtain promising generations. When we assign overly high coarse- and medium-granularity ratios on the face area, i.e. large patch size, the models also cannot recover the face details well. When we set the patches in the face region as fine-granularity ones and reduce their patch size, the artifacts can be further alleviated, where the generated images reveal clearer details. Consequently, our model using (40%, 40%, 20%) yields a better reconstruction and lower bpp. Despite the superiority, due to the complexity of small faces, there is still a need for more targeted design.

Weakness 2: The comparison in Figure 4

Response 2: We apologize that some results are missing in Figure 3. This is because our initial view posited that the analysis of the Kodak dataset would suffice to ascertain the efficacy of all the methods. Therefore, in Figure 4, on the DIV2K dataset, we select the variable-rate method CTC and GAN-based method according to the performance on the Kodak, which are related to ours (variable-rate and GAN-based). The BPG as a classical method on specific compression rates is applied as a reference. Besides, to provide more comprehensive comparisons, we introduce another two metrics FID and KID to help measure their performance. We appreciate your comments that the complete comparisons on DIV2K are more important than those on Kodak actually. In the revised manuscript, we have included the additional comparisons in Figure 4 (revised manuscript, \textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}). Our Control-GIC surpasses almost all the methods across most distinct metrics and exhibits a significant improvement in NIQE. We also include more generative models (MRIC [Ref1] and MS-ILLM [Ref2]) in Figure 4. As we can see, these two methods are trained for specific R-D points, which achieve relatively higher quantitative performance in LPIPS and DISTS. However, even though, our method exhibits comparable performance in FID and KID, and better NIQE than these two most state-of-the-art methods. Here, for simple clarity, we provide training steps (M), BD-rate (BD-LPIPS) results on the DIV2K dataset, and encoding/decoding time (s) on the Kodak dataset for quick comparison using VVC as an anchor, as follows:

It can be seen that the variable-rate approach represented by SCR does not perform as well as models trained on a single point (e.g., M&S), while CTC obtains superior performance with a larger number of parameters (399M, computed from the official open-source code), but with a very large drop in model efficiency (the encoding time is $38\times$ times longer and the decoding time is $40\times$ times longer than M&S). MRIC and MS-ILLM trained separately for each compression rate obtained very close BD-rate savings with our Control-GIC, outperforming other methods. Our method achieves the fastest encoding/decoding time, which is $7\times$ faster than MS-ILLM and $4\times$ faster than MRIC in encoding, and $3\times$ faster than MS-ILLM and $1.5\times$ faster than MRIC in decoding. Moreover, the single-point training methods (e.g., HiFiC, CDC, MRIC, MS-ILLM) require independent training of $n$ models for $n$ R-D points. Our proposed model requires only a single training session that enables compression across various bitrates, with the total training steps being substantially reduced to 0.6 million steps. By comparison, our method can achieve a promising balance among training costs, inference speed, and BD-rate saving.

Weakness 3: Performance saturation when the bpp increases

Response 3: As you pointed out, in most cases, when bpp increases to a certain extent, the improvement in objective metrics gradually becomes very marginal. Besides, the difference in reconstructed images by high bitrates in human visual perception also decreases to very low. To more intuitively illustrate such differences, in Figure 7 (revised manuscript, \textcolor[RGB]{0, 205, 205}{highlighted\ in\ cyan}), we calculate the difference maps between compressed images and original one, one can see that, as the bpp increases, there is also a decrease in information loss, which indicates an improved capability of the model to for better image details recovery.

[Ref1] Eirikur Agustsson et al., Multi-realism image compression with a conditional generator. CVPR, 2023.

[Ref2] Matthew Muckley et al., Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models. ICML, 2023.

Dear Reviewer P96B:

Hoping this message finds you well. Your comments and the provided insights are very valuable to our work. Besides previous responses and revisions, we now add more experiments and analyses, including 1) well-developed theoretical support and detailed descriptions of the granularity assignments; 2) results of additional datasets; and 3) exploration of the extremely low bitrates. We will attach these analyses and additional experiments to our final version. As the rebuttal-discussion period is nearing its end, could you please review our response to see if it addresses your concerns? Your timely feedback will be extremely valuable to us. Could you read and let us know if there are more questions? We would much appreciate the clarification related to soundness, presentation, and contribution. Your decision is of utmost importance to us, and we earnestly hope that you will consider the significant contribution of this research to the field of image compression. Thank you very much!

Best regards,

All the authors

Weakness 4: Results on CLIC2020

Response 4: Thanks for your suggestion. We have provided the comparison on the CLIC2020 dataset, where the R-D performance and qualitative performance are shown in Figure 10 (Appendix, \textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}). Here, we provide the results of several representative methods for comparison.:

It can be seen that Control-GIC achieves superior performance in most metrics over M&S and variable-rate method CTC. Compared to generative compression methods which are trained separately for multiple R-D points, our Control-GIC still maintains competitive performance, which validates that our method can achieve optimal trade-off between flexibility and effectiveness.

Questions (1): Justification on the mask ratio setting.

Response (1): In our method, when the granularities of image patches changes, the characteristics of input data processed by the model also change. In actual, for each specific granularity ratio combination of data, the model learns as a new task. Therefore, the model tends to suffer from the catastrophic forgetting among multiple combinations, which is a common challenge in deep learning tasks. Regard to our controllable variable bitrate approach, it is necessary to ensure the model for peak performance given a granularity ratio while mitigating the performance degradation for other ratios.

During experimental process, we have researched various combinations for training the model. However, many of them cannot result in even acceptable reconstruction. Then, we rethink the optimization for our model as a geometric optimization problem. Imagine multiple points on a line segment; the point that minimizes the total distance to all other points is the midpoint. Similarly, in our context, we aim to position our model on the R-D curve such that its parameters are as close as possible to those of a model trained at any given bitrate point. The optimal point for this is the midpoint, which serves as a balanced and versatile reference point for model performance across different bitrates. As a result, we get the ratio setting (fine: 50%, medium : 40%, and coarse: 10%), which can yield optimal results.

Questions (2): Explanation for the bpp range.

Response (2): Similar to the explanation in Response 3, where the lowest bitrate ($<0.05$ bpp) of our method corresponds to a fully coarse-grained partition, i.e. $(r_1, r_2, r_3)$ = (0, 0, 100%) for fine, medium, and coarse. We use Mao et al. [Ref3], a VQGAN-based method also for very low bitrate, as a reference. The quantitative results can also be seen in Response 3.

[Ref3] Qi Mao, et al., Extreme image compression using fine-tuned vqgan models. DCC, 2024.

Weakness 3: Comparison with VQGAN methods

Response 3: Thanks for your valuable suggestion. The main reason is that our Control-GIC and these methods are different from the bpp ranges and bitrate flexibility.

1) Different bpp ranges. [Ref3] and [Ref4] address the challenge of extremely low bitrate compression with tailored strategies. The encoder in [Ref3] employs 16x downsampling and clusters a 16384-sized codebook into {2048, 1024, 512, 256, 128, 64, 32, 16, 8}, managing bitrate by adjusting codebook sizes. Similarly, the encoder in [Ref4] uses 16x downsampling, discards about 40% of features, and clusters the codebook of size 16384 into {16, 64, 256}, controlling bitrate through codebook clustering as well. To clarify the bitrate calculations, we assume an input image resolution of $256\times256$ pixels, with a uniform distribution of codes within the codebook, meaning all indexes have equal bitstream lengths. For [Ref3], the encoder yields $16\times16$ feature maps with codebook sizes varying from 8 to 2048 ($2^3$ to $2^{11}$), translating to the bit transmission range is [$16\times16\times3$, $16\times16\times11$], and the bpp interval of [0.011, 0.043]. For [Ref4], with a $16\times16$ feature map retaining about 60% of feature points and codebook sizes from 16 to 256 ($2^4$ to $2^8$), the bit transmission range is [$16\times16\times0.6\times4$, $16\times16\times0.6\times8$], resulting in the bpp interval of [0.009, 0.019]. For [Ref5], the upper limit of the bitrate reported in its paper does not exceed 0.035.

Differently, our Control-GIC employs a multi-grained encoder with variable downsampling factors—$4\times$ (fine), $8\times$ (medium), and $16\times$ (coarse)—and a fixed codebook size of 1024. We manage the bitrate by adjusting the ratio of these granularities, offering a significant advantage over the aforementioned methods. The lowest bitrate of our method corresponds to a fully coarse-grained partition, yielding a $16\times16$ feature map, while the highest bitrate corresponds to a fully fine-grained partition, resulting in a $64\times64$ feature map. With a codebook size of 1024 ($2^{10}$), the range of bits required for transmission is [$16\times16\times10$, $64\times64\times10$], and the bpp interval is [0.040, 0.625], catering to a broad range of compression requirements and ensuring an optimal balance between compression efficiency and image quality.

2) Bitrate flexibility. These methods train the models separately for specific R-D points with Lagrange multiplier $\lambda$-weighted R-D loss, each corresponding to an individual $\lambda$. In this way, multiple fixed-rate models are necessitated to vary bitrates, leading to dramatic computational costs and inefficient deployment to cater to diverse bitrates and devices. In contrast, our method is capable of fine-grained bitrate adaption while preserving high-perceptual fidelity reconstruction in a unified model.

To investigate the performance of our method, we have conducted experiments on extremely for extremely low bitrate compression (<0.05bpp). Since the source codes and pre-trained checkpoints of [Ref3]-[Ref5] are unavailable, we report the best-approximated results of [Ref3] on Kodak and CLIC2020 datasets based on the public paper, where quantitative results are as follows:

Our method achieves the best LPIPS on both datasets with lower bpp, validating its superiority. Moreover, we also provide several visualizations of compressed images in Figure 11 (Appendix, \textcolor[RGB]{184,134,11}{highlighted\ in\ gold}), where the results demonstrate that our method can be well-generalized to low bitrates and maintain vivid reconstruction.

[Ref3] Qi Mao, et al., Extreme image compression using fine-tuned vqgan models. DCC, 2024.

[Ref4] Naifu Xue, Qi Mao, et al., Unifying Generation and Compression: Ultra-low bitrate Image Coding Via Multi-Stage Transformer. ICME, 2024.

[Ref5] Zhaoyang Jia et al., Generative Latent Coding for Ultra-Low Bitrate Image Compression. CVPR, 2024.

Weakness 2: More comparisons in Figure 4.

Response 2: We apologize that some results are missing. We have included the additional comparisons in Figure 4 (revised manuscript, \textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}). Our Control-GIC achieves comparable performance in most metrics and exhibits a significant improvement in NIQE. Notably, compared to SCR and CTC, Control-GIC achieves finer granular flexibility for bitrate control while preserving obvious preferable perceptual quality. Besides, since conventional CNN-based NIC methods, e.g. M&S (Hyperprior) and SCR, optimize for the R-D trade-off using pixel-wise MSE loss, one can see that they produce relatively higher PSNR than HiFiC and CDC as well as ours.

We also include more generative models (MRIC [Ref1] and MS-ILLM [Ref2]) in Figure 4. These two methods are trained for specific R-D points, which achieve relatively higher quantitative performance in LPIPS and DISTS. However, even though, our method exhibits comparable performance in FID and KID, and better NIQE and model efficiency than these two most state-of-the-art methods. Here, for a simple clarity, we provide training steps (M), BD-rate (BD-LPIPS) results on the DIV2K dataset and encoding/decoding time (s) on the Kodak dataset for quick comparison using VVC as an anchor, as follows:

It can be seen that the variable-rate approach represented by SCR does not perform as well as models trained on a single point (e.g., M&S), while CTC obtains superior performance with a larger number of parameters (399M, computed from the official open-source code), but with a very large drop in model efficiency (the encoding time is $38\times$ times longer and the decoding time is $40\times$ times longer than M&S). MRIC and MS-ILLM trained separately for each compression rate obtain very close BD-rate savings with our Control-GIC, outperforming other methods. Our method achieves the fastest encoding/decoding time, which is $7\times$ faster than MS-ILLM and $4\times$ faster than MRIC in encoding, and $3\times$ faster than MS-ILLM and $1.5\times$ faster than MRIC in decoding. Moreover, the single-point training methods (e.g., HiFiC, CDC, MRIC, MS-ILLM) require independent training of $n$ models for $n$ R-D points. Our proposed model requires only a single training session that enables compression across various bitrates, with the total training steps being substantially reduced to 0.6 million steps. By comparison, our method can achieve a promising balance among training costs, inference speed, and BD-rate saving.

[Ref1] Eirikur Agustsson et al., Multi-realism image compression with a conditional generator, CVPR, 2023.

[Ref2] Matthew Muckley et al., Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models, ICML, 2023.

Weakness 1: Clarify the details on the granularity division in Section 3.1

Response 1: We are sorry that some details are missing, which makes you confused. As shown in Figure 2 (revised manuscript), given an image $x$ divided into serial patches, the encoder $E$ first calculates the entropy values of all patches and sorts them from low to high. We define three types of granularities (coarse, medium, high) for these patches. Once model trained, we can obtain an index frequency table and assign the codes to each index during the entropy encoding process based on this table. By calculation, for a codebook with 1024 codes, the average code length of all the indices is $L=10.3875$. Supposing an input image with the size of $H\times W$, the amount of its assigned indices range is [$H/16\times W/16$, $H/4\times W/4$], which corresponds to full coarse-grained and fine-grained patch division. For each combination of granularity ratios ($r_1$, $r_2$, $r_3$), we can get its theoretical values of indices and masks by:

$\mathbf{Bpp_{Indices}}=\frac{\frac{H}{4}\times \frac{W}{4}\times r_1+\frac{H}{8}\times \frac{W}{8}\times r_2+\frac{H}{16}\times \frac{W}{16}\times r_3}{(H\times W)}\times L =\frac{(16r_1+4r_2+r_3)\times L}{256}$

$\mathbf{Bpp_{Mask}}=\frac{\frac{H}{4}\times \frac{W}{4}\times r_1+\frac{H}{8}\times \frac{W}{8}\times r_2}{H\times W} =\frac{4r_1+r_2}{256}$

where the total bpp is $(\mathbf{Bpp_{Indices}}+ \mathbf{Bpp_{Mask}}) =\frac{(16L+4)r_1+(4L+1)r_2+r_3}{256}, r_1\geq0, r_2\geq0, r_3\geq0, r_1+r_2+r_3=1$. Consequently, we can generate a query table for the theoretical bpps with ($r_1$, $r_2$, $r_3$). When receiving a user-given bpp, the model can automatically search the closest theoretical bpp in the query table and assign corresponding granularity ratios. In our investigations, we found that the error between the two values is less than 0.02, and the actual value is always slightly smaller than the theoretical value. Notably, our model allows user to modify the granularity ratios to mitigate such an error and produce better results. Here, we provide a simplified query table that describes the correlations between theoretical bpp and granularity ratios, as an example. Our model enables a very meticulous control of bitrates. We also add the details in our Appendix A.3 (\textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}).

Weakness 2: Discussion with VQGAN-based methods.

Response 2: Here, we would like to discuss with existing VQGAN-based compression methods [Ref3] - [Ref5], to clarify the differences and our contributions.

First, Mao et al. [Ref3] and UIGC [Ref4] control the bitrate by adjusting the codebook size, which necessitates the fine-tuning on the model for each codebook size to achieve variable bitrates. GLC [Ref5] introduces VQVAE in the hyperprior model to mine the semantic visual components and guarantee the reconstruction fidelity, which also requires multiple trainings with different $\lambda$ to realize multiple bitrates. In contrast, our method only requires training once and does not involve any fine-tuning, which enables to produce the compression results across a broad spectrum of bitrates.

Second, these methods are motivated by the representative and generative ability of VQGAN, which ignores explicitly exploring the local image characteristics and the differences between patches (e.g., local information density). Our method correlates the information density of local image patches with their granular representations. Hence, we can flexibly determine a proper allocation of granularity for the patches to achieve dynamic adjustment for VQ-indices, resulting in desirable compression rates. Our method assigns different granularities to regions of different complexity and achieves fine control of the bitrate through the dynamic adjustment of granularities.

Third, to our knowledge, the mentioned knowledge above focuses on extremely low bitrate compression (<0.05bpp), whereas our method is prone to support a more general bit rate range within 0.1~0.6bpp. Besides, to investigate the effectiveness of our method at such a low bitrate. In Appendix A.6, we set the granularity of all patches as coarse ones, i.e., the granularity for fine, medium, and coarse are $(r_1, r_2, r_3)$ = (0, 0, 100%), resulting in the theoretical lowest bitrate. We report the quantitative results compared to [Ref3] in Bpp and LPIPS on the Kodak and CLIC2020 datasets, which are shown in Table 3 (Appendix A.6, \textcolor[RGB]{184, 134, 11}{highlighted\ in\ gold}).

One can see that our method achieves the best LPIPS on both datasets with lower bpp, validating its superiority. Moreover, we also provide several visualizations of compressed images in Figure 11 (Appendix A.6, \textcolor[RGB]{184,134,11}{highlighted\ in\ gold}), where the results demonstrate that our method can be well-generalized to low bitrates and maintain vivid reconstruction.

[Ref3] Qi Mao, et al., Extreme image compression using fine-tuned vqgan models. DCC, 2024.

[Ref4] Naifu Xue, Qi Mao, et al., Unifying Generation and Compression: Ultra-low bitrate Image Coding Via Multi-Stage Transformer. ICME, 2024.

[Ref5] Zhaoyang Jia, et al., Generative Latent Coding for Ultra-Low Bitrate Image Compression. CVPR, 2024.

Weakness 1: More comparisons with other GIC methods.

Response 1: We have compared our method with two additional latest GIC methods, MRIC [Ref1] and MS-ILLM [Ref2] on both Kodak and DIV2K datasets, where the results are shown in Figure 3 - 5 (revised manuscript, \textcolor[RGB]{65, 105, 225}{highlighted\ in\ blue}), respectively. One can see that, these two methods are trained for specific R-D points, which achieve relatively higher quantitative performance in LPIPS and DISTS. However, even though, our method exhibits comparable performance in FID and KID, and better NIQE and model efficiency than these two most state-of-the-art methods. Here, for simple clarity, we provide training steps (M), BD-rate (BD-LPIPS) results on the DIV2K dataset, and encoding/decoding time (s) on the Kodak dataset for quick comparison using VVC as an anchor, as follows:

It can be seen that the variable-rate approach represented by SCR does not perform as well as models trained on a single point (e.g., M&S), while CTC obtains superior performance with a larger number of parameters (399M, computed from the official open-source code), but with a very large drop in model efficiency (the encoding time is $245\times$ times longer and the decoding time is $85\times$ times longer than ours). MRIC and MS-ILLM trained separately for each compression rate obtained very close BD-rate savings with our Control-GIC, outperforming other methods. Our method achieves the fastest encoding/decoding time, which is $7\times$ faster than MS-ILLM and $4\times$ faster than MRIC in encoding, and $3\times$ faster than MS-ILLM and $1.5\times$ faster than MRIC in decoding. Moreover, the single-point training methods (e.g., HiFiC, CDC, MRIC, MS-ILLM) require independent training of $n$ models for $n$ R-D points. Our proposed model requires only a single training session that enables compression across various bitrates, with the total training steps being substantially reduced to 0.6 million steps. By comparison, our method can achieve a promising balance among training costs, inference speed, and BD-rate saving.

[Ref1] Eirikur Agustsson et al., Multi-realism image compression with a conditional generator, CVPR, 2023.

[Ref2] Matthew Muckley et al., Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models, ICML, 2023.

Dear Reviewer o1HZ:

Hoping this message finds you well. Your comments and the provided insights are very valuable to our work. Besides previous responses and revisions, we now add more experiments and analyses, including 1) well-developed theoretical support and detailed descriptions of the granularity assignments, and 2) results from additional datasets. We will attach these additional experiments and analyses to our final version. As the rebuttal-discussion period is nearing its end, could you please review our response to see if it addresses your concerns? Your timely feedback will be extremely valuable to us. Could you read and let us know if there are more questions? We would be very grateful! Your decision is of utmost importance to us, and we earnestly hope that you will consider the significant contribution of this research to the field of image compression. Thank you very much!

Best regards,

All the authors