CMC-Bench: Towards a New Paradigm of Visual Signal Compression

Dear Reviewer kYAX,

Thank you for your positive comments, which greatly motivated us. Regarding the weaknesses and questions you are concerned about, we summarize them as Q1-Q4:

Q1: Defination of 1,000 times compreesion as ULB. Has previous work reached it?

Since the original image is in RGB 8-bit format, i.e. 24 bpp, we set 0.024 bpp as ULB. This 1,000 compression rate, is related to the current development of codecs. Generally speaking, the compression rate of codecs doubles every ten years and is now 500 times, so we set 1,000 as the goal to be achieved.

In the past, some models did achieve 1,000 times [1][2][3], but their results were poor and almost completely different from the original image. It can be said that CMC is the first to give satisfactory results at 1,000 times compression.

Q2: More I2T models for benchmark.

We have added four advanced models, including two I2T (Gemini, InternLM-XComposer) and two T2I (Clandestine, SUPIR). Their performance is shown in the radar map attached in appendix section A.7.2. Note that we will maintain this project in the long term, and the performance of mainstream LMMs on CMC-Bench will be updated on the leaderboard every month.

Q3: std performance for each T2I metric.

We originally wanted to attach the standard deviation in the main text, but had to remove it due to page width. I am happy to update this part here.

The three sub-columns in each column of the above table represent the effects of NSI, SCI, and AIGI. Compared with the scores in the main text Table 4, we find that the standard deviation of the model is relatively uniform. Basically, there is no encoder/decoder with a huge difference between the upper and lower limits.

Q4: Open source all data.

Yes, We will make all the contents of CMC-Bench public, including image data, subjective annotations, and end-to-end test pipelines. We sincerely hope that it can inspire LLM researchers and drive the evolution of the codec paradigm.

Reference

[1] Xue, Naifu, et al. Unifying Generation and Compression: Ultra-low bitrate Image Coding Via Multi-stage Transformer. Arxiv, 2024.

[2] Gao, Junlong, et al. Cross modal compression with variable rate prompt. IEEE TCSVT, 2024.

[3] Gao, Junlong, et al. Rate-distortion optimized cross modal compression with multiple domains. IEEE TCSVT, 2024.

Best Regards

CMC-Bench Author Team

Q7: Are the authors concerned that this paper’s originality might be questioned? (looks like a combination of existing works)

In general, CMC-Bench has certain novelty for both issues you are concerned about. (1) Data source. CMC-Bench has only 1,000 reference images taken from the previous dataset, and 58,000 distorted images are all generated by itself. Most importantly, these images have 160,000 quality annotations, all of which come from human subjective evaluation. Therefore, most of the dataset is original. (2) Whether it is a combination of existing models. CMC-Bench is not a simple connection of existing models. We have comprehensive considerations on bitrate, parameter settings, and reference image formats, such as the verification of parameter λ in Figure A5. If I2T+T2I are directly connected without any operation, you can refer to the results of PICs [1] and Image2Paragraph [2], which are disastrous. In short, CMC-bench provides an effective combination and surpasses VVC for the first time, which is original.

Most importantly, we propose an end-to-end evaluation pipeline, not just a compression algorithm. Any LMM developer (both T2I and I2T) can plug in and test performance easily. It can expand the influence of the codec community and directly benefit from the rapid development of LLM. I think this contribution is more important than one codec.

Q8: Further discussion on the balance between consistency and perceptual quality, and polish certain expressions.

Thank you for your suggestion. We have incorporated the response in Q5, namely consistency-perception trade-off, [3][4] into the manuscript, you may check appendix section A.7.3. Regarding writing, we invited a native English-speaking professor from Canada to revise the paper at the character level.

Reference

[1] Lei, Eric, et al. Text+ sketch: Image compression at ultra-low rates. arXiv preprint arXiv:2307.01944. 2023.

[2] ShowLab. Image2Paragraph. https://github.com/showlab/Image2Paragraph. 2023.

[3] Blau, Yochai, et al. The perception-distortion tradeoff.In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6228–6237, 2018.

[4] Blau, Yochai, et al. Rethinking lossy compression: The rate-distortion-perception tradeoff.In International Conference on Machine Learning, pages 675–685. PMLR, 2019.

Best Regards

CMC-Bench Author Team

Dear Reviewer 1Puj,

Thank you for the positive comments on CMC-Bnech. In particular, thanks for your character-level review of our article. We are honored to receive such a responsible review. We will reply to your Question from Q1-Q7, and then to Weakness as Q8.

Q1: Full name and corresponding unit in Table 1.

Sure. We have added the full name of dis/ref in the title of Table 1. Please check the latest manuscript.

Q2: Figure 5 represents 12 models but only six methods are shown on the chart.

For aesthetic reasons, it would be messy to plot all 12 T2I models on the radar chart. Thus we only visualized 6 representative models in Figure 5. The detailed performance of all models is placed in Table 3 below.

According to your suggestion, we also changed the title of Figure 5 to 6 models.

Q3: Typo in Section 4.2.

Thanks for addressing such a hidden typo! We have corrected it in the latest manuscript.

Q4: The three colors in the histogram in Figure A3 are not all explained within the figure.

In fact, there are only two colors. For better visualization, we set the transparency of them to 0.5. The third color you see is the overlap of the two colors.

Q5: Balancing consistency and perceptual quality when establishing the evaluation metrics, and ablation to determine the final weights

We set the ratio of consistency and perceptual quality to 2:1 for the following reasons: (1) Consistency is more important than perception. As it is feasible to enhance low-quality, high-consistency images (like using super-resolution); while improving a high-quality, low-consistency image is challenging. (2) Consistency and perception have different fluctuation ranges. Here we use TOPIQ-FR and TOPIQ-NR respectively. Table 3/4 shows that the former has a smaller fluctuation range. Therefore, for fairness, its weight needs to be increased.

We added ablation experiments to demonstrate why equal (1:1) or higher (3:1) weights are incorrect. First, Figure A8-A13 shows that under ULB (the first two rows of images), the results obtained by methods such as RealVis/SSD-1B are consistent with the original image, while the methods focusing on Perception, such as PG20/PG25, perform poorly. Under ELB (the last two rows), the results obtained by all methods are not completely consistent with the original image, so PG25/PG20 should be ranked higher by humans.

The leaderboard above shows that if a 1:1 weight is used, PG25/PG20 will rank second/third respectively under ULB, which places too much emphasis on Perception; if a 3:1 weight is used, RealVis will win the championship under ELB, which places too much emphasis on Consistency. Therefore, we believe this ratio is most consistent with the subjective evaluation results of humans.

Q6: Further adjust the organization of the main text and appendix.

Yes, we have made revisions. We have added more experimental settings in the main text and moved some data analysis to the appendix section A.7.

Q5: Bias during the human annotation.

Yes, bias is an important factor. Therefore, we calculated the correlation of subjective scores between the 20 subjects, represented by the Spearman Rank Correlation Coefficient (SRCC). Due to space limitations, we only present the results for the first 10 subjects as shown in the table below:

According to the ITU standard [4], an SRCC of 0.6 or higher indicates that a subject has been diligent in their scoring, while an SRCC of 0.7 or higher indicates that two subjects have similar opinions. An SRCC of 0.9 or higher is considered too high (lacking differences). The table above shows that the SRCC for the subjects is within an acceptable range, with no outliers that are too low and no duplicated data that is too high. Therefore, the bias issue has been effectively addressed.

Reference

[1] Sun, Peize, et al. Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation. arXiv. 2024.

[2] Mentzer, Fabian, et al. "High-fidelity generative image compression." Advances in Neural Information Processing Systems 33. 2020.

[3] Yang, Ruihan, and Stephan Mandt. Lossy image compression with conditional diffusion models. Advances in Neural Information Processing Systems 36. 2024.

[4] I.T.Union. Methodology for the subjective assessment of the quality of television pictures. ITU-R Recommendation BT.500-11, 2002.

Best Regards

CMC-Bench Author Team

Dear Reviewer CJXY,

We appreciate your constructive feedback. Your guidance is crucial in advancing our work, especially in giving some in-depth insight beyond the experiment. On the technical level, we have summarized your opinion about weakness as Q1-Q3 and questions as Q4-Q5, then responded to them one by one:

Q1: Give some valuable non-trivial insight, such as “which type of T2I architecture would perform better for CMC task.”

Thank you for your feedback. Based on your request, we have drawn the following conclusions beyond that "CMC is superior to traditional codecs":

1. Is the AR or the Diffusion architecture better for CMC tasks? Generally speaking, the Diffusion architecture is better. This is not because the AR performance metrics are insufficient, but because generating pixels one by one leads to a higher complexity. We acknowledge that the AR model is more controllable and may sometimes achieve better performance than the Diffusion model, but it requires a massive amount of parameters for the T2I model and longer decoding times. Work [1] shows that AR generation of a 384*384 image takes more than 6 seconds, which is enough time to generate several 1024-resolution images with a Diffusion model whose size is less than 1B. Considering the real-time nature of visual communication, we believe that the Diffusion model is more suitable for CMC.
2. Among the Diffusion architectures, which one is better? This requires a case-by-case discussion, including the Restoration, SDXL, and SD architectures. The leaderboard in Figure 6 shows that on the ULB, the order is Restoration > SD > SDXL, because Restoration fine-tunes based on the original image, SD uses the original image as the starting point to draw, and SDXL, compared to SD, has more freedom to make changes, which can lead to excessive modifications. On the ELB, the order is SDXL > SD, because without a reference image, Restoration is directly unavailable. Compared to SDXL's aggressive strategy, which becomes an advantage, it can produce more details.

Due to space limitations, we decided to add these analyses to the appendix section A.7.1 to inspire the codec community's thinking on this issue.

Q2: Technical algorithm, or direct insight in the proposed benchmark to address the inconsistency issue.

Yes, as a benchmark paper, its focus is indeed not on the technical algorithms, but it still gives direct insights for inconsistency problems. Firstly, CMC-Bench has proven that the cascaded approach of I2T + T2I can directly surpass the state-of-the-art VVC. Note that this is different from the previous conclusion that generative codecs are superior to traditional codecs. This is because previous methods required fine-tuning processes such as VQ-VAE-2 [2] and DDIM [3], whereas the approach based on LLM is completely training-free. Secondly, CMC-Bench has obtained effective evaluation standards through a large amount of human subjective preference data, as seen in Table 2. Take consistency as an example, the lpips commonly used to evaluate codec performance have a correlation of less than 0.6 with human subjective scores, while the data-driven by TOPIQ has a correlation with human preferences that exceeds 0.9. Compared to traditional performance indicators, it provides a more accurate direction for optimization in the inconsistency problem.

To avoid the suspicion of overclaiming, we have modified the expression of the sentence in the abstract.

Q3: Benchmark reliability. For example, the increasing/decreasing in Figure 7.

This is an interesting question. The fluctuation of the FID is due to the four different modes of CMC, of which the first two modes (Text and Pixel, i.e., the ELB in Q1) grant the T2I model a greater degree of editing freedom, at 0.8; while the latter two modes (Image and Full, i.e., the ULB in Q1), in order to improve consistency, reduce the freedom to 0.5. As we emphasize in our paper, consistency and perception are a dilemma, and the codec task must first ensure consistency, so we have made this setting of permissions. Therefore, you will see that most of the consistency metrics in Figure 7 increase with the bpp, while perception shows an unusual drop between the second and third points.

Q4: Specific example about solving the inconsistency issue.

Yes, we have added examples in Figure A17 of the Appendix, please check the main PDF. The figure shows that while the consistency of CMC may not be perfect, it still significantly outperforms VVC. Note that CMC is targeted at Ultra-low bitrate, although it still has shortcomings, other models are almost unable to work in such a challenging scenario. Therefore, we believe that CMC provides the first acceptable baseline solution for inconsistency issues.

Q5: PSNR result is not reported.

The experiment follows the evaluation principle of covering all high-mid-low levels. Since SSIM is already available for low-level, PSNR is not shown in the main text.

We have added the PSNR results as you requested, see Figure A10 in the appendix.

Q6: The RD curve of GPT-4o + RealVis in Fig.7 looks weird.

This is an interesting question. The curve in Figure A4 appears unusual because the y-axis range is small. When compared to the -axis in Figure A5 below, it becomes clear that the changes are very weak. We are demonstrating that as the number of words in the I2T model increases, the performance changes very little, almost like minor oscillations. This ensures that the length of the text will not affect the fairness of the benchmark.

Q7: The training does not include images generated by other methods, such as traditional codecs or any learning-based codecs. Wouldn't the indicators trained on this data be biased?

Yes, we used GPT-4o+12 T2I as training data for the following reasons: (1) No other I2Ts: The differences between I2Ts are not large, as shown in the annotations on Figure 5. The performance of different I2Ts differs by at most 0.05, and the performance of different T2Is differs by 1. Therefore, we did not use data like Qwen+12 T2I or LLaVA+12 T2I. (2) No other codec: The initial weights of the IQA model are already trained on compressed images (JPEG, JP2K, VVC), thus there is prior knowledge. From Table 2, both FR and NR exceed 0.9. With such high performance, there is almost no deviation in the IQA field. [3]

Q8: The method for obtaining the compression rate.

We set the compression rate based on VVC QP=53. You can refer to the description in Appendix Section A.6.

Q9: Randomness in CMC-based codecs.

As we replied in Q3, the randomness of CMC is not that serious. The larger error bar is because we introduced different λ parameters in the training data. We sincerely apologize again for the misunderstanding.

Q10: If CMC-based codecs outperform traditional codecs in objective evaluations? Even if CMC outperforms others, do they really have practical applications in real-world compression scenarios?

Outperform traditional metric: Yes. Note this work faces an extremely compressed scenario. (In other words, it is unfair to compare the fidelity of two images when CMC is compressed 10,000 times and the traditional method is compressed 100 times. We need to compare them at the same bpps). CMC has the following two fidelity advantages over existing work: (1) At a compression ratio of 1,000 times, namely 0.024bpp, the traditional method can only give the outline of the image, while CMC can restore the approximate information. In other words, even if CMC has a small probability of "you are not yourself", the traditional compression will most likely produce "you are pixel"; (2) Under a compression ratio of 10,000 times, namely 0.002bpp, no traditional method can achieve this extreme condition, while CMC can at least semantically reconstruct the image and give the main object, background, color and other information consistent with the original image.

Real-world application: Currently, CMC can be used in the following two scenarios. (1) Communication: In severe scenarios, channel resources are extremely limited, such as deep sea and space; or there are too many devices, that is, hundreds of devices in IoT share a local area network. In this case, traditional compression methods cannot adapt to such a low bit rate, and image communication can only be carried out through CMC; (2) Storage: According to statistics from mainstream social media, 10% of visual information contributes to 99% of views, and most images are "junk data". For these images that are rarely clicked but not suitable for deletion, their storage will consume considerable resources. Therefore, they can be compressed in CMC format and decompressed when needed. With the advancement of LMM, models with lower complexity have emerged in recent years. We believe that CMC can move from these two applications with low latency requirements to future real-time scenarios.

Reference

[1] Lei, Eric, et al. Text+ sketch: Image compression at ultra-low rates. arXiv preprint arXiv:2307.01944. 2023.

[2] ShowLab. Image2Paragraph. https://github.com/showlab/Image2Paragraph. 2023.

[3] I.T.Union. Methodology for the subjective assessment of the quality of television pictures. ITU-R Recommendation BT.500-11, 2002.

Best Regards

CMC-Bench Author Team

Dear Reviewer kYAX,

Thank you for your detailed comments, and your recognition of the benchmark section. Since the Weakness and Question you mentioned have some overlap, we have merged them into 10 questions and responded to them point by point.

Q1: This paper does not propose a new model but instead (1) creates a mixed dataset by collecting images from several existing datasets and (2) combines existing I2T and T2I models to evaluate their coding performance.

Thanks for your question. Actually, most of the CMC dataset is original. (1) Only 1,000 reference images were taken from the existing dataset you mentioned, while 58,000 distorted images were all generated by ourselves. Most importantly, these images have 160,000 quality annotations, all of which come from human subjective evaluation we conducted. (2) CMC-Bench is not a simple concatenation of existing models. We have comprehensive considerations in terms of bitrate, parameter settings, and the format of reference images, such as the validation of the parameter λ in Figure A5. If the I2T+T2I connection is directly made without any operations, you can refer to the negative results of PICs [1] and Image2Paragraph [2]. In summary, CMC-Bench has provided an effective combination instead of a simple connection and has surpassed VVC with novelty.

Q2: The reported -FR and -NR values are the weighted average of multiple metrics, but it is unclear how the proposed weighting ensures a reasonable assessment.

Our weighting comes from two indicators, TOPIQ-FR and NR, in a 2:1 ratio. Their combination has been strictly verified, see Q1. The rationality is reflected in (1) both indicators are consistent with human subjective preferences, see Table 2, and the correlation coefficient is above 0.9; (2) the weight ratio is reliable, see Table 3/4, where FR and NR have different fluctuation ranges, the former has a smaller fluctuation range. Thus we apply this ratio for a fair comparison.

Q3: Is the large gap between the upper and lower bounds reasonable? Can reliable conclusions be drawn from this? How the rankings in Fig. 6 were determined?

We apologize for any misunderstanding we may have caused. In fact, the discrepancy stems from changes in the λ parameter, not instability within the model itself. The bar chart and table differ in that the bar chart tested multiple configurations of λ, ranging from 0.2 to 0.8, in order to find the optimal setting; the table, however, is based on a single λ value, as the optimal obtained above, to ensure fairness. As shown in Figure A6, the variation in parameters from 0.2 to 0.8 is the reason for the discrepancy; once fixed, the model's performance is much more stable, whose std is shown in the following table:

It can be seen that under the same λ, std is not as large as the figure error bar. As we fixed the λ during evaluation, the reliability is ensured. The rankings in Figure 6 are derived from a 2:1 mix of Consistency and Perception, as described in Q2.

Q4: Subjective evaluation between traditional and CMC-based codecs.

The reason why we use the objective function is for end-to-end evaluation, to ensure that the community can reproduce our benchmark. Therefore, we first let people score, and then use the objective function to fit. (Table 2 shows that the correlation between the two is 0.94, which is a good fit, that is, objective evaluation is able to replace subjective)

To further prove it, we also added subjective evaluation as you requested, comparing advanced codecs and CMC, see appendix Figure A9 and section A.7.4. We show the results of the two codecs and let five experts choose their preferences. The majority's choice is considered better. Overall, CMC still has a greater advantage.