Bridging Compressed Image Latents and Multimodal Large Language Models

[W1] The motivation of this paper seems to lack practical significance. The primary goal of image compression is to reduce image resolution. [...] This resolution (224x224) does not incur significant transmission overhead in current network communications.

We would like to respectfully clarify that the primary goal of image compression is NOT to reduce image resolution, but to reduce image storage or transmission costs while preserving essential information. Our approach is specifically designed to adapt the compressed image latents to facilitate MLLM-based recognition tasks while minimizing the transmission cost.

It is also a misconception that transmitting images of size 224x224 does not incur significant transmission overhead. The reasons are two-fold:

• The volume of images/videos for recognition tasks is growing explosively. Many end devices with limited resources send images to the cloud for recognition. Given this large-scale image/video transmission across various networks, compression remains necessary to meet the bandwidth and energy consumption requirements even for images of size 224x224, not to mention higher resolution images. These observations motivate the ongoing international standardization projects on image/video coding for machines in ISO/IEC MPEG [R1] and ISO/IEC & ITU-T JPEG [R2], underscoring the importance of image/video compression for machines whether they are traditional recognition networks or modern MLLMs.

• Additionally, there is an increasing number of MLLMs exploring video-based applications, where videos are often sub-sampled temporally as a sequence of images. This trend also calls for efficient compression of images/videos for MLLM-based tasks.

[W1] Some works explore using high-resolution images as input, they usually crop high-resolution images into several low-resolution sub-images.

We like to clarify that cropping and compression are two distinct mechanisms with different objectives. Simply cropping high-resolution images into low-resolution patches does not effectively reduce the transmission overhead. We stress that significant bandwidth is still required even for cropped sub-images if no efficient compression is applied.

[W1] The authors do not explain the advantages of using compressed images as MLLM input over exploring more compact Image Encoders [R3].

Token reduction methods (such as [R3]) differ fundamentally from our approach. They aim to reduce the number of visual tokens in MLLMs to lower computational complexity, and do NOT transmit visual tokens in compressed form. In contrast, our work addresses a fundamental preliminary step. Since the image must be transmitted to a cloud server for MLLM processing, we focus on bridging the gap between compressed image latents and their suitability for downstream MLLM and image reconstruction tasks.

Ideally, one might propose generating visual tokens on the end device and using token reduction techniques as a compression method to reduce transmission bandwidth. However, this approach would impose significant computational demands on the end device, making it impractical for coding for machines scenarios, where the primary goal is to offload heavy feature extraction computations to the cloud.

Notably, our method and the token reduction technique could potentially complement each other to develop a more efficient system. For instance, our approach allows to save transmission bandwidth and reduce cloud-side complexity by eliminating the need for image decoding, then token reduction techniques can further optimize complexity on the cloud.

[W2] The proposed method requires training different transform-necks and codecs for different downstream tasks.

We clarify that our proposed method does NOT require re-training for every downstream task. First, since one single MLLM is capable of performing multiple tasks [R4], our trained system inherently supports multi-task inference. Second, our training framework, which back-propagates the surrogate loss at the output of visual encoder rather than that of the downstream MLLM, enables the resulting transform-neck and/or image codec to be applied directly to EVEN DIFFERENT MLLMs sharing the same visual encoder, WITHOUT the need for retraining. This feature further distinguishes our approach from many existing coding for machines methods that indeed require task-specific retraining.

We thank the reviewer for the insightful comments. We hope our responses have addressed the concerns satisfactorily, and would greatly appreciate that the reviewer considers increasing the rating on this pioneering attempt to perform image coding for MLLM-based tasks.

References

[R1] "Use cases and requirements for Video Coding for Machines," ISO/IEC JTC 1/SC 29/WG 2 output document, N00190, 2022.
[R2] "JPEG AI Use Cases and Requirements," ISO/IEC JTC 1/SC 29/WG 1 output document, N100724, 2024.
[R3] W. Li et al., "TokenPacker: Efficient Visual Projector for Multimodal LLM", arXiv 2024.
[R4] H. Liu et al., "Visual instruction tuning", NeurIPS, 2024.

Dear Reviewer, thank you again for your thoughtful review of our paper. We have carefully addressed your concerns and comments in our previous rebuttal response to the extent possible. As the end of discussion phase is fast approaching (Nov. 26 AoE), we look very much forward to your further comments (if any). If you find our responses satisfactory, we would kindly ask your consideration for increasing your rating on this pioneering attempt to perform image coding for MLLM-based tasks.

[W1] The paper's approach relies on the assumption that the downstream MLLMs will use the same pre-trained CLIP visual encoder.

Our proposed framework is absolutely NOT restricted to specific visual encoders. While our main experiments (Figure 3) are evaluated based on MLLMs that adopt the popular CLIP ViT visual encoder, the additional results in Section 4.6 and Figure 8 demonstrate the applicability of our method to two MLLMs with different visual encoders: (1) mPlug-Owl2 [R1] with a custom-trained ViT visual encoder, and (2) Osprey [R2] with a ConvNeXt-based visual encoder. We hope this clarification makes the matter clearer to the reviewer.

[W2] The paper does not provide a comprehensive comparison with existing image compression methods beyond the context of MLLMs.

In the paper and supplementary document, we have provided the comparison with the existing image compression methods beyond the context of MLLMs, including ELIC [R3] (a learned image codec) and VVC Intra coding [R4] (the state-of-the-art traditional codec for intra image coding). Both are optimized for human perception. The former is labeled as "Reconstruction" in the performance comparison of Figure 3, and the comparison with the latter is included in Section A.2 of the supplementary document. Our proposed method, tailored for MLLMs, outperforms both of these codecs significantly.

[Q1] Will you plan to release the code?

Yes, we shall release the source code upon the acceptance of the paper.

We thank the reviewer for the insightful comments. We hope our responses have addressed the concerns satisfactorily, and would greatly appreciate that the reviewer considers increasing the rating on this pioneering attempt to perform image coding for MLLM-based tasks.

Reference

[R1] Q. Ye et al., "mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration." arXiv, 2023.
[R2] Y. Yuan et al., "Osprey: Pixel understanding with visual instruction tuning." CVPR, 2024.
[R3] D. He et al., "Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding." CVPR, 2022.
[R4] B. Bross et al., "Overview of the versatile video coding (vvc) standard and its applications." IEEE TCSVT, 2021.

Dear Reviewer, thank you again for your thoughtful review of our paper. We have carefully addressed your concerns and comments in our previous rebuttal response to the extent possible. As the end of discussion phase is fast approaching (Nov. 26 AoE), we look very much forward to your further comments (if any). If you find our responses satisfactory, we would kindly ask your consideration for increasing your rating on this pioneering attempt to perform image coding for MLLM-based tasks.

[W1] The proposed method appears to be a general image compression approach, lacking unique design elements specifically tailored for MLLMs.

First, we like to clarify that it is NOT our intent to design a specific image compression system solely for MLLMs. As the title "Bridging Compressed Image Latents and Multi-modal Large Language Models'' of this paper suggests, we focus on how to adapt the compressed image latents produced by an existing image compression system to suit the downstream MLLMs (and image reconstruction) tasks. To this end, we have developed three application scenarios (see Figure 1 of the paper), two of which involve not only the MLLM task but also the image reconstruction task for human perception. Note that many real-world applications (e.g. surveillance) call for the flexibility that the compressed image latents be adapted to and used for both the machine and image reconstruction tasks. Our framework is able to enable this feature.

Second, our system incorporates several novel components specifically for MLLMs: (1) a surrogate loss that combines the distillation loss for knowledge transfer and the cross-entropy loss to connect visual features to the text domain for MLLM-based tasks. Notably, for them to be effective, these two loss functions are applied via (2) a progressive training strategy. Last but not least, (3) the concept of back-propagating the surrogate loss from the output of the visual encoder that precludes large MLLMs in the training of our framework. To our best knowledge, these ideas are proposed for the first time to adapt the compressed image latents to MLLM-based tasks. We would be happy to discuss and cite any prior works from the reviewer that address similar issues.

[W1] They do not provide specific comparisons regarding the reduction in training costs.

Our proposed framework enables efficient training with a surrogate loss back-propagated through only the visual encoder. In contrast, naively adopting the existing coding for machines techniques, which rely on back-propagating a task loss through the downstream recognition networks (MLLMs, in our case), would lead to extremely larger computational complexity. Taking a 7B MLLM model as an example, the sheer model size is more than 20 times that of the 300M CLIP visual encoder. With our limited compute resources, i.e. RTX 4090 with 24 GB of VRAM, we could not include such a large MLLM for end-to-end training.

[W2] The proposed SURROGATE LOSS is not novel.

We respectfully disagree with the reviewer's statement regarding the lack of novelty of the surrogate loss. The training framework involving the proposed surrogate loss is illustrated in this figure. To achieve high downstream MLLM performance, we propose an alternative to the straightforward yet undesirable approach of directly back-propagating a task loss through the entire MLLM. In fact, our proposed surrogate loss is imposed at the output of visual encoder, with two distinct loss terms: (1) the distillation loss for aligning the visual encoder output representations between using uncompressed image and transformed latents as input, and (2) the cross-entropy loss for bridging the visual features to the text domain. This dual-loss approach ensures effective latent transformation and has not been explored in prior work. Our finding is that relying solely on a single metric, such as the distillation or cross-entropy loss, leads to sub-optimal performance, as evidenced by our experiments in Section 4.5 of our paper.

[W3] The proposed method has limited applicability, as it requires alignment with specific vision encoders.

Our proposed framework is absolutely NOT restricted to specific visual encoders. While our main experiments (Figure 3) are evaluated based on MLLMs that adopt the popular CLIP ViT visual encoder, the additional results in Section 4.6 and Figure 8 of our paper demonstrate the applicability of our method to two MLLMs with different visual encoders: (1) mPlug-Owl2 [R1] with a custom-trained ViT visual encoder, and (2) Osprey [R2] with a ConvNeXt-based visual encoder. We hope this clarification makes the matter clearer to the reviewer.

[Q1] Why are different MLLMs chosen for each task instead of evaluating multiple MLLMs on each task?

Our experiments involve multiple tasks with different MLLMs to show a wide spectrum of MLLM-task pairing for validating the effectiveness and generalizability of our proposed method. We believe our experiment setup is suitable to demonstrate this. As per the suggestion, we additionally provide the results of our proposed method on an additional task with two different MLLMs, Honeybee [R3] and Shikra [R4], in Section A.5 of our revised supplementary document.

[Q2] The motivation for using different loss functions at various stages across all epochs is not clearly explained.

Our training framework integrates the cross-entropy and distillation losses to achieve an effective latent transformation for downstream MLLMs. As demonstrated in Section 4.5 and Figure 6(a) of our paper, using either loss individually results in suboptimal performance. Additionally, Figure 7 of our paper provides further insights into the roles of each loss function: the cross-entropy loss focuses on minimizing the feature matching errors in the foreground object regions while the distillation loss is to minimize the global feature matching errors.

We thank the reviewer for the insightful comments. We hope our responses have addressed the concerns satisfactorily, and would greatly appreciate that the reviewer considers increasing the rating on this pioneering attempt to perform image coding for MLLM-based tasks.

Reference
[R1] Q. Ye et al., "mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration." arXiv, 2023.
[R2] Y. Yuan et al., "Osprey: Pixel understanding with visual instruction tuning." CVPR, 2024.
[R3] J. Cha et al., "Honeybee: Locality-enhanced projector for multimodal llm." CVPR, 2024.
[R4] K. Chen et al., "Shikra: Unleashing multimodal llm’s referential dialogue magic." arXiv, 2023.

Dear Reviewer, thank you again for your thoughtful review of our paper. We have carefully addressed your concerns and comments in our previous rebuttal response to the extent possible. As the end of discussion phase is fast approaching (Nov. 26 AoE), we look very much forward to your further comments (if any). If you find our responses satisfactory, we would kindly ask your consideration for increasing your rating on this pioneering attempt to perform image coding for MLLM-based tasks.

[W1] Why we need the proposed method of compressing image latent instead of existing token pruning/merging work on MLLMs such as crossget [R1].

Token reduction methods (such as [R1]) differ fundamentally from our approach. They aim to reduce the number of visual tokens in MLLMs to lower computational complexity, and do NOT transmit visual tokens in compressed form. In contrast, our work addresses a fundamental preliminary step. Since the images must be transmitted to a cloud server for MLLM processing, we focus on bridging the gap between compressed image latents and their suitability for downstream MLLM and/or image reconstruction tasks.

Ideally, one might propose generating visual tokens on the end device and using token reduction techniques as a compression method to reduce transmission bandwidth. However, this approach would impose significant computational demands on the end device, making it impractical for coding for machines scenarios, where the primary goal is to offload heavy feature extraction computations to the cloud.

Notably, our method and the token reduction technique could potentially complement each other to develop a more efficient system. For instance, our approach allows to save transmission bandwidth and eliminates the need for image reconstruction, while token reduction techniques further optimize the computational complexity on the cloud side.

[W2] The proposed method shows inconsistent performance degradation on different benchmarks.

The fact that the performance degradation varies across benchmarks/tasks is expected and has been observed in previous coding for machines literature [R2, R3]. These variations stem from the different sensitivity of the downstream task network to the image's compression artifacts. We stress that our proposed method outperforms the Reconstruction baseline across different tasks despite some variations, as shown in Figure 3 of our paper.

We thank the reviewer for the insightful comments. We hope our responses have addressed the concerns satisfactorily, and would greatly appreciate that the reviewer considers increasing the rating on this pioneering attempt to perform image coding for MLLM-based tasks.

Reference
[R1] D. Shi et al. "Crossget: Cross-guided ensemble of tokens for accelerating vision-language transformers," ICML 2024.
[R2] Y-H. Chen et al. "Transtic: Transferring transformer-based image compression from human perception to machine perception." ICCV, 2023.
[R3] H. Li et al., "Image compression for machine and human vision with spatial-frequency adaptation", ECCV 2024.

Dear Reviewer, thank you again for your thoughtful review of our paper. We have carefully addressed your concerns and comments in our previous rebuttal response to the extent possible. As the end of discussion phase is fast approaching (Nov. 26 AoE), we look very much forward to your further comments (if any). If you find our responses satisfactory, we would kindly ask your consideration for increasing your rating on this pioneering attempt to perform image coding for MLLM-based tasks.