Efficient Large Multi-modal Models via Visual Context Compression

We appreciate the reviewer's constructive suggestions. We address the concerns raised on a point-by-point basis, including additional benchmarks on the global response PDF. We will include the new results in our revised manuscript.

For weakness 1: "more discussions and investigations on why other visual compressors are significantly worse than average pooling", we thanks for the reviewer's suggestion. In line 262-263, we analyzed that "they are ineffective when applied to training because the in-training attention scores are unstable.". Our insight is that while advanced compressors (e.g., attention-based token pruning) excel in inference-only scenarios (See bottom rows in Table 2), the simple pooling method performs better during training (See top rows in Table 2). We hypothesize that this is because training advanced compressors, such as attention-based pruning, necessitates (1) differentiable token selection and (2) stable attention mechanisms of LM Transformers.

For weakness 2: "scale up to larger models like 13B, 34B, or other structures such as Mini-Gemini [1]", we thanks for the reviewer's suggestion. To show the scalability, we scale the method up to 13B model, and observe consistent performance and efficiency improvements.

To show the generalizability to different structures, we supplement the experiment on Mini-Gemini (MGM-2B). Our four-stage compression training delivers comparable results with a 17% increase in training efficiency and significantly fewer vision tokens as input. We plan to incorporate additional structures in future versions.

We thanks for the reviewer's comments on Weakness 3. It's worth noting that our method not only improves training efficiency by 16% but also enhances average performance by 0.5% across 13 benchmarks. To address the cumbersomeness, we are exploring the development of a simple training schedule (linearly decreasing the compression ratio over time; like learning rate schedule), aimed at balancing training efficiency and model simplicity.

We sincerely hope our rebuttal addresses your concerns, and we look forward to your feedback. Your response will motivate us to refine our paper into a more solid version.

Kind regards,

Authors of Paper 2615

We appreciate the reviewer's constructive suggestions. We address the concerns raised on a point-by-point basis, including additional benchmarks on the global response PDF. We will include the new results in our revised manuscript.

To address weakness 1, we follow the reviewer's suggestion, and add the evaluation of multi-modal tasks in Table 1 of Global Response, especially MMBench and MMMU (A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI)[49].

For weakness 2, we thanks for the reviewer's comment on information loss. We have discussed this in the limitation section of original paper. The naive compression may not be ideal for tasks requiring dense comprehension, grounded reasoning, and OCR/text capabilities. Nonetheless, our staged training approach, which does not involve compression in the final stage, is capable of managing tasks that require dense visual information. We provide analysis and visualization in Global Response.

To address weakness 3.1, we try to implement the experiment of larger 448×448 input resolution. However, we note that the default openai-CLIP-336 encoder can only input 336x336 and interpolating positional embedding to 448x448 degrades the performance, resulting in -9% drop in GQA. We are experimenting the dynamic high resolution of LLaVA-NeXT (its training code has not yet been made public as of Aug/8) by employing openai-CLIP-224 to encode four 224x224 images derived from a 448x448 image. We plan to include the large input experiment in the revision once our new results are available.

For weakness 3.2, we thanks for the reviewer's suggestion. As LLaVA-NeXT does not make the training code public upon the submission of rebuttal, we are in the progress of reproducing the code. We supplement the experiment of testing the method on larger model of LLaVA-13B in Table 3 of Global Response, and we can achieve 177% compression rate and accelerate training efficiency by 15% while keeping the performance. We will include the new results in our revision.

For weakness 4, we thanks for the reviewer's suggestion. We aimed at providing comprehensive analysis of design space. We will provide a simple version for the practical use in the near future. For instance, we might implement a universal training schedule that gradually reduces the compression ratio over time.

We thanks for the reviewer's comments for the minor issues. We address them as follows.

Q1. We thanks for the reviewer's comment. #Tokens means the summation of number of visual tokens in each LLM transformer layer. It is inversely proportional to CR.

Q2. We mentioned this in line 242: "When reporting average performance in Tab. 5, the score of MME is normalized by 2000, as its range is from 800 to 2000". The average performance is simply the mean of the (normalized when it is MME) individual performances.

Q3 (1) multi-stage compression. We thanks for the reviewer's suggestion. The multi-stage compression is a great idea. We follow the suggestion and experiment with the setting of multi-stage (layer 0-3: stride=1; layer 4-11: stride=2; layer 12-23: stride: 4; layer 24-31: stride: 8), resulting in a CR of 267%. We compare this to single-stage compression (on layer 8 stride 8) with a CR of 266%.

The multi-stage compressor exhibits strong performance when applied directly during inference, outperforming layer 8 stride 8 by 3% under same CR. However, it's surprising that training the multi-stage compressor yields only a marginal average performance improvement of 0.2%. We analyze that the complexity of multi-stage operations makes the LLM more challenging to train. We will have more future analysis.

Q3 (2) video extension. We thanks for the reviewer's suggestion. For extension of modalities, we have experimented on video-language understanding (based on video-LLaVA). We observe consistent enhancements (an average improvement of +0.4% across three video datasets and a 9% reduction in training time) over the baseline using our new training setting. We will include the new results in our revision.

We sincerely hope our rebuttal addresses your concerns, and we look forward to your feedback. Your response will motivate us to refine our paper into a more solid version.

Kind regards,

Authors of Paper 2615

We appreciate the reviewer's constructive suggestions. We address the concerns raised on a point-by-point basis, including additional benchmarks on the global response PDF. We will include the new results in our revised manuscript.

For weakness 1, we have included 2D-Conv and C-Abstractor (from Honeybee) as a baseline in Table below. The stride of 2D-Conv and C-Abstractor are the same with other compressor to ensure fair comparison under same compression ratio (CR). However, we found that (1) The performance of convolution-based compressors is significantly lower, with 2D-Conv achieving 55.1%, C-Abstractor achieving 50.6%, and 1D Pool reaching 60.4% in average accuracy across 13 benchmarks. (2) there is overhead parameters of conv. We analyze that it is hard to optimize the extra convolution kernel within LLM at the same time (this is an underexplore area).

It is worthy mentioning that Honeybee focus on the design of projector outside LLM while our focus is the visual context compression within LLM. Therefore, our method has the potential to accelerate the inference and training of Honeybee (i.e., keep Honeybee's 2D Conv on projector before LLM, and add our 1D pool compressor on visual tokens within LLM). We will include these new results in our revision.

For question 1, we thank the reviewer for the comments. We add experiment with 2D pooling in the same Table above. As results, 1D pooling perform slightly better than 2D pooling (+0.2%) under the same compression ratio. This could be due to the visual tokens being processed in a 1D manner within several LLM layers. Besides, we opted for 1D pooling because it allows for more adaptable compression ratio. For instance, using a 1D pool with a stride can reduce the number of tokens by 2x, whereas a 2D pool with a stride of 2 reduces the token count by 4x. We will include these new results in our revision.

For question 2, we appreciate the reviewer's feedback. K is set to 2 with stride set to 8 in Table 2, resulting in a compression ratio of 556% to approximate [41]’s 514% (fixed CR) for a fair comparison of compressor.

For question 3, we thank the reviewer for the comments. The previous validated four benchmarks are considered representative because they assess diverse capabilities: GQA evaluates visual understanding, SQA focuses on scientific knowledge across 26 topics, MME tests perception and cognition through 14 subtasks, and MM-Vet measures ensembling skills in recognition, OCR, math, and spatial awareness (encompassing other benchmarks like VQAv2, COCO, and TextVQA).

In Table 1 of GLOBAL RESPONSE PDF, we have included all 13 benchmarks in our ablation now and evaluated based on the average performance across these benchmarks. The evaluated 13 benchmarks include not only VQA but also multimodal benchmarks. We achieve consistent conclusions with our original paper for both the ablation study and final results. We will include these new results in our revision.

We sincerely hope our rebuttal addresses your concerns, and we look forward to your feedback. Your response will motivate us to refine our paper into a more solid version.

Kind regards,

Authors of Paper 2615

Dear reviewers,

Please carefully read the rebuttal and the other reviews, then reply to the authors indicating whether your questions/concerns have been addressed. If not, please specify which questions/concerns remain unresolved so that the authors have a fair opportunity to address them before the author-reviewer discussion period ends in approximately 36 hours from now (August 13th, 11:59pm AoE).

Best, AC