Compress & Cache: Vision token compression for efficient generation and retrieval

We thank the reviewer for their time and feedback provided. We hope our responses below address their remaining concerns.

Q1. On confunding effects of LoRA fine-tuning C&C improving over the baseline in several cases. How much of this performance improvement (both over LLaVA and other methods) can be attributed to the LoRA fine-tuning and how much to C&C. I believe that a comparison between a LLaVA model fine-tuned via LoRA using a similar procedure (as much as possible) and C&C would provide clearer insights.

A1. We believe the reasons behind the observed behaviours are multifold: (1) As shown in Table 1 in the paper, multiple compression methods report for certain datasets improvements over the uncompressed LLaVA-1.5 baseline (e.g. [28] on SQA and VisWiz, [3,28] on VisWiz, [3] on MMB). This suggests that the compressor acts as a filter that filters out some of the distractors, focusing more on prominent objects/concepts. (2) Performance on these datasets tends to vary between adjacent checkpoints during training, and the last checkpoint is not always the best one across all datasets, despite being the best on average. (3) The rest is indeed down to the LoRA finetuning, as the result of the experiment you suggested showcases below:

Q2. I believe that the dataset size comparison between EVA-CLIP and C&C on line 271 is unfair, as it compares the size of the dataset used for training this specific CLIP model versus the size of a fine-tuning dataset.

A2. What we meant is the size of the dataset used for contrastive training (which is optimal for retrieval tasks). We will make this clear, thank you.

Q3. Table 1, for MMB, the best-performing method with 36 tokens is Matryoshka Multi, but its entry is not bolded. And a small typo in line 47 of the supplementary, "performs" should be "performance".

A3. Thank you! We have fixed them in the updated manuscript.

Q4. How are the summary tokens initialized?

A4. The summary tokens are initialized randomly. We explored multiple choice (e.g, zero-init, character-based init) but ultimately didn't notice a significant difference.

Q5. While the authors provide a FLOPs-based comparison between indexing and inference in Figure 5, I wonder how the inference speed would vary (and compare to other token pruning methods) under different scenarios, e.g., when the image is given by the user, and no retrieval is performed. Would it then be more efficient to use a matryoshka method compared to two VLM inference steps?

A5. Following your suggestion, we benchmark the LLaVA-1.5 7B model on a RTX 4090 GPU. Each result is averaged over 100 runs following a warm-up period.

Original LLaVA model cost: 0.0587 sec/image (out of which 0.00353 sec spent for the vision encoder)

Caching cost: 0.0584 sec/image

Online C&C cost (16 tokens): 0.00158 sec/image

Matryoshka (16 tokens): 0.00521 sec/image

Online C&C cost (4 tokens): 0.000406 sec/image

Matryoshka (4 tokens): 0.004001 sec/image

Caching cost + Online C&C (16 tokens): 0.0599 sec/image

Caching cost + Online C&C (16 tokens): 0.0588 sec/image

Notice that the practical measurements closely align with the expected approximated improvements. Moreover, in an online manner, once the features are cached, we are notably faster at building the context.

Furthermore, our method can also make use of a sliced LLM to speed up the compression. To showcase this, we train a model with the following configuration: a LLaVA-OV 0.5B generator with a compressor instantiated using the first 25% of the LLM's layer, as opposed to 100%. This results in a 4x faster and smaller compressor. To recover the performance drop, we distill the features produced by the full-sized compressor. The results for 16 tokens (i.e. 45x compression rate, 729/16) are presented in the Table below:

This means that we can reduce the compression time by up to 4x without notable performance degradation.

Q6. The paper states that they aim for a 1:1 sampling ratio between LLaVA 665k and CC3M. Do the authors aim for an overall 1:1 ratio, or do they strive to maintain this ratio in each batch? Would a different sampling ratio improve performance?

A6. We aim for a 1:1 sampling ratio globally, not within a batch. The batches themselves are simply constructed to maximize throughput, grouping the samples by length. We did try a few different sampling ratios. The model is generally robust in the range 1:1 - 3:1 (665K:CC3M).

Q7. It would be interesting to know the total runtime of the LoRA fine-tuning process.

A7. A training run took 40 hours on 24 GPUs.

Q8. To my understanding, the first-stage LoRA adapters are optimized for both the contrastive and autoregressive objectives, while the second-stage adapters are optimized only for the autoregressive objective. Is this correct?

A8. That's correct, we will make this clearer in the updated paper.

We thank the reviewer for their feedback and for recognizing the novelty of our approach. We hope our responses below address their remaining concerns.

Q1. The illustrations can be further refined for clarification and better readability. For example, Figure 2 does not show that the proposed method does not require finetuning the entire LLM, and the stage-specific LoRA modules are also missing from this figure.

A1. Thank you for your suggestion. We have updated the figures and will include them in the updated manuscript.

Q2. Regarding the storage perspective, the proposed method needs to store a summarized version of images with shape $R^{k' \times d}$, where $d$ is the hidden dimension of LLM. In contrast, the original LLaVA-1.5 only need to store features with shape $R^{k \times d_v}$, where $d_v < d$ is the dimension of the vision encoder output. It would be better to clarify this difference to illustrate the actual compression difference, rather than just comparing the number of tokens.

A2. You are generally right, if we opt to store the vision features prior to the projection layer that maps them to the input dimension of the LLM then generally $d_v < d$. The caveat is that we will have to rerun the projector in this case. For the LLaVA-1.5 7B model, $d=4096$ and $d_v=1024$ while for the smaller LLaVA-OV 0.5B model $d_v$ is a bit larger than $d$, with $d=896$ and $d_v=1152$.

We will make this distinction clearer in the updated paper.

Dear Reviewer h3Po,

Please note that submitting mandatory acknowledgement without posting a single sentence to authors in discussions is not permitted. Whether your concerns have been addressed or not, please do tell the authors.

Thanks,

AC

We're grateful for your time and valuable feedback, and for recognizing the novelty of our approach! We hope our responses below address your remaining concerns.

Q1. The method's core strength is also a limitation. It is not suitable for real-time applications (e.g., live video analysis) where pre-processing is impossible. While this is an intrinsic property of the paradigm, it constrains the method's scope of application.

A1. Our main focus is indeed on settings naturally suitable for offline indexing (e.g. RAG-based applications).

However, we note that our method is sufficiently versatile, and the cost can be in part alleviated by combining our method with techniques like distillation and LLM slicing. To showcase this, we train a model with the following configuration: a LLaVA-OV 0.5B generator with a compressor instantiated using the first 25% of the LLM's layer, as opposed to 100%. This results in a 4x faster and smaller compressor. To recover the performance drop, we distill the features produced by the full-sized compressor. The results for 16 tokens (i.e. 45x compression rate, 729/16) are presented in the Table below:

Q2. The paper claims C&C is a general method, yet all experiments are confined to the LLaVA family. The LVLM landscape includes diverse architectures (e.g., Qwen-VL, CogVLM) with different vision-language interfaces. The effectiveness of the "self-compression" mechanism is not guaranteed to transfer across these architectures. By limiting validation to a single model family, the paper's claim of generalizability remains insufficiently supported by empirical evidence.

A2. Thank you for your suggestion. We would like to note that we have already validated the efficacy of our approach using two different architectures, LLaVA-1.5 and LLaVA-OneVision (LLaVA-OV). Although they partly share their name, they have different architectures and are trained on different data. Vision encoder: CLIP ViT-L/14 at 336px (LLaVA-1.5) vs SigLIP ViT-SO400m/14 at 384px (LLaVA-OV); LLM: Vicuna-1.5 (LLaVA-1.5) vs Qwen2 (LLaVA-OV); Input: single patch and image at fixed 336px resolution (LLaVA-1.5) vs multi-patch (i.e. high resolution) 384-2304px operating range with multi-image support; Num vision tokens: 576 tokens per image (LLaVA-1.5) vs 729 tokens per patch (LLaVA-OV).

In the main manuscript, the LLaVA-OV model was evaluated on the VisRAG suite. For consistency, in the table above, from answer A1, we report results on a shared suite of datasets. The same conclusions hold, our compressed models match and even outperform the original model in some cases.

We note that many of the existing alternative models (e.g., QwenVL) are trained on closed-source data, making the retraining of the compressor under a fair setting impractical.

We thank the reviewer for their feedback and for recognizing the novelty of our approach. We hope our responses below address their remaining concerns.

Q1. The most significant weakness of the proposed method is the lack of runtime latency results on the real hardware. The FLOPs analysis it good but it remains in theoretical computation. A comprehensive testing on GPU or other computing devices will be more convincing.

A1. We reported FLOPs estimates because the timings themselves are subject to the specific implementation. Following your suggestion, we benchmark the LLaVA-1.5 7B model on a RTX 4090 GPU. Each result is averaged over 100 runs following a warm-up period.

Original LLaVA model cost: 0.0587 sec/image (out of which 0.00353 sec spent for the vision encoder)

Caching cost: 0.0584 sec/image

Online C&C cost (16 tokens): 0.00158 sec/image

Online C&C cost (4 tokens): 0.000406 sec/image

Notice that the practical measurements closely align with the expected approximated improvements: 576/16 = 36 theoretical vs 0.0587/0.00158 = 37.1, respectively 576/4 = 144 theoretical vs 0.0587/0.000406 = 144.5. The practical gains are slightly higher because, in our case, the vision encoder doesn't need to be rerun.

Q2. For any compression work, the main purpose is to maintain performance and reduce computation. Even if the performance is good, it is still unknown if the robustness is affected. Some works report the result variance, or test the method in perturbed or adversarial input to show how robust the compressed model is. My concern on this paper is how the C&C performs when in such scenario?

A2. Thank you for your suggestion. Following [1] we evaluate our approach under a various set of perturbations, e.g: zoom blur, elastic transformation, pixelation, JPEG compression, shot noise, brightness jitter, contrast jitter, Gaussian noise, etc. For brevity, we include below the results in terms of relative performance drop on a subset of them. Notice that both approaches, with and without compression, have similar robustness strength.

(negative values denote cases where the performance increases marginally post-transformation).

For completeness, we also report for a subset of transformations how the performance degradation changes under varying augmentation strengths (levels 3 and 5). The same conclusions hold true.

[1] Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models, Chen et al, NeurIPS 2023