Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters

We thank the reviewer for finding our work has “no major weaknesses from a technical standpoint” and stating they are willing to “champion the paper as is.” We agree that we could include additional literature review and discussion on existing adaption computational techniques, as we elaborate below. 

it could benefit from a more in-depth discussion of related work on adaptive compute

We have added additional related work that focuses on this type of VLM adaptive compute, i.e., dynamically adjusting which tokens are processed within the LLM to reduce inference cost. Below is a excerpt from our revised manuscript which includes this type of adaptive technique.

Another approach to reducing inference cost is adaptive token processing, where the compute dedicated to certain tokens at inference is varied Jain et al. (2024). Most of these methods prune visual tokens within the LLM due to their lower attention scores compared to the prompt, system, etc., tokens (Chen et al., 2024; Wan et al., 2024), a heuristic commonly found in regular text-only LLM KV cache reduction techniques (Zhang et al., 2023; Oren et al., 2024). Finally, while we focus our paper on image-based VLMs, a host of works (Xu et al., 2024; Shen et al., 2024) discuss token compression for video processing using VLMs.

Please let us know if you have any additional questions or concerns!

We thank the reviewer for taking the time to read our paper and appreciate the reviewer finding our scaling laws a novel perspective for improving inference efficiency in VLMs. The general concerns of the reviewer surround our token compression algorithm, which we address below.

Query-based token compression (Quecc)

We first would like to highlight that the main focus of our work is characterizing the trade-off between tokens and parameters via scaling laws. We observed that extreme token compression is necessary for compute optimal inference, along with other insights as also appreciated by all the reviewers. Through query-based token compression, we wanted to simply highlight the key requirements for extreme token compression, one of them being selecting relevant tokens based on the actual input query. We did this by incorporating query in existing algorithms like TokenPacker. However, we address specific comments about query-based compression below and in the general response.

The method is quite similar to LLaMA-Vid, the paper didn't compare it in its experiment and didn't show the differences between them.

While LLaMA-VID indeed uses query in their algorithm, their architecture only uses one query-aware token compared to ours which adds query information into each visual token. Our key goal in the query-based token compression was to simply validate one of the insights from our scaling laws - selecting relevant tokens based on the user query is necessary under extreme compression.

That said, we have added a comparison of the performance of our algorithm compared to LLaMA-VID in the general response which shows that our method is competitive with LLaMA-VID on numerous tasks despite using a weaker vision encoder. We would like to additionally highlight the increased computation required for LLaMA-VID, stemming from their text projector, which for most cases is an independent BERT/QFormer-style model, which also requires training. In addition, the core differences that differentiate our method from LLaMA-VID are as follows: not directly using a text decoder for query injection, utilizing learnable convolution instead of average pooling for visual embedding, and injecting query information into all visual tokens instead of having a context token and content tokens.

The paper could also compared with VoCo-LLaMA

For comparisons with VoCo-LLaMA, as mentioned in the general response, we believe that it is not a fair baseline in our setting due to differences in the general token compression methodology. In summary, VoCo-LLaMA involves processing all the visual tokens through LLM to get a cached image representation, which gives inference efficiency gains only when the inference is done again on the same image. However, under the common setting of varying images, VoCo-LLaMA does not lead to inference gains (unless the text query is super long) as all the visual tokens are to be processed by the LLM.

Could you explain more about the discrepancy of the result between previous work and your work in Log-Linear Relation between Error and Number of Visual Input Tokens? Why limited downstream benchmarks lead to the discrepancy?

Many papers claim that their token compression algorithms can “effectively reduce the number of visual tokens while achieving comparable or even better performance.” However, we believe this is due to reporting results only on a few selective benchmarks. In our scaling laws, we averaged the performance across a host of nine common VLM evaluation benchmarks (from lmms-eval [1]) and observed a consistent decrease in performance as the number of visual tokens was reduced from 576. That said, the performance does indeed remain the same or comparable to using all the tokens on select benchmarks within our evaluation suite, corroborating our experiments with those reported in the literature. However, when averaged, there is a consistent log-linear drop in performance with inference FLOPs/tokens.

Is the Convolutional Downsampling necessary?

There is a lack of ablation study proof for both User Query Information Injection and Convolutional Downsampling.

As demonstrated in our ablations in the general response, convolutional downsampling helps improve existing token compression algorithms on certain tasks. We have also explored the user query information injection and convolution downsampling impacts in the shared response, in which the combination of both helps mitigate certain drawbacks of each individual component.

Dear Reviewer jj75,

In our shared and reviewer response, we have tried to answer your questions and concerns. To summarize, we

• Increased the core focus of our paper to its scaling laws and insights instead of the compression algorithm
• Compared against LLaMA-VID and VoCo-LLaMA
• Added an ablation study for our compression technique.

We hope our response addresses all of your concerns, and we look forward to hearing back from you. We are more than happy to answer any lingering questions or points of confusion.

Thank you for the clear and detailed rebuttal. It's quite a great job of introducing the scaling laws for VLM efficiency, but I'm concerned about whether the opinion that you mentioned ‘Inference optimal VLMs need only one token but larger models’ is impractical to reach, a larger model may be heavier to inference, which may against the efficiency of token compression, although this may be a trade-off between fewer token numbers and larger model. Additionally, in Table 1, my personal opinion is pointless to compress vision tokens to 1, because the vision information is completely lost for all baselines, with regard to the score on all benchmarks, comparing in this unrealistic scenario would not gain any confidence in the robustness of the purposed method. I suggest you compare it with pure text tokens if you intend to do so.

Dear Reviewer jj75,

As today is the last day for the discussion phase, we humbly request you to review our response to your concerns and comments and hope that it allows you to assess the work more positively.

We are glad the reviewer found our scaling laws insightful. We address their concerns, which mostly surround the query-based token compression algorithm below.

Query-based token compression (QueCC)

We first would like to highlight that the main focus of our work is characterizing the trade-off between tokens and parameters via scaling laws. We observed that extreme token compression is necessary for compute optimal inference, along with other insights as also appreciated by all the reviewers. Through query-based token compression, we wanted to simply highlight the key requirements for extreme token compression, one of them being selecting relevant tokens based on the actual input query. We did this by incorporating query in existing algorithms like TokenPacker. However, we address specific comments about query-based compression below and in the general response.

LLaMA-VID [2] and VoCo-LLaMA [3] have already done token compression in extreme regimes, which is impressive. The author should compare their performance with QueCC. It seems that QueCC is inferior to VoCo-LLaMA on benchmarks such as GQA and MME.

The employment of cross-attention in QueCC to compress information is common

In the general response, we discuss key differences between these works and our motivation for adding a section on query-based compression in our work. We have added a comparison with LLaMA-VID and show that our approach performs competitively to theirs and outperforms on multiple benchmarks even while using a weaker image encoder. More importantly, this also validates the key point we wanted to make— the importance of selecting relevant tokens based on the query (note that LLaMA-VID also uses query).

For comparisons with VoCo-LLaMA, as mentioned in the general response, we believe that it is not a fair baseline in our setting due to differences in the general token compression methodology. In summary, VoCo-LLaMA involves processing all the visual tokens through LLM to get a cached image representation, which gives inference efficiency gains only when the inference is done again on the same image. However, under the common setting of varying images, VoCo-LLaMA does not lead to inference gains (unless the text query is super long) as all the visual tokens are to be processed by the LLM.

There is a lack of ablation experiments, especially the analysis of depth-wise 2D convolution and the injection of text-embedding.

Although the main focus of this paper is not the proposed algorithm but the various insights derived from the scaling laws of VLMs, we have added the ablation experiments as requested by the reviewer in the general response above. In summary, our ablations empirically validate the importance of using both convolutions and query injection, especially under extreme token compression.

The authors’ scaling law seems to have no direct relationship with the method they proposed.

We are sorry if the connection did not come out clearly and have updated the manuscript (start of Section 4) to make this more clear. Our scaling laws highlight that the compute optimal inference requires extreme token compression (e.g., just using 1,4, or 16 tokens). Intuitively, at such extreme compression, it becomes necessary to select only the relevant tokens based on the actual user query. We empirically validate this insight, by incorporating query-based compression over existing token compression algorithms.

Dear Reviewer WUho,

In our response above, we have tried to address all your concerns and questions. To summarize, we have

• Added comparisons to and highlighted key differences with LLaMA-VID and VoCo-LLaMA
• Performed ablations on our method that show the importance of both the convolution and query injection component
• Made it clearer that the main focus of our paper is our scaling laws and the insights derived from them.

We hope our response addresses all of your concerns and hope to hear back from you soon. Please let us know if you have any additional questions.

Dear Reviewer WUho,

As today is the last day for the discussion phase, we humbly request you to review our response to your concerns and comments and hope that it allows you to assess the work more positively.

We thank the reviewer for their time and feedback on our manuscript. We are glad that the reviewer found our scaling law results insightful. We address the concerns below.

Given that many real-world applications require fine-grained visual understanding, the proposed compression method may not fully address these demands

We agree with the reviewer that the proposed query-based compression is only for tasks where extreme token compression is compute optimal (which has been clarified in the manuscript as even observed by the reviewer). We have updated the manuscript to clarify this in the algorithm section as well. For fine-grained visual understanding tasks like OCR, using all the visual tokens gives the compute optimal performance as shown in Figure 3b. This is because of the huge performance drop even with a slight token compression.

Finally, we also would like to highlight that our query-based token compression algorithm is just meant to validate some of the insights from our scaling laws. It is not meant to be a core contribution of our work.

their claim that "only one visual token is needed" may be overly naive given the trend in VLM research toward advancing fine-grained visual comprehension capabilities

While we agree with the reviewer that there has been a lot of recent effort toward pushing the capabilities of VLMs on hard tasks like fine-grained visual understanding (e.g., OCR, document understanding), we would like to point out that this does not undermine the importance of improving the efficiency of VLMs for general visual understanding and reasoning tasks. These tasks still and will continue to encompass a significant proportion of downstream use cases of VLMs (e.g., monitoring systems, autonomous driving, etc.). On real-world edge devices, inference efficiency can be the deciding factor between using a technology or not. We believe that our findings will be of significant importance to practitioners, especially since decreasing the model size is the current approach/norm used to reduce inference cost and is heavily suboptimal, as shown by our findings.

We acknowledge that our claim is specifically conditioned on visual reasoning tasks rather than fine-grained comprehension tasks such as OCR. While we have highlighted this distinction at multiple places in the text (introduction, main body), we agree that the title may inadvertently generalize our findings. We are open to revising the title to better reflect the scope and applicability of our results.

We thank the reviewer for raising some concerns about the presentation of some claims. We hope our response is able to highlight the significance of our findings and resolve some of the reviewer’s concerns. Please let us know if you have any additional questions!

We thank the reviewer for increasing their score! To clarify, we used the term "inference efficiency" loosely to refer to inference cost only. We are more than happy to answer any other lingering doubts or questions related to the efficiency perspective.

We thank the reviewer for carefully reviewing our manuscript and are glad that they found the scaling laws “interesting” and the analysis of the trade-offs of LLM size and visual token count “comprehensive” and providing a “deeper understanding of VLM optimization”. We address your concerns below

The main concern is the generalization of the scaling law. The paper focuses on visual reasoning tasks and may not fully explore other types of tasks where a single visual token might not be sufficient…..

Will one visual token be enough for other tasks (e.g, VLMs for detection)? ……

We thank the reviewer for highlighting a potential point of confusion for readers. Although we have addressed this in the original manuscript in Section 3.4 (Scaling laws for OCR tasks), where we consider tasks that require detailed image analysis (e.g., reading the text in document), we have made this point more explicit in the updated manuscript.

As predicted by the reviewer, we indeed observe in Fig. 3b that for such tasks, it is compute optimal to prioritize the number of visual tokens over LLM parameters; the opposite of what should be done for visual reasoning tasks. In other words, for OCR tasks, compute optimal inference requires using all the visual tokens while minimizing the LLM size to fit the given fixed inference budget. We have now highlighted the variation in optimal scaling behavior with various types of tasks in our Introduction section on L83.

Different architectures might yield different optimal points in the trade-off between the number of visual tokens and LLM size, which could limit the applicability of the results. It's unclear how these laws would apply to other VLM architectures

It's uncertain how well this extreme compression would generalize to unseen data or datasets with different characteristics.

Will the proposed scaling law generalize to other VLM architectures? The authors only conducted experiments on one type of VLM.

VLM architecture can vary in multiple ways: (a) use of different LLM architecture, (b) use of different projectors between visual embeddings and LLM input (e.g., InstructBLIP, Qwen-VL, different token compression algorithm), and (c) visual encoder free VLMs like Chameleon [1].

However, note that ultimately all VLMs operate on the same general principle— a large number of visual tokens are to be processed by LLM, generated either via an image encoder (in projector-based VLMs like LLaVA, InstructBLIP or Qwen VLM) or via an image tokenizer in encoder free VLMs like Chameleon [1].

Thus while we agree with the reviewer that the exact point of optimality might vary slightly across the architectures (e.g., instead of 1 token being optimal, it might be 4 or 16), the key message will continue to hold that compute optimal inference requires trading off the visual tokens for a bigger LLM size, and the number tokens required at the optimal point are very small. We explain this in detail for each of the possible ways of modifying architecture below.

(a) Variation with LLM architecture: In general in scaling laws literature [2, 3], scaling insights developed in one particular type of models/training recipes (while keeping other factors like architecture design constant), transfer seamlessly when other factors are changed, like the architecture design. For example, from Chinchilla scaling laws, we find that pretraining data tokens to model parameters ratio should be around 20x, which empirically holds almost always for any sensible LLM architecture design (e.g., choice of hidden embedding, number of layers, etc.)

In this work, we used the Qwen family of LLMs, as they have LLMs varying from 0.5B to 14B, all trained on a similar pretraining mixture. This allowed us to ensure that other factors like LLM pretraining data quality or pretraining recipes do not lead to confounding effects in our scaling laws. We hypothesize that when using newer families of LLMs, the key trends will continue to hold as model quality improves. Better models will increase LLM quality parameter value in our scaling laws, making compute optimal VLMs further emphasize LLM parameter count. In addition, testing the scaling law for various LLM families may be computationally infeasible, and we believe our current explorations already provide valuable contributions.

Dear Reviewer 78vE,

As the end of the discussion phase is approaching, we were wondering if you had any additional questions or concerns.

To summarize, in our response above, we have addressed all your concerns mainly around generalization of our scaling laws to different architectures. We added a discussion of why it will generalize and also added new results that empirically validate the same by showing performance on other VLM architectures followed similar scaling trends as in our original manuscript.

We appreciate your time and valuable feedback which has improved our work, and we hope to hear back from you soon.

Dear Reviewer 78vE,

As the end of the discussion phase is approaching, we look forward to hearing back from you regarding our response. We hope our shared and reviewer responses have been able to address your questions, and we are more than happy to address any lingering questions or concerns.

Dear Reviewer 78vE,

As today is the last day for the discussion phase, we humbly request you to review our response to your concerns and comments and hope that it allows you to assess the work more positively.

Ablations for Token Compression Algorithm:

Reviewers WUho and jj75 also requested ablations of our query-based compression that shows the impact of (a) user query information injection, and (b) the convolutional downsampling on performance. We report the ablations for both components below.

Based on the results shown below, it can be seen at extreme levels of compression that combining query and convolution can magnify the benefits of either adding only query or only convolution, e.g., TextVQA performance at token count one increased by 0.7 percentage points (pp) with both convolution and query while using only one of the components led to at most 0.2 pp increase. In addition, combining the two can mitigate performance drops that are associated with utilizing only query or convolution, as seen in MMB at one token where using only convolution drops performance by more than 1 pp but performance can not only be restored but also improved when adding query, eventually outperforming the baseline by 0.7 pp; a similar situation can be seen for MME.

[1] [2311.17043] LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

[2] [2303.15389] EVA-CLIP: Improved Training Techniques for CLIP at Scale

[3] [2406.12275] VoCo-LLaMA: Towards Vision Compression with Large Language Models

[4] [2403.15388] LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

[5] [2407.02392] TokenPacker: Efficient Visual Projector for Multimodal LLM