We thank Reviewer 1uHD for the detailed and constructive comments! We appreciate the reviewer’s recognition of our strengths in well-motivated design, comprehensive evaluations, novel VTU algorithm, and more controllable tradeoff in efficiency and accuracy. We hope the following response addresses the remaining concerns and fosters meaningful discussion.

W1,Q1: Comparison to additional merging/pruning methods

We thank the reviewer for pointing out additional baselines to compare with. Below we add the comparison with DiffRate[1] and PiToMe[2], together with VisionZip[3] which is suggested by reviewer iv6c. We show that DyMU achieves competitive overall performance while being both training-free and dynamic in length.

Reference:

• [1] Chen, Mengzhao, et al. "Diffrate: Differentiable compression rate for efficient vision transformers."
• [2] Tran, Chau, et al. "Accelerating transformers with spectrum-preserving token merging."
• [3] Yang, Senqiao, et al. "Visionzip: Longer is better but not necessary in vision language models."

W2,Q2: Visualization of the merged token regions

This is a great suggestion! Specifically, such a visualization can help further validate the dynamic-length merging by showing that less complex areas are more aggressively merged. Since OpenReview does not allow figures or external links in the rebuttal, we are unable to include it at this stage. We will add this visualization in our next revision.

We thank reviewer 1uHD once again for the insightful feedback and constructive discussion, which have made this work a stronger submission. We will incorporate the additional results and visualizations as discussed during the rebuttal.

We thank Reviewer iv6c for the detailed and constructive comments! We appreciate the reviewer’s recognition of our novelty in complexity-conditional token reduction. We hope the following response addresses the remaining concerns and fosters meaningful discussion.

W1, W6: Inference Time Comparison

W1.1: Additional Analysis on Wall-clock Inference Times

We thank the reviewer for the suggestion! We provide additional analysis on wall-clock inference times, summarizing our findings below. We continue to see speedup in terms of inference times in most settings, however, we find that the inference time reduction is less consistent with token count and FLOPs, particularly when measuring end-to-end duration. This discrepancy has also been observed in prior works, such as Table 5 in [1] and Table 7 in [2]. We provide a detailed discussion in the next section on the underlying reasons and practical challenges.

W1.2: Deep Dive into the Gap between FLOPs and Inference Time

In this section, we elaborate our findings on why there is a non-trivial gap between FLOP reduction and actual inference time gains, especially for algorithms like DyMU that fundamentally alter the computation of attention. We also explain why we believe FLOPs is a more robust metric for comparing theoretical/algorithmic efficiency, as is also a standard practice in prior works [3][4][5][6].

• Controllability: From definition, FLOPs quantifies algorithmic compute independent of hardware, batching, parallelism, and system load. Wall-clock inference time can vary significantly due to deployment configurations and parallel jobs running on the same node, which are practically difficult to control on shared school servers.

• Implementation Gap: In practice, wall-clock inference time is also heavily dependent on the actual implementations and low-level optimizations of the computation functions. This may lead to significant discrepancies between FLOPs and inference time.

  • Here we show a concrete example, where we compare two implementations of computing a N×N attention matrix from two (N,D) vectors. In Version 1, we simply do one matrix multiplication using torch.matmul to get the N×N matrix. In Version 2, we first split the input vectors into N/4 chunks, do matrix multiplication on the (N/4, N/4) chunks and add them back; As shown below, this results in identical FLOPs but very different wall-clock times. This is because torch.matmul is heavily optimized for large matrix multiplications, leading to longer inference times when multiple torch.matmul calls are made on smaller matrices, despite having the same total FLOPs.

    	#torch.matmul	matrix size per multiplication	MFLOPs	Wall-Clock Time (ms)
    Version1	1	576	339.74	1.374
    Version2	16	144	339.74	2.311

  • In our case, the VTU operation requires decomposing the matrix multiplication into smaller sub-matrices, which inflates the inference time although having a better theoretical efficiency in terms of FLOP. (See the “Implementation Notes” section in the supplementary code for more details.)

  • This demonstrates that, to fully translate the benefit of theoretical FLOPs improvements to actual inference time improvements may require non-trivial work such as implementing new kernels, which themselves can be a separate paper such as Flash-Attention[7].

Therefore, in this work, we focus on formally deriving the theoretical efficiency gains and presenting empirical results on FLOPs, token length and end-task performance. But we do acknowledge that optimizing wall-clock inference time of DyMU through dedicated low-level kernel development is an important follow-up work.

Reference:

• [1] Xing, Long, et al. "Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction."
• [2] Wen, Zichen, et al. "Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?."
• [3] Bolya, Daniel, et al. "Token merging: Your vit but faster."
• [4] Chen, Mengzhao, et al. "Diffrate: Differentiable compression rate for efficient vision transformers."
• [5] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models."
• [6] Shang, Yuzhang, et al. "Llava-prumerge: Adaptive token reduction for efficient large multimodal models."
• [7] Dao, Tri, et al. "Flashattention: Fast and memory-efficient exact attention with io-awareness."

W2: Additional Comparison Baselines

We thank the reviewer for pointing out additional baselines for comparison!

• A kind reminder that we have already included PyramidDrop in Table 2, Row 11, labeled as “PDrop.”
• In the following table, we show additional comparison for the following baselines: VisionZip[1], DiffRate[2], and PiToMe[3]. We show that DyMU achieves competitive overall performance while being both training-free and dynamic in length.

Reference:

• [1] Yang, Senqiao, et al. "Visionzip: Longer is better but not necessary in vision language models."
• [2] Chen, Mengzhao, et al. "Diffrate: Differentiable compression rate for efficient vision transformers."
• [3] Tran, Chau, et al. "Accelerating transformers with spectrum-preserving token merging."

W3: Merging Per-Layer vs Merging Last-Layer

Compared to post-encoder merging, per-layer merging offers two key benefits:

• (1) Additional efficiency gains within the ViT encoder (as demonstrated in the ToMe paper);
• (2) Progressively merging a smaller number of tokens per layer helps preserve more visual concepts better than merging a large number of tokens in the final layer.

To further validate (2), we provide an additional ablation study as follows, in which we introduce a new DyMU variant that performs merging in the last three layers. Note: Due to the nature of bipartite matching, each layer can merge at most half of all tokens. Therefore, merging only in the last layer is insufficient to meet the target number of remaining tokens. Instead, we merge across the last three layers to reach the same average target token count, i.e., 94 for DyMU-low. This setting can be considered a “less extreme” version of post-encoder merging. Even under this configuration, as shown in the table below, we observe a notable performance drop compared to per-layer merging, particularly on the hallucination task (POPE).

W4,W5: Limitations on Semantically-Dense Images

We are fully aware of this limitation and have acknowledged it in Section 5, where we refer to such cases as token-sensitive tasks, exemplified by TextVQA. In fact, as shown in Table 2, we find this to be a common limitation across all training-free methods. We also agree that incorporating additional benchmarks, such as DocVQA, could further validate this limitation. And future work is needed to explore complexity-aware token merging for semantically dense images, such as multimodal documents.

Nevertheless, we would like to highlight a unique advantage of DyMU not present in previous work: its compatibility with external tools. As illustrated in Figure 6, DyMU can be used in conjunction with OCR tools to address text-dense tasks, while still introducing efficiency gains through dynamic-length token merging. Incorporating agentic tool use into the token merging framework may offer a promising direction for handling the diverse merging needs across different tasks.

Q1: Compatibility with FlashAttention

We currently provide two implementations of VTU: V1 uses the exact matrix decomposition, as derived in Section 3.2, which is more efficient in terms of FLOPs but empirically slower in wall-clock time; V2 uses a direct matrix expansion approximation, which has higher FLOPs but achieves faster wall-clock performance. Among the two, V2 is compatible with FlashAttention. More details can be found in the “Implementation Notes” section of the supplementary code.

Q2: Reconstruction Metric for Image Complexity

We believe this is an interesting idea and a very promising future direction! Specifically, a reconstruction-based metric captures more low-level visual details, which could be particularly useful for identifying complexity in semantically dense images, as discussed above. Moreover, we believe that combining similarity-based metrics (which focus on high-level semantics) with reconstruction-based metrics (which emphasize low-level details) could result in a more general and effective metric for diverse tasks.

We thank Reviewer 2zYg for the constructive and encouraging feedback! We appreciate the reviewer’s recognition of our detailed explanations and comprehensive experiments on the proposed methods. We hope the following response addresses the remaining concerns and questions.

W1: Clarification on the token illustration in the method figure

We thank the reviewer for the suggestion. We believe the reviewer is referring to the red and blue squares in the method overview figure (Figure 2), which indicate the “visual tokens” in the transformer layers. These tokens indeed represent the same characteristics but with different values. We use two colors to better illustrate the bipartite matching algorithm, which divides the tokens into two groups for matching. The red and blue tokens indicate tokens in different groups which will potentially be merged together. We will add additional explanation in the figure caption to clarify this in the next revision.

We thank Reviewer TS2b for the detailed and constructive comments! We appreciate the reviewer’s recognition of the strengths of our comprehensive experiments and the novelty of both DToMe and VTU in enabling training-free dynamic-length token merging. We hope the following response addresses the remaining concerns and fosters meaningful discussion.

W1: Overall Computational Cost

W1.1: Additional Analysis on Wall-clock Inference Times

We thank the reviewer for the suggestion! We provide additional analysis on wall-clock inference times, summarizing our findings below. We continue to see speedup in terms of inference times in most settings, however, we find that the inference time reduction is less consistently correlated with token count and FLOPs, particularly when measuring end-to-end duration. This discrepancy has also been observed in prior works, such as Table 5 in [1] and Table 7 in [2]. We provide a detailed discussion in the next section on the underlying reasons and practical challenges.

W1.2: Deep Dive into the Gap between FLOPs and Inference Time

In this section, we elaborate our findings on why there is a non-trivial gap between FLOP reduction and actual inference time gains, especially for algorithms like DyMU that fundamentally alter the computation of attention. We also explain why we believe FLOPs is a more robust metric for comparing theoretical/algorithmic efficiency, as is also a standard practice in prior works [3][4][5][6].

• Controllability: From definition, FLOPs quantifies algorithmic compute independent of hardware, batching, parallelism, and system load. Wall-clock inference time can vary significantly due to deployment configurations and parallel jobs running on the same node, which are practically difficult to control on shared school servers.

• Implementation Gap: In practice, wall-clock inference time is also heavily dependent on the actual implementations and low-level optimizations of the computation functions. This may lead to significant discrepancies between FLOPs and inference time.

  • Here we show a concrete example, where we compare two implementations of computing a N×N attention matrix from two (N,D) vectors. In Version 1, we simply do one matrix multiplication using torch.matmul to get the N×N matrix. In Version 2, we first split the input vectors into N/4 chunks, do matrix multiplication on the (N/4, N/4) chunks and add them back; As shown below, this results in identical FLOPs but very different wall-clock times. This is because torch.matmul is heavily optimized for large matrix multiplications, leading to longer inference times when multiple torch.matmul calls are made on smaller matrices, despite having the same total FLOPs.

    	# torch.matmul	matrix size per multiplication	MFLOPs	Wall-Clock Time (ms)
    Version 1	1	576	339.74	1.374
    Version 2	16	144	339.74	2.311

  • In our case, the VTU operation requires decomposing the matrix multiplication into smaller sub-matrices, which inflates the inference time although having a better theoretical efficiency in terms of FLOP. (See the “Implementation Notes” section in the supplementary code for more details.)

  • This demonstrates that, to fully translate the benefit of theoretical FLOPs improvements to actual inference time improvements may require non-trivial work such as implementing new kernels, which themselves can be a separate paper such as Flash-Attention[7].

Therefore, in this work, we focus on formally deriving the theoretical efficiency gains and presenting empirical results on FLOPs, token length and end-task performance. But we do acknowledge that optimizing wall-clock inference time of DyMU through dedicated low-level kernel development is an important follow-up work.

Reference:

• [1] Xing, Long, et al. "Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction."
• [2] Wen, Zichen, et al. "Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?."
• [3] Bolya, Daniel, et al. "Token merging: Your vit but faster."
• [4] Chen, Mengzhao, et al. "Diffrate: Differentiable compression rate for efficient vision transformers."
• [5] Chen, Liang, et al. "An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models."
• [6] Shang, Yuzhang, et al. "Llava-prumerge: Adaptive token reduction for efficient large multimodal models."
• [7] Dao, Tri, et al. "Flashattention: Fast and memory-efficient exact attention with io-awareness."

W1.3: How VTU can achieve better tradeoff in performance and efficiency

We elaborate on how VTU can effectively help achieve a better trade-off in both DyMU-low and DyMU-high settings.

• As shown in the Table below, in DyMU-low settings (where token reduction is large) VTU shows more significant improvement in performance and introduces moderate increases in absolute FLOPs (although nearly doubling);
• On the other hand, in DyMU-high settings, removing VTU results in much smaller performance drop; so it’s safer to remove VTU under this setting to gain more efficiency.

W2: Applicability to CNN and Non-RoPE models

We agree that this would be an interesting extension: to explore whether the high-level idea can be applied to CNNs and non-RoPE models. For DToMe, since the fundamental similarity measurement relies on the key vectors of visual tokens, it is not directly applicable to CNN architectures. For VTU, it is not directly applicable to non-RoPE layers due to the use of absolute positional embeddings. Nevertheless, within the claims of this paper, we emphasize that the proposed method remains broadly applicable to most mainstream VLMs, where ViT-based vision encoders and RoPE-based LLM backbones are widely used.

W3: Hyperparameter Search Cost is Minimal, No Finetuning Needed

We would like clarify that:

• The offline threshold calculation is inference-only; no fine-tuning is performed to find the layer-specific r_i (Line 122).
• In practice, threshold finding on a CLIP ViT-L encoder using 250K images can be completed within 1 hour on 2 H100 GPUs.
• There is no strict restriction on the size or type of dataset. Any sufficiently diverse dataset is acceptable, and only images (without annotations) are required.

W4: VTU Approximation in Equation 10

The reviewer raised an interesting discussion on the approximation of the “re-merging” part in the VTU algorithm. To clarify, this approximation is required for obtaining the expected efficiency derived in S3.2, so no direct alternatives can be compared under the same computational complexity. And we show empirically in Figure 5 and Table 6 that this VTU operation successfully leads to performance gains. However, we agree that exploring more sophisticated re-merging methods, such as using similarity-based weighted averaging, would be an interesting direction for future work, albeit at the cost of some additional computation.

Dear AC and Reviewers,

Thank you for facilitating the discussion and for your time and effort throughout the review process!

We greatly appreciate the reviewers’ engagement and look forward to any further feedback and discussions.

Best regards, DyMU Authors