Video Token Merging for Long Video Understanding

Thank you for your positive and valuable comments. Please find our responses below.

---

Exploration

Compared to image token merging methods, token merging for video is relatively under-researched. In this work, we investigate various video token merging methods and finally propose a learnable VTM that outperforms all previous techniques. Like our algorithm, there are many conventional algorithms in the video computer vision community that proposed simple but effective algorithms, such as 3D-CNN, inflated 3D kernel, (2+1)D CNN and etc. Hence, we believe that the exploration of various basic token merging methods has its benefits for the research community and the proposed learnable VTM algorithm will serve as a strong baseline. However, as the reviewer suggested, we will reduce Section 3.3 and include more analysis on learnable VTM.

Training of learnable VTM

Learnable VTM is optimized to reduce the classification losses on both main path and auxiliary path. Also, the entire process is differentiable because we merge the matched tokens by using average pooling as in the standard token merging. Partitioning and matching processes do not have to be differentiable for end-to-end backpropagation. These processes do not update the features but just determine which tokens to merge. We will include this explanation in the revision.

Stretched claims

We agree with the reviewer. We will revise the draft to explain the experimental results precisely.

Citations

We will revise the draft as suggested.

Notation C and D

We will revise the notation more clearly.

Throughput

Yes, we measure the FPS.

Training

In learnable VTM, we apply the same video classification loss to both predictions from the main path and the auxiliary path. Therefore, the saliency estimation module is optimized to assign high saliency scores to the important tokens to reduce the classification loss during training, since the tokens with high saliency scores have more influence in the guided attention in the auxiliary path.

Figure 4

Yes, tokens in the auxiliary path are also merged according to the matching results from the main path.

---

We will address all these comments faithfully in the final paper. If you have additional comments, please let us know. Thank you again.

We do appreciate your constructive comments and will address them faithfully in the final paper. Please find our responses below.

---

Difference from CTS

CTS is not similar to the proposed algorithm, since it is not even a token merging method. CTS is a semantic segmentation algorithm which shares some neighboring tokens expected to belong to the same class in the segmentation network. Hence, CTS does not have any similar key idea such as learning the token saliency or merging tokens. We will do our best to answer your concerns if you provide more detailed points.

Design for video domain

Compared to the images, there is much more redundant or noisy information in the videos, which deteriorates the quality of video understanding. The proposed learnable VTM is a domain-specific algorithm for long-context scenario. To reveal the discriminative information among redundant tokens in the video, we develop the learnable VTM which learns the saliency of each token and keeps the salient tokens, thereby increasing the influence of these tokens in the following attention processes after merging. Also, we explored the motion-based VTM, which selects the tokens with large motions as the target tokens, and thus this token merging method is only applicable to the video.

Training time

Below, we compare the training time. The training time is slightly increased, but not doubled. We use 8 Tesla V100 GPUs and PyTorch for the experiments as stated in L258.

Attention as saliency score

The objective of saliency estimation is to assign high scores to the tokens corresponding to important objects and low scores to them corresponding to backgrounds or noisy information. However, $\sum QK^t$ would assign high scores to the tokens which have many similar tokens regardless of their semantics. Therefore, we employ the saliency estimation module for the proposed learnable VTM.

---

If you have any additional concerns, please let us know. We will address them faithfully. Thank you again for your constructive comments.

1. Design for video domain

The author, although exploring motion-based VTM, ultimately adopts a learnable VTM. In the learnable VTM, I do not see any special design for the video.

2. Training time

Could you explain why maintaining two paths during training does not result in a relationship of twice as much?

We do appreciate your constructive comments and will address them faithfully in the final paper. Please find our responses below.

---

Token merging for video

To the best of our knowledge, there are only a few techniques for video token merging, such as TESTA(EMNLP2023), ToMe(ICLR2023), and VidToMe(CVPR2024). However, all of them are straightforward applications of the standard token merging to video input. Specifically, TESTA and VidToMe employ the same token partitioning, matching, and merging scheme of ToMe. In contrast, we explored multiple token merging methods for video and proposed the learnable token merging method which shows good performances on various long-video classification datasets.

Saliency

Learnable VTM uses the estimated saliency scores to divide the tokens into target token set and source token set. The saliency score estimation aims to select a greater number of important tokens (e.g. tokens corresponding to important objects in the scene) as target tokens, thereby increasing the influence of these tokens in the following attention processes after merging.

As in the standard token merging, we merge tokens based on their similarity. Hence, even though some tokens corresponding to the important objects are selected as target tokens, they are not merged with irrelevant tokens, and thus we do not lose much important information during the merging process. It is also shown in Figure 5. We can see that tokens are merged more in the background area than in the salient area.

Also, as stated in L206-208, we sample the target tokens based on the probability which is computed from the estimated saliency scores. Hence, it is less likely that the tokens with low saliency scores are selected as target tokens, but not always. As shown in Figure 3, some tokens corresponding to backgrounds or unimportant objects can be selected as target tokens.

Comparison

Please note that ToMe (ICLR2023) does not provide the results on the long video classification datasets. Kinetics only includes 10s video clip which is spatially heavy and without many temporal dependencies. From previous literature, it is easy to receive good performance with a single frame (What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets, CVPR 2018), which is very different from the scope of this paper. As stated in L154-155, our naive VTM is a straightforward application of ToMe which exploits the standard token merging method as intact as possible. As the reviewer pointed out, the memory usage and throughput of various VTM methods that we explored in this paper are almost the same. However, with the same memory and speed, a learnable VTM shows better performances than other VTM methods. Moreover, the proposed algorithm outperforms the conventional long-video understanding algorithms with large margins even though it has much better memory-efficiency and speed.

Long video understanding

For evaluation, we just follow the standard protocol in the field of long video understanding. For fair comparison, we use the same datasets (LVU, COIN, and Breakfast) and the same input setting as in conventional algorithms (ViS4mer, S5) for long video understanding.

High performance of VTM

In the long video understanding datasets such as LVU, there is a lot of redundant and noisy information in a video. Therefore, as shown in Table 1, the transformer network does not yield reliable prediction results because those redundant tokens may hinder the feature refinement in the attention layers. However, when tokens are merged properly, the influence of important tokens in the attention process increases and thus the network can yield better performances, as shown in our experimental results. This phenomenon has also been revealed in the recently published token selection work (Selective Structured State-Spaces for Long-Form Video Understanding, CVPR 2023).

Motion vector computation

As stated in L194-195, motion decoding takes only 0.3 milliseconds for each frame which is negligible, since the motion is converted to the token resolution $H \times W$ by using average pooling.

Training time

We list the training time for the LVU and COIN datasets below. The training is finished within a few hours. We use 8 Tesla V100 GPUs and PyTorch for the experiments as stated in L258.

Re-training, fixed compression ratio

We agree with the reviewer. We will include this discussion in the revision.

---

If you have any additional concerns, please let us know. We will address them faithfully. Thank you again for your constructive comments.

Thanks for the detailed rebuttal, I really appreciate your efforts. I will raise my rating for now to a 4. I have some further doubts that I'd appreciate clarification on.

Saliency

I think I understand this better now but I really don't think saliency is the right word. From what I understand, the point of this is to learn some scoring function to do a better job of partitioning, since the hypothesis of the paper is that the partitioning scheme of ToMe is suboptimal. Maybe "learned partition" coudl be a better word? Saliency is too broad / vague of a term.

Training Time

When I asked for the training time, I meant compared to standard ToMe (no learned module) and a baseline with no merging. From the results of the original ToMe paper they demonstrated very large speedups on training time. It seems that learnable VTM would remove those gains, which would be a big reason to not use this method. Can you comment on this?

Long Video vs Short Video

I still don't really see why this method is tailored for long video at all, and Reviewer 7bwU agrees. Nothing about learnable VTM specifically is more effective than other methods with more frames . Furthermore, as I mentioned, you can train a model on Kinetics-400 with 64 frames as well and we should be able to notice some sort of improvement. What exactly about this method is specialized for long videos and thus merits not evaluating on Kinetics (where there are more baseline results)?

Thank you for your positive review and insightful suggestions, all of which will be addressed faithfully in the final paper. Please find our responses below.

---

Difference from EVEREST

The proposed learnable VTM is quite different from EVEREST. EVEREST measures the cosine similarity of tokens at the same spatial location in consecutive frames for more efficient mask generation which is used for masked video autoencoder training. Therefore, by its design, it does not consider the content of each token for saliency prediction. In contrast, learnable VTM employs the auxiliary path and the saliency guided attention to learn the saliency of each token. Therefore, the saliency estimation module is optimized to assign the high saliency scores to important tokens to reduce the classification loss in the auxiliary path.

Learnable VTM variants

Below, we compare the variants of learnable VTM. In 'From $X$’, we estimate the saliency scores from token features $X$ instead of key vectors $K$. In `Multiply’, we use $Q_\mathrm{aux}K_\mathrm{aux}^t\odot\mathbf{1}S^t$ inside Softmax in Eq.(10) for the saliency guided attention. Here, $\odot$ denotes the Hadamard product. Both alternatives show decent scores. However, the proposed learnable VTM yields better performances and thus we employ it as our method.

Impact of $L$

Yes, as the reviewer pointed out, $L_i$ is an important hyper-parameter for both efficiency and effectiveness of the proposed algorithm. Therefore, we analyze it in Table 6. However, the performance gain of learnable VTM is not just from the selection of $L$, because the other VTM methods in Table 1 yield low scores at the same $L$ setting. We will discuss this in the revision.

$L>60$

We evaluate the proposed algorithm with $L=60$ since conventional algorithms use 60 frames for their evaluation. Below, we list the performances of learnable VTM with $L=100$ setting. We see that the proposed algorithm yields good results in this setting as well.

Saliency visualization

We will include the visualization results in the final copy.

Figure 5

In Figure 5, we visualize the token merging results at the last VTM block, as stated in L298-299. We will revise this more clearly in the revision.

L145-147 and typos

We will revise them thoroughly. Thank you for your suggestion.

---

If you have any additional concerns, please let us know. We will address them faithfully. Thank you again for your positive comments.