Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner

W1 Detailed Description of the Video Token Rearrangement Process

Our goal is to enable the pre-trained video encoder and alignment projector, which are configured to process a fixed number of frames $N$, to handle longer video sequences without retraining. The challenge arises because directly concatenating tokens from multiple passes of the encoder can lead to temporal inconsistencies, as the encoder's internal representations are not designed to naturally accommodate increased temporal variety.

To address this, we propose a video token rearrangement technique that preserves temporal consistency. The key idea is to interleave frames across multiple subsequences and then rearrange the resulting tokens to reflect the correct temporal order. Here's a step-by-step description:

1. Sampling Frames: Given a video sequence with $m\times N$ frames, we aim to process it using the fixed-capacity encoder. We divide the total frames into $m$ subsequences, each containing $N$ frames.

2. Constructing Subsequences: Instead of dividing the video into consecutive chunks, we sample frames in an interleaved manner to capture temporal dynamics across the entire video. For the $i$-th subsequence $(i=0,1,\cdots, m-1)$, we select frames at positions: $FrameIndices = \{i+jm| j=0,1,\cdots, N-1\}$. This means each subsequence contains frames spaced $m$ frames apart, ensuring that all parts of the video are represented.

3. Processing Subsequences: Each subsequence $\mathbf{X}_{v,i}$ is processed independently through the frozen video encoder and projector:

   $Z_{v,i} = VideoEncoder(X_{v,i})$, $H_{v,i} = Projector(Z_{v,i})$. This yields $m$ sets of visual tokens $\mathbf{H}_{v,i}$, each corresponding to a subsequence.

4. Reassembling Tokens: We rearrange the tokens from all subsequences to reconstruct the full temporal sequence:

   • For each frame position $k = 0, 1, \ldots, mN-1$, we identify the subsequence $i = k \bmod m$ and the position within that subsequence $j = \lfloor \frac{k}{m} \rfloor$.

   • The token for frame $k$ is then $\mathbf{H}_{v,i}[j]$.

   • This process results in a new token sequence $\mathbf{H}'_v$ that aligns with the original temporal order of the video frames.

Q2 Pseudocode Illustration of the Video Token Rearrangement Process

To further clarify, we provide pseudocode for the video token rearrangement process:

# Assume we have a list of frames: frames = [frame_0, frame_1, ..., frame_{mN-1}]
# Number of frames per subsequence: N
# Number of subsequences: m

# Step 1: Initialize subsequences
subsequences = [[] for _ in range(m)]

# Step 2: Distribute frames into subsequences in an interleaved manner
for idx, frame in enumerate(frames):
    subseq_idx = idx % m
    subsequences[subseq_idx].append(frame)

# Now, each subsequence[i] contains frames at positions:
# subsequence[0]: frames at indices [0, m, 2m, ..., (N-1)*m]
# subsequence[1]: frames at indices [1, m+1, 2m+1, ..., (N-1)*m+1]
# ...
# subsequence[m-1]: frames at indices [m-1, 2m-1, 3m-1, ..., mN-1]

# Step 3: Process each subsequence through the encoder and projector
tokens_list = []
for subseq in subsequences:
    Z_v = video_encoder(subseq)  # Visual tokens from the encoder
    H_v = projector(Z_v)         # Projected features
    tokens_list.append(H_v)

# Step 4: Rearrange tokens to reconstruct the full sequence
H_v_prime = []
for frame_idx in range(len(frames)):
    subseq_idx = frame_idx % m
    token_idx = frame_idx // m
    H_v_prime.append(tokens_list[subseq_idx][token_idx])

# H_v_prime now contains tokens ordered according to the original frame sequence

Explanation of Why Our Method Leads to Better Temporal Consistency

Interleaved Frame Sampling: By constructing subsequences where frames are sampled every $m$ frame (i.e., frames are spaced $m$ frames apart within each subsequence), we ensure that each subsequence matches the temporal structure the encoder expects.

Problem with Concatenation: Simply concatenating tokens from multiple encoder passes can introduce temporal discontinuities because the encoder processes each chunk independently, without awareness of the temporal context across chunks.

Comprehensive Temporal Coverage: Interleaving frames allows each subsequence to capture temporal information spread throughout the entire video, rather than just a segment.

Compatibility with the Encoder's Training, Frozen Encoder and Projector: Since the encoder and projector are kept frozen, it's important to provide inputs that align with their training conditions.

W1 More Baseline Video-LLMs

Additional State-of-the-Art Comparisons: We have expanded our experiments to include PLLaVA, demonstrating our method's generalizability:

There are two observations: INTP provides consistent improvements across all benchmarks, and the gains are achieved without any training or additional resources.

Regarding Other Recent Models: Video-LLaMA 2 and Flash-VStream were released in June 2024. According to ICLR's policy on contemporaneous work (4-month window), these works fall outside the required comparison scope for our submission. However, our method's success with both Video-LLaVA and PLLaVA suggests that INTP's core techniques are model-agnostic and could potentially benefit these newer architectures as well.

W2 Suggestion for Improving Readability.

Thank you for this suggestion to improve clarity. We agree that including the number of frames provides important context for performance comparison. We have updated Table 2 (highlighted in red).

W3 Comparison to other LLM sequence extension Method, such as LongVA

Thank you for suggesting this comparison with LongVA. While both works address the challenge of extending context length in vision-language models, there are fundamental differences in approach and practical implications:

Training-Free vs Training-Required:

LongVA requires substantial training:

• 2 days on 8 A100 GPUs
• Estimated hardware cost: $150K-200K
• Additional training data needed

INTP (Our method is a free lunch):

• Zero training required
• Implementation through code modification only
• No additional data or computing resources needed
• Immediate deployment capability

Our training-free approach democratizes long-context video understanding.

W4 Extention on More Benchmarks

New Results on MVBench:

These results further validate our method's effectiveness across different evaluation scenarios.

Regarding Other Benchmarks:

• VideoMME: Released in June 2024, falls outside ICLR's 4-month window for contemporaneous work.
• MoVQA: We attempted evaluation but the evaluation code and date are not publicly available.

W1: Calcification on Performance

We would like to emphasize several important points:

• Cost-Free Performance Gains: While the improvements may appear modest numerically, they are achieved without any training, additional data, or computational costs. This "free lunch" improvement is significant considering:

• No additional GPU hours required (compared to thousands of hours needed for training Video-LLMs)

• No need for additional video-language paired data

• No architectural modifications

• Consistent Improvements Across Models: Our method generalizes well to stronger Video-LLM baselines like PLLaVA: |Method | MSVD-QA | MSRVT-QA | ActivityNet-QA| |---|---|---|---| PLLaVA7B | 76.6 | 62.0| 56.3 PLLaVA7B + INTP | 76.9 | 62.6| 57.1 Video-LLaVA | 70.7 | 59.2 | 45.3 Video-LLaVA + INTP | 72.0 (+1.3) | 61.4 (+2.2) | 48.9 (+3.6)

• Practical Impact: The improvements are particularly meaningful considering:

  • They are achieved through a plug-and-play approach that can be immediately applied to existing models
  • The method enables processing of 4x more frames (from 8 to 32) without any training
  • The gains are consistent across different architectures and benchmarks

• Memory Efficiency: Beyond accuracy improvements, INTP also provides memory optimization through KV-cache compression, making it practical for deployment.

---

W2 Calcification on Novelty

While NTK interpolation is indeed established for LLMs, our key contribution lies in the systematic integration of multiple components to enable training-free video length extension:

• We demonstrate how NTK interpolation can be effectively combined with token rearrangement for video understanding
• Our ablation studies (to be added in revision) show that neither component alone is sufficient
• The synergy between these components enables processing 4x more frames without training

While we demonstrate our method using video encoders, the core principles apply more broadly to any Video-LLMs:

• Many Video-LLMs (including those without explicit video encoders) face similar frame-length constraints in their visual encoding components
• The token rearrangement technique we propose can be adapted to any fixed-length visual encoder
• Our results on both Video-LLaVA and PLLaVA (using only ViT, Projector and Pooling layer for video tokens) demonstrate this generalizability

Our contribution lies not in inventing new base components, but in developing a complete, practical solution for extending Video-LLMs to handle longer sequences without training. This addresses a significant practical need in the field.

---

W3 Validation on other Benchmarks

Thank you for this valuable suggestion. We need to clarify two important points:

Technical Scope and Limitations:

Our current method reliably extends Video-LLMs from 8 to 32 frames, as demonstrated in our experiments. While theoretically possible to process more frames, our ablation studies show:

Performance degrades beyond 32 frames, likely due to limitations in the NTK-based interpolation. This finding identifies an important direction for future research in extending Video-LLM capabilities.

Regarding Long Video Benchmarks:

The suggested benchmarks (VideoMME-Long, MLVU, LongVideoBench) were released in June 2024. According to ICLR's policy on contemporaneous work (4-month window), these benchmarks fall outside the scope of required comparisons for this submission. However, we appreciate the suggestion and are actively working on evaluating our method on these benchmarks. we are still trying to implement the experiments on those benchmarks. We hope we can get the results within a couple of days.

Dear Reviewer,

Thank you for dedicating your time reviewing our paper. As the discussion period deadline is approaching, we kindly invite any further comments or concerns you might have. Your feedback has been immensely valuable to us in refining the paper.

Best,

The Authors

W1: Generalization to other Video-LLMs

Thank you for this important comment. We agree that demonstrating broader applicability is crucial. We have expanded our experiments to include PLLaVA [1], another state-of-the-art Video-LLM, and found that INTP generalizes well across different architectures. The results are shown below:

Our method's success with both Video-LLaVA and PLLaVA suggests that INTP's core techniques (token rearrangement and context window extension) are model-agnostic and can benefit various Video-LLM architectures. We will include these results in the revised paper.

[1] PLLaVA: Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning arXiv (29 Apr 2024)

---

W2 Clarification on Ablation Studies

Thank you for suggesting these important ablation studies. We have conducted comprehensive experiments to analyze each component's contribution, with results shown below:

These results reveal several key insights: Necessity of NTK-aware Interpolation: Without NTK-aware scaling, the model fails to generate meaningful responses. This demonstrates that proper position embedding interpolation is critical for extending the context window while maintaining model coherence. Importance of Token Rearrangement: Removing the token rearrangement module leads to significant performance drops (3.8% on MSVD-QA, 4.4% on ActivityNet-QA), indicating its crucial role in maintaining temporal relationships when processing longer sequences. Synergistic Effect: The full INTP system shows consistent improvements across all benchmarks, suggesting that both components work synergistically to enable effective long-video understanding.

We consider the token rearrangement and LLM backbone interpolation as complementary components that must work together - the former handles temporal token organization while the latter enables the processing of the extended sequence.