Exploring the Design Space of Visual Context Representation in Video MLLMs

Thank you for your insightful suggestions. We have outlined our responses to your concerns below. If you have any additional questions, please feel free to let us know. We will do our best to address them.

Weakness: Comparison to SOTA results on the benchmarks. It is understood that the goal of this paper is not to obtain SOTA results on the evaluated benchmarks, but rather to perform an analysis on the design space of visual context representation in video MLLMs. Nevertheless, it would be informative to the reader to have the information of how far the experimental recipes presented in this paper are from the SOTA.

R1: We follow the reviewer's suggestion and compare our result (achieved with 120 frames and 49 visual embeddings per frame) to the SOTA results on the benchmarks.

From the comparison, it is evident that our results initially show a significant gap from the SOTA, primarily due to differences in training data. To address this, we incorporated Video178K, a subset of training data in [2], to replace the instructions used in our original experiments (denoted as Ours+Video178K). This adjustment yielded substantial performance improvements: our results on VNBench and LongVideoBench surpassed the SOTA, while Video-MME became comparable to the SOTA. However, the performances on MLVU and Event-Bench remain below SOTA levels. The disparity on MLVU and Event-Bench can be attributed to the fact that the model in [2] is built upon a powerful image MLLM pre-trained on approximately 8M image instructions and mixed 3.2M image instructions during video instruction tuning, which is a scale not yet reflected in our experimental setup. Nonetheless, these results underscore the effectiveness of our proposed experimental recipes and their potential when paired with richer data. In the future, we plan to incorporate more diverse and extensive image instructions to further enhance our video MLLM's capabilities and close the remaining gaps with SOTA.

[1] Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution

[2] Video Instruction Tuning With Synthetic Data

[3] Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding

Q1: Figure 2b and 3b: In addition to the performance curves on each benchmark, it would be valuable to also plot the average over all of them.

R1: Thanks for the suggestion and we have plotted the average performance in the revised version.

Q2: Section 3: At the other end of the scaling spectrum, it would be interesting to also report the performance without any visual token at all (M=0) to see what would be the comparative performance of a "blind" MLLM.

R2: Following the reviewer's suggestion, we trained a "blind" MLLM and evaluate it on the benchmarks without any visual tokens as the input. The results are summarized below:

From these results, we draw the following conclusions:

• Event-Bench and VNBench are the most reliant on the visual input.

• Comparing the results in the above with Section 3, the model achieves substantial gains across all benchmarks when visual inputs are included.

These findings highlight the critical role of visual tokens in enabling video MLLMs to achieve high performance on diverse benchmarks.

Q3: Minor typo: L325: "increasing the frames"

R3: Thanks for your carefully readining and we will check the writing of the whole paper.

Dear Reviewer WRue,

Thanks for your careful reading of our paper. We have made several improvements to our paper based on your feedback. Since the rebuttal stage is coming to an end, we are more than happy to hear your comments and address any of your further concerns during the remaining time.

Best,

Authors

Dear Reviewer WRue,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. In response to your comments, we have carefully conducted additional experiments and revised our paper accordingly:

1. We conducted additional experiments to compare our results with the SOTA.

2. We included the results of a "blind" video MLLM without frames as input.

3. We made detailed revisions to the manuscript based on your other recommendations.

We would be grateful if you could let us know whether our revisions and responses address your concerns, or if there are additional questions or clarifications you would like us to provide. Since the rebuttal stage is coming to an end, we are more than happy to hear your comments and address any of your further concerns during the remaining time.

Best,

Authors

W3: Since this paper aims to study the scaling effect, conducting a sufficient number of experiments with training models is crucial to mitigate the impact of randomness bias. The authors should clarify the number of models trained for each data point presented in experiments.

R3: Thank you for pointing out the importance of conducting sufficient experiments to mitigate randomness bias. However, due to our limited computational resource, we can not train multiple models for every data point. Thus, before we start training, we take rigorous steps to validate the robustness of our experimental settings to test the reliability of our findings:

(1) Robustness Validation with Multiple Seeds:

To confirm the consistency of our results, we trained the model with a widely used configuration <T,M>=<32,196> using five different random seeds. The results are as follows:

These results demonstrate minimal variability in both loss and accuracy, with a particularly low standard deviation for the loss (0.0002). This confirms that our training process is robust and not significantly affected by randomness. Furthermore, the fluctuations in loss are smaller than those in accuracy on downstream tasks. Therefore, we used the loss as a stable metric to estimate the parameters of the scaling law function.

(2) Predictive Validation of the Scaling Law:

To further validate the reliability of our scaling law, we performed predictive validation by holding out the last data point, refitting the scaling function using the remaining points, and then predicting the loss for the hold-out point. The predicted loss was subsequently compared with the actual loss. For example, when fitting the scaling function for the visual embeddings in Section 3.2.1, we excluded the data point at $M=196$ and refitted the curve to obtain $L_M = 0.46, M_0 = 1.04\times 10^{-5}, \alpha_M = 0.09$, with $R^2 = 0.906$. We then use this refitted curve to predict the loss at M=196, obtaining a loss of 0.678, which closesly matches the actual loss of 0.679. We repeat this process for both embedding scaling and frame scaling, and the results are summarized below:

These results demonstrate that the predicted losses consistently align with the real losses, with prediction errors below 0.01. This validates the reliability of our scaling law and ensures the robustness of our findings.

In summary, while we acknowledge the constraints of not training multiple models for every data point, the combination of multi-seed robustness tests and predictive validation of our scaling law provides strong evidence of the reliability of our results. These efforts effectively mitigate randomness bias and support the conclusions presented in our study.

W2: In the experiments depicted in Figure 2 and Tables 1 & 2, the authors utilize the same T = 32 (L168) for all experiments, which results in video tokens of varying lengths within the same LLM context window size. It is generally assumed that longer tokens within a pretrained LLM context window size lead to better performance. How can it be determined whether longer visual embeddings after the vision encoder or longer visual tokens in the LLM contribute more significantly to performance improvements?

Thanks for the insightful question. Actually, there are two methods that can increase the number of visual embeddings produced by a ViT-based vision encoder: (1) increasing the resolution of the frame, or (2) increasing the number of patches in each frame. To compare the values of longer visual embeddings after the vision encoder versus longer visual tokens input to the LLM, we conducted two sets of experiments based on these two methods.

(1) Increasing the resolution of the frames:

To increase the encoded resolution of the frames while keeping other factors the same, we utilize SigLIP-base-patch16-256 and SigLIP-base-patch16-512 as the vision encoders and design the following three experiments:

• Exp1 (Baseline): Standard setup with 256 visual tokens after the vision encoder and 16 tokens (setting kernel size as 2) input to the LLM.
• Exp2: Increasing the number of tokens after the vision encoder to 1024 while keeping the number of tokens input to the LLM constant at 16.
• Exp3: Keeping the number of tokens after the vision encoder at 256 while increasing the number of tokens input to the LLM to 64.

From these results, we observe:

• Exp3 consistently improves performance across all benchmarks, demonstrating the benefit of increasing the number of visual tokens input to the LLM.
• Exp2 shows slight performance degradation on some benchmarks compared to Exp1, which is expected since more tokens after the vision encoder introduce greater information loss during token compression/sampling to fit within the fixed LLM input size.

(2) Increasing the number of patches per frame:

Here, we compare three setups:

• Exp1 (Baseline): Standard setup with 49 tokens after the vision encoder and 16 tokens (setting kernel size as 2) input to the LLM.
• Exp2: Increasing the number of tokens after the vision encoder to 196 while keeping the number of tokens input to the LLM constant at 16.
• Exp3: Keeping the number of tokens after the vision encoder at 49 while increasing the number of tokens input to the LLM to 49.

From these results, we observe:

• Exp2 shows consistent performance degradation across all benchmarks. The reason is the same as in the above experiment: more visual tokens after the vision encoder means that we may lose more information during token sampling/compression in order to keep the number of visual tokens input to LLM the same.
• Exp3, which increases the number of tokens input to the LLM, improves the performance on most benchmarks, further emphasizing the benefits of directly optimizing the LLM input size.

The results from both sets of experiments consistently show that increasing the number of visual tokens input to the LLM is significantly more effective than increasing the number of visual tokens after the vision encoder. The findings highlight the importance of optimizing the LLM input for improved performance, as excessive compression of tokens after the vision encoder leads to information loss that cannot be recovered downstream.

Thank you for your insightful suggestions. We have outlined our responses to your concerns below. If you have any additional questions, please feel free to let us know. We will do our best to address them.

W1: Is the design space for embedding and frame operations too narrow? The findings within this design space may be biased towards different LLM backbones and vision encoders. We suggest that the authors conduct experiments across a variety of LLM backbones and vision encoders to ascertain the generalizability of their findings. Or at least one more different LLM backbone and vision encoders.

R1: Thanks for the insightful suggestion. We utilize other representative backbone models: LLaMA3-8B and CLIP-ViT-L-336px. Experiments are conducted under the same settings as described in Section 4. The experimental results in the following Table show that both the minimum loss and the optimal benchmark performance are achieved with 120 frames and 49 visual embeddings per frame, consistent with the conclusion in Section 4. This demonstrates the generalizability of our conclusion and are expected to inspire future developments in video MLLMs.

Dear Reviewer 4WaN,

Thanks for your careful reading of our paper. We have made several improvements to our paper based on your feedback. Since the rebuttal stage is coming to an end, we are more than happy to hear your comments and address any of your further concerns during the remaining time.

Best,

Authors

W2: Based on the above question, if this paper is just an experiment report, readers cannot obtain enough inspiration from this paper.

R2: Thank you for the question. Actually, prior research on long video understanding has primarily focused on optimizing training data or model architecture, our work highlights a critical yet under-explored problem: how to effectively represent a video under a constrained context window? We find that the trade-off between the number of frames and visual embeddings has a large impact on the model performance. However, current video MLLMs often neglect it and rely on large-scale hyper-parameters searches for the two factors, which is costly and does not effectively guide the design of video-MLLMs.

Our study aims to fill this gap, investigate the scaling effect of visual context length, and explore the trade-off between the number of frames and visual embeddings. Inspired by prior work on scaling laws [1, 2, 3, 4], we propose a formulation and conduct extensive experiments. The main contribution and insights in our paper are as follows:

• Scaling law for visual context length: We present a scaling law that predicts the performance of video MLLMs based on visual context length.
• Impact of frame and embedding representation strategies: We analyze how different strategies for frame and token representation affect the performance of video MLLMs, offering practical insights for optimization.
• A principled framework for balancing frames and embeddings: We propose a general approach for balancing the number of frames and visual embeddings and generalize it to various visual context window lengths in the updated version.

These contributions go beyond reporting experimental setups and results, providing a broader understanding and practical guidelines for designing video MLLMs.

[1] Training Compute-Optimal Large Language Models

[2] Scaling laws for neural language models

[3] Scaling laws for transfer.

[4] Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance

Question: Besides the question in Weakness, how can you determine the function formation of Eqs. 3, 4, 5, and 6? Just observe from figures 2a, 3a, and 4?

The scaling law's general formulation often adopts a power-law function, typically expressed as $L=kx^{\alpha}+c$, where $k, \alpha$, and $c$ are parameters to fit and $x$ can be model parameters or data volume [1,2,3]. We follow this formulation in defining Eq. 3 and Eq. 4. Initially, we applied a power-law form to Eq. 5, as in Eq. 3 and Eq. 4, but the fitted $R^2$ indicated a poor fit. We then shifted to a linear function, which yielded significantly improved fit. This process of testing different functional forms and selecting the best fit has been a standard practice in scaling law studies [3]. For Eq. 6, we utilized a commonly adopted scaling law function to express relationships between two factors, following established approaches from prior work [2,3]. While other functional forms could potentially be explored, the selected equations yielded sufficiently low fitting errors to justify their use.

[1] Training Compute-Optimal Large Language Models

[2] Scaling laws for neural language models

[3] MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies

Thank you for your insightful suggestions. We have outlined our responses to your concerns below. If you have any additional questions, please feel free to let us know. We will do our best to address them.

W1: My main concern about this paper is the generalization of the final conclusion. As we all know, video MLLM is very large and complex. This paper uses only one model structure (SigLip + Qwen2-7B) to conduct experiments on several benchmarks. Is the final best setting (Topt≈118 and Mopt≈51) generalizable to other models and benchmarks? If not, must we conduct several experiments to fit a new curve to find the best factor combination?

R1: Thanks for the insightful question. Although the video MLLMs are large and complex, their architectures have been gradually converging toward a common structure: vision encoder+connector+LLM, where the primary components are vision encoder and LLM. To verify the generalization of our conclusion in Section 4, we utilize other representative base models: LLaMA3-8B and CLIP-ViT-L-336px. Experiments are conducted under the same settings as described in Section 4. The experimental results in the following table show that both the minimum loss and the optimal benchmark performance are achieved with 120 frames and 49 visual embeddings per frame, consistent with the conclusion in Section 4. This demonstrates the generalizability of our conclusion and is expected to inspire future developments in video MLLMs.

Additionally, Eq. 7 presents a more general conclusion derived from our experiments, and it is easy to generalize across different settings. Specifically, by varying the visual context window length $L$ from 4K to 126K (a range representative of most LLMs), we can compute the corresponding $M_{\text{opt}}$, $T_{\text{opt}}$, and their ratio $\frac{T_{\text{opt}}}{M_{\text{opt}}}$ in the following table.

To verify this conclusion, we conduct additional experiments using Qwen2-7B and SigLIP with a visual context window length of 4K. The results in the following table show that the lowest loss and optimal performance are achieved with 96 frames and 36 visual embeddings per frame, which closely aligns with the predicted $T=97, M=41$. Due to memory constraints, we will leave the validation of longer context lengths for future work.

Dear Reviewer UYUt,

Thanks for your careful reading of our paper. We have made several improvements to our paper based on your feedback. Since the rebuttal stage is coming to an end, we are more than happy to hear your comments and address any of your further concerns during the remaining time.

Best,

Authors

Dear Reviewer UYUt,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. In response to your comments, we have carefully conducted additional experiments and revised our paper accordingly:

1. We verified the optimal settings using other representative base models (LLaMA3-8B and CLIP-ViT-L-336px).

2. We extended our analysis to generalize the conclusion across visual context lengths ranging from 4K to 128K.

We would be grateful if you could let us know whether our revisions and responses address your concerns, or if there are additional questions or clarifications you would like us to provide. Since the rebuttal stage is coming to an end, we are more than happy to hear your comments and address any of your further concerns during the remaining time.

Best,

Authors

Dear Reviewer UYUt,

We sincerely appreciate your time and efforts in reviewing our paper. We have carefully addressed the suggestions and concerns raised by the reviewers. Specifically, we verified our conclusions using other representative base models and extended them to accommodate various visual context lengths.

As the rebuttal deadline approaches, we kindly seek your feedback on the revised version of the paper. Your insights are invaluable in improving our work, and we greatly appreciate your timely response.

Thank you once again for your time and thoughtful review.

Best regards,

Authors