PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance

Thanks for the valuable comments. We appreciated you giving positive feedback on our paper. All your concerns are addressed below. It is appreciated if you have any further feedback on our response.

Q1.1: How the capability of CLIP affects the model performance is not discussed.

We agree with the reviewer's point that the text-image matching performance of CLIP determines the upper limit of our model. In fact, we have attempted to break through this bottleneck in the paper. For example, we found that CLIP's 77-token encoding context struggles to handle long questions or multi-turn conversations. Therefore, we have proposed an additional CLIP context extension method. As shown in Table 5, this modification to CLIP significantly enhances performance.

On the other hand, the choice of text-image matching encoder actually depends on the visual encoder used by the image-domain pretrained MLLM. For example, since LLaVA-Next uses CLIP-L14-336, we must use the corresponding aligned text encoder. Otherwise, if we were to use a new text-image matching encoder, we would have to start from scratch with a multi-stage pretraining, which is highly inefficient. As an alternative option, in Q1.2, we use another image-domain pretrained MLLM, LLaVA-Qwen, which employs a more powerful text-image matching encoder, SigLIP.

Q1.2: Will a stronger text-image matching encoder further improve the model's performance?

To address the reviewer's concern, we have also implemented PPLLaVA on LLaVA-Qwen and compared it with several other models. LLaVA-Qwen uses the more powerful text-image matching encoder, SigLIP, which may synergize with our model to bring further improvements. Due to time constraints, we trained on our small datasets (300k LLaVA-1.5 image data + LLaVA-Hound-300k video data, totaling 600k), sampling 64 frames per video. The results are as follows (all compared models are based on LLaVA-Qwen2):

It can be observed that PPLLaVA remains highly competitive on the LLaVA-Qwen baseline. Despite having much less training data and shorter context length than LLaVA-One-Vision, it still achieves comparable performance. To be honest, we acknowledge that the changes from LLaVA-Qwen to LLaVA-Next are quite significant. Not only has the visual encoder changed from CLIP to SigLIP, but the LLM has also shifted from Vicuna to the much stronger Qwen2. Therefore, it is difficult for us to quantitatively determine how much of the improved results come from the stronger text-image matching encoder. However, from another perspective, on the LLaVA-Next baseline, PPLLaVA shows relative improvements of 0.23 and 2.6 on VCGBench and VideoMME, respectively. On LLaVA-Qwen, the relative improvements are 0.31 and 5.7. The relative improvements exclude the impact of the LLM in some degree, indicating that the combination of a stronger text-image matching encoder and PPLLaVA indeed further improves the model's performance.

Dear Reviewer 1Y1n,

We want to send you a friendly reminder that the stage of discussion will be completed soon.

It is greatly appreciated if you are willing to reconsider your score based on our responses, and we really want to know whether our responses address your concerns. If there is any other concern that we could not address in the response, please feel free to let us know and we would be happy to provide further explanation.

With sincere regards,

Authors of Paper 1865

Thank you for the valuable comments. We are sorry for some misunderstandings on our paper. We believe that our response can help clarify and address some potential misunderstandings. We have addressed all of your concerns point by point in two pages. It is appreciated if you have any further feedback on our response.

Q1: Confusion about Video Caption.

We agree with the reviewer’s point that the caption prompts provide limited information, which may affect the method's performance. In fact, both our training and test sets contain a large number of caption samples. For example, in the VideoChatGPT-Bench, over 50% of the questions are caption-style prompts such as "What is happening in the video?" and "What is the sequence of events in the video?". Despite this, our method still performs well on VideoChatGPT. We speculate that when encountering holistic captioning prompts, our method might sample the entire video more uniformly and even adaptively extract video features that are helpful for summarizing the video (due to the large number of caption training samples).

We understand that this may not fully address the reviewer's concern. Therefore, following the reviewer's suggestion, we tested our model on the ActivityNet caption. Due to the lack of comparable methods, we only compared our model with our Average Pooling baseline and the AdaptivePooling3D without CLIP guidance. The testing process followed the same procedure as ActivityNet-QA using GPT evaluation (we adopted the score because the accuracy of captioning is difficult to measure). The results are as follows:

The results show that our Prompt-Guided Pooling method still outperforms AdaptivePooling3D, indicating that our approach remains effective for captioning tasks. However, we must acknowledge that the improvement is not as large as observed in other datasets, suggesting that the limited information provided by caption prompts does have some impact on PPLLaVA's performance.

Q2: Comparison with LongVA and Kangaroo.

Thanks for the reminder. The reason we didn't make the comparison earlier is that our model was based on Vicuna-7B, while LongVA and Kangaroo used more powerful LLMs. To ensure a fair comparison, we retrained our model based on LLaVA-Qwen2. The same as the experiment in Reviewer 1Y1n’s Q1.2, we trained on our 600k corpus (300k LLaVA-1.5 image data + LLaVA-Hound-300k video data, totaling 600k) with 64 frames per video. The results are as follows:

As can be seen, under the new baseline, PPLLaVA still achieves very competitive results. Despite the disadvantages in context length and training data, PPLLaVA achieves the optimal result on VCGBench, while also demonstrating comparable performance on Video-MME. Notably, our method still achieves the highest result on Video-MME-Long, indicating that PPLLaVA not only maintains the highest efficiency and fastest throughput, but also excels in long video understanding performance.

We speculate that when encountering holistic captioning prompts, our method might sample the entire video more uniformly and even adaptively extract video features that are helpful for summarizing the video.

Q3.1: Novelty Concerns: the big boost of the performance might come from fine-tuning on 1.3M multimodal data and the prompt-guided pooling seems a bit incremental from my personal perspective.

Aside from the method itself, there are indeed many significant factors that affect performance, such as data and LLM. As shown in Q2, changing the baseline LLM significantly improved performance. However, this does not affect our evaluation of the method's inherent effectiveness. To demonstrate this, we select a few methods from the original manuscript for fair comparison below. They use the same Image LLM, with identical LLM sizes, similar amounts of instruction tuning data, and none of them use multi-image data:

As can be observed, without multiimage data, PPLLaVA still demonstrates strong performance across various test sets under fair comparison, with significant advantages on most benchmarks. These advantages are sufficient to demonstrate that our improvements stem from the new pooling technique. Factors beyond the method itself, such as more data and stronger LLMs, will affect all methods, so they do not impact our evaluation of the method's inherent effectiveness.

Q3.2: Novelty Concerns: So I am a bit worried that if we conduct the same level of training with a stronger LLM backbone that has a long content window, the model can learn this query-related attention itself, so the proposed core idea, prompt-guided pooling, seems sub-optimal.

A stronger LLM backbone and sufficiently long context can indeed solve many problems. For example, the strategies used in LLaVA-Next-Video and LLaVA-OneVision is to directly fill the long LLM context with video tokens, letting the LLM itself interpret the user query and the video. However, this does not conflict with our approach:

(1) One major advantage of our method is that it can improve performance while enhancing efficiency. Filling the context with video tokens can indeed guarantee performance, but long contexts lead to extremely long throughput times, which is not practical for real-world applications. PPLLaVA ensures good performance while providing fast response times.

(2) No matter how large the LLM’s context is, it can only accommodate a limited number of video frames. Our method, however, allows more video frames to be input into the same context, thus improving long video understanding performance.

(3) Following the reviewer’s suggestion, in Q2, we fine-tuned our method on a stronger LLM. As seen, our method still achieves good results with the more powerful LLM, demonstrating the versatility of our approach and idea.

Dear Reviewer vFqm,

We want to send you a friendly reminder that the stage of discussion will be completed soon.

It is greatly appreciated if you are willing to reconsider your score based on our responses, and we really want to know whether our responses address your concerns. If there is any other concern that we could not address in the response, please feel free to let us know and we would be happy to provide further explanation.

With sincere regards,

Authors of Paper 1865

Thank you for the comments. We are shocked and disheartened by the reviewer's reasons for rejecting our paper. We believe there exist some misunderstandings. We have addressed all of your concerns point by point below. It is appreciated if you have any further feedback on our response.

Q1: PPLLaVA does not show leading results compared to some training-free compression methods like SLOWFAST-LLAVA, on benchmarks such as MSVD and MSRVTT.

（1）We use the GPT-3.5-turbo version 0613, which is consistent with previous works like LLaVA-Next-Video, ST-LLM, and Chat-UniVi, while SlowFast-LLaVA uses version 0125. The version of GPT has a significant impact on test results, and version 0125 is more friendly to open-ended VideoQA. To demonstrate this, we re-tested our MSVD and MSRVTT answers using version 0125, and the results are as follows:

As seen, with the same GPT version, we achieved better results than SlowFast-LLaVA with higher efficiency. Additionally, multiple-choice question answering can mitigate fairness issues related to GPT evaluation. Unfortunately, SlowFast-LLaVA lacks results from more specialized Video LLM benchmarks like MvBench and VideoMME.

（2）MLLMs have numerous benchmarks, and it is unfair to dismiss an entire model simply because it does not achieve leading results in one or two benchmarks. For example, LLaVA-Next-Video performs poorly on MvBench, but we would not dismiss LLaVA-Next-Video based on that alone. It's essential to assess models comprehensively across multiple benchmarks of different kinds.

Q2: The use of Direct Preference Optimization (DPO) and Proximal Policy Optimization (PPO) lacks innovation. Also, LLaVA-Next-Video also achieve great accuracy improvements using above methods, but this paper does not highlight any unique advantages of these methods within PPLLaVA.

We have never claimed that the innovative use of DPO is our main contribution in the paper. Rather, what we want to emphasize is the compatibility of PPLLaVA with DPO (Line 701), and the richer choices provided by our codebase (Line 119). As can be seen:

PPLLaVA + DPO achieved the largest growth. This demonstrates the good adaptability between PPLLaVA and DPO. But in any case, this is just a secondary aspect of the paper. We argue that rejecting our paper simply because the subordinate content "lacks innovation" is unfair.

Q3: Difference with Involution

Traditional convolutions share kernels in the spatial dimensions, while Involution shares kernels across channels. In PPLLaVA, the kernel used during pooling is not shared spatially, which indeed makes it similar to Involution. However, PPLLaVA does not have spatial-specific features. Therefore, PPLLaVA employs an operation similar to convolution, but it is neither a true convolution nor Involution.

Q4: Have any ablation studies been conducted using regular 3D convolution for pooling instead of prompt-guided methods?

Using regular 3D convolution without prompt guiding is equivalent to nn.AdaptiveAvgPool3d, which is essentially the content of PLLaVA. This comparison with PLLaVA is already provided in the original paper.

Q5: In Table 5, why is the context length for average pooling set to 576 instead of 1024?

The meaning of average pooling is to directly pool the temporal dimension (T) into 1, so the context length is equal to the image context length. In LLaVA-Next, a single frame image is encoded into 24x24, which results in 576 tokens.

Q6: Can PPLLaVA extend to multimodal generative models?

Sorry, we are not experts in the generation field. We speculate that PPLLaVA could be applied to video editing. By inputting an editing prompt and the corresponding video, PPLLaVA could provide more accurate and generalized editing text for diffusion model. Due to PPLLaVA’s efficiency, it would not significantly increase the editing time.

Dear Reviewer Qkbp,

We want to send you a friendly reminder that the stage of discussion will be completed soon.

It is greatly appreciated if you are willing to reconsider your score based on our responses, and we really want to know whether our responses address your concerns. If there is any other concern that we could not address in the response, please feel free to let us know and we would be happy to provide further explanation.

With sincere regards,

Authors of Paper 1865

To be honest, it was difficult to remain calm after reading the reviewer's comments. So many of the issues have been mentioned multiple times in the paper, yet the reviewer still writes "I cannot see". Regardless, we appreciate the reviewer's efforts. We have addressed all of your concerns point by point below. We sincerely hope that the reviewer will take the review process seriously, so that the efforts of everyone involved are not wasted.

Q1: I cannot see why the approach should be applied to videos instead of image-level applications. Will the method also be applied to image-level applications? is video necessary?

（1）We have emphasized multiple times in the paper! Videos, especially long videos, introduce a large number of tokens, while the LLM's context can only accommodate a limited number of video frames, which necessitates compressing the video tokens. On the other hand, a single image has a limited number of tokens, so models after LLaVA 1.5, by cutting the grid, even increase the number of tokens to achieve better visual understanding. Therefore, PPLLaVA is more needed in videos than in images.

（2）We do have PPLLaVA's results on images! And we even mentioned this in the abstract!!! The image results can be found in Table 6 of the main text. After conducting video instruction tuning, PPLLaVA did not compromise image performance; on the contrary, it achieved better results with higher efficiency.

Q2: I cannot see the comparison of performances when applying the method to long/short videos. Also, how to define "long video"? 1 hour long?

(1) The comparison between long and short videos is already provided in Table 3 of the main text!!! In Table 8, we have also conducted an additional ablation study on whether PPLLaVA contributes to long video understanding!

(2) Most of the benchmarks in the Video LLM field consist of short videos ranging from a few seconds to a few minutes. However, in VideoMME, the "long" video duration can reach 30 minutes to 1 hour, which can absolutely be considered long video within the field.

Q3: I think LLAVA is for images, instead of video. While if the authors want to apply LLAVA to videos, why not do like this: a. locate the related frames; b. VQA on frames. I cannot see the necessity of the three steps way.

Almost all Video LLM papers are built upon image-domain LLMs like LLaVA. Before forming the idea, we "think" the reviewer is expected to at least read one or two papers in our field.

Q4: What if the questions are open-ended questions without an answer?

In training, questions without answers are not included. During testing and real-world usage, if the questions are open-ended questions without an answer, PPLLaVA would try to provide the most reasonable response depending on the question and video content. What if the questions are open-ended questions without an answer for ChatGPT?

Q5: why we need "Prompt-Guided Pooling"? I cannot see the necessity of prompt here.

We've already addressed this point countless times in the paper!!!! Prompt=user question=user instruction in our context. Direct pooling often leads to performance degradation, whereas pooling while extracting video content relevant to the user’s question helps avoid this performance loss.

Thanks for the valuable comments. We appreciate you giving positive feedback on our paper. All your concerns are addressed point by point. We believe that our response can help clarify and address some potential misunderstandings. It would be appreciated if you have any further feedback on our response.

Q1: Improving the diagrams.

We appreciate the reviewer for the suggestion. As suggested by the reviewer, in the new manuscript, we have updated Figure 2 by adding more explanations and context to enhance clarity.

Q2: The fairness.

We understand the reviewer's concern. In the original manuscript, we had marked the results with DPO, while also retaining the results where PPLLaVA did not use DPO on all datasets to ensure a fair comparison. In fact, DPO only significantly improves the model's performance on dialogue tasks. For Video Question-Answering tasks that do not rely on GPT evaluation, DPO does not have much effect.

To further demonstrate the fairness of the comparisons in the paper, we separately compare a few methods below. They use the same Image LLM, with identical LLM sizes, similar amounts of instruction tuning data, and none of them use DPO. The following results are all from the original manuscript:

As can be observed, even without DPO, PPLLaVA demonstrates strong performance across various test sets under fair comparison, with significant advantages on most benchmarks. These advantages are sufficient to demonstrate that our improvements stem from the new pooling technique. The use of DPO or additional data (multi-image) is aimed at achieving the best possible results, and we have provided results without these methods or data. Therefore, it does not affect the fairness of the basic comparison.

Q3: Implementation on existing Video LLM.

That's a very interesting question! Currently, many top-performing MLLMs tend to directly input all video tokens into the LLM, filling up the context, as seen in LLaVA-Next-Video and LLaVA-One-Vision. As a result, PPLLaVA can also be seamlessly implemented on these methods, reducing visual context to increase throughput or support more video frames as input. To test the effect of this approach, we conducted a tiny experiment with LLaVA-Next-Video by initializing PPLLaVA with LLaVA-Next-Video's weights and performing instruction tuning on VideoChatGPT-100k. The results are as follows:

As can be seen, when implemented on Video LLMs, our method significantly improves efficiency without causing a severe drop in performance. In fact, it leads to notable performance improvements on some benchmarks. However, the results are not as strong as those directly obtained with LLaVA-Next. We speculate that this is because Video LLMs have already been trained to fit video instruction data, and further fine-tuning on video instruction data may lead to overfitting or disrupt the knowledge acquired from the original image-text pretraining.

Dear Reviewer tF5Y,

We want to send you a friendly reminder that the stage of discussion will be completed soon.

It is greatly appreciated if you are willing to reconsider your score based on our responses, and we really want to know whether our responses address your concerns. If there is any other concern that we could not address in the response, please feel free to let us know and we would be happy to provide further explanation.

With sincere regards,

Authors of Paper 1865