We sincerely thank the Reviewer gCco for their thoughtful and constructive feedback. We appreciate your recognition of our work addressing the significant challenge of leveraging existing video datasets to improve instruction tuning. We are glad that you found our writing clear and our experiments extensive. In response to your valuable feedback, we have conducted multiple additional experiments, including training two new models, and have updated the manuscript accordingly.

1. Missing Experiments: Validation of Label Filtering and Inclusion of Ground Truth Labels

We appreciate your suggestion to provide validation on the effectiveness of our label filtering process and the rationale for including ground truth (GT) labels in the answers. In response, we conducted additional experiments to validate the parser-verifier and to analyze the correlation between the presence of GT labels in the answers and the correctness of those answers. These results have been added to the revised manuscript.

(i) Parser-Verifier Validation: We randomly selected 500 videos per dataset to validate the parser and prompted GPT-4o to verify whether the label appears in the response (True Positives). To test for False Positives, we selected 500 videos per dataset that the parser filtered out and again used GPT-4o to check for any incorrectly filtered instances.

Table: Verifying the function of the Parser-Verifier. (left) the percentage of filtered answers (answers that were categorized to have had the gold labels in them) that had the ground-truth labels. (right) percentage of filtered-out data (answers that the parser-verifier determined the labels were not in the generated answer) that had the gold labels.

• Filtered Dataset (True Positives): Across all datasets (Kinetics700, STAR-Benchmark, FineDiving), the parser correctly identified the presence of the label in over 97% of the cases across all cycles, indicating high precision.
• Filtered-Out Dataset (False Positives): Only <8% of the filtered-out answers had the correct GT labels. STARB has multiple labels/answers, which is why it had the highest rate of misidentified labels.

These results validate the effectiveness of our parser-verifier in accurately identifying the inclusion of GT labels in the generated answers.

(ii) Correlation Between GT Labels and Correctness: We investigated whether including GT labels in the answers correlates with higher correctness. We randomly selected 500 videos per dataset and categorized the answers based on whether they included the GT label. We then prompted GPT-4o to assess the correctness of these answers.

Table: Validating that label presence is correlated with the correctness of the answer. (left) The correctness rate is when the label is present in the generated answer. (right) correctness rate when the label is not present in the generated answer. In both Kinetics and FineDiving, labels are closely related to the answer, so we have a low correctness rate for answers without labels. In STAR, they are less related, so there is a slightly lower correctness rate even for answers containing the GT labels.

• Answers Containing GT Labels: Over 95% of these answers were correct across all datasets, indicating a strong correlation between the presence of GT labels and answer correctness.
• Answers Not Containing GT Labels: Only about 10% of these answers were correct, suggesting that the absence of GT labels in the answer often leads to incorrect responses. We believe that the low percentages of Kinetics and FineDiving are due to how similar the answers and labels are.

The author's reply addressed some of my concerns. So I decided to increase my rating. However, considering that the method proposed in this paper still requires dedicated human efforts, it is not a fully automated workflow. It still has obvious limitations that need to be noted.

We sincerely thank Reviewer ke7n for their thoughtful and constructive feedback. We appreciate your recognition of the elegance and effectiveness of our method, as well as your positive comments on our writing and ablation study. In response to your valuable feedback, we have conducted multiple additional experiments, including training two new models, and have updated the manuscript accordingly. Your insights have significantly helped us improve the clarity and depth of our paper, and we address your concerns and questions in detail below.

1. Comparison with Video-LLaVA Baselines and Dataset Details

The Video-LLaVA+ was generated using the same questions in Video-STaR but in a method similar to what you discussed in Weakness 2. This experiment aims to show that the performance improvements are not due to the increased dataset size but due to the CoT QA generations.

We understand that Reviewer ke7n feels that our Gemini distillation baseline is too weak, so we created a more robust comparison and updated our manuscript accordingly.

Improved Video-LLaVA^Gemini Baseline:

• Model Initialization: Instead of starting from the Video-LLaVA pretrained model, we initialized from the SFT variant, which has already seen approximately 100K videos. This provides a stronger baseline.
• Increased Labeled Data: We labeled an additional 8K videos using Gemini, totaling 10K videos for fine-tuning. While labeling more videos is ideal, it is cost-prohibitive (approximately $1,000 USD for 10K videos), limiting our ability to scale this further.
• Training Procedure: We fine-tuned the model on a mixture of 50% Gemini-labeled data and 50% from the original Video-LLaVA mixture to mitigate catastrophic forgetting.

This improved baseline is more competitive and provides a better comparison to our method. The results are shown in the updated tables below.

Table: Results of improved baseline (Gemini Distillation) on the adapted datasets

Table: Results of improved baseline (Gemini Distillation) on TempCompass

2. (a) Lack of experimental description, theoretical analysis, or other deep analysis of some assumptions.

We appreciate your suggestion to provide more detailed experimental descriptions and analyses. In response, we have conducted additional experiments and analyses to validate our claims and provide deeper insights into our assumptions. These will be included in the revised manuscript.

(i) Higher Possibility of Hallucination in Label Rationalizations:

We conducted a new experiment to test hallucinations and demonstrate that this is mostly an issue during Label Rationalization, while the hallucination rate in Answer Generation is low. This experiment is included in the revised manuscript. While a ~10% hallucination rate in Label Rationalization may appear to be problematic, Label Rationalization is only used to bootstrap Answer Generation. The final dataset does not include any Label Rationalization-based generations, so hallucinations will not impact the final model performance.

We randomly selected 500 instances of Answer Generations and 500 instances of Label Rationalizations per cycle and per dataset (Kinetics700, STAR-Benchmark, and FineDiving). We utilized GPT-4o as an evaluator by providing it with the question, the model’s answer, and the sampled video frames (ensuring alignment with our model’s inputs). We asked GPT-4o to identify any textual descriptions in the answers that did not appear in the video. The results of these experiments can be seen here:

Table: Hallucination rate in Answer Generation and Label Rationalization Across Cycles. For kinetics, we did not use label rationalization. This experiment is included in our revised manuscript.

The higher hallucination rate in Label Rationalizations arises because the model may rely solely on the provided labels and neglect the video content. In contrast, during Answer Generation, the model must reference the video to produce coherent answers, reducing the likelihood of hallucinations. As such, almost no (~1%) hallucination rate can be seen in Answer Generation, while in Label Rationalization, ~10% hallucination rate can be observed. In Kinetics, we did not use Label Rationalization as direct Answer Generation had a high initial conversion rate.

(ii) Generations Containing Ground Truth Labels Have Higher Correctness Probability

Another core assumption is that answers that contain ground-truth labels are more likely to be correct. To show this is the case, we conducted the following experiment. These results are included in the revised manuscript. We randomly selected 500 answers that contain and do not contain the gold labels, per dataset and cycle. We then used GPT-4o to assess the correctness of these answers :

Table: Validating that label presence is correlated with the correctness of the answer. (left) The correctness rate is when the label is present in the generated answer. (right) correctness rate when the label is not present in the generated answer. In both Kinetics and FineDiving, labels are closely related to the answer, which is why we have such a low correctness rate for answers without labels. In STAR, they are less related, so there is a slightly lower correctness rate even for answers containing the GT labels.

• Answers Containing GT Labels: Over 95% of these answers were correct across all datasets, indicating a strong correlation between the presence of GT labels and answer correctness.
• Answers Not Containing GT Labels: Only about 10% of these answers were correct, suggesting that the absence of GT labels in the answer often leads to incorrect responses. We believe that the low percentages of Kinetics and FineDiving are due to how similar the answers and labels are.

If Reviewer AVhL would like us to run additional experiments testing other aspects of our work, we are open to doing so.

We sincerely thank the Reviewer AVhL for their thoughtful and constructive review of our work. We appreciate the reviewer's recognition of the importance of our topic and the strengths they have highlighted. In response to their valuable feedback, we have conducted additional experiments and updated the manuscript accordingly. The reviewer's insights have significantly helped us improve the clarity and depth of our paper, and we have addressed their concerns and questions in detail below.

1. (a) Technical novelty

To the best of our knowledge, our work is the first to explore the potential for self-improvement in LMMs, specifically focusing on the video modality and emphasizing the self-training of video understanding capabilities. While our approach draws inspiration from STaR, we believe that our method presents significant advancements specific to multi-modality. Unlike text-based data, video datasets often contain labels or metadata but lack question-answer (QA) pairs. This fundamental difference required us to redefine the problem and develop new techniques tailored to videos. Some differences we had to introduce to adapt to videos are:

• Multimodal vs Text-only. While STaR is applied only to text-only LLMs, Video-STaR is applied to Large Multi-modal models, specifically video (and easily translatable to image-LMMs).
• Answers vs Labels. While STaR is provided directly with QA pairs, we only had video labels to guide generation. As such, we have shown that answer generations containing labels result in more correct answers, not only CoT (see Weakness 2, response (ii), and the new experiment testing answer correctness).
• Addressing video-specific challenges. We designed the temporal reasoning and perception Q-A generation for videos, including object comparisons, where objects at different times and locations are compared, and grounded captioning. Please refer to the revised manuscript Table 8 for a complete list of all the design question types.
• Developed a parser-verifier specific to video data: STaR checks if the final answer is correct. However, most video datasets do not have QA pairs. Instead, we must first identify the predicted labels in the generated text. For this, we trained many NER models or used regex/BERT embeddings to identify the labels and then compare them correctly to the gold labels.
• Video-Specific, down-stream applications: We demonstrated the utility of Video-STaR to adapt LMM to downstream applications, like acting like a judge in Olympic diving events.

These adaptations aim to tackle the different video-specific challenges. In video, there are many challenges related to generating conversations related to video content. Self-training could alleviate these challenges, particularly when leveraging readily available (or generable) supervision.

1. (b) The method is unclear

We added a detailed description of the Parser-Verifier to the appendix (please see App. Sec. F). If any other part of our work needs clarification, we will happily update those sections.

Dear Reviewer AVhL,

Thank you for your response and feedback. We want to emphasize the contribution of this work. Collecting video instruction tuning datasets is incredibly time-consuming, requiring annotators to view videos and generate question-answer (QA) pairs manually. Unlike images, which can be reviewed in a few seconds, videos demand significantly more time and effort. Therefore, we believe self-training is a promising approach to introducing human supervision to the video-SFT dataset. Currently, no self-training methods are available for video-LMMs, and Video-STaR represents the first exploration in this research direction. Video-STaR can be further scaled up to internet-scale datasets using any available metadata (e.g., YouTube transcriptions, video text) or using auxiliary models (e.g., OWLv2 to generate bbox, OCR, etc).

Regarding point 2, where you mentioned a "lack of experimental description, theoretical analysis, or other deep analysis of some assumptions,". As a result, we tested our main assumptions:

1. Labels in Answer Generation Lead to Higher Correctness Probability: In our rebuttal experiments, we found that answers containing the correct label have a correctness rate exceeding 90%.
2. Answer Generation Has Fewer Hallucinations Than Label Rationalization: Our experiments showed that Answer Generation had a negligible hallucination rate (<3%).
3. Parser-Verifier Clarification and Validation: The revised manuscript includes detailed descriptions, new figures, tables, and experiments validating the parser-verifier.

Could you please let us know what aspects of our work we need to provide more information about experimental description, theoretical analysis, or other deep analysis of some assumptions? In the remaining rebuttal time, we would like the opportunity to improve our work further.

Thank you once again for your valuable input.

Best regards,

The Authors

Dear Reviewer AVhL,

Thank you once again for your thoughtful feedback and for taking the time to review our work. As today is the final day for discussion, please let us know if you have any additional questions or require further clarification. We are eager to address any remaining concerns to improve our manuscript.

Thank you for your time and consideration.

Best regards,
The Authors

We sincerely thank the reviewer for their thoughtful and detailed feedback. We are pleased that you found our approach original, of high quality, clear, and significant to the field. In response to your valuable comments, we have conducted additional experiments and updated the manuscript accordingly. We address your concerns and questions below.

1. Computational Intensity

While our method does involve additional computation, we believe it remains practical and scalable for several reasons:

(i) Compared with human-annotated instruction-tuning data and other data curation methods, Video-STaR doesn’t need human effort or expensive APIs (which also introduce implicit computational overhead) to distill frontier models.

(ii) Once generated, the dataset can be re-used by other models.

(iii) Practitioners can adjust the number of self-training cycles or limit the size of the datasets used based on available computational resources. This flexibility allows for practical implementation without prohibitive overhead.

(iv) Recent trends in the field indicate that smaller models (e.g., less than 3 billion parameters [1,2]) are achieving competitive performance, which can reduce computational requirements. Although we did not employ techniques like Low-Rank Adaptation (LoRA) in our current implementation, such methods can be utilized in future work to decrease GPU memory usage and computational costs significantly.

The self-training process in Video-STaR is inherently scalable. To adapt LMMs to new and diverse task, the scale required is easily accessible to most academics, where gathering ~100K SFT data should suffice. However, Video-STaR could also be utilized on a much larger scale (100M>) by frontier companies to further improve their models. This scale is only accessible to a few large companies.

In the revised manuscript, we discuss potential strategies (Sec. G) —such as using smaller models and parameter-efficient training methods—to mitigate computational intensity for various use cases.

[1] LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding [2] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

2. Hallucination in Rationalization

To address Reviewer MNYz’s concern, we conducted a new experiment to test hallucinations and demonstrate that this is mostly an issue during label rationalization. This experiment is included in the revised manuscript. We found that hallucination rates in label rationalization are relatively low, at ~10%. However, hallucination rates in answer generation are negligible (~1%) and do not increase over cycles, even though the model is trained on label rationalizations. While a ~10% hallucination rate in Label Rationalization may appear to be problematic, Label Rationalization is only used to bootstrap Answer Generation. The final dataset does not include any Label Rationalization-based generations, so hallucinations will not impact the final model performance.

Label rationalization is only used during self-training cycles to inject complex reasoning capabilities into the model, and the answer generation is used to ensure the model would not be greatly influenced by the potential hallucination in the Label rationalization stage. This is why, in the final training stage, we exclusively use direct answer generation, ensuring that any hallucinations from rationalization do not impact the final model’s performance. We detail the experiment below:

We randomly selected 500 Answer Generations and Label Rationalizations per cycle and dataset and set out to determine how many of them were hallucinated. To do this, we prompted GPT4o with the question, answer, and video. We made sure to sample the same frames our model sampled for improved alignment and asked it to identify if any textual descriptions do not appear in the video.

Table: Hallucination rate in Answer Generation and Label Rationalization Across Cycles. For kinetics, we did not use label rationalization. This experiment is included in our revised manuscript.

3. Generalizability of Label Rationalization

We only perform label rationalization if direct answer generation has failed (refer to Fig. 2), which ensures only the question requiring additional reasoning will be rationalized.

Dear Reviewer MNYz,

Thank you for taking the time to review our work and for your thoughtful feedback. We have conducted additional experiments and updated the manuscript accordingly.

We hope that these revisions resolve the concerns you raised. If there are any remaining questions or concerns, please let us know, and we will be happy to address them, including running additional experiments or updating the manuscript.

Thank you again for your valuable insights,

The Authors

Dear Reviewer MNYz,

Thank you once again for your thoughtful feedback and for taking the time to review our work. As today is the final day for discussion, please let us know if you have any additional questions or require further clarification. We are eager to address any remaining concerns to improve our manuscript.

Thank you for your time and consideration.

Best regards,
The Authors

Experimental description, theoretical analysis, or other deep analysis of some assumptions:

1. Hallucination Rates in Label Rationalization and Answer Generation (Reviewers AVhL, MNYz, gCco):

We conducted experiments measuring hallucination rates in both Label Rationalization and Answer Generations, and confirmed that hallucinations are primarily present during Label Rationalization (~10% rate), whereas Answer Generation exhibits a negligible hallucination rate (~1%). Since Label Rationalization is only used to bootstrap Answer Generation and is not included in the final training data, it does not adversely affect the model's performance (see Table below or Figure 13 in the appendix).

Table: Hallucination rate in Answer Generation and Label Rationalization Across Cycles. For Kinetics, we did not use Label Rationalization. This experiment is included in our revised manuscript.

2. Correlation Between Ground Truth Labels and Answer Correctness (Reviewers MNYz, AVhL, gCco, ke7n)

We validated our assumption that answers containing gold labels are more likely to be correct. Our experiments showed that over 95% of answers containing gold labels were correct, while only about 10% of answers without gold labels were correct. This strong correlation justifies filtering generations that include gold labels and supports the effectiveness of our parser-verifier (see Table below or figure 12 in the appendix).

Table: Validating that label presence is correlated with the correctness of the answer. (left) The correctness rate when the label is present in the generated answer. (right) correctness rate when the label is not present in the generated answer. In both Kinetics and FineDiving, labels are closely related to the answer, which is why we have such a low correctness rate for answers without labels. In STAR, they are less related, so there is a slightly lower correctness rate even for answers containing the gold labels.

3. Validation of the Parser-Verifier Effectiveness + Additional Details (Reviewers MNYz, gCco, AVhL, ke7n)

We performed additional experiments to assess the accuracy of our parser-verifier in identifying the inclusion of GT labels in generated answers. The parser-verifier correctly identified the labels in over 97% of cases (true positives), and less than 8% of the filtered-out answers actually contained gold labels (false positives). These results confirm the high effectiveness of our parser-verifier in ensuring high-quality training data. (see Table below or Figure 14 in the appendix).

To clarify how the Parser-Verifier functions, we added Sec F to the appendix, including Figure 11 and Table 8, which details all the label types identified in Video-STaR and their corresponding parser, verifier, and threshold.

Table: Verifying the function of the Parser-Verifier. (left) the percentage of filtered answers (answers that were categorized to have had the gold labels in them) that had the ground-truth labels. (right) percentage of filtered-out data (answers that the parser-verifier determined the labels were not in the generated answer) that had the gold labels.

Additional experiments

4. Improved Baselines (Reviewers gCco, ke7n)

To strengthen our baseline comparisons, we improved the Video-LLaVA^Gemini baseline by initializing from the fine-tuned Video-LLaVA model, increasing the amount of Gemini-generated data, and including 50% of the original Video-LLaVA dataset (to mitigate forgetting). Despite these enhancements, our method still outperformed the improved baseline. This reinforces our approach’s practical advantages and effectiveness (see Tables bellow or Tables 3-5 in the manuscript).

Table: Results of improved Gemini Distillation baseline (Video-LLaVA^Gemini) on the adapted dataset

Table: Results of improved Gemini Distillation baseline (Video-LLaVA^Gemini) on TempCompass

5. Ablations to demonstrate the utility of each component (Reviewers AVhL, ke7n)

We conducted an additional ablation to evaluate the individual contributions of Label Rationalization and Answer Generation. Specifically, we added a model trained only with Label Rationalization, which resulted in lower performance compared to Video-STaR. This highlights the importance of both components, with Answer Generation playing a crucial role in reducing hallucinations and improving answer correctness.

We hope these additional experiments addressed the concerns raised and strengthen the contributions of our paper. We have updated the manuscript accordingly to reflect these improvements.

Sincerely,

The Authors