Thank you very much for your quick response!

In most cases, non-speech events play a role in assisting video understanding, where the information of the video is mainly derived from visual and speech. But in some cases, non-speech events are also very necessary, such as the laughter of the audience (outside the video), the sound of car horns (outside the video), and so on.

By string matching, we crudely count speech-related and non-speech atomic events for all videos in the test set. The speech-related events are about the human voice, and the non-speech events are like music, wind blowing, bird chirping et al. The rough statistics are listed below. On average:

• Each video contains 34.2 atomic events.
• Within these, there are 6.1 audio-related atomic events per video, including 4.6 speech-related events and 1.5 non-speech events.

The visual element provides more information than the audio element, but the latter also accounts for a certain portion, especially the human voice.

For off-screen audio, there are a lot of cases. For example, this is a video in our benchmark: https://www.youtube.com/watch?v=t3Y2eYBos4k

Sometimes the speaker appears out of frame or does not show his full body within the video. And non-speech events like the music are off-screen.

It is hard to detect off-screen audio events in the whole benchmark. We just manually and randomly watched 10 videos. We estimate that about 10%-20% of the speech content is said by speakers outside the video frames. For non-speech events, human-related events such as children crying or people screaming are usually in the video, while other events like music are always off-screen.

For Q2 and Q4, we will update the paper later.

Thank you very much for your thoughtful and constructive feedback on our paper. We sincerely appreciate the time and effort you dedicated to reviewing our work, and will revise our paper carefully according to your valuable suggestions. Below, we will respond to each of your points in detail.

• Weakness 1

  • We apologise for the lack of clarity. We will revise and improve the clarity of this paper carefully according to the valuable feedback and suggestions from the reviewers.

• Weakness 2

  • In DPO training, the learning rate is 5e-6. We use 64 GPUs to train the model and the batch size per GPU is 1. The coefficient $\lambda$ of gDPO in Eqn. 11 in the paper is set to 0.1. Actually, a suitable learning rate is important for DPO training. We have searched for a lot of learning rates: 5e-5, 2e-5, 1e-5, 5e-6, 2e-6 and 1e-6, and 5e-6 works best. Too large or too small learning rate may result in training failure. We will include all these details in the revision of our paper. We also promise to release all our datasets, source code and model checkpoints to achieve full reproducibility of our paper, if it is accepted.
  • The DPO loss is initially about 0.8 and almost reaches zero in the end, while cross entropy loss for the ground truth fluctuates around 1.5. We will provide the training curves in our revised paper.

• Weakness 3

  • There may be a confusion here. LoRA Proxy doesn't prevent catastrophic forgetting. It's rebirth tuning that alleviates the problem.

    Actually, the LoRA proxy is a more beneficial method for performance improvement in mrDPO compared to directly training the same LoRA, especially after over 4 rounds of gDPO.

• Weakness 4

  • Thanks for your advice. We will conduct the Elo test for our benchmark and add the analysis results to the revised paper. We're now trying our best to conduct the Elo test and provide the results to the reviewers as soon as possible.
  • Besides, we conduct a case study at https://github.com/Video-SALMONN-2/Video-SALMONN-2.github.io/blob/main/rebuttal/README.md to prove the performance of our method.

• Weakness 5

  • The details of our test set are given below:

  • Video Source	#Sample	Average Duration	#Vocab.	#Word	Average Caption Length	Average Atomic Event Num.
    Youtube	483	51 seconds	17137	296,938	615 words	34

  • We will include these details in our revised paper.

• Weakness 6

  • The captioning results of our Video SALMONN 2 model and the AuroraCap model (SOTA model on the VDC benchmark) are shown below. All the results are tested by gpt-4o-08-06.

  • 	Camera (Acc%/Score)	Short (Acc%/Score)	Background (Acc%/Score)	Main Object (Acc%/Score)	Detailed (Acc%/Score)
    AuroraCap (7B)	43.5/2.27	32.07/1.68	35.92/1.84	39.02/1.97	41.30/2.15
    Video-SALMONN 2 (7B)	35.81/1.94	28.77/1.52	40.83/2.13	40.67/2.12	36.67/1.96

    Task	VDC Score (Acc%/Score)
    AuroraCap (7B)	38.21/1.98
    Video-SALMONN 2 (7B)	36.55/1.93

  • As shown in the tables, on the VDC benchmark, our Video-SALMONN 2 model can achieve competitive results compared with the AuroraCap model, a contemporary work on detailed video caption (without audio) with SOTA performance on the VDC benchmark.

    Note that the VDC benchmark is a pure visual benchmark, which means the videos in the VDC benchmark are all videos without any audio. Since Video-SALMONN 2 was not trained on any silent video data in the SFT and mrDPO stages, the video captions it produces for VDC are potentially not of the best quality. We believe video-SALMONN 2 can generate better captions if there are paired audio tracks that come along with the videos in the VDC benchmark. We observed that video-SALMONN 2 generates severe hallucinations in the captioning silent video. Here are a few examples. Although this could be resolved by retraining the model with silent video, the goal of this paper is to study videos with paired audio.

    {
        "question": "How are the different viewpoints presented?",
        "answer": "through frequent angle shifts",
        "pred_answer": "Through visual actions and continuous audio commentary."
    },
    {
        "question": "Who or what is guiding the individual through the crafting process",
        "answer": "manual",
        "pred_answer": "The man's voice in the audio."
    },
    

We have uploaded the revised paper. The revised parts are marked blue.

For Q2: The training curves are provided in Appendix G (Lines 878-907), and a pointer in Lines 384-385.

For Q4: We conduct Elo rating to rank the video captioning capability of Gemini-1.5-pro, GPT-4o, our visual base model, and the final video-SALMONN 2. Parameters of the Elo rating system are provided below:

We collected 360 pairs of captions generated by different models as simulated matches, where each model generates 180 video captions. Annotators are provided with the video and two corresponding video captions generated by two models, respectively, and they are expected to select the better caption based on its completeness and correctness.

Compared with the Elo experiments in AuroraCap, we use a smaller K-factor. This is because we use a much smaller number of simulated matches (360 vs 2778) due to the time constraint of the rebuttal period and each model wins multiple times against any other model. Therefore, we cannot increase the K-factor, otherwise the rating will not converge. We repeated the collected comparison 8 times to form 2880 matches to get stable rating results. The final Elo rating results and the global caption error rates of each model are provided below:

The results are sufficient to demonstrate the consistency between our proposed automatic metric and human perception. We have updated the paper with the discussion and results in Appendix H (Lines 910-940), and a pointer in Lines 430-431. We are still conducting larger-scale Elo experiments and will update the results once completed.

The results on the VDC benchmark are also provided in Table 9 (Lines 983-995).

Thank you very much for your thoughtful and constructive feedback on our paper. We sincerely appreciate the time and effort you dedicated to reviewing our work, and will revise our paper carefully according to your valuable suggestions. Below, we will respond to each of your points in detail.

Weakness 1: Lots of details about the method seem to be missing or not clearly stated, making it hard to understand the method

• Weakness 1.1

  • In the audio alignment and SFT stages, the "Cross Entropy Loss" is computed on the labelled texts based on "Next Token Prediction". In the audio alignment stage, the model is trained on speech recognition and audio captioning tasks.

  • In the mrDPO stage, we apply a revised DPO loss (gDPO). The loss is computed on $L_{gDPO}(\pi_{\theta}; \pi_{ref}) = -E_{(x,y_{win},y_{lose})\sim {D}}\left[\log \sigma \left(\beta \log \frac{\pi_{\theta}(y_{win}\mid {x})}{\pi_{ref}(y_{win}\mid {x})} - \beta \log \frac{\pi_{\theta}(y_{lose}\mid {x})}{\pi_{ref}(y_{lose}\mid {x})}\right)\right] + \lambda E_{(x, y_{gt})\sim D_{human}}\log \pi_\theta(y_{{gt}}|x)$

  • Here, the first term is the classical DPO loss, where $\pi_{\theta}$ is the policy model, $\pi_{ref}$ is the reference model. $y_{win}$ and $y_{lose}$ are the preferred sample and the dispreferred sample for the input video $x$. The second term is the classical cross-entropy loss on the labelled captions. We will include this standard DPO loss in the revision of the paper.

• Weakness 1.2

  • During LLM generation, we apply top-p sampling to obtain video captions for one video. Top-p sampling, also known as nucleus sampling, is a technique used in text generation that selects from a subset of the most probable tokens based on a cumulative probability threshold p.

    In our implementation, we set the top-p value to 0.9 and the temperature value of 1.0 when sampling captions, and we sample two captions for one video to construct the training set of DPO.

• Weakness 1.3

  • We used the following pipeline to get the atomic events of the video and calculate the metrics (shown in Appendix E):

    First, we have the groundtruth caption the video. We use GPT-3.5 or GPT-4o to split the groundtruth caption of this video into several atomic events, where we use GPT-4o for the test set and GPT-3.5 for the RL training set. Atomic events need to be simple and clear and describe only one particular event. Here are some examples of the atomic events:

    ["Man in yellow shirt holding a clear plastic water bottle", "Man pouring water into a tall, clear glass", "Backdrop with dark blue wall and starry pattern", "Audio mentions a steep bridge in Japan", ...]

    Then, the list of atomic events and the caption to be evaluated are simultaneously fed into GPT-3.5, to find out if there exist any atomic events in the list that are missed or hallucinated (L850). Finally, the quotient between the number of missing or hallucination events and the number of all events in the video is the final missing or hallucination rate (L853).

  • The global caption is to describe the entire video, and the local caption is to describe the video at a specific time interval (L275-278).

• Weakness 1.4

  • We manually observed the caption of the model's output on the test set and directly counted the problematic pattern of the model's output, and subsequently directly removed data containing the same pattern during rebirth tuning. The patterns are something like "[Video Description]: ...", "###Audio Description\n...", and some non-English descriptions and strange characters that are unnatural. Here is a case of unnatural caption: https://github.com/Video-SALMONN-2/Video-SALMONN-2.github.io/blob/main/rebuttal/README.md#unnatural-case

• Weakness 1.5

  • The final model is indeed obtained by training another DPO after rebirth tuning. We'll make it clearer in the revised version.

Dear Reviewer MeEe,

Thank you for your invaluable feedback on our paper. We submitted our detailed rebuttal a week ago, addressing all the concerns you raised. We greatly appreciate the time and effort you invest in the review process, and we would be grateful if you could take a moment to review our rebuttal.

We hope our clarifications and revisions demonstrate the value and significance of our work. Thank you for considering our rebuttal, and we look forward to your feedback.

Best regards,

The Authors

Thank you very much for your thoughtful and constructive feedback on our paper. We sincerely appreciate the time and effort you dedicated to reviewing our work, and will revise our paper carefully according to your valuable suggestions. Below, we will respond to each of your points in detail.

• Weakness 1

  • This internal benchmark is carefully checked by humans, and the evaluation pipeline is also manually checked. Therefore, we can ensure the high quality of the internal benchmark. We also promise to release all of our datasets, source code and model checkpoints, if the paper is accepted.

• Weakness 2 and Question 1

  • When processing with audio and video, we sample 1 frame per second for visual input and utilize the entire audio track for audio input. As a result, our model struggles to handle long videos with paired audios, since the use of the complete audio track leads to an excessively long audio token sequence that can exceed the memory limit. That is why we only evaluate the VideoMME Short subset. In comparison, most public video LLMs evaluated on these datasets do not have paired audio input.

    To avoid this difficulty, to test VideoMME "Medium" and "Long" sets, we discard the audio track and only input the model with the video frames. Besides VideoMME, we also test NeXT-QA, MVBench and VideoVista. These test sets are all video QA benchmarks. For VideoVista, we also discard audio for overly long video when testing.

    The QA results are shown as follows. Note that the training set of VideoChat 2 includes NeXT-QA training data, while video-SALMONN 2 did not use any of these datasets during training.

    Model	VideoMME Overall Acc%	NeXT-QA Acc%	MVBench Acc%	VideoVista Acc%
    Video-LLaVA (7B)	39.9	-	-	53.8
    Long-LLaVA (9B)	52.4	-	54.6	-
    VideoChat 2 Mistral (7B)	42.3	78.6	61.3	54.9
    video-SALMONN 2 (7B)	54.1	71.4	51.4	65.3

    We compare our model with other video LLMs that are of a similar size. As shown in the table, our model gets competitive results on several video QA benchmarks, indicating a relatively strong ability of video understanding. Specifically, we can get relatively high scores on VideoMME and VideoVista, which include paired audio-video data, which demonstrates the advantages of audio-visual LLM.

• Question 2

  • We test our model on the MSR-VTT data and count the length of our model's captions. Unfortunately, the average length of our model's captions is 527 words, while the average of the labelled captions is only 9 words, leading to both BLEU-4 and CIDER results being less than 1. We think that it is not suitable to use these metrics and datasets designed for brief captions to evaluate the ability to generate detailed captions by our model.

    Besides MSR-VTT and Vatex, we also test our model on the recent VDC benchmark as Reviewer aYgd suggested, which is designed for detailed caption evaluation. The captioning results are shown as follows:

    	Camera (Acc%/Score)	Short (Acc%/Score)	Background (Acc%/Score)	Main Object (Acc%/Score)	Detailed (Acc%/Score)
    AuroraCap (7B)	43.5/2.27	32.07/1.68	35.92/1.84	39.02/1.97	41.30/2.15
    Video-SALMONN 2 (7B)	35.81/1.94	28.77/1.52	40.83/2.13	40.67/2.12	36.67/1.96

    Task	Average VDC Score (Acc%/Score)
    AuroraCap (7B)	38.21/1.98
    Video-SALMONN 2 (7B)	36.55/1.93

    Our model is able to outperform the SOTA results in two of the five categories of VDC. In terms of the final score, our model can achieve competitive results as well compared with the SOTA on VDC. This indicates the captioning capabilities of our model.

• Question 3

  • Yes, we will release the internal benchmark upon acceptance, as well as the code, model checkpoints, and training data.

Dear Reviewer 7DpV,

Thank you for your invaluable feedback on our paper. We submitted our detailed rebuttal a week ago, addressing all the concerns you raised. We greatly appreciate the time and effort you invest in the review process, and we would be grateful if you could take a moment to review our rebuttal.

We hope our clarifications and revisions demonstrate the value and significance of our work. Thank you for considering our rebuttal, and we look forward to your feedback.

Best regards,

The Authors

Thank you very much for your thoughtful and constructive feedback on our paper. We sincerely appreciate the time and effort you dedicated to reviewing our work, and will revise our paper carefully according to your valuable suggestions. Below, we will respond to each of your points in detail.

• Weakness 1 and Question 1

  • We test the models on two video question-answering benchmarks: NExT-QA and VideoMME. Since our model is trained on paired audio videos of around 1 minute, we test the models on the "Short" set of VideoMME to reflect changes in the model's QA capabilities more accurately.

  • 	NExT-QA	VideoMME Short
    Ours-Visual Base	72.8	67.2
    Ours-DPO Round 3	72.0	67.3
    Ours-DPO Round 6	70.2	65.3
    Ours-Rebirth Tuning	71.1	67.6
    Ours-Rebirth Tuning + 1 Round DPO	71.4	67.0

  • In the first few rounds of mrDPO, the model's QA capability remained largely unchanged. However, after 6 rounds of DPO, performance on the QA benchmark dropped, although the model's caption became more complete. After rebirth tuning, the model regained performance on the QA benchmark.

• Weakness 2 and Question 2

  • First, in order to ensure the stability and reproducibility of our test results, we have fixed the random seed and set the top_p and temperature to 0.1 and 0.0 respectively when requesting GPT models.

  • As for our evaluation pipeline, while this automated evaluation process may have inaccurate evaluations, we have manually checked each part of our evaluation pipeline and ensured the process is as reliable as possible through prompt engineering.

• Weakness 3 and Question 3

  • As promised in the paper abstract, we will open-source the test data, training data, model checkpoints and source code once the paper is accepted. As for the results on other open datasets, we test our model on the recent VDC benchmark [1] as Reviewer aYgd suggested. The captioning results are as follows:

  • 	Camera (Acc/Score)	Short (Acc/Score)	Background (Acc/Score)	Main Object (Acc/Score)	Detailed (Acc/Score)
    AuroraCap (7B)	43.5/2.27	32.07/1.68	35.92/1.84	39.02/1.97	41.30/2.15
    Video-SALMONN 2 (7B)	35.81/1.94	28.77/1.52	40.83/2.13	40.67/2.12	36.67/1.96

  • 	Average VDC Score (Acc/Score)
    AuroraCap (7B)	38.21/1.98
    Video-SALMONN 2 (7B)	36.55/1.93

  • The AuroraCap 7B model is the SOTA model on the VDC benchmark, and our model can achieve competitive results. Note that the VDC benchmark is a pure visual benchmark. Our model's performance on this benchmark is degraded by the lack of paired audio input since it is trained only on paired audio-visual data.

    In addition to the captioning results, we also test our model on open video QA benchmarks including VideoMME, NeXT-QA, MVBench and VideoVista, as Reviewer 7DpV suggested:

    When testing the "medium" and "long" subsets of VideoMME, since the audio signal is overly long and exceeds the limit, we discarded the audio signal and only provided uniformly selected video frames to our model. Similarly, when testing VideoVista, we discarded the audio when testing videos longer than 2 minutes. The QA results are:

    Model	VideoMME Overall Acc%	NeXT-QA Acc%	MVBench Acc%	VideoVista Acc%
    Video-LLaVA (7B)	39.9	-	-	53.8
    Long-LLaVA (9B)	52.4	-	54.6	-
    VideoChat 2 Mistral (7B)	42.3	78.6	61.3	54.9
    video-SALMONN 2 (7B)	54.1	71.4	51.4	65.3

• Response to Weakness 3 and Question 3 (Continued)

  • Although the training data of VideoChat 2 includes the NeXT-QA training set, video-SALMONN 2 was not trained using any of these datasets, video-SALMONN 2 is still able to achieve competitive results on these video benchmarks.

    Regarding using the MSR-VTT benchmark, we believe the BLEU and CIDER metrics often used for MSR-VTT are unsuitable for evaluating video-SALMONN 2. This is because the detailed captions generated by video-SALMONN 2 have an average length of 527 words, while the reference captions provided by the MSR-VTT benchmark have an average length of only 9 words.

• Weakness 4 and Question 4

  • The detailed analysis is shown in Sec. 5.2, and the results are shown in Figure 4 (a). Here we summarise the results in the table below.

  • Model	Total Error Rate%
    Model before DPO	53.6
    Single-round DPO	40.8
    Multi-round DPO	23.8
    Multi-round DPO, with LoRA Proxy	21.3
    Multi-round DPO, with LoRA Proxy, gDPO regularization	16.3

  • Multi-round DPO can achieve significant improvement compared to single-round DPO, with a total error rate reduction of about 17%. In multi-round DPO, the error rate will reduce by 2% if the LoRA proxy is used for training. Since the classical DPO loss might not be very stable during multi-round training, another 5% absolute error rate reduction is received by using gDPO in training.

• Weakness 5 and Question 5

  • Since the experimental resource consumption of mrDPO is very high, it is difficult to make comprehensive comparisons. However, we find that 5 or 6 rounds of mrDPO may be sufficient as the performance improvements are reduced, as shown by the "Proxy-gDPO" line and the "Direct-gDPO" line in Fig. 4.

    Regarding the increased computational overhead of multiple LoRA proxies, it can be eliminated using an implementation trick. Specifically, we merge the LoRA module into the backbone LLM before each DPO round, as shown in Eqn. (10) in the paper. Therefore, in each round of DPO training, there is only one LoRA proxy hooked up to the LLM, and the memory and computational costs are always the same as using a single LoRA adaptor. During inference, we similarly merge the LoRA proxy into the model, which results in a model without any LoRA adaptor.

    We train the model for 1k steps in each DPO round, which is enough for convergence. This takes about 6 hours in our experiments.

• Weakness 6 and Question 6

  • We do consider the synchronisation of audio and video in model training. Besides global video captioning, we also train our model to conduct local video captioning, which is to describe the paired audio and video events within a certain time interval. An example of local video captioning is shown below:

    Prompt: Please describe the content of the video and audio from 00:11 to 00:38, focusing on the key events, visual elements, and sounds.

    Response: From 00:11 to 00:38, the video continues to show the individual handling various pieces of paper and woodcuts, with the person explaining the history and significance of the woodcuts. The setting remains the same workshop environment with the ...

    We also test our model using another internal benchmark, which is an audio-visual QA test set focusing on audio-visual synchronization. There are mainly two types of questions in this benchmark:

    • When the speaker mentions some specific things, what happens in the video?
    • When some specific things happen in the video, what does the character says?

    The accuracy of our model is shown below. Gemini-1.5's result is also shown:

    	Synchronized Audio-Visual QA Acc%
    Gemini-1.5-pro	88.9%
    Video-SALMONN 2	65.8%

    Although the performance of our model on the Synchronized Audio-Visual QA is worse than Gemini-1.5-pro, the accuracy is still impressive considering its small model size and much smaller training data. Note that there is no other model to evaluate on this internal benchmark at the moment due to the lack of general speech and audio understanding ability of many public video LLMs. We will provide further case studies and will include the Synchronized Audio-Visual QA results in the revised paper.

[1] Chai, Wenhao, et al. "AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark." arXiv preprint arXiv:2410.03051 (2024).

Dear Reviewer zUKj,

Thank you for your invaluable feedback on our paper. We submitted our detailed rebuttal two weeks ago, addressing all the concerns you raised. We greatly appreciate the time and effort you invest in the review process, and we would be grateful if you could take a moment to review our rebuttal.

We hope our clarifications and revisions demonstrate the value and significance of our work. Thank you for considering our rebuttal, and we look forward to your feedback.

Best regards,

The Authors