Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization

We sincerely thank the reviewer for their time and thoughtful feedback. We are glad to see the overall positive remarks, particularly regarding the scope of this work for real-world applications and the novelty of our method. We have carefully addressed all points below and would be happy to clarify further if any questions remain.

W1. What is the performance on the LVLM after RRPO on the cases that the model answered correctly before?

To evaluate this, we measure how many samples previously answered correctly by the base model remain correct after RRPO training. Let $B$ be the set of indices correctly answered by the base model, $R$ be the set correctly answered by the RRPO-trained model, and $T$ is the set of all indices in the evaluation set. We define retention_score = $100-(\frac{|B|}{|T|} - \frac{|B \\cap R|}{|T|}) \times 100$ .

As shown below, both RRPO and DPO retain a very high percentage (97.2-97.9) of these samples across all benchmarks. However, the average improvement in performance in case of RRPO is much higher compared to DPO (2.26 vs. 0.46).

W2. Does the difference between coarse-grained and fine-grained reward matter? Gradient comparison between DPO and RRPO.

Difference between coarse-grained and fine-grained reward: Fine-grained reward modeling is crucial for capturing subtle distinctions in response quality and correctness, especially when the model needs to understand why one response is preferred over another. This has far-reaching implications and is very crucial in domains where even minor errors can have significant consequences, for example, in medical, legal, and safety-critical applications.

DPO provides only a coarse signal by treating the entire response as preferred or non-preferred. It does not indicate where the error occurred or which sub-sequence of the entire response contributed to the preference decision. As a result, the model receives limited guidance during alignment, which can lead to inefficiencies in learning. In contrast, our fine-grained reward modeling in RRPO explicitly identifies and localizes errors, providing more targeted supervision. As shown in Table 3 of the main paper, this leads to consistent and substantial improvements across all setups compared to DPO and other baselines, demonstrating the importance of fine-grained alignment.

Gradient comparison between DPO and RRPO; Claim in Line 128: To support our claim in Line 128, we provide a theoretical analysis in Lines 149–174. Specifically, we show that the upper bound of the DPO gradient $\\nabla \_\\text{DPO}$ (see Eq. 10) scales with the total length of the response pairs as $|y^+| \+ |y^-|$. In contrast, the upper bound of the RRPO gradient $\\nabla \_\\text{RRPO}$ (see Eq. 9) scales with the length of the differing concepts i.e., $2NL$ where $N$ is the number of differing concepts between $y^+$ and $y^-$, and $L$ is the average token length of each differing concept.
Since $|y^+| \+ |y^-| \>\> 2NL$, the upper bound of $\\nabla\_\\text{DPO}$ is significantly larger than that of $\\nabla \_\\text{RRPO}$.

Furthermore, the term $\\nabla\_{\\text{D}\_\\text{TKL}}$ in RRPO gradiebt is negative (see Line 167), which makes: $\\nabla\_{\\text{RRPO}} \<\< \\nabla\_{\\text{DPO}}$.

We also provide empirical evidence in Figure 5, which shows that the large gradient in DPO leads to significant divergence from the model’s initial state, unlike RRPO. Importantly, RRPO not only diverges less but also achieves better performance than DPO, which is a highly desired trait at post-training.

W3. Implementation details, e.g., value of $\\alpha$

All key implementation details, including training configurations and hyperparameters, are provided in Appendix E. As reported in Table S6, the value of $\\alpha$ was empirically set to 0.01 for VideoChat2 and LLaVA-Video, and 0.05 for LongVU. Please note that we also share the full codebase with default configurations to support reproducibility.

W4. As part of future research direction, when the key object for understanding is masked, if the model still makes the correct prediction, it can be treated as a hallucination, which is not preferred and should be mitigated? Then, how can your method be further improved to identify such cases?

As a potential direction for future work, our method could be extended to identify and mitigate false positives arising from hallucinations. One possible approach is to adopt a guided masking strategy, where we explicitly track which objects or segments are masked. This would allow for more informed verification during training and help detect cases where the model relies on spurious cues. That said, implementing such a strategy may require richer annotations or additional visual grounding models during data generation, which introduces added complexity.

We sincerely thank the reviewer for their time and thoughtful feedback. We are glad to see the overall positive remarks, particularly regarding the well-executed experimental setup and the generalizability of our approach. We have carefully addressed all points below and would be happy to clarify further if any questions remain.

W1. Can our approach be extended to image tasks?

While our approach currently focuses only on videos, it can be easily adapted to image tasks with minor modifications. For example, in that case, only spatial perturbations (e.g., random masks to occlude parts of the image) would be applied to generate negative responses, due to the absence of a temporal dimension. In fact, as shown in our ablation study in Table 5 (main paper), spatial perturbation alone is also quite effective even on videos. The rest of the pipeline, including data preparation, reward formulation, and training procedure, remains largely unchanged.

W2. Qualitative comparisons.

Qualitative examples are available in the root directory of our github repository (please see the link in the main paper) under a file named QualitativeComparisons.md. These examples further demonstrate the effectiveness of our method in comparison to the baselines on diverse video tasks including fine-grained temporal understanding, comprehensive video understanding, and hallucination. A few qualitative results are also provided in Fig. 1.

W3. Training time comparisons and inference speed

We present a training time comparison of our method RRPO against other preference optimization baselines in the table below. We use LongVU-7B as the base model and train on a single node of 4 A100 80GB GPUs. As shown, the training times across methods are close to each other, demonstrating that training time required for RRPO is in the same range as others. Minor variations in training time can be attributed to fluctuations in data loading, server load, and other similar factors at the time of training.

We also evaluate the inference speed of all the methods. As expected, there is no difference across them, since none introduce architectural modifications that would impact runtime. We report inference speed under various generation lengths for completeness, in the table below.

W4. Evaluation on higher-resolution scenarios

We would like to clarify that the evaluation benchmarks used in this work include videos with a wide range of spatial and temporal resolutions. Specifically, all benchmarks include videos with high pixel resolution (see table below). The temporal duration varies by task, for example, long-video understanding benchmarks feature videos of an hour long, while short-video tasks include videos of a few seconds to several minutes long. A summary of video characteristics is provided below. Table 3 in the main paper demonstrates strong performance across all these diverse benchmarks, indicating that our method is effective across various video resolutions (Res.) and durations (Dur.).

We sincerely thank the reviewer for their time and thoughtful feedback. We are pleased to see the positive remarks, particularly on the rigorous empirical and theoretical analysis, the innovative fine-grained reward modeling, and the scalability of our proposed approach. We have carefully addressed all points below, please let us know if any questions remain.

W1. Densely packed mathematical notations and acronyms.

Due to space constraints, some mathematical expressions are written in-line, which we acknowledge can reduce readability. We will use the additional page allowed in the camera-ready version to reformat these expressions into separate lines wherever possible to improve clarity.

Regarding the acronyms RS (Random Shuffling), LS (Local Shuffling), and GS (Global Shuffling), their definitions are provided in Lines 190–193 under Section 4. Following the reviewer’s suggestion, we will revise the text to make such acronyms clearer and easier to read.

W2. Formatting & Style

We thank the reviewer for their attention to detail. We will ensure that these minor formatting inconsistencies are corrected in the camera-ready version.

Q1. Risk of bias or alignment drift due to the usage of GPT-4o?

We would like to clarify that GPT-4o is not used as the learning target, so concerns about bias or alignment drift do not apply in our setting. To illustrate the process, consider a video captioning example provided in Fig. 4 of the main paper. Suppose the ground-truth response of a video is: “The video features a group of people playing soccer on a sandy field.” Now assume the LVLM generates the response as: “A group of people is seen playing volleyball on a beach covered in sand.”. We use GPT-4o to extract the key incorrect concepts from the LVLM-generated response, in this case, “volleyball” vs. “soccer” and “beach” vs. “field”. We then rewrite the ground-truth response by incorporating these incorrect concepts to produce a closely aligned non-preferred response (which are used at fine-grained alignment training) such as: “The video shows a group of people playing volleyball on a sandy beach.”

In this way, the model is aligned with the ground-truth or preferred responses, which are obtained from the instruction-tuning dataset while explicitly contrasting its own mistakes. Thus, GPT-4o serves only as a tool for identifying and manipulating semantic differences, not as a learning target. Further details on the instructions used in processing the responses are provided in Appendix D.

Q2. Robustness of the identification of incorrect concepts when scaling to noisy or less structured video-language datasets?

We find that identifying key concept pairs $(y^+, y^-)$ using LLMs is fairly robust in practice, particularly as current LLMs such as GPT4o variants have become increasingly capable at handling such language understanding tasks. To further ensure reliability of this process, we extract incorrect concepts sentence by sentence, rather than treating them as monolithic texts. Our method successfully processes responses from a range of diverse video-language datasets, including VideoChat, VideoChatGPT, EGO-QA, TGIF-QA, and WebVid-QA, demonstrating its generalizability. While we acknowledge that LLMs can still make occasional errors in this identification process, this remains the most practical and scalable approach. Manual annotation or human-in-the-loop revision, though potentially more accurate, would be prohibitively expensive and infeasible at scale. Given these trade-offs, our approach offers a strong balance between scalability and perfection.

We sincerely thank the reviewer for their time and thoughtful feedback. We are pleased to see the overall positive remarks, especially regarding the clarity of writing, the motivation behind fine-grained reward modeling, the comprehensive evaluations, and the theoretical analysis. We have carefully addressed all points below, please let us know if any questions remain.

W1. Can synthetic perturbations in generating negative responses capture the model’s true hallucination behavior?

Our experiments on the data generation pipeline (see Sec. 5, Q7) show that mild perturbations are particularly effective in generating negative responses that capture true hallucination and error patterns without introducing unrealistic artifacts such as overfitting to artificial failure modes or misaligned negative responses. As a result, our method improves accuracy by 4.8% to 9.8% on hallucination tasks. Our study further reveals that the absence of any perturbations leads to diminished model performance due to poor generalizability (please see Table 5).

W2. Could the requirement for non-preferred responses to maintain structural similarity with the ground-truth responses (for fine-grained alignment) limit the model’s generalizability to diverse response patterns?

Maintaining structural similarity is not restricted to only the non-preferred responses. In principle, we could rephrase the ground-truth responses to match the semantic structure of the non-preferred ones for fine-grained alignment. However, we adopted a more conservative approach, keeping the ground-truths fixed and modifying only the non-preferred responses, to avoid potentially noisy alignment targets. To further enrich error diversity multiple responses can also be obtained for one original sample with varying perturbations and sampling temperatures. We avoided this direction in order to stay within a fixed computational budget, where we opted for sample diversity across instances rather than generating multiple variants per instance. Our results in Table 3 show that our approach for generating negative responses is effective and generalizes well across a variety of video tasks including comprehensive short- and long-video understanding, hallucination tasks, and fine-grained temporal reasoning.

W3. Experiments on failure cases or domain shifts.

Failure cases: Examples of failed cases are available in the root directory of our github repository (please see the link in the main paper) under a file named FailedCases.md. We notice that depending on the capabilities of the base models, RRPO-aligned models can still make mistakes or exhibit hallucinations in detailed video description tasks. We also observe that RRPO-aligned models may still exhibit limitations in long video understanding tasks, primarily due to architectural constraints of the base model in processing extended frame sequences, as well as computational constraints during RRPO training that limit the use of long video inputs.

Domain shift: Given that the underlying general-purpose LVLMs are trained on massive web-scale corpora during their pre-training and post-training phase, including millions or billions of diverse videos and images, and trillions of language tokens, drawing a clear boundary around what constitutes a “shift” is non-trivial. Moreover, in many cases, the details of the pre-training datasets are not public. Nevertheless, we believe that the diverse set of benchmarks we evaluate on, spanning a wide range of video tasks including video hallucination, comprehensive short- and long-video understanding, and fine-grained temporal reasoning, provides a strong proxy for robustness under diverse conditions.

W4. Discussion on limitations.

Our RRPO method introduces fine-grained reward modeling, which requires preference pairs to maintain structural symmetry, unlike the coarse-grained approach used in DPO, which imposes no such constraint. While fine-grained alignment requires an additional step in data preparation, modern LLMs (e.g., variants of GPT-4o, Gemini, LLaMA, Qwen) are increasingly adept at handling such language tasks, making this step relatively easy to automate. More importantly, given the strong performance gains achieved through our fine-grained alignment strategy, we believe this additional step is a worthwhile trade-off. We will include this discussion in the final version of our paper.

Q1. Is there a trade-off between hallucination suppression and general language-vision capabilities?

While the gains on hallucination benchmarks are more pronounced than those on non-hallucination tasks, we do not observe a trade-off where reducing hallucinations comes at the cost of general capabilities. In fact, the fine-grained reward modelling in RRPO loss is explicitly designed to suppress hallucinations without compromising overall language-vision performance, unlike DPO. As shown in the table below, RRPO improves performance in both hallucination (VideoHallucer) and general benchmarks (MVBench, VideoMME, MLVU) while DPO improves performance (while lower than RRPO) on hallucination benchmarks but leads to a drop in general capabilities.