MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models

Dear Reviewer DZHj,

We sincerely thank you for your thorough feedback: the motivation of this paper is clearly stated and the effectiveness of the method. We address your questions below and have incorporated all feedback in the revised version. All new material added to the revised manuscript has been highlighted in red text for better visibility.

Q1: Can the authors provide a more specific way to compute $A_{target}$ and $A_{sum}$ in L306?

A1: Apologies for not clearly explaining the calculation methods for $A_{sum}$ and $A_{target}$. As described in Line 150, $x$ represents the input, which includes images $v$ and text $t$, where $v$ may consist of multiple images. When $x$ is input into the model, we calculate the attention values of the model's output and the text $t$ tokens relative to the image $v$ tokens, obtaining the attention value for each image. The total attention of the image $v$ pointed to by the instruction (i.e., $t$) is defined as $A_{target}$, while the total attention across all images is defined as $A_{sum}$. In Figure 2, taking the first example as an illustration, we input five images and visualized the concatenated attention map of the five images. The total attention for the first image corresponds to $A_{target}$, and the total attention for all five images corresponds to $A_{sum}$.

Q2: Which are the layers used to extract the attention $A_{target}$ and $A_{sum}$?

A2: Good question! During our experiments, we observed an interesting phenomenon: the differences in attention distribution across different images are most pronounced in the intermediate layers of the model, while this phenomenon is less evident in the initial and final layers. Therefore, we chose the intermediate layers of the model as the layers for calculating $A_{target}$ and $A_{sum}$.

Q3: In section 3.3.3, what are the parameters we need to optimize in the proposed method?

A3: The LLaVA-based LVLMs typically have three components: a vision encoder, an MLP projector, and an LLM. During the DPO process, the vision encoder of the LVLM remains frozen, and the remaining parameters (MLP projector, LLM) are optimized.

Q4: In Table 1 and Table 2, why do the authors not include "RLHF", "HA-DPO" and "POVID" in InternLM-XC2.5?

A4: Thanks for your comment! Since the authors of POVID and HA-DPO have open-sourced their trained models based on LLaVa, we directly tested their released models and report in Table 1 and Table 2. Following your suggestion, we conducted additional experiments by applying DPO to InternLM-XC2.5 using the datasets provided by POVID and HA-DPO. As shown in Table 1, the performance improvements for applying POVID and HA-DPO on multi-image benchmarks were limited. This suggests that while these previous papers are designed for single-image scenarios, their direct application to multi-image settings may not yield significant gains. We believe that the limitations observed in these experiments highlight the unique challenges posed by multi-image hallucinations and the need for specialized techniques like our proposed MIA-DPO approach.

Q5: In Table 1, why LLaVA-RLHF, HA-DPO, and POVID are worse than LLaVA?

A5: While previous works (LLaVA-RLHF, HA-DPO, and POVID) are effective methods for addressing single-image hallucinations, they do not consider multi-image scenarios. Thus, these methods are often trained on datasets that exhibit certain biases on single-image data and may not align well with the diverse and complex nature of multi-image understanding abilities. So in some cases, the application of these techniques can even degrade the original multi-image capabilities of the underlying LLaVA-v1.5 model.

Dear Reviewer DZHj,

We deeply appreciate your efforts in reviewing our paper, and we are particularly grateful for the invaluable suggestions that have greatly enhanced its quality.

In response to your suggestions, we provided more details on computing $A_{target}$ and $A_{sum}$ (Q1 and Q2) and introduced which parameters need to be optimized in our method (Q3). Subsequently, we presented additional baselines for IXC2.5, including the performance of POVID and HA-DPO on IXC2.5 (Q4). Finally, we explained why LLaVA-RLHF, HA-DPO, and POVID perform worse than LLaVA.

Could you please read our rebuttal and consider adjusting the score in light of our response? Thank you for your time and effort!

Best,

Authors

Dear Reviewer DZHj,

Thank you again for your time and valuable feedback on our paper. We have carefully addressed all your comments in our rebuttal and would greatly appreciate it if you could review our response and consider adjusting your score accordingly.

Your support means a lot to us, and we are grateful for your consideration.

Best regards,

Authors

Dear Reviewer DZHj,

I hope this message finds you well. I’m writing to kindly follow up on the review process for our paper. We deeply value your expertise and would greatly appreciate it if you could provide your feedback and scoring at your earliest convenience to help progress the review.

We humbly hope for your positive consideration and support in evaluating our work. Your recognition of the effort we’ve put into this research would mean a great deal to us.

Thank you so much for your time and thoughtful review.

Best regards,

Authors

Dear Reviewer DZHj,

I hope this message finds you well. As we are in the final stages, we would greatly appreciate your valuable feedback and scoring at your earliest convenience.

We sincerely understand how busy you must be, and we are deeply grateful for the time and effort you’ve already dedicated to reviewing our paper. Your support and thoughtful feedback are extremely important to us, and we would be thankful for any comments you can provide.

Thank you once again for your time and consideration.

Best regards,

Authors

Dear Reviewer DZHj.

The deadline is approaching. Now two reviewers post their response to our rebuttal, and our paper received mixed results (one 6 rating and one 5 rating). We have not received any response from you. Can you read our rebuttal and post your comments? Thanks.

Q4: It would be interesting to see how model size affects the attention patterns and overall performance.

A4: Good question! To explore the effectiveness of larger models, we apply the MIA-DPO with the LLaVa-v1.5-13B model. We observe that the attention distribution patterns of the 13B model are largely consistent with those of the 7B model. We have included the results of MIA-DPO + LLaVa-v1.5-13B on multi-image and single-image benchmarks in Tab. 16 and Tab. 17. These results further validate our conclusions about the effectiveness of MIA-DPO, demonstrating that it can consistently improve the performance of larger size models.

Q5: Additionally, some analysis of failure cases would help understand the method's limitations.

A5: We have provided some erroneous cases in Fig. 11. The cases presented are from the multi-image QA of the MMMU benchmark. From the model's responses, we can observe that although MIA-DPO has improved the model's multi-image understanding and reasoning capabilities, the model may still make errors when encountering questions from out-of-domain knowledge (e.g., fine-grained plant classification, medical image processing), which is not present in our training data. We believe extending the training data to more diverse domains will alleviate these failure cases.

Q6: For the post-selection filtering, how did you choose the specific metrics (perplexity, length ratio, edit distance)? Were other metrics considered?

A6: The use of perplexity is based on the observation that when the model outputs a high perplexity, it indicates a lack of confidence in its responses, making it more prone to generating hallucinations. The length ratio is employed to ensure that there is no significant disparity in length between the chosen and rejected samples, as such differences could negatively affect the optimization of the loss function. Edit distance is used to prevent the chosen and rejected samples from being too similar, as overly similar samples (e.g., "apple" and "apples") could hinder the model’s ability to learn meaningful patterns. While our selected metrics have proven effective, other techniques, such as semantic similarity metrics, could be explored to further refine the post-selection process. These additional metrics can help ensure the diversity of the selected data.

Q7: Your method improves most on the Mantis benchmark (+11.1%). Any insights into why the gains are particularly large there?

A7: Good question! The IXC2.5 model itself is relatively advanced with strong learning capabilities, which may explain the substantial performance gains achieved after applying MIA-DPO. On the other hand, we find that the original IXC2.5 model lacks instruction-following abilities when answering questions in the Mantis benchmark. This often resulted in the model providing lengthy descriptions instead of concise answers for multiple-choice questions as shown in Fig. 12. After applying MIA-DPO, the model prefers to answer the question that meets the format requirement, which contributed to its higher scores on Mantis.

Dear Reviewer mzdh,

We sincerely thank you for your thorough feedback: identifying the key challenges, avoiding costly human annotation or API calls, clear improvements, and thorough ablations. We address your questions below and have incorporated all feedback in the revised version. All new material added to the revised manuscript has been highlighted in red text for better visibility.

Q1-1: My main concern is about the attention-based selection method. While using attention patterns to detect hallucinations is interesting, the threshold selection feels somewhat ad-hoc. The authors set different thresholds based on image counts (0.7/0.6/0.5/0.5 for 2/3/4/5 images), but there's limited justification for these specific values. I wonder if these thresholds could be learned automatically rather than manually set.

A1-1: Thank you for your valuable feedback. Our attention-based selection method is inspired by our observation that attention distributions become more dispersed as the number of images increases. This necessitates the use of different thresholds to effectively identify relevant information. While we acknowledge the potential benefits of automatic threshold learning, our current approach is based on statistical analysis (see Fig. 5): before setting the threshold, we analyze the attention ratio distributions for 1k samples of each of the three data formats. Based on these distributions, we set the threshold to retain 50%-70% of the samples. Furthermore, our experiments demonstrate that our MIA-DPO is relatively robust to different threshold ranges (see A1-2). We agree to the value of exploring automated techniques to optimize threshold selection and will consider this as a promising direction for future research.

Q1-2: Have you analyzed how sensitive the results are to the attention thresholds?

A1-2: Thanks to your suggestion, we have conducted multiple ablation experiments with different threshold ranges, see Tab. 13 and Tab. 14. We observe that our MIA-DPO is relatively robust to different threshold ranges, and our default choices (0.7/0.6/0.5/0.5) performs slightly better than other choices.

Q1-3: What happens if you use uniform thresholds across different image counts?

A1-3: We have added the baseline that uses uniform thresholds, and results are shown in the fourth row of Tab. 13 and Tab. 14. The experimental results demonstrate that using a uniform threshold can negatively impact performance. This observation is due to the attention ratio distributions varying significantly depending on the number of images (see Fig. 5), which indicates that a one-size-fits-all threshold is not optimal. By adjusting the threshold based on the number of images, we can improve the model's ability to generate accurate and coherent responses.

Q2: The relationship between augmentation strategies and model performance isn't fully explored. While the authors try three formats (sequence, grid collage, pic-in-pic), there's limited analysis of how different ratios of these formats affect results. Some discussion of which formats work better for what types of queries would be valuable.

A2: Valuable feedback! First, we conducted ablation experiments for each data type, as presented in Table 3 of the paper. We explored the effects of using each data type individually, and the results show that the performance improvements from using each data type alone are similar but do not surpass the case where all three types of data are combined. Additionally, based on your insightful suggestion, we have included ablation studies on different data proportions in Tab. 13 and Tab. 14. We adjust the proportions of the three data types and evaluate the results. The results indicate that the model's performance on both multi-image and single-image tasks remains stable, with only minor fluctuations. This demonstrates the robustness of our model across different data proportion settings. In conclusion, none of the data types showed a particularly significant advantage over others. When used in combination, the three data types achieve better performance than any single type used individually.

Q3: The ablations, while informative, don't completely isolate the impact of different components.

A3: Thanks for your suggestion. In Tab. 15, we have conducted ablation experiments on different components of post-selection. Specifically, we individually used perplexity (ppl), length ratio, and edit distance as the sole components of post-selection for the experiments. The experiments show that the performance of using each component individually is similar, but the combined use of all three methods yields the best results.

Dear Reviewer mzdh,

We deeply appreciate your efforts in reviewing our paper, and we are particularly grateful for the invaluable suggestions that have greatly enhanced its quality.

In response to your suggestions, we conducted ablation experiments on both different and unified attention thresholds(Q1). Additionally, we further explored augmentation strategies, examining the impact of different data ratios on the model(Q2). We also broke down the different components for more in-depth ablation studies(Q3). Subsequently, we tested the effect of model size on our method(Q4) and presented some failure cases(Q5). Finally, we analyzed the post-selection process and potential alternative approaches(Q6), as well as the significant performance improvements observed on the Mantis(Q7).

Could you please read our rebuttal and consider adjusting the score in light of our response? Thank you for your time and effort!

Best,

Authors

Dear Reviewer mzdh,

Thank you again for your time and valuable feedback on our paper. We have carefully addressed all your comments in our rebuttal and would greatly appreciate it if you could review our response and consider adjusting your score accordingly.

Your support means a lot to us, and we are grateful for your consideration.

Best regards,

Authors

Dear Reviewer mzdh,

I hope this message finds you well. I’m writing to kindly follow up on the review process for our paper. We deeply value your expertise and would greatly appreciate it if you could provide your feedback and scoring at your earliest convenience to help progress the review.

We humbly hope for your positive consideration and support in evaluating our work. Your recognition of the effort we’ve put into this research would mean a great deal to us.

Thank you so much for your time and thoughtful review.

Best regards,

Authors

Dear Reviewer mzdh,

I hope this message finds you well. As we are in the final stages, we would greatly appreciate your valuable feedback and scoring at your earliest convenience.

We sincerely understand how busy you must be, and we are deeply grateful for the time and effort you’ve already dedicated to reviewing our paper. Your support and thoughtful feedback are extremely important to us, and we would be thankful for any comments you can provide.

Thank you once again for your time and consideration.

Best regards,

Authors

Thank the authors for addressing all my concerns, I'll keep my score.

Dear Reviewer ZLYy,

We sincerely thank you for your thorough feedback: an efficient pipeline for constructing and filtering multi-image preference data. We address your questions below and have incorporated all feedback in the revised version. All new material added to the revised manuscript has been highlighted in red text for better visibility.

Q1: The authors mention at the beginning of the paper that existing multi-image SFT methods can degrade performance on single-image tasks while improving multi-image performance. However, they do not compare their approach with these methods in the experimental section.

A1：Thank you for your valuable feedback. To address your concern, we conducted experiments comparing MIA-DPO with existing multi-image SFT methods, including Mantis and MMDU. The results, presented in Tab. 11 and Tab. 12, demonstrate that MIA-DPO effectively improves multi-image performance without compromising performance on single-image benchmarks. In contrast, direct SFT on multi-image data can lead to a slight degradation in single-image performance. This highlights the advantage of MIA-DPO in maintaining a balance between both tasks.

Q2: Although the MIA-DPO method can easily convert data into SFT format, the authors do not compare the performance changes of the model after SFT using the same source data.

A2: Valuable feedback! Based on your insightful suggestion, we have added experiment in Tab. 9 and Tab. 10. We convert the chosen DPO data into the SFT format, and fine-tune the model using merely the NLL loss term in Eq. (5). The results show that, compared to directly SFT with the chosen data (second row), MIA-DPO (third row) achieves more significant performance improvements on multi-image benchmarks, which demonstrate the importance of negative samples in the DPO loss.

Q3: When evaluating the model's single-image performance, only MMMU is used, without assessing performance on other widely recognized benchmarks such as MMStar, MMBench, and OCRBench.

A3: Since the MMMU is a benchmark that includes both single-image and multi-image QA, we have placed it in Tab. 1. We present the testing results for seven single-image benchmarks (e.g., MMBench, MMStar, SQA, POPE, MMB) in Tab. 2. In response to your suggestion, we have added the results of OCRBench to Tab. 2.

Q4: For evaluating the model's multi-image performance, the authors could use the Sequence, Grid Collage, and Pic-in-Pic methods to construct multi-image VQA test sets from existing single-image VQA data, directly assessing improvements in sequence confusion and element interference.

A4: Good suggestion! Your feedback is highly valuable to us. We construct a VQA test set of 500 questions using images and questions from LLaVA-665k that are mutually exclusive with the MIA-DPO training data. This test set includes questions with 2 to 5 images per question, allowing us to directly assess improvements in sequence confusion and element interference. By evaluating the pre- and post-DPO versions of LLaVA and IXC2.5 on this test set, we observed accuracy improvements of 5.8% and 1.9%, respectively (see Tab. 18). These results further validate the effectiveness of MIA-DPO in enhancing multi-image understanding. We plan to release this VQA test set in the final version of our paper to facilitate future research in this area.

Q5: While handling more images is crucial for large vision-language models, the paper does not analyze whether MIA-DPO can still provide stable benefits when the number of simultaneously appearing images increases significantly (e.g., 10+ images).

A5: Thank you for your insightful suggestion. We agree that processing extremely large numbers of images is indeed a valuable direction for future work. We constructed a VQA test set (the construction steps are the same as A4) consisting of 50 questions each for 4, 6, 8, and 10 images. We report the performance of IXC 2.5 + MIA-DPO in Tab. 19. Our MIA-DPO consists of improving the multi-image understanding abilities as the number of images increases. However, the performance of LVLMs, such as IXC2.5, on extremely large numbers of images will also be limited by factors like context window size. As a result, the performance gains from MIA-DPO will gradually diminish with an increasing number of images. These findings show potential for future research on long-context abilities, such as ROPE extrapolation on LVLMs.

Q6: Can experiments be conducted on more advanced open-source large vision-language models or those explicitly supporting multi-images, such as the Qwen-VL series and Tarsier?

A6: The Qwen-VL series and Tarsier are both highly valuable works in the field, and we have added the discussion with them in the related work section. However, given the limited time of the rebuttal period, we plan to include the results of combining these models with MIA-DPO in future work. To validate the performance of MIA-DPO on advanced open-source large vision-language models, we primarily used the IXC2.5 model, which supports multi-image and video processing. Applying MIA-DPO to this model demonstrates the robustness of our method on advanced models.

Dear Reviewer ZLYy,

We deeply appreciate your efforts in reviewing our paper, and we are particularly grateful for the invaluable suggestions that have greatly enhanced its quality.

In response to your suggestions, we first explored the comparison between our method and other multi-image SFT methods (Q1) and conducted ablation experiments by converting MIA-DPO data into SFT data (Q2). Additionally, we provided results on more benchmarks (Q3). Subsequently, we constructed a multi-image VQA test set and performed tests (Q4). Furthermore, we examined the model's performance as the number of images increased (Q5) and discussed its performance in advanced models(Q6).

Could you please read our rebuttal and consider adjusting the score in light of our response? Thank you for your time and effort!

Best,

Authors

Dear Reviewer ZLYy,

Thank you again for your time and valuable feedback on our paper. We have carefully addressed all your comments in our rebuttal and would greatly appreciate it if you could review our response and consider adjusting your score accordingly.

Your support means a lot to us, and we are grateful for your consideration.

Best regards,

Authors

Dear Reviewer ZLYy,

I hope this message finds you well. I’m writing to kindly follow up on the review process for our paper. We deeply value your expertise and would greatly appreciate it if you could provide your feedback and scoring at your earliest convenience to help progress the review.

We humbly hope for your positive consideration and support in evaluating our work. Your recognition of the effort we’ve put into this research would mean a great deal to us.

Thank you so much for your time and thoughtful review.

Best regards,

Authors

Dear Reviewer ZLYy,

I hope this message finds you well. As we are in the final stages, we would greatly appreciate your valuable feedback and scoring at your earliest convenience.

We sincerely understand how busy you must be, and we are deeply grateful for the time and effort you’ve already dedicated to reviewing our paper. Your support and thoughtful feedback are extremely important to us, and we would be thankful for any comments you can provide.

Thank you once again for your time and consideration.

Best regards,

Authors

Dear Reviewer ZLYy.

The deadline is approaching. Now two reviewers post their response to our rebuttal, and our paper received mixed results (one 6 rating and one 5 rating). We have not received any response from you. Can you read our rebuttal and post your comments? Thanks.

Dear Reviewer ZLYy,

I hope this message finds you well. As the deadline is fast approaching, I would like to kindly inquire about the status of your feedback on our rebuttal. We have received responses from three reviewers, with mixed ratings (two 6 ratings and one 5 rating), but we have not yet received any comments from you. If possible, could you kindly review our rebuttal and provide your feedback at your earliest convenience?

Thank you very much for your time and consideration.

Best regards,

Authors

Q2-3: Tables 2 and 17 reveal inevitable performance degradation in single-image scenarios.

A2-3: This is not true. Due to the model being trained with DPO on multi-image data, there are naturally some fluctuations in single-image benchmark performance, with some metrics improving and others declining. However, when considering the overall performance across multiple single-image benchmarks, the model's single-image performance remains at its original level, and shows a significant advantage compared to the results obtained using only NLL loss.(LLaVA 51.6, LLAVA+NLL 49.9, -1.7%, MIA-DPO 51.7, +0.1% in Table 10). Should we consider the average results on the 8 datasets, or just look at the decreasing result on one dataset and ignore the other seven?

Q2-4: Given the manual and complex nature of MIA-DPO training, would a method like COSA [1] be more efficient and scalable, improving performance during pretraining at minimal cost?

A2-4: Our method is completely different from COSA [1], so we believe there is no meaningful basis for comparison between the two. COSA is designed for video description, while our method focuses on achieving model alignment through DPO. COSA is a pre-training strategy, whereas we focus on post-training. COSA primarily proposes a new video understanding model, whereas our method aims to mitigate multi-image hallucinations and achieve model alignment.

[1] COSA: Concatenated Sample Pretrained Vision-Language Foundation Model

Dear Reviewer ZLYy,

We are sorry that we have not received a response from you since we last response to your questions. Do you have any further concerns? We will do our utmost to address them. Given the mixed results with two ratings of 6 and one of 5, we would be grateful if you could clarify any remaining concerns that we may not have fully addressed. We truly look forward to your response. Thank you again for your time and thoughtful feedback.

With sincere appreciation,

The Authors

Dear Reviewer ePWn,

We sincerely thank you for your thorough feedback: the paper is clearly written, the proposed method is well-motivated, easy to reimplement, and the experiments are well organized. We address your questions below and have incorporated all feedback in the revised version. All new material added to the revised manuscript has been highlighted in red text for better visibility.

Q1:My biggest concern is, the proposed synthetic method is designed to deal with the interference from unrelated images, however, which seems only a corner case in LVLMs. In practice, typically all images in the question prompt contribute to the final answer. It is a great challenge (but more important) to generate preference data for such real multi-image prompt at scale without much cost. The paper does not involve the topic. Therefore, I think the title "Multi-Image Augmented Direct Preference Optimization" may somewhat overclaim the contribution.

A1: We agree that our synthetic method indeed only requires considering parts of (rather than all) images to infer the final answer. Despite its limitations, our MIA-DPO approach has demonstrated significant improvements on general single-image and multi-image benchmarks. This suggests that our method can effectively address general multi-image hallucination issues even when not explicitly designed for the most complex scenarios. And we will present reasonable explanations in A2. We acknowledge your suggestion that generating preference data that requires considering all input images is an important direction for future research. However, this is a significant challenge, requiring strong models (e.g., GPT-4o) or human expertise as the annotator, which is both costly and time-consuming. We are committed to exploring more sophisticated techniques (e.g., incorporating domain-specific knowledge to generate more accurate and informative preference data) to handle complex multi-image prompts in future work. Although without involving the complex multi-image prompts, we believe that our work represents a valuable first step and provides a strong foundation for future research in multi-image preference alignment.

Q2: To improve the work, I think the authors may defend their contributions by analyzing the root of the improvements on general benchmarks. For example, an interesting topic may be "does the proposed synthetic DPO dataset also help to suppress more general hallucinations other than those in Line 212?"; if yes, why?

A2: Thank you for your valuable feedback! Here we provide more detailed explanations about why MIA-DPO enhances model performance on general benchmarks and suppresses more general multi-image hallucinations. About Training Paradigm:

• MIA-DPO's training process necessitates the processing of a significantly larger number of image tokens, improving the model's ability to take multiple images as inputs.
• Our careful selection of chosen and rejected samples in the DPO framework further enhances the model's ability to discern subtle differences and nuances between images. About Hallucination Suppression:
• Our synthetic DPO data is meticulously designed to address a diverse range of multi-image hallucination scenarios, including those involving unrelated images, conflicting information, and ambiguous prompts. By exposing the model to these challenging examples, MIA-DPO effectively learns to identify and mitigate such hallucinations. In conclusion, MIA-DPO's unique training paradigm and data generation strategy empower models to handle complex multi-image prompts to some extent, ultimately leading to more robust and accurate models.

Q3: According to Table 3, parts of the improvements are resulted from data cleaning (especially on MMMU and BLINK), which is not the unique technical contribution of the paper.

A3: The post-selection procedure, which involves removing outliers and low-quality data, is an integral part of our method. This step, combined with our multi-image prompt construction and attention-aware selection techniques, leads to a substantial improvement in the overall quality of the DPO dataset. From an average perspective, the primary driver of our method's superior performance on multi-image benchmarks (Table 3) is our multi-image prompt construction and attention-aware selection techniques, rather than data cleaning alone.

Q4: In Eq 5, the NLL term plays a role of supervised fine-tuning with only chosen answers. It is an important baseline (i.e. fine-tuning with isolated NLL term) which can reflect the effectiveness of rejected samples, but not provided in the experiment.

A4: Excellent suggestion! We have incorporated a new baseline (fine-tuning with only the NLL term). Results are presented in Tables 9 and 10 for multi-image and single-image benchmarks, respectively.

• On multi-image benchmarks (Table 9), fine-tuning with the NLL term (second row) yields a performance improvement over the LLaVA-1.5 baseline. However, MIA-DPO (third row) consistently outperforms the NLL-only baseline, demonstrating the significant contribution of negative samples to model improvement.
• On single-image benchmarks (Table 10), the NLL-only baseline (second row) leads to a performance degradation compared to LLaVA-1.5, highlighting the potential risks of incorporating multi-image data during SFT may adversely affect performance on single-image tasks. By contrast, MIA-DPO (third row) maintains performance parity with LLaVA-1.5, thanks to the KL-divergence loss constraint in Eq. (3). This further demonstrates the advantages of MIA-DPO over the NLL-only baseline.

Q5: In Eq 4, please clarify how to calculate R(y) in detail. Is the summation of attention values computed over all layers, attention heads and tokens subject to the given image/region?

A5: Apologies for not clearly explaining the calculation of R(y). First, we used the attention from the middle layer as the experimental subject (since the phenomenon is most evident in the middle layer). For sequence data, we calculate the total attention value of the image pointed to by the instruction as A_target, and the total attention value of all images as A_sum. For grid collage and pic-in-pic data, we calculate the total attention value of the region in the image pointed to by the instruction as A_target, and the total attention value of the entire image as A_sum. The ratio of A_targe

Dear Reviewer ePWn,

We deeply appreciate your efforts in reviewing our paper, and we are particularly grateful for the invaluable suggestions that have greatly enhanced its quality.

In response to your suggestions, we first analyzed the issue of related images in Q1, then discussed the root of the improvements and the data cleaning problem in Q2 and Q3, respectively. Additionally, in Q4, we provided ablation experiments on NLL, and in Q5, we provided a detailed explanation of the calculation method for R(y).

Could you please read our rebuttal and consider adjusting the score in light of our response? Thank you for your time and effort!

Best,

Authors

Dear Reviewer ePWn,

Thank you again for your time and valuable feedback on our paper. We have carefully addressed all your comments in our rebuttal and would greatly appreciate it if you could review our response and consider adjusting your score accordingly.

Your support means a lot to us, and we are grateful for your consideration.

Best regards,

Authors

Dear Reviewer ePWn,

I hope this message finds you well. I’m writing to kindly follow up on the review process for our paper. We deeply value your expertise and would greatly appreciate it if you could provide your feedback and scoring at your earliest convenience to help progress the review.

We humbly hope for your positive consideration and support in evaluating our work. Your recognition of the effort we’ve put into this research would mean a great deal to us.

Thank you so much for your time and thoughtful review.

Best regards,

Authors

Dear Reviewer ePWn,

I hope this message finds you well. As we are in the final stages, we would greatly appreciate your valuable feedback and scoring at your earliest convenience.

We sincerely understand how busy you must be, and we are deeply grateful for the time and effort you’ve already dedicated to reviewing our paper. Your support and thoughtful feedback are extremely important to us, and we would be thankful for any comments you can provide.

Thank you once again for your time and consideration.

Best regards,

Authors

Q1: I think the significance of an improvement varies from different fields and methods. For ImageNet classification and COCO detection, each 1% gain could be valuable. But it is LVLMs -- for example, for 7B model on MMMU, LLaVA-v1.5 is 35.1, InternLM-XC2.5 is 41.4, and recently Qwen2-VL-7B reaches 54.1, which increases 19% within only one year! It is the power of data and engineering. With the rapid increasing of baselines, for a research paper I value technical contributions more than simply benchmark gains (unless the improvement is as much as that of Qwen2-VL). In other words, I do not think the result (itself) in Table 1 and Table 2 is "significant", if the paper does not address a technically impressive problem.

A1: We disagree with the reviewer’s assessment that a 3-4 point improvement is insignificant. While it is true that models like Qwen2-VL-7B have demonstrated remarkable performance gains, this does not weaken the value of our improvements. A 3-4 point improvement on key benchmarks, especially when achieved through a low-cost, simple, and easily realizable pipeline like ours, is not trivial.

We cannot disguise replacement of the concept. C > A does not mean that A + B > A is not significant. The QWen2-VL has a different architecture, pre-training/SFT data, and training schedule from our baseline (LLaVA 1.5, IXC 2.5). Our method is general and we promise to further try to apply our method to recent baselines (e.g., QWen2-VL). In addition, we would like to state that LLaVA (accepted to NeurIPS 2023) and IXC (accepted to NeurIPS 2024) are valuable baselines. The release time of QWen2-VL's technical report and ICLR 2025 submission deadline is less than one month, so we did not report the results of QWen2-VL. Do you think it is reasonable to weaken the contribution of a paper by using a tech report that has not been peer-reviewed and is released in the same period?

In the field of machine learning, incremental improvements are often the result of highly effective techniques that can be deployed more easily and at a lower cost. It is important to note that these gains, while perhaps smaller than those of Qwen2-VL-7B compared to LLaVA, are still meaningful and demonstrate the practical efficacy of our approach, especially considering the constraints and trade-offs involved.

Other reviewers have clearly acknowledged that our proposed method delivers noticeable improvements while maintaining single-image performance (reviewer mzdh). In particular, MIA-DPO demonstrates clear enhancement in multi-image tasks, without sacrificing single-image metrics like MMMU (reviewer ZLYy). Our experiments also validate the effectiveness of the method, showing strong performance against baselines (reviewer DZHj). In any case, we believe our approach makes a valuable contribution by delivering tangible, measurable improvements.

Q2:The authors may argue that their method obtains consistently improvements on two base models, which may generalize to more baselines. Yeah, it may be true, however, does not surprise me very much -- since more multi-image prompts (Sec 3.3.1) with proper data cleaning (Sec 3.3.2) are introduced, multi-image benchmarks are expected to improve (Table 1), and some single-image benchmarks drop due to distribution changes (Table 2). But, the key issue is the synthetic method (which I think is the core technical contribution) is too straight forward, only dealing with the simplest case in multi-image QA. I do not think in the paper gives new academic insights, except for a few engineering practice. It does not mean the paper has no contributions, but may not meet the standard of a top conference like ICLR.

Why I do not think there is a significant improvement

I think the significance of an improvement varies from different fields and methods. For ImageNet classification and COCO detection, each 1% gain could be valuable. But it is LVLMs -- for example, for 7B model on MMMU, LLaVA-v1.5 is 35.1, InternLM-XC2.5 is 41.4, and recently Qwen2-VL-7B reaches 54.1, which increases 19% within only one year! It is the power of data and engineering. With the rapid increasing of baselines, for a research paper I value technical contributions more than simply benchmark gains (unless the improvement is as much as that of Qwen2-VL). In other words, I do not think the result (itself) in Table 1 and Table 2 is "significant", if the paper does not address a technically impressive problem.

The authors may argue that their method obtains consistently improvements on two base models, which may generalize to more baselines. Yeah, it may be true, however, does not surprise me very much -- since more multi-image prompts (Sec 3.3.1) with proper data cleaning (Sec 3.3.2) are introduced, multi-image benchmarks are expected to improve (Table 1), and some single-image benchmarks drop due to distribution changes (Table 2). But, the key issue is the synthetic method (which I think is the core technical contribution) is too straight forward, only dealing with the simplest case in multi-image QA. I do not think in the paper gives new academic insights, except for a few engineering practice. It does not mean the paper has no contributions, but may not meet the standard of a top conference like ICLR.

About my solution

It is an example to explain which problem I think is significant (and timely to the current development of LVLMs), not a reason to reject the paper (sorry for the potential misunderstanding). I mean, if the paper aims to synthetic "true" multi-image data and obtains improvements like those in Table 1, I would definitely vote acceptance. I do not agree it is analogous to the early versions of diffusion models; at least to me, diffusion models have nontrivial technical contributions regardless of some shortcomings.

Concerns on $L_{NLL}$

The paper analyzes multi-image hallucinations. In the proposed DPO framework, the negative samples should take responsibility for multi-image hallucination suppression. However, in Table 9, if $L_{NLL}$ individually has already obtained most gains, I am afraid whether the claim is valid.

A2: Ablation experiments on SFT and data cleaning have already demonstrated that "multi-image prompts (Sec 3.3.1) with proper data cleaning (Sec 3.3.2)" are not the primary reasons for the performance improvements achieved by our method. Additionally, the statement "some single-image benchmarks drop" is also inaccurate. As shown in Table 10, our method even achieves an improvement in average single-image performance on LLaVA. Should we consider the average results on the 8 datasets, or just look at the decreasing result on one dataset and ignore the other seven?

Besides, we think the reviewer mistakes in identifying the academic insights and novelty of a paper. We recommend the reviewers to read [1].

We believe novelty or academic insights should not be limited to designing technical complex algorithms (which looks like the understanding of the reviewer ePWn), because:

• A comprehensive analysis that is not found by previous papers can also be worth publishing. We conducted an in-depth analysis of the types of multi-image hallucinations and their differences from single-image hallucinations, and explored the underlying mechanisms of multi-image hallucination generation through attention.

• A simple and effective approach can also be worth publishing. Our MIA-DPO is easy to implement, low-cost, and efficient, and can be scaled to different models and larger datasets.

[1] Novelty in Science: A guide for reviewers. Michael J. Black.

Q3: I mean, if the paper aims to synthetic "true" multi-image data and obtains improvements like those in Table 1, I would definitely vote acceptance. I do not agree it is analogous to the early versions of diffusion models; at least to me, diffusion models have nontrivial technical contributions regardless of some shortcomings.

A3:

We would like to reiterate that our method is not limited by the lack of "true" multi-image data; rather, it achieves significant performance improvements through an efficient and low-cost pipeline. It is a novel approach to multi-image alignment that achieves significant improvements without incurring the high computational costs associated with large language models (LVLMs). The essence of our method lies in its low-cost, low-computation design, which provides an effective solution for multi-image tasks without the need for expensive inference or data construction.

While we acknowledge that generating true multi-image data could be an interesting direction, we believe it is important to recognize the practicality and impact of our current approach. Synthesizing multi-image data using large models such as 7B or 70/72B LVLMs, as you suggested, may indeed be useful, but it comes with significant inference costs and potential hallucinations from the models, which could diminish the reliability and accuracy of the generated data. Moreover, such approaches would likely increase the complexity and cost of the system, which runs counter to the advantages we emphasize in our work—simplicity and cost-effectiveness.

We understand your point, but we would like to clarify that our analogy to the early versions of diffusion models was not intended to make an in-depth comparison between the two methods. Rather, it was simply to illustrate a broader principle: many influential works are not perfect in the beginning, but that does not diminish their significance or the value they provide as starting points for future improvements.

We believe the core message here is that innovations often start with simple, effective, and easy-to-follow solutions, and that does not make them any less important. By dismissing our approach simply because it may be optimized further in the future, we risk overlooking the current value it already provides. Can we weaken the contribution of one paper simply because it may be optimized further in the future (but not verified)?

Q4: Concern on NLL... In the proposed DPO framework, the negative samples should take responsibility for multi-image hallucination suppression. However, in Table 9, if $L_{NLL}$ individually has already obtained most gains, I am afraid whether the claim is valid.

A4: Adding $L_{NLL}$ to $L_{DPO}$ is a standard step in DPO-based alignment optimization. Just like we must add Layer Normalization after the multi-head self-attention layer of the Transformer, we also must add the $L_{NLL}$ to $L_{DPO}$. We just follow previous works to add the $L_{NLL}$ to improve the stability of DPO training. For example, the authors of LLAMA-3 also add the $L_{NLL}$ to their DPO loss, see page 16 of the LLAMA-3 tech report [1].

We assume you are junior in this area and lack the essential background in understanding the DPO algorithm, so you are not aware that this is a standard step to DPO, thus ask the same question over and over again. But it doesn't matter, we are glad to explain why adding the $L_{NLL}$ is a standard step for you.

• Why do we add the $L_{NLL}$ term?

As we mentioned in our first version (line 354), the $L_{NLL}$ term is used for improving the stability of DPO training. In Eq. (3), the DPO loss consists of $x_{1} = \frac{\pi_{\theta}(y_{w}|x)}{\pi_{\text{ref}}(y_{w}|x)}$ and $x_{2} = \frac{\pi_{\theta}(y_{l}|x)}{\pi_{\text{ref}}(y_{l}|x)}$. Ideally, we want $x_{1}$ to have a large value for chosen data, and $x_{2}$ to have a small value for rejected data. However, due to the theoretical limitations of DPO [2], the gradient $\frac{\partial{L_{DPO}}}{x1}$ is smaller than $\frac{\partial{L_{DPO}}}{x2}$, which means the DPO optimization towards decreasing the value of $x_{2}$ is easier than increasing the value of $x_{1}$. This is terrible because it means that the model is not learning from the positive examples, but is just overly penalizing the negative examples. So, the solution is to add the $L_{NLL}$ term to increase the likelihood of the chosen data. We want to emphasize again that adding $L_{NLL}$ is standard practice for improving the stability of DPO training. Just like the gradient will explode if Layer Normalization is not added to the Transformer, current mainstream DPO algorithms all mention that they have added $L_{NLL}$, such as LLAMA 3 [1].

[1] The Llama 3 Herd of Models, Meta AI.

[2] Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

• Can we only use the $L_{NLL}$ without the $L_{DPO}$?

Which do you think is more important: improving on multi-image datasets while preserving the capabilities of single-image benchmarks, or improving only on multi-image datasets and significantly decreasing on a single image? As clearly shown in Table 10, using only NLL loss results in a significant decrease in the model's single-image performance (LLaVA 51.6 → LLAVA+NLL 49.9, a -1.7% drop). This directly refutes your argument that NLL loss alone accounts for the majority of performance improvements.

In contrast, our DPO method, compared to SFT, leverages negative data and the KL divergence term in the policy model (Eq. 3), which not only leads to greater improvements in multi-image understanding (LLaVA +3.0%, IXC2.5 +4.3%) but also preserves the original single-image reasoning ability (LLaVa 51.6, MIA-DPO 51.7, a +0.1% increase). These are significant gains that the NLL-only approach simply can’t achieve.

While NLL loss may improve performance on multi-image benchmarks to some degree, the magnitude of the improvement and the preservation of single-image ability are both far superior in our approach. The evidence is in the numbers, and the results speak for themselves. We encourage you to carefully review the data presented in our previous response, as it clearly highlights the limitations of NLL loss and the advantages of our proposed method.

We are glad to answer your further questions to address your concerns! And again, we want to ask you the following questions:

• Do you think it is reasonable to weaken the contribution of a paper by using a tech report that has not been peer-reviewed and is released in the same period?

• Should we consider the average results on the 8 datasets, or just look at the decreasing result on one dataset and ignore the other seven?

• Can we weaken the contribution of one paper simply because it may be optimized further in the future (but not verified)?

• Which do you think is more important: improving on multi-image datasets while preserving the capabilities of single-image benchmarks, or improving only on multi-image datasets and significantly decreasing on a single image?

• Many existing works in the NLP/LLM area have demonstrated that applying RLHF (e.g., using the DPO loss) is better than using the SFT (NLL loss) only, and our observations on multi-image understanding also verify this point. Why do you still think merely using NLL loss is enough?