Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization

Dear Reviewer 8fTK,

Thank you for your feedback. We provide our responses to your questions as following.

Question 1: Lack of Clarity in Problem Formulation: The paper raises two theoretical limitations of existing vision-oriented preference optimization methods, but the explanation is abstract and lacks intuitive grounding. The impact of issues like mismatched partition functions or fixed-response supervision is unclear without concrete examples. Clarifying these points in plain terms with illustrative scenarios would make the problem statement more accessible and compelling.

Response:

We sincerely thank you for your critical feedback. We acknowledge the importance of providing an intuitive explanation of these theoretical issues discussed. While these theoretical aspects are challenging to quantify, we will strive to elucidate these limitations with clearer and more comprehensible language.

1. Mismatched Partition Function

In the context of multimodal scenarios, the partition function $Z(m,x)$ in DPO is defined as $\sum_y\pi_{ref}(y\vert m,x)\exp(\frac{1}{\beta}r(m,x,y))$. This term integrates over all possible responses y given the input image $m$ and text $x$, and captures the overall quality of the response space conditioned on $(m,x)$. However, since $Z(m,x)$ requires computation over all possible responses, it is intractable. DPO resolves this by constructing preference data pairs, e.g., $(m,x,y_w)$ and $(m,x,y_l)$, and subtracting the implicit rewards of these pairs. This subtraction between the implicit reward of the chosen response $y_w$ and the implicit reward of the rejected response $y_l$ cancels out the shared partition function $Z(m,x)$.

In vision-oriented preference optimization approaches, the preference data pairs are constructed with different visual inputs $m_w$ and $m_l$, while keeping the response $y_w$ identical. The goal is to encourage the model to better understand and prioritize visual content. However, the partition functions $Z(m_w,x)$ and $Z(m_l,x)$ differ due to the distinct visual inputs, i.e., $m_w$ and $m_l$. Consequently, these terms cannot be directly canceled out as in standard DPO. Prior studies have overlooked this issue, failing to account for the influence of the partition function and leading to inconsistencies between the theoretical formulation and practical implementation. This mismatch results in suboptimal optimization outcomes.

In contrast, our proposed Symmetric Multimodal Preference Optimization (SymMPO) addresses this issue by pairing responses corresponding to different images under the same textual input as each other’s rejected responses. By leveraging identical multimodal inputs across paired data, SymMPO naturally cancels out the partition function terms, ensuring consistency between theory and implementation.

2. Indirect Preference Supervision

Standard multimodal DPO optimizes preference by leveraging triplets $(m,x,y_w,y_l)$, where the model is guided to generate $y_w$ over $y_l$ given the image $m$ and text $x$. In vision-oriented preference optimization methods, preference pairs are constructed with different visual inputs $m_w$ and $m_l$, but identical responses $y_w$. The intention is to improve the model’s focus on visual content through comparative learning. However, this approach deviates from the original design of DPO, which relies on response-based supervision signals to directly enhance model performance. As a result, these methods struggle to effectively improve the model’s visual understanding capabilities.

In comparison, SymMPO adopts the same direct supervision strategy as original DPO. By treating responses associated with different images under the same textual input as symmetric preference pairs, SymMPO achieves direct preference supervision for vision-oriented optimization. This ensures alignment with the theoretical foundation of DPO while effectively enhancing the model’s preference optimization across varying visual inputs.

Question 2: Missing Evaluation on Standard Benchmarks: The method is not tested on widely used LVLM benchmarks such as MMBench, MME, POPE, SEED, or MM-Vet. Without results on these datasets, it is difficult to assess the method’s general effectiveness or practical relevance across diverse tasks.

Response:

Thank you for this suggestion regarding evaluation on standard benchmarks.

Our primary research goal with SymMPO is to specifically address and mitigate multimodal hallucination. Consequently, we focused on leading hallucination-centric benchmarks，including HallusionBench, Object-HalBench, MMHal-Bench and AMBER, which have been widely adopted in prior work [1-6] aimed at mitigating MLLM hallucination through DPO. Additionally, we evaluated our method on MMStar, which assesses the model’s ability to combine visual information with textual reasoning to answer questions correctly. This aligns closely with our goal of improving joint vision-language understanding while reducing hallucination.

The benchmarks you suggested can be categorized as follows:

MMBench, SEED, and MM-Vet serve as excellent and widely recognized benchmarks for assessing general multimodal understanding in MLLMs rather than hallucination issues. To the best of our knowledge, these benchmarks have not been utilized in prior studies related to hallucination mitigation.

MME and POPE, in contrast, are highly relevant benchmarks for evaluating object-level hallucination. However, since these benchmarks are less frequently adopted by prior studies in this domain, we prioritized using five widely adopted benchmarks in our current experiments. Below, we provide a comparative table summarizing benchmark usage in existing work versus our work.

As shown, most of existing work only employed three hallucination-centric benchmarks, whereas ours incorporates four. We believe this more comprehensive evaluation setup better validates our model's generalization capability. We fully recognize the relevance of these benchmarks, especially MME and POPE, and plan to include them in future evaluations to further demonstrate SymMPO's generalizability and effectiveness across a broader range of tasks and datasets.

Question 3: Limited Model Generalization Evidence: All experiments are conducted on LLaVA, with no evaluation on other strong models like DeepSeek or MiniGPT-4. This narrow scope raises concerns about overfitting and limits claims of generalizability. Broader model-level validation is needed to support the method’s robustness.

Response:

Thanks for your question. We selected the LLaVA family for evaluation based on the following considerations:

Firstly, many related studies on multimodal DPO [1-6] have consistently utilized the LLaVA family as their primary evaluation baseline. By centering our experiments on LLaVA, we ensure direct comparability with these established body of works. Notably, two recent CVPR 2025 and ICLR 2025 publications [4, 6] employed the same evaluation approach, focusing on the 7B and 13B LLaVA variants to assess model generalization capabilities.

Secondly, the LLaVA family represents a foundational and widely-used class of MLLMs, characterized by a classic MLLM architecture (a vision encoder connected to a language model via a projection layer). Experimental results obtained from LLaVA are therefore more likely to generalize to other MLLM models.

We fully acknowledge the importance of demonstrating SymMPO’s effectiveness on structurally diverse MLLM families to strengthen claims of generalizability, and plan to explore them in future work.

References

[1] Fei Wang, et, al.; “mDPO: Conditional Preference Optimization for Multimodal Large Language Models”. EMNLP 2024.

[2] He, Lehan, et al. "A topic-level self-correctional approach to mitigate hallucinations in mllms." arXiv preprint arXiv:2411.17265 (2024).

[3] Yu, Tianyu, et al. "Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness." CVPR 2025.

[4] Yang, Zhihe, et al. "Mitigating hallucinations in large vision-language models via dpo: On-policy data hold the key." CVPR 2025.

[5] Fu, Jinlan, et al. "Chip: Cross-modal hierarchical direct preference optimization for multimodal llms." ICLR 2025.

[6] Sarkar, Pritam, et al. "Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment." ICLR 2025.

Dear Reviewer rbzU,

Thank you for recognizing the value of our paper, which is really encouraging to us. We also greatly appreciate your thoughtful comments on the minor issues, as they provide valuable suggestions for improvement. We will carefully revise the manuscript in accordance with your recommendations.

We address the specific questions raised in your review below.

Weakness: the coefficient-related hyperparameters seem a bit unusual, such as $\gamma$=1e-4. I am curious about how these values were selected. Perhaps some ablation studies on this aspect would be helpful. If time is too limited to provide them now, I hope they can be included in the final version of the paper.

Response:

Thank you for your question.

For hyper-parameters $\beta$, $\eta$ and $\delta$, we use the same setup following prior works:

• We set $\beta$=0.1 directly.
• Consistent with mDPO and OPA-DPO, we use $\eta$=1.0 and $\delta$=0.

For the hyper-parameters, including learning rate, $\lambda$ and $\gamma$, we select their values through grid search. Specifically, we searched within the following ranges:

• Learning rate: [5e-5, 5e-6, 5e-7],
• $\lambda$: [0.1, 0.3, 0.5, 0.7, 0.9],
• $\gamma$: [1e-2, 1e-3, 1e-4, 1e-5].

After performing the grid search, the best results were obtained with learning rate=5e-6, $\lambda$=0.5, and $\gamma$=1e-4.

From our experiments, we observe that too small $\gamma$ values lead to weak preference margin consistency regularization and limit the effectiveness of SymMPO, and too large $\gamma$ values impose excessively strong regularization which interferes with preference learning.

Unfortunately, due to character limitations, we are unable to provide detailed comparative results of our grid search. However, we fully acknowledge the value of sensitivity analysis on these hyper-parameters and promise to include them in future updates to the paper.

Question 1: Could the authors provide more details about the "contrastive image construction"? Specifically, what does "pair each image with its nearest neighbor based on cosine similarity" mean? Does this imply that is sourced from other images within the dataset? If so, what would happen if the dataset is small and no sufficiently similar image can be found?

Response:

Thanks for your question. As you consider, the paired image is indeed sourced from the same dataset. The "contrastive image construction" operates as follows:

1. Feature Extraction: For every image in our source dataset, we first extract its visual feature vector using the CLIP model.
2. Similarity Search: We then compute the cosine similarity between this feature vector of the target image and the feature vectors of all other images in the dataset.
3. Pairing: The image with the highest cosine similarity score is selected as its "nearest neighbor" of the target image. This pair of images is then used for the contrastive training process of SymMPO.

We fully acknowledge the point you raised: if the dataset is small or lacks sufficient visual diversity, it may be challenging to find a meaningful "nearest neighbor" that satisfies the requirements of our contrastive strategy.

In fact, to justify the robustness of our framework, we have evaluated our model with multiple contrastive image construction methods: (1) Black; (2) Cropped; (3) Noisy; and (4) Synthetic (see Section 4.5). The first three represent naive approaches, while the Synthetic method is our contribution (detailed in Section 4.5).

Figure 4 reveals that despite its sensitivity to the data distribution, our "Similar" method (nearest neighbor pairing) outperformed all alternatives approaches, none of which depend on data distribution. This success likely stems from two factors:

1. The paired images are derived from real data and share semantically meaningful features with the target image.
2. Even when the paired image is suboptimal due to the absence of sufficiently similar images, it can still function as a random contrastive sample, providing more or less contrastive signals for vision enhancement.

In the future, we plan to refine this contrastive image construction method. For instance, we can apply a similarity threshold for the "Similar" method and fall back to the "Synthetic" method (the second-best performer) in cases where no adequately similar pair exists. This hybrid approach offers a robust solution for datasets of varying sizes and diversity levels.

Question 2: It appears that the rejected responses are generated by the model being trained, while the preferred responses are generated by DeepSeek-v3 (based on captions generated by Qwen2.5-VL-32B). If this is correct, the performance upper bound of the proposed method should theoretically align with Qwen2.5-VL-32B, right? In that case, it would be helpful if the authors could include the performance of Qwen2.5-VL-32B in Table 1 for comparison.

Response:

Thank you for this insightful question regarding our experimental design and analysis. You are correct in observing that the preferred responses in our preference data are generated through a pipeline where captions are sourced from Qwen2.5-VL-32B and rewritten by DeepSeek-V3.

Regarding the upper bound, the assertion that the performance of Qwen2.5-VL-32B represents an upper bound of our trained model may not be entirely accurate, as the rewriting process is performed by DeepSeek-V3 rather than Qwen2.5-VL-32B. As discussed in Section 4.3, the image captions generated by Qwen2.5-VL-32B may lack fine-grained detail or contain hallucination, which could lead to incorrect consistent/inconsistency claim judgments by DeepSeek-V3 and consequently affect the final response quality. For instance, when the base model’s initial responses include erroneous claims not explicitly mentioned in the captions, DeepSeek-V3 may fail to correct them and propagate these mistakes during rewriting due to its lack of direct image context access. In contrast, Qwen2.5-VL-32B, with its powerful vision understanding capability, could potentially recognize and correct such errors if used for rewriting. Consequently, the synthetic preferred responses do not directly reflect the end-to-end capabilities of Qwen2.5-VL-32B. These additional processing steps and adjustments mean that the performance upper bound of our proposed method with SymMPO is likely lower than the raw outputs of Qwen2.5-VL-32B. Here, we clarify that we chose DeepSeek-V3 over Qwen2.5-VL-32B for response rewriting because initial experiments showed that Qwen2.5-VL-32B's rewritten responses exhibited weaker alignment with the base model's outputs in both format and style. In contrast, DeepSeek-V3 better maintains the required stylistic consistency, which led to its selection as our rewriting model.

Regarding comparisons, directly comparing Qwen2.5-VL-32B with our method would be less meaningful, as Qwen2.5-VL-32B would undoubtedly outperform other baselines by a significant margin. This is due to its more advanced architecture, larger parameter scale, and training on more extensive, diverse, and proprietary datasets. Thus, our work focuses on improving multimodal preference optimization within the LLaVA-1.5 family, consistent with existing studies.

We understand the potential value of including Qwen2.5-VL-32B's performance as a reference point. While such a comparison may fall outside the immediate scope of our evaluation goals, we recognize the value it could add for broader contextual understanding. We will consider exploring and including such results in future iterations of this work.

Question 3: Building on the previous question, if the preferred responses are generated by DeepSeek-v3, they are effectively off-policy and, according to the analysis in OPA-DPO, may be difficult to learn. What would happen if the authors first conducted SFT on these preferred responses? Would this improve the results? (I fully understand that time during the rebuttal period is limited, and I will not fault the authors if experimental results cannot be provided at this stage.)

Response:

This is an exceptionally insightful question, and we sincerely thank you for raising this critical point. We also deeply appreciate your understanding of the time constraints during the rebuttal phase.

You are absolutely correct that the analysis in OPA-DPO highlights a key challenge for DPO when working with off-policy data, particularly when there is a significant stylistic or distributional gap between the model's own outputs and the preferred responses. This gap can result in a large reverse KL-divergence penalty, which may hinder the effectiveness of preference learning.

To address this issue, we designed our rewriting process (outlined in Section 4.1) with specific practice to mitigate this challenge. We explicitly instructed LLM rewriter to preserve the original format and style of the base model's outputs as much as possible. This design aims to minimize the distributional gap and ensure that the preference data remains as "close-to-on-policy" as feasible.

While this approach reduces the off-policy challenge to some extent, we acknowledge that it is an indirect solution and may not completely eliminate the issue. We strongly agree with your suggestion that first conducting supervised fine-tuning (SFT) on the preferred responses before applying DPO represents a more direct and principled solution. Fine-tuning the policy model on these preferred responses would effectively shift its output distribution closer to the target distribution defined by the preference data.

Unfortunately, due to the time constraints of the rebuttal period, we were unable to conduct experiments to validate this approach. Nonetheless, we fully recognize the importance of your suggestion and are excited to explore this in future research.

Dear Reviewer rbzU,

Thank you for your encouraging feedback! We are delighted that our responses addressed most of your questions and concerns, and we greatly appreciate your thoughtful suggestions.

Due to the character limitations of the rebuttal, we were unable to include the results of the sensitivity analysis of hyper-parameters. Here, we provide the supplementary results to offer more insight into the choices behind our hyper-parameter settings.

We searched learning rate in the range [5e-5, 5e-6, 5e-7]:

We searched $\lambda$ in the range [0.1, 0.3, 0.5, 0.7, 0.9]:

We searched $\gamma$ in the range [1e-2, 1e-3, 1e-4, 1e-5]:

Based on the results above, we selected the following as the overall best hyper-parameter: learning rate =5e-6, $\lambda$=0.5, $\gamma$=1e-4. As noted in the rebuttal, we will incorporate this sensitivity analysis and corresponding discussions into the future revisions of our paper to enhance its clarity and completeness.

Dear Reviewer PFAD,

Thank you for recognizing our work! We will address your questions as follows:

Concern 1: SymMPO introduces additional computational overhead compared to standard DPO, potentially increasing the computational cost by several times.

Question 1: How does the SymMPO approach compare with other methods in terms of training time and scalability, particularly when working with larger datasets or more complex multimodal inputs?

Response:

Thank you for raising this concern regarding the computational overhead of SymMPO. We fully understand the importance of evaluating training time and scalability, particularly when working with larger datasets or more complex multimodal input.

To address this directly, we conducted a direct comparison of training time of DPO, mDPO, and SymMPO using DeepSpeed Zero-3 under the same experiment settings. For instance, we trained on a dataset consisting of 21.4k preference pairs over 2 epochs, with a uniform batch size of 64. Due to computational resource limitations, these experiments were conducted on 2 NVIDIA A100-40G GPUs with CPU offload, which is slower than the standard experimental conditions outlined in the paper. The training time comparisons are as follows:

• DPO: 9.6 hours (52 seconds per iteration)
• mDPO: 10.4 hours (56 seconds per iteration)
• SymMPO: 13.0 hours (70 seconds per iteration)

The increased training time of SymMPO primarily stems from the additional computational overhead required for symmetric sampling and optimization. These operations were specifically designed to address limitations of existing methods, i.e., (1) Non-rigorous Objective Function and (2) Indirect Preference Supervision. As demonstrated in Table 1 of our paper, experimental results confirm that our approach achieves superior preference alignment. We remain committed to further exploring optimization techniques to enhance SymMPO's scalability while maintaining its theoretical rigor and effectiveness.

Thank you for raising this excellent concern. We will address it by including a detailed discussion and analysis of this issue in the revised manuscript.

Concern 2: The two limitations mentioned in the Introduction essentially refer to the same issue: the inherent conflict between the existing vision-oriented preference learning and the standard DPO framework. Could the authors further explore SymMPO, particularly in terms of whether aspects beyond pairwise rewards might help address existing limitations?

Response:

Thank you for raising this insightful concern. While the two limitations mentioned in the Introduction both stem from challenges in aligning vision-oriented preference learning with the standard DPO framework, they address distinct aspects of this misalignment:

1. Non-rigorous Objective Function: This limitation highlights the theoretical inconsistency due to mismatched partition functions when input pairs differ in visual content. The inability to cancel out the partition function results in optimization that deviates from the theoretical foundation of DPO.
2. Indirect Preference Supervision: This limitation focuses on the supervision signal used in vision-oriented preference learning, where responses are indirectly supervised through visual input comparisons rather than directly optimized via response preferences. This indirect supervision reduces the effectiveness of preference optimization in enhancing multimodal understanding.

SymMPO is specifically designed to address both issues. Through symmetric pairwise preference learning, it cancels out the partition function mismatch by pairing different visual inputs under the same textual context, ensuring theoretical consistency. Simultaneously, it restores direct preference supervision by leveraging response pairs as rejected responses, aligning with the original design intent of DPO.

Regarding exploring aspects beyond pairwise rewards, SymMPO introduces the preference margin consistency regularization to quantitatively regulate the preference gap between symmetric pairs. This extends optimization beyond simple pairwise comparisons by imposing constraints on the magnitude of preference differences, encouraging more stable and nuanced learning.

Question 2: I looked at the code provided in the anonymous repository, but it seems that it cannot be executed. Could the authors upload a runnable version of the code for reference?

Response:

Thank you for bringing this to our attention. We sincerely apologize for the inconvenience you experienced while attempting to execute our code.

To address this, we have thoroughly reviewed and updated the anonymous repository to ensure its usability. Specifically:

1. The README.md file has been revised to include clear, step-by-step instructions for setting up the environment, installing dependencies, and running the code.
2. We have added a small demo dataset alongside a corresponding script, allowing users to quickly validate the core functionality of our pipeline.

Unfortunately, due to storage limitations of Github, we were unable to include the full training dataset. However, we are committed to ensuring full reproducibility. Upon acceptance of the paper, we will publicly release the complete dataset, all model weights, and any remaining resources needed to replicate our results.

We hope this update resolves the issue, and we appreciate your understanding and feedback on this matter. Please do not hesitate to reach out if further issues arise.

Dear Reviewer PFAD,

Thank you for the time and effort you have devoted to reviewing our paper!

With the discussion period now extended, we would welcome the opportunity to clarify any remaining questions or concerns you might have. Please let us know if there is any aspect of the work you would like us to elaborate on.

We look forward to your reply and greatly appreciate your feedback.

Dear Reviewer PFAD,

There are only 24 hours left in the discussion period. We have done our best to address your questions in the rebuttal, but have not yet received your response. If you have any further questions or concerns, we would be happy to provide clarification.

We look forward to your reply and greatly appreciate your feedback.

Dear Reviewer okF2,

Thank you for recognizing our work. We greatly appreciate your insightful comments and suggestions. Below, we provide detailed answers to your questions:

Question 1: Whether other models can be used to enhance pair screening and acquisition (Besides CLIP).

Response:

Thank you for this insightful question. Yes, since the primary objective of our contrastive image construction is to pair images that are visually similar but have subtle differences, any model capable of capturing fine-grained visual features and discriminating between nuanced differences is a potential candidate for this task.

For example, self-supervised vision models such as DINO [1] and DINOv2 [2] could serve as highly effective alternatives to CLIP. While CLIP is designed to align visual and textual representations, models like DINOv2 are optimized particularly for extracting high-quality, fine-grained visual features through self-supervised learning. This specialization makes DINOv2 especially well-suited for identifying images with minor visual differences.

Additionally, advanced image segmentation models, such as SAM (Segment Anything Model) [3] can segment objects or regions within an image, enabling region-level feature extraction. These features could then be used to pair images with specific region-level similarities.

We acknowledge that while CLIP served as a robust and practical choice for our current implementation, investigating alternative feature extractors is an excellent suggestion for enhancing SymMPO. We will consider this direction in our future work.

Question 2: Which specific stage or component of the "Caption-Anchored Claim Extraction-and-Rewriting" pipeline (Section 4.1) is primarily responsible for Fine-Grained task performance deficit? Is the limitation rooted in the initial caption generation, which may lack fine-grained detail, or does it stem from the subsequent claim extraction and rewriting process?

Response:

Thank you for your thoughtful question. You are correct that the performance deficit on fine-grained tasks primarily stems from the initial caption generation stage, which serves as the foundational step in our "Caption-Anchored Claim Extraction-and-Rewriting" pipeline.

Based on our extensive experiments, we found the caption generation process, powered by the Qwen2.5-VL-32B model, exhibits two key limitations that hinder fine-grained task performance:

1. Lack of Fine-Grained Detail: The prompt used in current pipeline is primarily designed to generate global scene descriptions, rather focusing on producing fine-grained details. As a result, the captions often miss critical nuances or specific information. This lack of detailed captions impacts the subsequent claim extraction process. As a result, the generated claims tend to be overly generic, lacking the specificity required for fine-grained tasks [4]. For example, if the model only describes the overall scene without recognizing detailed features such as the shape of a bird’s feet, wings or beak, it would struggle to accurately classify its species.
2. Propagation of Hallucinations: Hallucination remains a significant challenge in current MLLMs. The Qwen2.5-VL-32B model sometimes generates captions containing hallucinations. In these instances, the subsequent claim extraction process may inadvertently incorporate these inaccuracies into claims, leading to distorted or erroneous preference data.

In summary, these limitations in the initial caption generation stage can trigger a cascading effect throughout the preference data construction pipeline, ultimately compromising the quality of the generated preference data, particularly for tasks requiring fine-grained recognition. In the future, we plan to explore solutions to address this challenge.

References:

[1] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." ICCV 2021.

[2] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." TMLR 2024.

[3] Kirillov, Alexander, et al. "Segment anything." ICCV 2023.

[4] Yucheng Shi, et, al.; “Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data”. ICLR 2025.