Automated Multi-level Preference for MLLMs

Q1. The contribution of the paper heavily relies on the preference fine-tuning algorithm, showing limited innovation beyond this aspect.

A1. In this paper, we introduce the Automated Multi-level Preference (AMP) framework, which involves generating high-quality multi-level preference datasets and the effective learning objective for MDPO. These novel designs ensure AMP achieve promising performance on several hallucination benchmarks. Moreover, we conduct the first hallucination benchmark in multi-round dialogues and devise the relevant metrics, which may stimulate future research.

---

Q2. The method does not demonstrate significant improvements on the LLaVA-Bench benchmark.

A2. Despite a slight performance degradation on the LLava-Bench, our AMP shows significant improvements compared to both general MLLMs and RLHF methods on other benchmarks, $e.g.$, 3.09 (FGAIF) -> 3.23 on MMHal-Bench, 3.71/3.70 (SILKIE) -> 4.21/4.21 on MRHal-Bench, and 85.8 (POVID) -> 87.2 on POPE.

---

Q3. The method's performance on the adversarial tasks of the POPE benchmark is moderate, suggesting a need to reconsider the impact of MDPO on model robustness and how to balance performance and robustness.

A3. The principle of RLHF is explicitly enlarging the probability of superior responses while decreasing the probability of inferior responses. Compared to the conventional autoregressive function, loss in RLHF ensures MLLMs to fit the characteristics of the dataset at the sacrifice of generation ability/robustness. In the future, we will consider integrating the advantages of autoregressive function to maintain the robustness of MLLMs.

Q1. It is recommended to provide more quantitative information on the preference dataset generated by the automated dataset generation pipeline. For instance, the authors could use a subset of the dataset to demonstrate the similarity results compared to human annotators.

A1. Thanks for raising this good discussion. We provide more intrinsic evaluation for the automated multi-level preference dataset and the auto-check mechanism.

Evaluation of the automated multi-level preference dataset. During rebuttal, we estimated the inconsistency rate of our AMP dataset to be 2.25% (through manual evaluation on 2000 random samples). The 2.25% inconsistency rate is significantly lower than the human (14.40% inconsistency) and GPT-4V (11.95% inconsistency) annotations. Below is an example of an inconsistent case in our AMP dataset, where $N_i$ denotes the $i$-th noun chunks:

Standard Response: A little girl ($N_1$) with a purple jacket ($N_2$) is flying a kite ($N_3$).

Response A: A little girl ($N_1$) dressed in purple jacket ($N_2$) is flying a kite ($N_3$) on the lawn, surrounded by many people (other information, including some hallucinations denoted by italic).

Response B: A young girl ($N_1$) dressed in purple clothes ($N_4$, similar to $N_2$ but not exactly the same) is flying a kite ($N_3$).

Response A accurately predicts these noun chunks and thus gets a higher score, leading to A>B. However, the actual ranking is B>A because A has some hallucinations.

Moreover, we conduct another experiment to validate the superiority of our AMP dataset. We mix the AMP and human/GPT-4V data for training the model. Table 1 in the PDF document shows that as the proportion of human/GPT-4V annotated data increases, the hallucination rate rises accordingly.

---

Q2. In this paper, the authors conduct experiments on three hallucination benchmarks and only one general benchmark. To verify the more general applicability of the method, additional experiments are needed on general benchmarks such as TextVQA, GQA, and IconQA.

A2. Compared to the baseline, metrics of the MLLM fine-tuned with our MDPO are improved by 3.3 (58.2 -> 61.5) and 1.8 (62.0 -> 63.8) on two general benchmarks, TextVQA and GQA, respectively.

---

Q3. In Table 1, the authors compare several MLLMs and RLHF-based MLLMs across MMHal-Bench, MRHal-Bench and LLaVA-Bench. However, the baseline model should be more up-to-date. Could you compare it with more current models such as LLaVA-v1.6, DeepSeek-VL, or MiniCPM-V?

A3. We present the performance LLaVA-v1.6, DeepSeek-VL, and MiniCPM-V in Table 3 (PDF document). As shown in Table 3, our method significantly exceeds general MLLMs on hallucination benchmarks, $e.g.$, +0.13 (LLaVA-V1.6) on MMHal-Bench, +0.27/+0.28 (LLaVA-V1.6) on MRHal-Bench, verifying the effectiveness of preference learning and our AMP pipeline.

---

Q4. Assume we have 3 preference samples: A, B, C. Using the MDPO algorithm, we need to calculate the loss for AB, AC, and BC and then update the parameters. However, why do we need to calculate the loss for BC? Sample B may contain hallucinations; does this affect the model's learning of correct preferences?

A4. First, loss for BC provides more comparisons among hallucination examples, resulting in a better performance. Second, to alleviate the adverse effects caused by hallucinations in B, we only introduce the penalty term for AB and AC and exclude the penalty term for BC.

We also conduct another experiment for loss BC. As reported in Table 4 (PDF document), the loss for BC brings extra performance improvement.

---

Q5. This paper does not enhance the visual capabilities of the model. However, in the case study, several OCR tasks and the AMP-MEG model can successfully recognize. Can the authors explain why MDPO algorithm can improve this aspect of ability?

A5. Improvement on OCR is because our AMP dataset actually contains some preference pairs that are centric to OCR task. Below is an example:

Prompt: What does the sign read?

Correct Response: NO TIPPING THIS AREA IS MONITORED BY 24 HOUR RECORDED CCTV.

Response A: NO TIPPING AREA IS MONITORED BY 24-HOUR RECORDED CCTV.

Response B: NO TIPPING MONITOR 24 HOUR RECORDED CCTV.

Response C: NO TIPPING.

Preference learning framework can increase the probability of correct responses while decreasing the probability of other responses, which enhances the OCR ability.

Q1. Lack intrinsic evaluation of the AMP dataset.

A1. Thanks for your advice. We provide more evaluation for the AMP dataset and the auto-check mechanism.

Evaluation of the AMP dataset. We estimate the inconsistency rate of our AMP dataset to be 2.25% (by manual evaluation on 2k random samples). The 2.25% inconsistency rate is significantly lower than the human/GPT-4V (14.40%/11.95% inconsistency) annotations. Below is an inconsistent case in our AMP dataset, where $N_i$ denotes the $i$-th noun chunks:

Standard Response: A little girl ($N_1$) with a purple jacket ($N_2$) is flying a kite ($N_3$).

Response A: A little girl ($N_1$) dressed in purple jacket ($N_2$) is flying a kite ($N_3$) on the lawn, surrounded by many people (other information, including some hallucinations denoted by italic).

Response B: A young girl ($N_1$) dressed in purple clothes ($N_4$, similar to $N_2$ but not exactly the same) is flying a kite ($N_3$).

Response A accurately predicts these noun chunks and thus gets a higher score, leading to A>B. However, the actual ranking is B>A because A has some hallucinations.

Moreover, we conduct another experiment to validate the superiority of our AMP dataset. We mix the AMP and human/GPT-4V data for training the model. Table 1 in the PDF document shows that as the proportion of human/GPT-4V annotated data increases, the hallucination rate rises accordingly.

Evaluation of the auto-check mechanism. If we remove the auto-check mechanism of the sampled 2k examples, the inconsistency rate increases from 2.25% to 17.45%. It shows that the auto-check mechanism is critical.

---

Q2. Missing comparison with rank-based preference alignment approaches.

A2. We make both conceptual and empirical comparisons between our MDPO and prior rank-based preference alignment approaches, showing that our method has several advantages, $e.g.$, higher performance. The details are as below.

Conceptual comparison. There are two main differences between our MDPO and other rank-based preference alignment approaches. First, our MDPO mitigates the challenge of distinguishing micro hallucinations in responses. Taking 3-level preference as an example, the comparisons of other methods are 'A>BC, B>C', while comparisons made by our AMP are 'A>B, A>C, B>C'. More specifically, our AMP splits 'A>BC' into 'A>B, A>C', which enables MLLMs to perceive the subtle differences between different responses. Second, our penalty term explicitly increases the probability of MLLMs generating good answers, ensuring the stability of the training process.

Empirical comparison. As reported in Table 2, our MDPO surpasses these two learning objectives, $i.e.$, [1], [2] on all hallucination benchmarks, $e.g.$, 3.01 -> 3.17 on MMHal-Bench.

---

Q3. Missing results of FGAIF on MRHal-Bench.

A3. On MRHal-Bench, the mentioned FGAIF achieves 3.77/3.79 score and 0.30/0.31 hallucination rate. In comparison, our AMP is better, achieving 4.07/4.06 score (the higher the better) and 0.20/0.15 hallucination rate (the lower the better). Notably, the results of FGAIF are based on our re-implementation, because it is still close-sourced in terms of data and code. We are not confident that our implementation is completely correct. Once FGAIF is publicly available, we will update its results and add the comparison to our repo on Github.

---

Q4. Artifacts of the using responses from different model size.

A4. Using responses from different model sizes does have some artifacts, $i.e.$, some inconsistency. However, the inconsistency rate (2.25%) is significantly lower than the GPT-4V (11%) and human annotations (14%). Please kindly refer to the responses to Q1 for estimation details and examples.

Moreover, we empirically validate that using same-family models is better than using different-family models. We build a dataset that mixes the responses from GPT-4O, Qwen-2 and LLaVA. Training model with this dataset (#3 in Table 2) achieves inferior results than our strategy, $e.g.$ , 2.89 (different family) versus 3.17 (ours) score on MMHal-bench.

Perturbation-based methods also improve the baseline ($e.g.$, +0.14 on MMHal-Bench), but is still inferior to our MDPO ($e.g.$, -0.34 on MMHal-Bench), as shown in Table 2. Our implementation details are as follows. We randomly change the noun, adjective, preposition, and numeral. We obtain answers of varying quality by controlling the proportion of perturbations (10%, 30%, 50% based on 4-level preference setting). We infer the hallucination pattern generated by random perturbation is different from the real MLLM and is thus not informative enough for preference learning.

---

Q5. Why not use KL-Divergence for penalty?

A5. KL-Divergence is identical to our penalty term. The mathematical proof is as follows.

Assume $\mathrm{P}(y_w)$ is the probability distribution of superior responses, $\pi_\theta(y_w|x)$ denotes the probability distribution of the model generating superior responses, KL-Divergence is: \mathbf{KL}[\mathrm{P}(y_w)||\pi_\theta(y_w|x)]=\sum_{y_w\in\mathcal{Y}} \left[\mathrm{P}(y_w) \log \frac{\mathrm{P}(y_w)}{\pi_\theta(y_w|x)} \right],\tag{1} where $\mathrm{P}(y_w)=[0,...,1,...,0]\in \mathbb{R}^{L}$ and $L$ is the length of vocabulary. Since $\mathrm{P}(y_w)$ is 0 at all positions except for the superior word, where Equ.1 is re-written: $\mathbf{KL}[\mathrm{P}(y_w)||\pi_\theta(y_w|x)]=-\log\pi_\theta(y_w|x),$ $\min\mathbf{KL}[\mathrm{P}(y_w)||\pi_\theta(y_w|x)]\Leftrightarrow\max\log\pi_{\mathrm{\theta}}\left(y_{w} \mid x\right).$ Therefore, our penalty term is actually KL-Divergence.

---

Q6. Minor

A6. The outer loop is 1 to K-2. We take the responses from the pre-trained MLLM as $R_0$.

---

[1] Zhu et al. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons.

[2] Song et al. Preference Ranking Optimization for Human Alignment.