Understanding Alignment in Multimodal LLMs: A Comprehensive Study

Thank you so much for your feedback. Please find our answers to your questions below.

Q1: What are the effects of scaling up BDHS in terms of data size or complexity on model performance?

• BDHS is used to generate the “rejected” responses, and therefore naturally scales with the number and distribution of ground truth/preferred responses. If the reference dataset grows with a limited increase in diversity, the generated BDHS responses could eventually theoretically become indistinguishable from the preferred ones (no inherent hallucinations triggered). Our algorithm would then discard any further growth of the dataset by evaluating the similarities through sequence embeddings. However, in our experiments with common alignment datasets, we did not observe saturation. To support this with data, we have added a new table (Table 16) to our appendix that compares larger BDHS dataset variants with their non-BDHS references, specifically for the full POVID dataset (17k samples) and our custom VLFeedback dataset with GPT-4V preferred responses (34k samples). In both cases, the BDHS variants outperform the non-BDHS datasets for the majority of benchmarks, particularly for LLaVA in the wild and MMVet, similar to the smaller 5k ablations we showed before.

Q2: What specific modifications could be made to BDHS to achieve state-of-the-art results on key benchmarks?

• The goal of our alignment approach is to reduce hallucinations rather than inject additional knowledge into the model. As a result, the observed improvements primarily reflect reductions in hallucinations rather than enhancements in domain-specific capabilities (e.g., mathematics). This focus is why the most significant gains are seen on hallucination benchmarks. For instance, on the POPE benchmark, our aligned model achieves SOTA performance and is highly competitive on Objhal-v, even against models trained on data generated by stronger sources like GPT-4V or human annotators. To further enhance the BDHS pipeline, one promising direction could involve integrating our method with ranking-based approaches, such as the online-DPO pipeline. In this framework, instead of deriving both preference pairs from the model in every iteration, we could use BDHS to generate samples for a portion of the training data. This hybrid approach could leverage the strengths of both methods, potentially driving further improvements across various benchmarks. We view this as an exciting avenue for future research.

We are very grateful for your detailed review and feedback. Please find our response to your questions below.

Q1: ... How to select the range of attention masking, and whether the alignment effect can be optimized by dynamic adjustment?

• Since we use BDHS to generate “rejected” responses, our primary goal was to ensure that these responses are semantically different from the “chosen” responses. To achieve this, we designed an iterative approach where a rejected response is added to our dataset only if it meets a specific threshold on the similarity metric (a hyperparameter determined through our experiments), ensuring it is not too similar to the chosen response. In this process, as you may have suspected, we observed that simple question-answer pairs typically require more iterations to produce a response that is sufficiently distinct from the chosen response. We conducted extensive ablation studies on the amount of masking (Appendix G) and compared the sensitivity of our approach to that of other state-of-the-art methods designed for similar tasks (POVID dataset). Our results demonstrate that our approach is both more effective and less sensitive to the hyperparameter. That said, exploring how the amount of masking required to (more) reliably introduce hallucination correlates with visual complexity is an interesting idea, and we plan to explore its impact in future research.

Q2: ... Could this similarity score lead to a tendency for models to oversimplify when faced with less common or more complex visual scenes?

• Our approach relies on the ability to measure the similarity between the embeddings of chosen and rejected responses. Most recent sentence embeddings perform well in this task, making our pipeline relatively robust to the specific choice of embedding. After evaluating multiple options, we empirically found all-mpnet-base-v2 to be an effective and computationally efficient choice. This computational efficiency is particularly important in an online setting. Furthermore, by employing SFT forcing, we avoid oversimplifying the rejected responses. In most cases, the model generates only part of the “rejected” response rather than rewriting the entire sentence. This is a crucial aspect of the BDHS algorithm, which ensures that the rejected responses remain minimally different from the chosen responses in terms of complexity and style, while being semantically distinct. For example, two responses might describe the color of an object using the same style, but one specifies the correct color while the other specifies an incorrect one. This design ensures that the model must comprehend the image in order to optimize the objective effectively.

Q3: ... Especially on small-scale models, is it possible that the method effect is not significant due to parameter limitations?

• We have not conducted experiments on smaller models. Initially, we started our experiments on weaker models, specifically the LLaVa 1.5 family. However, we decided against continuing with them because, although they were widely used by the community at the time, we found that weaker models had such a large margin for improvement that almost any intervention proved effective. For example, higher values of 𝛽 in the DPO setting consistently resulted in better performance. This suggests that diverging from the reference policy enhances the model’s capabilities, which may indicate underlying weaknesses in the base model. Instead, we focused our study on stronger models after the SFT stage to better highlight the benefits that the alignment stage can bring to multimodal LLMs. While we conducted some limited experiments on larger models (e.g., LLaVa 1.6 13b), these followed similar patterns to the 7b model, however were not comprehensive enough to include in the paper due to compute limitations. Based on our findings in this paper and our ongoing research in this area, we believe this approach is consistently effective in settings where we can observe LLM bias in multimodal LLM behavior.

Q4: To what extent do current hallucination evaluation benchmarks such as POPE and MMHALBench-V truly reflect model performance in real-world applications?

• This is certainly one of the areas our research identified as having significant room for exploration, which we discussed in Appendix B2-B4. We found our proposed MMHALBench-V to be an effective benchmark for evaluating hallucination; however, we acknowledge that there is still plenty of scope for further improvement. For the experiments presented in the paper, we carefully analyzed both the wins and losses associated with any observed advancements to ensure they genuinely reflected enhanced model performance. For example, although we observed substantial gains in results on ObjhalBench, we excluded those numbers due to the issues outlined in Appendix B3.

Thank you so much for your positive feedback and recognizing the contribution of our work. Please find the answer to your questions below:

Q1: Will the code be open-sourced to facilitate further research in this area?

• Yes, we are working on open sourcing the codebase.

Q2: How does the proposed approach ensure that the distribution of generated hallucination data aligns with real-world hallucination data distributions?

• Hallucinations in multimodal LLMs can arise from various root causes, with bias in the language model being a significant contributor. Our proposed approach, BDHS, is specifically designed to address this bias directly. While BDHS does not explicitly target other causes of hallucinations, such as insufficient real-world knowledge, we have observed overall improvements across multiple benchmarks, indicating a positive impact on reducing hallucinations. We have added some explanation to the conclusion to highlight your feedback on this area further. To further ensure that the distribution of generated hallucination data aligns closely with real-world distributions, we introduced an online version of BDHS. This online approach allows the model to dynamically adjust the hallucinations as it learns, ensuring that as the model's performance improves, the generated hallucinations are continually aligned with the improved model.