Aligning Large Multimodal Models with Factually Augmented RLHF

Thank you for your insightful feedback! We appreciate your recognition of our approach in tackling multimodal misalignment and enhancing RLHF with factual information. Your constructive comments of our method's effectiveness and contribution is highly valued. We address your questions below.

Concern 1: Are answers and questions in the MMHAL-BENCH dataset written by humans or constructed by automatic methods?

The questions and answers are created by humans. As introduced in Appendix C, we have selected 8 question categories $\times$ 12 object topics, and manually designed questions where prior LMMs usually make false claims about image contents (hallucinations).

Concern 2: The authors proposed that the evaluation indicates that this benchmark dataset aligns well with human evaluations, especially when scores are adjusted for anti-hallucinations. I didn’t find how to get this conclusion in the experimental section or other sections.

After designing the questions, we have tested LLaVA-13B and IDEFICS-80B on MMHal-Bench. We manually inspected 1) the answers produced by these two LMMs, and 2) their corresponding reviews given by GPT-4. When judging whether hallucinations exist in the LMM responses, we found that the agreement between human inspection and GPT-4 evaluation was 94% (Appendix C). It is worth noting that IDEFICS-80B was not involved when we designed the question-answer pairs, but the hallucination evaluation is still highly consistent between humans and GPT-4. These observations demonstrate that the automated GPT-4 evaluation in MMHal-Bench is well-aligned with humans.

Concern 3: The method section requires an overall overview of the relationships between the various parts of the method.

We thank the reviewer for the insightful suggestion. We have added an overview paragraph in Section 2.

Thank you for your helpful comments! We appreciate your recognition of the novel application of RLHF in the vision-language domain and the introduction of our MMHAL-BENCH benchmark. We address your questions below.

Concern 1: I could NOT find direct evidence that the proposed method reduces hallucinations and mitigates reward hacking.

Our proposed Fact-RLHF indeed has helped the model to reduce hallucination. After investigating the evaluation by GPT-4 on MMHal-Bench, we would like to clarify that our model trained with Fact-RLHF was actually at a disadvantage when being evaluated with open-ended questions. The reason is that RLHF encourages the model to make detailed, informative responses, which are often significantly longer than responses given by a model only trained with SFT, as well as human annotators. Therefore, our RLHF model may include some information that are valid statements about the image, but just not included in the human-generated answer or image annotations. Consequently, GPT-4 would mistakenly believe that our RLHF model had hallucinations. We observe that such biased evaluation exists in our RLHF-based models (both 7B and 13B), but not in our model only trained with SFT or other LMMs, since they are not further tuned by RLHF.

Let’s see some examples:

Example 1

• Image: https://farm1.staticflickr.com/3815/9340455006_25a885f7ef_o.jpg
• Question: Who in the picture is wearing trousers?
• Image content annotations: "Clothing", "Human body", "Person", "Man"
• Human answer: The woman in the image wears the pants.
• LLaVA-SFT+-13B answer: The woman in the picture is wearing trousers.
• GPT-4 rating for LLaVA-SFT+13B: 5, very informative, no hallucination
• LLaVA-RLHF-13B answer: In the picture, the woman is wearing trousers while sitting next to the man. The man is wearing shorts, and both of them are sitting in front of a campfire. This indicates that they are likely on a camping trip or enjoying an outdoor activity together. The presence of a campfire suggests that they are in a natural setting, possibly in a park or a campsite. The fact that they are both sitting in chairs around the fire implies that they are taking a break from their activities, perhaps to relax, socialize, or cook food. Their choice of clothing, with the woman wearing trousers and the man wearing shorts, suggests that they are dressed for a warm and comfortable outdoor experience. The woman's choice of trousers may provide her with more protection from the elements, such as cold temperatures or insects, while the man's shorts allow for better air circulation and freedom of movement.\n\nOverall, the image of a woman wearing trousers and a man wearing shorts sitting around a campfire represents a moment of leisure and enjoyment in the outdoors, where they can appreciate the natural beauty and serenity of their surroundings.
• GPT-4 rating for LLaVA-RLHF-13B: 1, somewhat informative, with hallucination
• Explanation: LLaVA-RLHF-13B correctly identifies the person “wearing trousers,” but also includes additional information that can be observed in the image. However, the additional information (such as the campfire and chairs) are not included in the human-generated answer or the annotations. As a result, GPT-4 incorrectly assigns a low score to the answer.

Example 2

• Image: https://c2.staticflickr.com/4/3501/4068897399_a729022693_o.jpg
• Question: Describe the image in detail.
• Image content annotations: "Window", "Clothing", "Person", "Building"
• Human answer: The image captures a bustling street lined with various shops and an adjoining sidewalk. Several pedestrians are present, with two individuals notably passing a store with a green sign saying "Hawkin's Bazaar". At this store's entrance, a man stands, overseeing a spacious window display that houses diverse items, highlighted by a red and white sign. Interestingly, no trees or parked cars are present in the street. The mood is energetic, showcasing individuals engaged in their daily routines and shopping pursuits.
• LLaVA-RLHF-13B answer: The image depicts a lively street scene with a group of people walking past a store called "Hawkins Bazaar." The storefront is adorned with Christmas decorations, and the store appears to be selling a variety of items, including toys and gifts. There are several people walking along the sidewalk, with some of them carrying handbags. The pedestrians seem to be enjoying their time, browsing the storefronts and engaging in conversation. The street is bustling with activity, and the store is likely attracting customers with its festive decorations and diverse selection of products.
• GPT-4 rating for LLaVA-RLHF-13B: 1, somewhat informative, with hallucination
• Explanation: Similarly in this example, the model mentions additional information about “Christmas decorations,” which is valid but not included in the annotations or human-generated answer. Therefore, GPT-4 fails to correctly assess the response.

(cont.)

(cont.)

Due to limited space, we cannot include too many examples in this response, but we will include more detailed analysis regarding the under-estimation of RLHF models’ performance in the manuscript revision.

To summarize, RLHF would enrich our LMM’s responses. However, informative and detailed responses by RLHF models sometimes cannot be matched with elements in the annotations and human-generated answers. Although we have tried our best to make human answers as comprehensive as possible, there could always be more to include. As a result, the hallucination rate of our RLHF model is overly estimated by GPT-4. If we take this inherent bias into account, RLHF is indeed reducing the hallucinations.

Concern 2: The paper did not study more realistic scenarios where additional information is unavailable or noisy.

Fact-RLHF only needs question or image related factual information to augment the reward model. It is relatively cheap to annotate such factual information [1][2]. We also expect automatic caption generation techniques such as CapFilt [3] or DSC [4] (i.e., from a different model) can help automate this process.

Concern 3: The paper did NOT elaborate on experimental setup (e.g., descriptions of each dataset and task & evaluation metrics)

Thanks for pointing this out. We have previously included the implementation details as well as the hyperparameters in Appendix I.

Regarding the description of each dataset, task and metrics, we have added more information in Appendix. Specifically, the three employed supervised fine-tuning datasets are VQA-v2, AK-VQA and Flickr30k as listed in Section 2.2. We use “Yes” or “No” queries from VQA-v2 (83k), multiple-choice questions from A-OKVQA (16k), and grounded captions from Flickr30k (23k). The 10k human preference data are paired outputs from the base 7B LLaVA model and we ask the Amazon Turker annotators to label which one contains less hallucinations. The details about the collection process is in Appendix G.

For each evaluation task, we report the accuracy for MMBench, which is a multiple-choice question benchmark consisting of 1031 questions. We report the F1 score for the POPE, which is a “Yes/No” question benchmark and consists of 3k questions in three categories (random, adversarial and popular). The LLaVA bench consists of 100 questions and will be evaluated by GPT4 against the outputs from text-only GPT4. Finally, we report the GPT4 score on MMHalBench, which has 96 questions as well targeting the hallucination level of each model.

[1] Chen, Xinlei, et al. "Microsoft coco captions: Data collection and evaluation server." arXiv preprint arXiv:1504.00325 (2015).

[2] Pont-Tuset, Jordi, et al. "Connecting vision and language with localized narratives." ECCV, 2020.

[3] Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International Conference on Machine Learning. PMLR, 2022.

[4] Betker, James, el al., “Improving Image Generation with Better Captions”

Thank you for your helpful comments! We appreciate your recognition of the novelty and significance of our work in applying RLHF to improve multimodal alignment in LMMs, as well as your acknowledgment of the importance of our newly proposed MMHAL-Bench. Your appreciation of these aspects validates the direction and relevance of our research. We address your questions below.

Concern 1: The key issue of this work may be lack of details

Concern 1: The supplementary file is useless for reviewing

We have updated the supplementary material to share all the details of our implementation. We will also open-source the code, model, and the collected datasets.

Concern 2: No information is given about labelers

Question 2: How much does it cost to collect FACT-RLHF annotations? How long does it task? How many labelers are involved?

We have added more information about the anonymized labelers in the Appendix G of the paper. We spend around $5000 to collect the annotations. It took around 1 a week on the MTurk platform, and 28 labelers were involved.

Concern 3: Is there any quantitative evaluation that focus on how much hallucination reduces?

As previous LLM hallucination benchmarks are usually limited to the yes/no questions, in this paper, we mainly quantitatively evaluate the hallucination of LLaVA-RLHF with our proposed MMHAL-BENCH dataset. Please see detailed description of the MMHAL-BENCH in Appendix C and the results in Table 6. LLaVA-RLHF can improve by 60% over other baselines, and reduce the hallucination rate of the original LLaVA by 27%.

Concern 4: It would be better to include some experimental results about the performance of the reward model.

We have added more information about the training and evaluation of the reward model.

Question 1: FACT-RLHF is sourced from QA pairs in the LLaVA, A-OKVQA and VQA-v2. Are they any other datasets to get more samples?

LLaVA-1.5 [1] is trained on LLaVA, A-OKVQA, VQA-v2, and many more datasets like TextCaps, RefCOCO, and VG. LLaVA-RLHF can also potentially benefit from those additional vision-language alignment data.

[1] Liu, Haotian, et al. "Improved baselines with visual instruction tuning." arXiv preprint arXiv:2310.03744 (2023).

Thank you for your comments! We appreciate your recognition of the significance of our work as the first LMM trained with RLHF and its state-of-the-art results on LLaVA-Bench and MMHAL-BENCH. We are glad that these key aspects of our research were well received and appreciated. We address your questions below.

Clarification 1: “They collect preference dataset by creating a set of varied questions…”

The MMHal is only used for evaluating the hallucination in LMM generations.

Clarification 2: “The model improves by 96%...”

Our method achieves an absolute performance of 96% of text-only GPT-4, not improves by 96%.

Concern 1: The RLHF model degrades slightly on the capability benchmarks.

While there is a slight degradation at the 7B scale, the RLHF significantly improves performance on the MMBench at the 13B scale.

Concern 2: Can you please cite the reference to the scaling law of LLaVA-RLHF mentioned in the paper?

We have added citations to the alignment scaling laws of [1,2].

Question 1: how do you get the factual information for these examples

Since the images are from the MS COCO dataset, they come with 5 captions for each image.

Question 2: clarify how is the image captioning data from COCO converted to instruction tuning

We did not use captioning data for instruction tuning. It is only used as factual augmentation for the reward model during the RL phase.

[1] Askell, Amanda, et al. "A general language assistant as a laboratory for alignment." arXiv preprint arXiv:2112.00861 (2021).

[2] Kaplan, Jared, et al. "Scaling laws for neural language models." arXiv preprint arXiv:2001.08361 (2020).