RAPPER: Reinforced Rationale-Prompted Paradigm for Natural Language Explanation in Visual Question Answering

Q1: The two-stage approach, while novel, adds complexity to the model. It would be beneficial to see a discussion on the trade-offs involved, such as computational cost. How does the complexity of the two-stage approach impact the computational efficiency of the model?

A1: We thank the reviewer for suggesting further analysis for the computational cost. To discuss the trade-off between computational cost and our model performance, we now present the total and trainable model parameters of Rapper and its ablated variations (as listed in the upper part of Table 3).

From the table below, we see that Rapper significantly improves the metrics of natural language generation (e.g., CIDEr, SPICE, and Rough-L) over both the Baseline (Rapper w/o KD and w/o RLNF, a one-stage approach) and Rapper w/o RLNF (a two-stage approach). It is worth noting that, the Baseline only deploys the proposed reasoning module $M$ that is based on BLIP2 [1] and has a total of 3.82B parameters, including 187M trainable parameters (i.e., Qformer, initialized from BERT [2]) and 3.63B freezed parameters (i.e., ViT [3] and OPT [4]). On the other hand, Rapper w/o RLNF and the full version of Rapper are composed of the proposed rationale generator $G$ and reasoning module $M$. For such 2-stage frameworks, both modules are based on BLIP2 and share 3.63B freezed parameters, with 187M trainable parameters for each (and thus a total of 374M trainable parameters). It can be seen that, the two-stage methods do require a larger number (2X) of trainable parameters than the one-stage method does. However, the increased amount is in the same order (187M), and it is remarkably less than the total amount of parameters (~4B).

As for the training/inference stages, the runtime estimates are also listed in the table below, which are performed on 8/1 Nvidia V100 GPU(s) on the VQA-X [5] dataset. From the table below, we see that the two-stage method of Rapper w/o RLNF requires 2X computation time for both training and testing, and the full version of Rapper needs additional 3X computation time due to the RL optimization scheme. Nevertheless, even with RLNF, training can be done in 18hrs without utilizing an advanced GPU environment. And, acceleration of LLM training/inference is still an active research topic in the community. We sincerely thank the reviewer for suggesting the above analysis, which will be added to the revised version for the completeness of our discussions.

[1] Li, Junnan, et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML 2023.

[2] Devlin et al. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL 2019.

[3] Dosovitskiy et al. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021.

[4] Zhang et al. Opt: Open pre-trained transformer language models. arXiv 2022.

[5] Park et al. Multimodal explanations: Justifying decisions and pointing to the evidence. CVPR 2018.

Q2: The paper focuses on VQA. How's performance when it was adapted to other vision-language tasks?

A2: We thank the reviewer for the suggestion. We are happy to point out that we did apply our Rapper on another VL-NLE (natural lange explanation) task of Visual Entailment (VE), as listed in the lower part of Table 1 of our main paper.

We note that, VE requires one to assign a label determining the relationship between a premise image and a textual hypothesis (i.e., “entailment”, “neutral”, and “contradiction”), while giving an explanation for supporting the aforementioned relationship [6]. And, VE is tyically applied in cross-modal retrieval, etc. applications. In Table 1 of our main paper, we report the performance on the e-SNLI-VE benchmark, in which we observed that our Rapper performed favorably against NLXGPT [7] and S3C [8] in terms of different NLG metrics (i.e., CIDEr, SPICE, and ROUGE-L). We will add the definition of VE with the above additional clarification to Sect. 4.1 of our main paper.

[6] Do et al. e-snli-ve: Corrected visual-textual entailment with natural language explanations. CVPRW 2020.

[7] Sammani et al. Nlx-gpt: A model for natural language explanations in vision and vision-language tasks. CVPR 2022.

[8] Suo et al. S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning. CVPR 2023.

Q1: Choices of LLM for knowledge distillation (KD).

A1: We thank the reviewer for raising this issue. We are glad to conduct additional experiments to assess the LLM choices for KD in our framework.

Instead of using LLaMA [1] in our proposed work, we now replace LLaMA with GPT-3.5 [2] for experiments, as detailed below. To obtain distilled rationales, we consider the rationales produced by GPT-3.5 as pseudo grounth for training the deployed generator $G$. As explained in Q5, such pseudo ground truth rationales are collected via performing in-context learning (ICL) on the pre-trained LLM of GPT-3.5. To assess the quality of such distilled rationales, we take image-question pairs and the generated rationales as the inputs to conduct VQA and calculate its accuracy, following the ablation studies reported in Fig. 4. That is, we apply the pre-trained multimodal large language model of mPlug-Owl [3] to take the image-question pair with the produced rationale to output the answer. Thus, the higher improved accuracy means the higher quality of rationales.

With the above experiments, we observe that further improved accuracy can be achieved (i.e., 69.84% with LLaMA vs. 71.35% with GPT-3.5). This suugests that our proposed framework is applicable to the use of SOTA LLMs if computational environments are allowed. We will be happy to add such suggested experiments and evaluations to Sec 4.3, verifying the robustness of our proposed framework.

[1] Touvron, Hugo, et al. Llama: Open and efficient foundation language models. arXiv 2023.

[2] OpenAI, 2022. Introducing chatgpt.

[3] Ye et al. mplug-owl: Modularization empowers large language models with multimodality." arXiv 2023.

Q2: More human evaluations on plausibility and faithfulness.

A2: We thank the reviewer for suggesting additional experiments for subjective evaluation. Following the evaluation setting/process applied in NLXGPT [4] and S3C [5] (i.e., two SOTAs on VQA-NLE), we randomly select 200 test samples from the VQA-X [6] dataset with correctly predicted answers. Then, subjective evaluation is performed by 3 different annotators. Note that each annotator has to select one out of 4 choices: yes, weak yes, weak no, and no, as a response to whether the explanation justifies the answer. And, these 4 decisions are numerically mapped to 1, 2/3, 1/3, and 0, respectively. With averaged results obtained for each method, we present the performance comparisons in the following table. From this table, we see our proposed Rapper is preferable by the users in terms of subjective plausibility and faithfulness assessment. This conclusion also aligns with the objective quantification evaluation of Table 1 presented in the main paper.

[4] Sammani et al. Nlx-gpt: A model for natural language explanations in vision and vision-language tasks. CVPR 2022.

[5] Suo et al. S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning. CVPR 2023.

[6] Park et al. Multimodal explanations: Justifying decisions and pointing to the evidence. CVPR 2018.

Q3: How to determine the optimal hyperparameters (e.g. threshold τ) for filtering pseudo rationales from the LLM?

A3: Following [7], we simply set the threshold τ for filtering pseudo rationales as 0.54. As for other hyperparameters such as the RL discount factor for reward $\gamma$ and entropy weight $\alpha$ for calculating the reward function by Eqn (9), we follow [8] and set $\gamma$ = 1.0 and $\alpha$ = 0.2.

[7] Changpinyo et al. All you may need for vqa are image captions. NAACL 2022.

[8] Ziegler et al. Fine-Tuning Language Models from Human Preferences. arXiv 2019.

Q2: While the method is sound, why not just use large vision-language models and perform a CoT style prompting (which inspires this method)? How do large vision language models (e.g., BLIP-family or LLAVA models) perform?

A2: We thank the reviewer for the positive feedback. We are glad to perform additional experiments as suggested. As noted in our response to Q1, we would like to point out that existing VL models like LLAVA [2] or BLIP-family [8, 9] do not support in-context learning (ICL). Nevertheless, we are happy to take the suggestion from the reviewer to assess the use of VL models with properly designed ICL. We now consider the VQA-X dataset and compare Rapper with Otter [10] (i.e., a VL model built on LLaMA7B). For Otter, we follow the ICL setting in PICa [11] to random sample 16 samples as the demonstration prompts for ICL.

• Filtered results on the VQA-X dataset

• Unfiltered results on the VQA-X dataset

From the above tables, we see that Otter with ICL did not achieve comparable explanation performances as ours for both filtered and unfiltered cases (i.e., explanations associated with correct answers or not). This suggests that simple combination of VL models with ICL would not sufficiently address the VQA with natural language explanation, and a properly design VL model like ours would be desirable.

[8] Li et al. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ICML 2023.

[9] Dai et al. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. NeurIPS 2023.

[10] Li et al. Otter: A multi-modal model with in-context instruction tuning. arXiv 2023.

[11] Yang et al. An empirical study of gpt-3 for few-shot knowledge-based vqa. AAAI 2022

Q3: Small impact of the proposed method in the ablation study table (e.g., w/ or w/o RLNF)

A3: We thank the reviewer for pointing this out, and we are glad to clarify the performance improvements. In the upper part of Table 3, we perform ablation studies to verify the effectiveness of KD and RLNF. From this table, we see that the introduction of KD produced improved performance over the baseline model (e.g., CIDEr: 112.1 -> 120.2 and SPICE: 22.3 -> 24.2). This confirms the use of KD for predicting plausible outputs. Since our RLNF is deployed to mitigate the hallucination problem based on plausible prediction, adding RLNF to KD only increased the performance with smaller margins (e.g., CIDER: 120.2 -> 124.0 and SPICE: 24.2 -> 24.5). However, such improvements are expected. Recall that, as we explained in Q1, our Rapper is based on the introduction of KD and RLNF for providing plausible and faithful language explanation, and we cannot simply utilize RLNF without enforcing the NLE to be plausible.

Nevertheless, even with simply KD introduced and without RLNF, our model already performed favorably against SOTAs like S3C [12] and NLXGPT [13] (e.g., list CIDER/SPICE numbers, as shown in Table 1). And the full version of Rapper (i.e., with KD and RLNF) achieved the best results, which confirms our proposed learning schemes.

[12] Suo et al. S3C: Semi-Supervised VQA Natural Language Explanation via Self-Critical Learning. CVPR 2023.

[13] Sammani et al. Nlx-gpt: A model for natural language explanations in vision and vision-language tasks. CVPR 2022.

Q4: A clear definition of “rationale” is not presented in the paper. Only mentioned that it is like an “intermediate” just like in chain-of-thought prompting.

A4: We thank the reviewer for the suggestion. Although rationale-based prompting techniques have been presented in [14, 15], a proper definition should be given in our paper for clarity purposes. More precisely, in our work, rationales are the text prompts that generated by the rationale generator $G$, which are injected with language-based facts and visual contents. As illustrated in Fig. 1, the reasoning module $M$ takes the learned rationales as the text prompts, together with the image and question inputs, realizing plausible and faithful NLE. We will add the above definition to the fourth paragraph of the introduction.

[14] Zhang et al. Multimodal chain-of-thought reasoning in language models. arXiv 2023.

[15] Krishna et al. Post hoc explanations of language models can improve language models. arXiv 2023.

Q1: It is well-known that chain-of-thought (CoT) can improve reasoning. The role of using rationale to improve implausibility and hallucination is unclear. Why generating rationale could mitigate implausibility/hallucination?

A1: We thank the reviewer for giving us the opportunity to clarify the raised issues.

We first explain how our work is fundamentally different from CoT promoting [1]. In NLP, the use of CoT has been a common technique, which utilizes a pre-trained LLM with manually-designed prompts with in-context learning (ICL) techniques, so that additional reasoning descriptions can be produced by that LLM. However, since the above technique is designed for language data, extending CoT to vision and language is not a trivial task. More specifically, it is not clear how to determine prompting conseutive examples which allow pre-trained vision-language models like LLAVA [2] to fully extract the information from both visual and text modality data for vision-language explanation tasks [3]. Another concern is that, since CoT (for NLP) requires manually selected prompts and does not provide additional facts related to the output, the output would suffer from hallucination [4]. For our Rapper, the deployed rationale generator is designed to exploit language-based facts and visual content from cross-modality data inputs (i.e., with our learning schemes via knowledge distillation (KD) and Reinforcement Learning from NLE Feedback (RLNF), as explained in the following paragraph). With the learned rationales as the text prompts, plausible and faithful natural language explanations (NLEs) for VQA can be performed. Based on the above explanations, we hope the reviewer could see why CoT cannot be directly applied for reasoning in vision-language tasks.

We now explain how rationale learning helps mitigate implausibility and hallucination problems. As noted in our paper, implausibility refers to the problem when NLEs are irrelevant to the questions or contradictory to the established supporting facts [5]. As discussed in Sect. 3.2 (Plausible NLE Generation) and depicted in Fig. 2(A), we tackle this problem by learning rationales enriched with language-based facts from a pre-trained LLM. That is, we deploy a rationale generator $G$ in Rapper, which is trained via distillating the knowledge from LLM. With such rationales with language-based facts serving as additional text prompts, together with the image-question pair as the inputs, the subsequent reasoning module $M$ can be trained to provide answers with plausible explanations.

On the other hand, hallucination refers to the problem when the output explanation is not related to the visual image [6]. As noted in Sect. 3.3 (Faithful NLE Generation) and illustrated in Fig. 2(B), this is tackled by the proposed Reinforcement Learning from NLE Feedback (RLNF) technique, which jointly trains $G$ and $M$ for enforcing the learned rationale to fully exploit the visual content. The motivation of our RLNF aligns with the hypothesis of Kadavath, et al. [7], which views a learning model towards a faithful one when it exhibits greater confidence in their output answers. In other words, with the proposed RL scheme and objective rewards, the outputs of our reasoning module $M$ need to be supported by the rationale (i.e., predticed by the generator $G$) containing the visual content from the input image. As confirmed in Table 2, our Rapper achieved the best performance on faithful NLE for VQA in terms of RefCLIPScore. We thank the reviewer again for giving us the opportunity to clarify how our Rapper mitigates the problems of implausibility and hallucination.

[1] Wei et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS 2022.

[2] Liu et al. Visual instruction tuning. NeurIPS 2023.

[3] Li et al. Mimic-it: Multi-modal in-context instruction tuning. arXiv 2023.

[4] Dhuliawala et al. Chain-of-verification reduces hallucination in large language models., arXiv 2023.

[5] Majumder et al., Knowledge-grounded self-rationalization via extractive and natural language explanations. ICML 2022.

[6] Ji et al. Survey of hallucination in natural language generation. ACM 2023.

[7] Kadavath et al. Language models (mostly) know what they know., arXiv 2022.