LLaMA-Adapter: Efficient Fine-tuning of Large Language Models with Zero-initialized Attention

Q4: I suggest including the additional ablation study on reducing the rank of LoRA

Thanks for your advice! It is very necessary to evaluate the performance of Alpaca-LoRA with fewer parameters comparable to LLaMA-Adapter. The default rank of Alpaca-LoRA is 8, and we respectively show the results with the ranks of 2, 4, and 16 in the table below. For a more comprehensive assessment, we evaluate their language processing capabilities on Open LLM benchmark [1]. It evaluates LLMs' generative abilities in four different tasks: AI2 Reasoning Challenge [2], HellaSwag [3], MMLU [4], and TruthfulQA [5]. Each task contains challenging data samples over a wide range of knowledge domains.

As shown above, lower ranks of LoRA can effectively reduce the learnable parameters from 8.4M to 1.0M, and slightly lower the training time from 1.5h to 1.48h. However, our LLaMA-Adapter with 1.2M parameters and 1h still attains the best average result, demonstrating a good trade-off between performance and training efficiency.

---

Q5: The rescaled attention scores in Eq. 7 no longer represent a probability distribution. Why not [softmax($S_a)\cdot g;$ softmax($S_b)\cdot (1-g)$]?

Sorry for the confusion caused. Here, we denote the attention scores of the newly added prompts as $S_a$, the existing word tokens as $S_b$, and the learnable gating factor as $g$. During training, we aim to preserve the pre-trained knowledge of the LLM, i.e., softmax$(S_b)$, and adaptively incorporate the new instruction cues, i.e., softmax$(S_a)$. Thus, we do not multiply any coefficient to softmax$(S_b)$ to prevent the pre-trained knowledge from being disturbed, i.e., preserving its originally pre-trained probability distribution. The zero-initialized $g$ is only multiplied to $S_a$ to adaptively determine how much new information is injected into LLMs.

---

Q6: While LLaMA-Adapter may reduce memory usage, it doesn't necessarily reduce computation, a potential constraint on mobile devices

Thanks for pointing out! Our claim concerning mobile devices is not rigorously precise. We have rectified it in the revised paper as "LLaMA-Adapter enables more efficient storage of large-scale language models on mobile devices".

---

Q7: How about inserting adaption prompts across all transformer layers?

As you suggested, we conduct an ablation study on the insertion layer numbers on ScienceQA in the table below. As shown, inserting at the last 30 layers is better than the early 30 layers or all layers. This indicates that it's more effective to inject the instructional cues into layers capturing higher-level semantics for lanuguage generation. There also exists an optimal insertion number from the higher layers, since too few layers might underfit the training data, while too many layers would adversely disturb the early encoding of input words. If one has limited resources to identify the best insertion layer number, simply inserting our modules into all transformer layers is generally a good solution.

---

Q8: The output program in Fig. 4 seems to produce repetitive numbers

Thanks for pointing out! Our generated program showcases correct logic, but places the “print()” function in the wrong place, which outputs repetitive numbers. As the output of LLMs exhibits some randomness, we can obtain the correct program by another running, as shown in Figure 4 of the revised paper.

Reference:

[1] https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/.

[2] Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv 2018.

[3] HellaSwag: Can a Machine Really Finish Your Sentence? ACL 2019.

[4] Measuring Massive Multitask Language Understanding. ICLR 2021.

[5] TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022.

We sincerely appreciate your careful and constructive reviews. We hope our response can address your concerns.

---

Q1: The concept of zero-initialized attention closely mirrors the zero-initialized convolution in ControlNet

Sorry for the confusion caused. Our zero-initialized attention shares similar high-level motivations with ControlNet, i.e., progressively injecting new knowledge into the pre-trained model. However, our method is very different in four aspects.

1. Target Task. ControlNet aims to inject visual conditions into pre-trained diffusion models, which conducts text-to-image generation. Our target is to incorporate pre-trained LLMs with new instruction signals (language or multi-modality), tuning an LLM into an instruction-following model.
2. Methodology. On top of the CNN-based U-Net, ControlNet zero-initializes traditional convolutional layers, and directly adds new conditions with the pre-trained features by residual connections. In contrast, considering the transformer architecture of LLMs, we propose to achieve zero-initialization within the self-attention layer by a learnable gating factor. Then, we inject the new instruction cues into LLMs naturally through the attention process, which adaptively controls the interaction between prompts and word tokens.
3. Parameter Efficiency. ControlNet creates an entire copy of U-Net's encoder for training, which contains over 300M learnable parameters. However, our method only contains 1.2M parameters for adaption prompts and zero-initialized gating factors (1.8M for the multi-modal variant), indicating superior parameter efficiency.

Therefore, the zero-initialized attention of LLaMA-Adapter is distinct from ControlNet in both ideas and technical details.

---

Q2: The method seems more like an engineering design rather than a groundbreaking technical contribution

Our LLaMA-Adapter is not just an engineering design, but brings new insight to the field of LLM-based instruction tuning. We summarize our academic contributions as follows.

1. Progressively injecting instruction cues. Existing methods do not involve explicit designs to control how much instruction knowledge is injected into LLMs. This might disturb the pre-trained weights of LLMs, especially at the early training stage. To this end, we propose to utilize a learnable gating factor within LLM's self-attention layers, which is initialized as zeros to progressively incorporate new instruction cues into LLMs. Such a parameter-efficient design contributes to a more stable training process (Figure 7 of the paper), and can be utilized for future instruction-tuning methods.

2. Generalizability for multi-modal reasoning. In addition to language instructions, by simply adding multi-scale image features, our LLaMA-Adapter can be extended to a visual instruction model and conduct multi-modal reasoning. In contrast, the previous Alpaca and Alpaca-LoRA only support language instruction-following. Also, we report our downstream performance to fine-tune conventional vision and language models (ViT, RoBERTa, CLIP), demonstrating superior generalization capacity.

Therefore, our LLaMA-Adapter not only verifies the importance of zero-initially injecting instruction cues, but also exhibits good generalizability over a wide range of application scenarios.

---

Q3: The main difference comparing to Alpaca-LoRA

It is true that Alpaca-LoRA can also achieve parameter- and time-efficiency (4.2M, 1.5h), even though our LLaMA-Adapter is more efficient (1.2M, 1h). More importantly, the main difference of our approach is the generalization capacity for multi-modal input. LoRA appends trainable rank decomposition matrices into each transformer layer, which entirely inherits the original structures of LLMs. Therefore, LoRA cannot be extended to process new input features. In contrast, our LLaMA-Adapter with adaption prompts and zero-initialized attention is generalizable to accept new input conditions for instruction following, e.g., the plugin with visual reasoning expertise. With our approach, LLaMA can be fine-tuned into a multi-modal LLM with competitive visual understanding capability to existing multi-modal LLMs (shown in Table 3 of the paper), which indicates our promising generalizability for wider application scenarios.

Dear Reviewer WSmj,

Thanks again for your great efforts and constructive advice in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We put a significant effort into our response, with several new experiments and discussions. We sincerely hope you can consider our reply in your assessment. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any.

Regards,

Authors

Q4: More explanation on the choice of the "number of insertion layers"

Sorry for the confusion caused. Inserting prompts with zero-initialized attention into higher transformer layers is better than earlier layers. This is because it's more effective to inject the instructional cues into layers capturing higher-level semantics for language generation. There also exists an optimal insertion layer number, since too few layers might underfit the training data, while too many layers would adversely disturb the early encoding of input words.

We conduct an ablation study for the insertion layer numbers on ScienceQA in the table below. As shown, inserting at the last 30 layers is better than the early 30 layers or all 32 layers. Also, if one has limited resources to identify the best insertion layer number, simply inserting our modules into all transformer layers would generally be a good solution.

---

Q5: In equation 7, what happens when g grows bigger than 1?

Thanks for pointing out! In actual implementation, we add a tanh activation function to the gating factor $g$, which regulates its scale to into -1$\sim$1. This well controls the impact (in both forward and reverse directions) of adaption prompts to the pre-trained attention mechanism. We have added the description in the revised manuscript.

---

Q6: The method has few learnable parameters. How about the possibility of learning large-scale data or tasks?

Thanks for your advice! It is very necessary to evaluate the scalability of our LLaMA-Adapter for more data and tasks.

1. More Instruction Data. By default, we utilize a combination of Alpaca's data (52K) and LLaVA-I (158K) for visual instruction tuning. Here, we progressively add more question-answering data to enlarge the instruction datasets of LLaMA-Adapter: the sampled 83K VQAv2 by LLaVA-1.5 and the entire 204K VQAv2. As shown in the table below, the increasing instruction tuning data contributes to better multi-modal reasoning results on all benchmarks, demonstrating our method’s favorable scalability to a larger amount of data. | Data Size | MME || | MMBench | | | | | | | LVLM-eHub | | | | | |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | LLaMA-Adapter | All |P | C | All | LR | AR | RR | FP-S | FP-C | CP | All | VP | VKA | VR | VC | | Alpaca+LLaVA-I |1222| 973 | 249 |39.5| 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 |0.67| 0.81 | 0.44 | 0.83 | 0.59 | | +VQAv2 (83K) |1256| 1007 | 249 |43.4| 22.9 | 44.7 | 31.3 | 46.7 | 46.9 | 50.3 |0.69| 0.84 | 0.42 | 0.88 | 0.63 | | +VQAv2 (204K) |1618| 1272 | 346 |60.1| 34.7 | 65.3 | 48.7 | 63.1 | 57.3 | 69.3 |0.72| 0.86 | 0.44 | 0.92 | 0.65 |

2. More Learning Tasks. As shown in the paper and rebuttal, our LLaMA-Adapter has superior generalization capacity over a wide range of tasks, including language instruction following, e.g., question answering and code generation, multi-modal reasoning tasks, e.g., visual question answering and image captioning, and fine-tuning conventional vision or language models. For example, in the table below, we report the specific results on the task of optical character recognition (OCR) on several benchmarks: DocVQA [12], TextVQA [13], and OCR-VQA [14]. As shown, our LLaMA-Adapter achieves competitive performance compared to other methods. The experiments indicate that, given a small number of learnable parameters, our approach is generalizable to comprehend diverse vision-language tasks. | Datasets | LLaMA-Adapter | LLaVA | MiniGPT-4 | BLIP2 | |---|:---:|:---:|:---:|:---:| | DocVQA | 8.13 | 6.26 | 2.65 | 4.75 | | TextVQA | 43.76 | 38.92 | 19.40 | 31.98 | | OCR-VQA | 38.12 | 23.40 | 16.85 | 38.85 |

Reference:

[12] DocVQA: A Dataset for VQA on Document Images. WACV 2021.

[13] Towards VQA Models That Can Read. CVPR 2019.

[14] OCR-VQA: Visual Question Answering by Reading Text in Images. ICDAR 2019.

Q2: More evidence to show the proposed method is still effective and efficient in the multi-modal domain

Thanks for your advice! There have been many advanced multi-modal LLMs released very recently within several months. They incorporate a larger number of vision-language data for visual instruction tuning. For example, InstructBLIP (16M) and LLaVA-1.5 (665K) [4] collect visual question-answering data from a wide range of tasks for joint training. Note that LLaVA-1.5 was released on 5th October 2023, which is later than ICLR’s submission deadline, 27th September 2023.

In contrast, our multi-modal LLaMA-Adapter reported in the paper only adopts the basic Alpaca and LLaVA-I as the instruction data (210K). To fully unleash the potential of LLaMA-Adapter, we further add VQAv2 (204K) [5] into the instruction data for jointly tuning our model, denoted termed as LLaMA-Adapter$^\dagger$ in the following table. We evaluate LLaMA-Adapter$^\dagger$ on four multi-modal benchmarks: MME, MMBench, LVLM-eHub, and MM-Vet as follows.

As shown above, our LLaMA-Adapter$^\dagger$ consistently attains better results than the original version. This indicates the potential of our approach to achieve better performance with more instruction-tuning data. Compared to the latest InstructBLIP and LLaVA-1.5, our approach can attain competitive results, while using less instruction tuning data and adopting parameter-efficient tuning. This demonstrates our method is still effective and efficient in the multi-modal domain.

---

Q3: More quantitative comparisons with Alpaca and Alpaca-LoRA

Thanks for your advice! Besides qualitative results, we have compared them on the GPT-4 evaluation benchmark [6] in Figure 5 of the paper, which utilizes GPT-4 to assess the response quality on 80 questions. As you suggested, we further evaluate the language processing capacity of the three methods on Open LLM benchmark [7]. It evaluates LLMs' generative abilities in four different tasks: AI2 Reasoning Challenge [8], HellaSwag [9], MMLU [10], and TruthfulQA [11]. Each task contains challenging data samples over a wide range of knowledge domains.

As shown in the table below, LLaMA-Adapter still achieves the best average performance than Alpaca's full fine-tuning and Alpaca-LoRA. This demonstrates the strong language instruction-following ability of our approach.

Reference:

[4] Improved Baselines with Visual Instruction Tuning. arXiv 2023.

[5] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.

[6] https://lmsys.org/blog/2023-03-30-vicuna/.

[7] https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/.

[8] Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv 2018.

[9] HellaSwag: Can a Machine Really Finish Your Sentence? ACL 2019.

[10] Measuring Massive Multitask Language Understanding. ICLR 2021.

[11] TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022.

We sincerely appreciate your valuable reviews and recognition of our work. We hope our response can address your concerns.

---

Q1: Treating BLIP-2 as the baseline to LLaMA-Adapter for multi-modal evaluation

Thanks for your advice! BLIP-2 is a good and widely adopted baseline for existing multi-modal LLMs. However, please note that it is not appropriate to judge the capability of our approach solely based on BLIP-2's MME performance. The two reasons are as follows.

1. Targeting on Different Capabilities. Our LLaMA-Adapter, along with the concurrent LLaVA and MiniGPT-4, all aim to acquire the visual instruction-following capacity, namely, following human instructions to generate human-like answers conditioned on image input. In contrast, BLIP-2 focuses on the fundamental vision-language understanding capability. Therefore, the former visual instruction models tend to output long and detailed sentences like human conversations of daily life, while the latter BLIP-2 prefers to generate accurate answers with concise language.

2. Evaluation Bias in MME. MME mainly evaluates the fundamental perception and cognition abilities of multi-modal LLMs, for which the ground-truth answer contains only one word: "Yes" or "No". As analyzed above, such short-answer evaluation is more advantageous to BLIP-2 than visual instruction models (LLaMA-Adapter, LLaVA, and MiniGPT-4). For example, MiniGPT-4 is instruction-tuned based on the pre-trained vision encoder and Q-Former of BLIP-2, while performing worse than BLIP-2 in MME, as shown in the table below. In addition, InstructBLIP [1] further conducts instruction tuning upon the pre-trained BLIP-2, and also underperforms BLIP-2 on MME. | Method | MME | | | MMBench | | | | | | | MM-Vet | | | | | | | |---|:-:|---|---|:-:|---|---|---|---|---|---|:-:|---|---|---|---|---|---| | | All | P | C | All | LR | AR | RR | FP-S | FP-C | CP | All |Rec | OCR | Know | Gen | Spat | Math | | BLIP2 | 1584 | 1294 | 290 | - | - | - | - | - | - | - | 22.4 |27.5 | 11.1 | 11.8 | 7.0 | 16.2 | 5.8 | | InstructBLIP | 1505 | 1213 | 292 | 33.9 | 21.6 | 47.4 | 22.5 | 33.0 | 24.4 | 41.1 | 26.2 |32.4 | 14.6 | 16.5 | 18.2 | 18.6 | 7.7 | | Mini-GPT4 | 1159 | 867 | 292 | 23.0 | 13.6 | 32.9 | 8.9 | 28.7 | 11.2 | 28.3 | 22.1 |27.5 | 11.1 | 11.8 | 7.0 | 16.2 | 5.8 | | LLaVA | 718 | 503 | 215 | 36.2 | 15.9 | 53.6 | 28.6 | 41.8 | 20.0 | 40.4 | 23.8 | 28.0 | 17.1 | 16.3 | 18.9 | 21.2 | 11.5 | | LLaMA-Adapter | 1222 | 973 | 249 | 39.5 | 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 | 32.8 |38.5 | 20.3 | 31.4 | 33.4 | 22.9 | 3.8 |

Therefore, the instruction-following ability of visual instruction models, e.g., LLaMA-Adapter, cannot be entirely reflected by MME scores. Instead, on other multi-modal benchmarks, e.g., MMBench [2] and MM-Vet [3], our approach exhibits significant advantages over BLIP-2 or BLIP-2-based instruction models (InstructBLIP and MiniGPT-4):

1. MMBench [2] comprehensively measures 20 vision-language reasoning tasks with three levels of ability dimensions. MMBench introduces a new CircularEval strategy by using ChatGPT, which converts free-form predictions into pre-defined choices to alleviate the bias for BLIP-2. As shown in the table above, our LLaMA-Adapter showcases a clear advantage to BLIP-2-based instruction models.

2. MM-Vet [3] examines 6 core vision-language capabilities with 16 kinds of integration. It introduces an LLM-based evaluator for open-ended outputs, which thoroughly measures the answering performance of visual instruction models. As shown in the table above, our LLaMA-Adapter also performs much better than BLIP-2.

Reference:

[1] InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv 2023.

[2] MMBench: Is Your Multi-modal Model an All-around Player?. arXiv 2023.

[3] MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities. arXiv 2023.

Dear Reviewer UT9s,

Thanks again for your great efforts and constructive advice in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We put a significant effort into our response, with several new experiments and discussions. We sincerely hope you can consider our reply in your assessment. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any.

Regards,

Authors

Q3: It's better to test the scalability of the method, e.g., using a larger base model or more instruction tuning data

Thanks for your suggestion! As you suggested, we evaluate the scalability of our LLaMA-Adapter from two aspects: base model size and instruction data size.

1. Base Model Size. In the original paper, we conduct all experiments with LLaMA-7B. To scale up the base model, we further implement LLaMA-Adapter with a larger LLaMA-13B, and compare the results on three benchmarks as follows. On most tasks, LLaMA-13B attains better performance than LLaMA-7B, which demonstrates that our approach can be flexibly adapted to larger LLMs for potentially better multi-modal instruction-following capacity. | LLM Size|MME || | MMBench | | | | | | | LVLM-eHub | | | | | |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | LLaMA-Adapter | All |P | C | All | LR | AR | RR | FP-S | FP-C | CP | All | VP | VKA | VR | VC | | w LLaMA-7B |1222| 973 | 249 |39.5| 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 |0.67| 0.81 | 0.44 | 0.83 | 0.59 | | w LLaMA-13B |1299|1021|278|40.1|13.9|47.4|23.7|46.1|34.2|51.2|0.64|0.82|0.39|0.85|0.50|

2. Instruction Data Size. By default, we utilize a combination of Alpaca's data (52K) and LLaVA-I (158K) for visual instruction tuning. Here, we progressively add more question-answering data to enlarge the instruction datasets of LLaMA-Adapter: the sampled 83K VQAv2 by LLaVA-1.5 [5] and the entire 204K VQAv2. As shown in the table below, the increasing instruction tuning data leads to better multi-modal reasoning results on three benchmarks, demonstrating our method’s favorable scalability to data size. | Data Size | MME || | MMBench | | | | | | | LVLM-eHub | | | | | |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:| | LLaMA-Adapter | All |P | C | All | LR | AR | RR | FP-S | FP-C | CP | All | VP | VKA | VR | VC | | Alpaca+LLaVA-I |1222| 973 | 249 |39.5| 13.1 | 47.4 | 23.0 | 45.0 | 33.2 | 50.6 |0.6675| 0.81 | 0.44 | 0.83 | 0.59 | | +VQAv2 (83K) |1256| 1007 | 249 |43.4| 22.9 | 44.7 | 31.3 | 46.7 | 46.9 | 50.3 |0.6925| 0.84 | 0.42 | 0.88 | 0.63 | | +VQAv2 (204K) |1618| 1272 | 346 |60.1| 34.7 | 65.3 | 48.7 | 63.1 | 57.3 | 69.3 |0.7175| 0.86 | 0.44 | 0.92 | 0.65 |

Reference:

[5] Improved Baselines with Visual Instruction Tuning. arXiv 2023.

We sincerely appreciate your detailed and insightful reviews. We hope our response can address your concerns.

---

Q1: Applying the method to newer LLMs, e.g., MPT, Falcon, and LLaMA-2, could make the paper stronger

Thanks for your advice! Due to the limited time period of rebuttal, we apply our LLaMA-Adapter (including the adaption prompts and zero-initialized attention) to the latest LLM: LLaMA-2-7B [1]. LLaMA-2 is a stronger LLM that outperforms LLaMA on a wide range of tasks. For a more comprehensive evaluation, we tune the multi-modal variant of our approach, and evaluate it on three widely-adopted benchmarks: MME, MMBench, and LVLM-eHub. As shown in the table below, our approach can achieve higher visual instruction-following results with the stronger LLaMA-2. This demonstrates the superior generalization capacity of our method for instruction-tuning different pre-trained LLMs.

---

Q2: It's better to add more evaluations, e.g., counterfactual reasoning [2] or object hallucinations [3]

Thanks for your advice! Besides the regular evaluation in Table 3 of the paper, it is very necessary to explore the counterfactual reasoning abilities and object hallucination issues of multi-modal LLMs. As you suggested, we respectively evaluate our multi-modal LLaMA-Adapter on two suggested benchmarks as follows.

1. C-VQA [2] for counterfactual reasoning. C-VQA contains 2k counterfactual question and answer pairs, which are collected from VQAv2 [3] and supplemented by ChatGPT. As shown in the table below, for three groups of questions, LLaMA-Adapter performs comparably to the concurrent LLaVA. Especially for the Numerical indirect questions, our approach achieves the best counterfactual reasoning results (34.3) and the least performance loss (5.6$\downarrow$) than all other models. Method|Numerical direct$\uparrow$ (Loss$\downarrow$)|Numerical indirect$\uparrow$ (Loss$\downarrow$)|Boolean$\uparrow$ (Loss$\downarrow$) -|:-:|:-:|:-: ViperGPT|80.6 (19.4$\downarrow$)|31.6 (68.4$\downarrow$)|21.6 (72.4$\downarrow$) LLaVA-7B|27.0 (9.9$\downarrow$)|25.0 (15.2$\downarrow$)|58.5 (4.8$\downarrow$) LLaVA-13B|24.8 (11.9$\downarrow$)|20.8 (21.2$\downarrow$)|56.3 (4.7$\downarrow$) LLaMA-Adapter-7B|30.1 (5.8$\downarrow$)|34.3 (5.6$\downarrow$)|45.8 (14.5$\downarrow$)

2. POPE [4] for object hallucinations. POPE evaluates object hallucinations as a binary classification task, and includes 500 images from MSCOCO with 6 questions per sample. As shown in the table below, for different evaluation settings, LLaMA-Adapter with LLaMA-7B attains competitive accuracy compared to other multi-modal LLMs with LLaMA-13B, which indicates our relatively stronger robustness to object hallucination problems. Method|Random|Popular|Adversarial -|:-:|:-:|:-: mPLUG-Owl-7B|53.30|50.63|50.67 LLaVA-13B|54.43|52.43|50.77 MM-GPT-7B|50.03|50.00|50.00 LLaMA-Adapter-7B|75.47|60.43|60.66

Reference:

[1] Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv 2023.

[2] What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models. arXiv 2023.

[3] Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.

[4] Evaluating Object Hallucination in Large Vision-Language Models. arXiv 2023.

Dear Reviewer EjGx,

Thanks again for your great efforts and constructive advice in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We put a significant effort into our response, with several new experiments and discussions. We sincerely hope you can consider our reply in your assessment. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any.

Regards,

Authors