BrailleVision: Text Instruction Tuning of LLMs to Improve Visual Skills

1. The importance of the studied problem in this paper is questioned, i.e., improving the visual ability of only text-based LLM. As described in the introduction, I think the most popular paradigm of MLLM is the first one, i.e., extending LLM to vision-language task, which is adopted by most existing MLLMs. In contrast, the mentioned second paradigm seems receiving much less attention in both academia and industry. For instance, the papers cited by the authors in the introduction are before 2024, and only one is published in 2023. I would suggest the authors to give more proofs to indicate the importance of the studied problem, otherwise, the contribution will be very limited.

2. The experimental section is not sufficient. If the authors think that text-based LLM is an optimal solution for multimodal tasks, more comprehensive comparisons are required. In particular, text-based LLM for VL tasks also requires an VLM as a supplement, thus its overall parameter-scale is in fact similar with existing end-to-end MLLMs. So more comparisons are needed, for instances, the comparisons with more advanced MLLMs on more MLLM benchmarks.

Minors:

1. Under the task background and study problem of this paper, the description of ``Current multimodal large language models (MLLMs) incorporate general-purpose LLMs through multimodal instruction tuning. These LLMs, however, lack prior vision centric text based training, potentially limiting their effectiveness'' seems not very suitable. At the first glimpse, I thought this paper is to study the VL instruction tuning for common MLLMs.

1. Poor presentation:

The paper mainly consists of two parts: the first is how to construct an instruction-tuning dataset, and the second is how the instruct-tuned LLMs can assist multimodal models.

(1) For the first part, the authors should pose some cases from the text instruction dataset in the main body of the paper rather than relegating them to the appendix. Otherwise, only through reading section 3, I can hardly understand what kind of data the authors aim to curate or why the curated data can achieve the authors’ goal.

(2) For the second part, the authors propose two ways to leverage the tuned LLMs. The second way is multimodal LLM, which is more intuitive and aligns with current prevalent methods. However, for the first way, i.e., LLM assisting vision models, it cost me a lot of time to figure out how the LLM helps visual classification and detection. A diagram illustrating this process would enhance clarity.

(3) Many of the expressions in the paper are irregular. For example, the notation ‘→’ used in Table 1 (M-7B → Mistral-7B) lacks clarity, and the actual name of the test dataset is not labeled in the caption of Table 7.

2. Comparative Analysis:

While I appreciate the motivation, I wonder which data for learning is more efficient and effective: vision-centric text data or vision-text data. Could the authors design an experiment to compare these two approaches?

For example, in Table 2, if I understand correctly, the authors utilize Mistral-7B fine-tuned with vision-centric text data. What would happen if Mistral-7B were fine-tuned with vision-text data of the same volume?

3. Inefficiency of Fine-SFT:

The method of fine-grained supervised fine-tuning (Fine-SFT) appears inefficient, as it necessitates calculating additional token weights.

1. It is not clearly presented what exactly the "visual skills" learned through text-only training are. It appears more like learning and fitting the input-output format and boosting instruction-following abilities in visual benchmarks, rather than actual perceptual abilities. The core challenge in visual tasks—perception, i.e., extracting semantic information from raw pixels—seems untouched, while task format and instruction following capabilities can be well learned through NLP instruction dataset.

2. The additional text-only training requires extra computation and annotated datasets. I question whether allocating an equivalent amount of computation for visual instruction tuning would yield more substantial improvements. Incorporating the visual datasets used for BrailleVision-360k generation (e.g., ImageNet, Ego4D, VQAv2) directly as visual instruction tuning data might also lead to significant performance enhancements.

3. Generating the BrailleVision-360k dataset is complex and requires several additional steps and dependencies (e.g., CLIP, BLIP). A simpler baseline could be considered and compared to verify the necessity of the proposed method: translating images in visual instruction tuning datasets (e.g., LLaVa 1.5 dataset) into captions to derive a text-only dataset. This baseline is more straightforward and direct, and it would be simpler to implement.

4. Writing and Typo Suggestions

• Line 048: "supervised finetuning with instruction following data (IFT)" should be revised to "Supervised fine-tuning (SFT)," which are more commonly used terms. In modern LLMs, an alignment stage (e.g., RLHF or DPO) is often also included.
• Line 084: A space is missing between two sentences.
• Line 097: The term "semantic knowledge" is unclear.

1. What I am concerned about is the text performance. Will the method proposed in this paper hurt the text capability of LLM?

2. Does vicuna contain the same amount of data in BrailleVision? If not, the experiment is unfair.

3. Most multimodal benchmarks are in-domain or traditional VQA. Why not validate on the latest MLLM benchmarks like MMbench and MMVet, which can better reflect the effectiveness of the method.

4. typos：Line 52 and 84 are missing a space