CogVLM: Visual Expert for Large Language Models

Thank you for your comprehensive review and important comments. Please allow us to address your points in detail:

1. Comparing the proposed method to earlier approaches such as PaLI, CoCa, and Flamingo may not be entirely fair. These prior methods do not incorporate the SFT stage, making it unclear how the model performs before this crucial phase.

This is a misunderstanding. We did not incorporate SFT to evaluate the benchmarks, so the evaluation is fair. See 2. of "response to common concerns" for details.

2. Since the pretrained image encoder and LM and went through VLM finetuning, their original behavior may have changed. I wonder what the visual eval (linear probe, zero shot) will be for this finetuned encoder, compared to original model. How LM performance got affected?

This is also a misunderstanding. During the pretraining, the parameters of both the ViT image encoder and the LLM were completely frozen, thus the ViT is not affected, because it is also frozen in SFT. LM is trainable during SFT with a very small learning rate of 1e-6 for 8,000 iterations, which might affect the language style but not much on knowledge.

We find your concerns are mainly from two important misunderstandings. Could you increase your rating if our responses clarify it?

Dear Reviewer: Thanks again for your careful and valuable comments! Since the rebuttal discussion is due soon, we’ll be appreciated to know whether our replies have addressed your questions. If there are any further clarifications required or any other concerns, please feel free to contact us. Many thanks!

Thank you for your comprehensive review and insightful comments. Please allow us to address your points in detail:

1. The idea is not that novel, compared with BEIT-3 and VLMo, which also introduces different modality expert structures, although this work makes some changes to make it work in the era of LLM.

Our response to this concern is in 1. of "reply to common concerns" above. The MoE is definitely not a new idea, but in the current LLM+ paradigm, it is very valuable to rediscover its effectiveness as a way to better scaling up.

2. The VQA/Image Caption model, Visual Grounding model and Chat model are three different models, I am wondering how the performance can be if all these models are a unified one? Since GPT-4V may be a unified one.

Previously, we did not unify the models for VQA, chat, and grounding tasks due to their diverse output requirements, such as single-word answers for VQA, lengthy analytical responses for chat, and coordinate-heavy answers for grounding. However, our latest experiments indicate that VQA and chat models can be effectively unified. We will soon release a unified checkpoint where simple prompt modifications like "Short answer:" and "Answer:" enable the model to perform strongly in both VQA and chat domains, as shown in our latest results in Table 1(The touchstone score increase from 663 to 740).

The grounding task is slightly different; our results show it slightly reduces the model’s captioning ability. For instance, performance on the COCO dataset dropped by 2 points and on the NoCaps dataset by 1.5 points, potentially due to the model’s limited understanding of coordinates. Conversely, grounding training improved the model's performance by 0.8 point on the VQAv2 dataset, likely due to enhanced object localization capabilities. We believe a fully unified model is achievable and are working towards this, with adaptations needed for grounding tasks, such as adding specific coordinate-representing words to the vocabulary and loosening parameters in the word embedding layer and LM head during training.

3. Although I appreciate the excellent performance it achieves, the visual backbone is ViT-e, and the input resolution is 490 * 490, also the parameter size of LLM doubles, which makes the comparason a little hard.

Our comparison is fair enough actually. It's common for multimodal models to upscale resolution for downstream tasks to enhance performance, similar to BLIP-2 (490 * 490) and Qwen-VL (448 * 448), with Google’s PaLI-X even using 896 * 896 resolution for image inputs. For ViT-E, we also experimented with the OpenCLIP-L visual encoder, comprising 300M parameters, and found its performance only slightly worse than that of ViT-E. For instance, the score on the VQAv2 test set was 83.4 with OpenCLIP-L compared to 84.7 with ViT-E, and 88.6 on the Visual-7W dataset versus 90.6 with ViT-E.

4. Do you have more experiment on the archtecture of visual expert module and more insight about which part of the layers should be shared and which module should have separate parameters?

In the ablation study part, we experimented this. The finding basically shows that the best way is to insert new parameters in each layer.

5. For the generalist performance, is it possible that a model can achieve best performance on both real-world chat and benchmark datasets? since this paper has three separate training procedures to make it best in each individual dataset. If there exist some gaps between different kinds of datasets, how can the architecture be designed to better address this problem?

Yes, we merged the training of chat and generalist on benchmarks, which is reported in the response to the second question. The results are even better on chat and basically the same for benchmarks (generalist).

6. Have you observed some new emergent ability in this strong MLLM?

The definition of emergent abilities in the multimodal domain is not entirely clear. However, in our recent evaluations, we have indeed observed that our model shows significantly greater improvements in challenging tasks compared to other models, beyond what we see in VQAv2. For example, in the OCR&Spat task defined in MM-Vet, our model scored 59 compared to 14 for InstructBLIP, and in the Rec&Know task, our score was 39 while MiniGPT-4 scored 0.

Dear Reviewer: Thanks again for your careful and valuable comments! Since the rebuttal discussion is due soon, we’ll be appreciated to know whether our replies have addressed your questions. If there are any further clarifications required or any other concerns, please feel free to contact us. Many thanks!

Thank you for your comprehensive review and insightful comments. Please allow us to address your points in detail:

1. However, both these two disadvantages proposed by the authors are just hypothesises, not compelling nor conclusive. First, the performance gap between BLIP-2 and PaLI-X cames from several possible differences between two framework (e.g., the visual encoder size, the way that visual encoder is pre-trained by). And both MiniGPT-4 and LLAVA have extremely little trainable parameters in the alignment between visual features and language features.

This point is about the effectiveness of deep fusion. The comparison between PaLI-X and BLIP-2 are just two examples of shallow / deep alignment methods, and not a strict claim. You are right that they cannot be compared directly. We will modify it in the final version to avoid confusion.

2. Some blanket statements are used. For instance, the author claims that NLP ability is weakened when jointly train the language model in image-text training. However, there are some evidences show that jointly training can benefit both vision task as well as language task, at least in some aspect (e.g., [1]).

It is not a blanket statement. In our paper, the next sentence of this statement is "According to PaLM-E [2]...", which runs experiments to prove the catastrophic forgetting in LLMs during jointly training. Specifically, Palm-E without freezing the language model parameters during multimodal training, showed a 15% drop in NLU capability and an 87% drop in NLG capability across 21 NLP tasks. Similar findings were observed in Flamingo, where not freezing the LLM resulted in an average 8% performance decrease across five multimodal benchmarks. Regarding the 'Sight Beyond Text' paper, it only trained with a smaller learning rate on approximately 600,000 data points, which likely led to minimal changes in the LLM parameters, potentially resulting in a significant gap from large-scale pretraining. Hence, it may not be a proper reference to prove no catastrophic forgetting in LLMs during jointly training. For hallucination problems, our model has a significantly higher F1 score on the challenging POPE dataset than other models, which confirms the statement in our paper. See "Results on hard benchmarks" for details.

[2] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi-modal language model. arXiv preprint arXiv:2303.03378, 2023.

3. Ablation studies cannot prove that deep alignment is better than and solves the issues of shallow alignment, due to the results of shallow alignment method with larger visual encoder (same parameters as vision encoder + vision adapter) are remain unknown. This proposed ablation study is very reasonable but unfortunately is very hard to implement, because the ViT (CLIP) is pretrained and the performance is not solely determined by the number of parameters. We cannot get a good ViT with arbitrary number of parameters easily for comparison.

However, we can still use a hypothetical comparison. In our paper, the "VE-full every 4th layer" setting (1.7B) performs much better that the "MLP-adapter"(shallow) setting (140M), (COCO CIDEr 131.2 vs 138.7). If there were a well-trained EVA-CLIP ViT of (5B+1.7B) parameters, it is not possible to increase that much in "MLP-adapter" setting, since the current "MLP-adapter" has already 5B parameters.

If you make the ViT trainable to increase the trainable parameters, the performance will not be better either, which is well-documented in previous works like flamingo[3] (Table 7 (xii) 68.4 ->64.5 after making vision encoder trainable). I hope this hypothetical comparsion can ease your concern.

[3] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." Advances in Neural Information Processing Systems 35 (2022): 23716-23736.

4. despite with much fewer iterations, the performance gap between ablation model and the final model is not that significant (e.g., in Table 6, CogVLM achieves 142.8 COCO CIDEr, only ~4 CIDEr score less that the results in Table 3). So does this phenomenone implies that too much iterations in the two-stage training process are unnecessary?

More training iterations are important. Regarding dataset evaluations, the COCO dataset is relatively simple and ground truth is relatively fixed in style and mostly includes common everyday scenes. Thus, continued training does not show significant improvement, indicating a need for more challenging benchmarks in the research community, like MM-Vet. We observed that our model's performance continued to grow with training, ultimately far surpassing other models. CogVLM achieved a new sota of 52.8 for MM-Vet, which will be updated in our paper. Please see our comment titled "Result for hard datasets" above.

5. The visual expert in CogVLM includes FFNs in both attention block and FFN block. Which one is more important for better performance?

The ablation experiments in Table 6 suggest that the FFN plays a slightly more significant role than the attention layer. Adding a visual expert every four layers also yielded competitive results. Therefore, choosing a model structure will depend more on the balance between desired model performance and training efficiency.

6. Will the corrected dataset be made publicly available?

Yes, we will release the dataset if you think this behavior can benefit the community.

Dear Reviewer: Thanks again for your careful and valuable comments! Since the rebuttal discussion is due soon, we’ll be appreciated to know whether our replies have addressed your questions. If there are any further clarifications required or any other concerns, please feel free to contact us. Many thanks!

Thank you for your comprehensive review and insightful comments. Please allow us to address your points in detail:

1. The main concern to me about this paper is its limited novelty and scientific merit. First of all, the dense interaction between vision and language tokens has been heavily studied prior to the so-called multimodal LLM era.

Our response to this concern is in 1. of "response to common concerns" above. The dense interaction for vision and language is definitely not a new idea, but in the current LLM+ paradigm, it is very valuable to rediscover its effectiveness as a way to better scaling up.

Furthermore, our research extends beyond the mere application of this idea. Through comprehensive ablation studies, we have rigorously examined the influence of various model structures and training strategies on the final outcomes. This includes an in-depth analysis of often-neglected components such as the design of RoPE for image and text sequences, the construction of attention masks, the equilibrium between trainable parameters and model efficiency, and the effect of image self-supervised loss.

It's important to emphasize that each experiment in the realm of large model training is resource-intensive. Therefore, our contribution in identifying and refining effective training methodologies holds significant value in the field.

2. I can hardly tell what the researchers should proceed to further improve the performance. Do we need better model design, or more data and computations?

Thank you for your advice. We will add a discussion section in the final version if accepted.

Yes, scaling up is definitely one of the most important ways to increase performance in our opinion. As shown in the Ablation study part, scaling up the model with visual expert is very effective, and the fewer the parameters in the visual expert, the worse the performance. As you say, how to ensure the expected performance after scaling up may be related to model design (like visual expert) or data etc., which is our aim for investigation.

3. the authors also used some in-house data, which I guess cannot be released to the public. Given the barrier of reproducing the reported results and also the limited insights delivered by this work, I am sharing a huge concern regarding the current trend of building multimodal LLMs manifested by this work or other related ones.

This is a misunderstanding. All the results except on touchstone are only trained on public datasets, and reproducible. We answer this in 2. of "response to common concerns" above.

Dear Reviewer: Thanks again for your careful and valuable comments! Since the rebuttal discussion is due soon, we’ll be appreciated to know whether our replies have addressed your questions. If there are any further clarifications required or any other concerns, please feel free to contact us. Many thanks!

This study leaves me somewhat unconvinced that the improvements are primarily due to the architectural alterations. I have three primary concerns regarding this:

1. The unavailability of training data raises questions. The approach takes advantage of substantial proprietary training data during pre-training, which introduces the possibility of test data leakage during this phase.

2. The use of an oversized visual encoder (4.8B) seems unfair in the comparison. It would be more insightful to use standard vision transformers like ViT-L, g, or G for a more balanced comparison. This calls into question whether the improvements are largely due to the colossal visual backbone.

3. The unavailability of the training code raises doubts about potential tweaks during the training process. It would be more reassuring if the full training procedure could be reproduced, eliminating any uncertainties about the process.

Therefore, unless these problems are solved, I think it is not unconvincing for publication, personally.

1. Firstly, the training data used for model pre-training is transparent in their paper, such as LAION and COYO-700M datasets. The majority of current multimodal models have utilized these datasets, either during the training of the visual encoder or in the image-text alignment phase. Your concerns about data usage are unfounded and somewhat offensive. Secondly, it's plausible that the research team implemented strict separation between training and testing data, ensuring that the data used during the testing phase is completely independent of the training set. You only need to use the model's demo to find that its effect is indeed very strong.
2. I think the use of a 4.8B parameter visual encoder was to explore and demonstrate the potential of model performance at a high parameter scale. This doesn't imply that smaller models couldn't achieve similar improvements. In fact, the authors have conducted a comparative analysis with an OpenCLIP-Large model, as highlighted in their response to reviewer c9aL. The outcomes of this comparison reveal only a marginal difference in performance (84.7 vs 83.4 on VQAv2), which further underscores the point that substantial advancements are not solely contingent on the size of the model.
3. The paper in question likely provides ample detail, enabling interested researchers to effectively replicate the model. Additionally, the authors have made significant efforts towards transparency and accessibility by releasing the training code and model checkpoints. For those interested in delving deeper into the specifics of their work. This proactive approach by the authors greatly facilitates further research and exploration in the field.

Therefore, since these problems are solved, I think it is convincing for publication, personally.