LLM2CLIP: Extending the Capability Boundaries of CLIP through Large Language Models

Technical:

1. Section 3.1: The poor performance of LLMs in raw retrieval tasks is expected since they are trained primarily for generation. The LLM2Vec method makes them suitable for embedding and discrimination tasks. However, the paper does not explore or compare against LLM2Vec-transformed models, which could have provided a stronger evaluation baseline.

2. Section 3.1: The first of the three major contributions, "NATIVE LLMS ARE INEFFECTIVE TEXT ENCODERS FOR CLIP," lacks novelty. As LLMs are generally known to perform poorly in encoding tasks without further adaptation, this result is not surprising and applies beyond just CLIP.

3. Section 3.1: I don’t undertsand clearly why the text experiments, such as those done on Llama, help infer encoding capability within CLIP. The text-only experiments don’t seem directly relevant to the cross-modal task that CLIP targets.

4. Section 3.1: The paper claims to introduce "a new metric: the MS COCO Caption Retrieve Accuracy (CRA)," but this metric is not actually novel.

5. Section 3.2: The second contribution is essentially a replication of LLM2Vec, tested with different datasets. Much of Section 3.2 could have been covered in the introduction or related work sections, as it does not offer significant new insights.

6. Section 3.1 and 3.2: Both sections 3.1 and 3.2 could have been avoided if the authors used the LLM2Vec encoder, which is already cited in the paper. The reasoning for not adopting this approach remains unclear, and the decision should be validated with ablation studies.

7. Section 3.3: The justification for not training the base LLM is weak. The claim that large batch sizes are required for effective training is not well-supported, especially since similar methods like SigCLIP have demonstrated success without large batch sizes. Additionally, the use of LoRA (Low-Rank Adaptation) could have been applied to fine-tune the LLM without losing the knowledge base much.

8. Line 283: The details of the adapter module are under-specified, hindering reproducibility.

9. Section 4.1: The training hyperparameters are also not fully disclosed, further limiting reproducibility.

10. Section 4.2: The claim that knowledge from LLMs unlocks "richer visual representations" is primarily tested on image-text retrieval benchmarks. However, the paper lacks evaluations on classification and purely visual representation tasks, which are essential to validate this claim.

11. Section 4.2: Evaluation on purely visual understanding tasks is even more critical since the other compared models have much smaller text encoders compared to the one used in LLM2CLIP.

Writing and Presentation:

1. Abstract: It could be more concise with more clarity. It seems generated directly from the main content using LLM.

2. Figure 1: It is not easy to understand due to the use of multiple sub-images, and the caption is poorly aligned with the figure. Breaking this figure into multiple simpler visuals with clear explanations would enhance comprehension.

3. Table 2: the value "9.2" highlighted in the first column seems to be an error, as it is highlighted as the highest, which contradicts the data presented.

The recipe proposed by the paper lacks several crucial details:

• What is the design of the LLM adaptor? The paper only states that it's composed of "several linear layers". Is it an MLP? How deep is it? What activation function is used?
  • Moreover, the fine-tuning hyperparameters are not specified. What optimizer, learning rate (and schedule), etc., was used?
• The LoRA-based fine-tuning uses the same parameters as LLM2Vec, but that paper does not experiment with, e.g., Mistral Nemo. Were the training parameters for Mistral-7B used in that case? Overall, I think it would be good to report the training parameters fully for reproducibility.

In addition to the lacking details, the claim that the Chinese language experiment is OOD because they only trained models using English data does not really hold, as Llama 3 was pretrained with multilingual data.

Presentation-wise, the paper was somewhat hard to read and in a state closer to a draft than to a good submission. The paper also uses some expressions that are more appropriate for a product release rather than a scientific article, e.g., "like a miracle", "super text encoder", "audacious step", etc. Some typos and presentation improvements are listed below:

1. (Table 2) In the CLIP + LLM2CLIP / Flickr30k (I2T), the performance is only 9.2, which I assume is a typo as it's bolded.
2. (Table 2) The bolding in the ShareGPT4V (T2I) column and the first block are incorrect. The highest value is on the bottom row, while the penultimate value is highlighted.
3. On line 473, there is a missing table reference (Table X).
4. Bolding issue in Table 7 / Seed Benchmarks (Video).
5. The numbers on line 387 (51.7/50.6) do not match what was reported in Table 4 (32.3/39.3).
6. On line 178, the authors refer to "T5-V", while the method in the cited paper is named "E5-V"
7. The references should be updated, e.g. imu2clip is an EMNLP findings 2023 paper, but its arXiv version is being cited
8. On line 477, it's Mistral NeMo, not "Namo"
9. Line 104, MS-COCO is not cited correctly
10. Line 232, "replacing CLIP's text encoder and the corresponding ViT with Llama-3 8B". This sentence is not clear
11. The metrics are unclear, e.g., what does "@X" mean in Table 3? Recall@X?

1. Big claims, that feel not supported enough.

   1. “To validate the knowledge transfer capabilities of the LLM, we designed an out-of-distribution (OOD) experiment by conducting an image-text retrieval task in a completely unfamiliar language” (page 9) -- Add details on whether the LLM was trained on Chinese data.

   2. Too many marketing words that distract from the science.

      1. You refer to the encoder as “super text encoder” (page 5) in section 3.3, and section 3.3 only.

      2. “transformative leap in performance” (page 10)

      3. “the magical power of LLM2CLIP became evident” (page 9) -- It’s not magic, it’s science.

2. Writing/presentation is poor.

   1. Same sentences are repeated 2-3 times (e.g. “CLIP is one of the most important multimodal foundational models today” (page 1)).

   2. Sections contain information from other sections.

   3. References are not in order of mention in the text

   4. Add more details to all tables. What models are shown (what LLM size)? Some tables show big improvements, but they don’t indicate the increase in number of parameters for the text encoder versus the baselines.

   5. For Table4, “Table 4: Ablation Study of LLM2CLIP. Here LLM2CLIP-S refers to the results trained on the original short caption dataset.” (page 8), specify what the “Ablation Study” is. What was tested/compared here?

   6. (nit) Abstract feels very long, and does not seem to be a summary.

   7. “51.7/50.6” (page 7, mentioned in section 4.2) These numbers don't exist in that table.

   8. Typos:

      1. “Mistral Namo” (page 9) -> Mistral-Nemo

      2. “In Table X, we” (page 9) -- forgot reference

3. Missing references to existing works.

   1. “switching to bidirectional attention improves its ability to capture contextual information” (page 5) -- cite a work that proofs this, or show it?

4. Missing motivation versus existing works.

   1. Why use a next-token-prediction model, and then train it using Masked Language Modeling (MLM), instead of using an MLM as initialization? Larger Bert-style models are not compared with, but they are also not mentioned as an alternative (or if you don’t think they are, why?).

   2. Why not compare with existing text embedding models that are used for RAG? Given the cheap claimed final CLIP-training, and freezing the language model, taking an off-the-shelf embedding model from HuggingFace would be a great baseline.

5. Missing ablations for choices.

   1. Do we really need MLM, or would an adapter (with next-token prediction) be enough to get better embeddings if we finetune on the caption dataset? Is the reason we see low performance on caption retrieval because the embeddings are generally bad (current claim) or because the captions are not aligned well with the original training data?

6. Missing details

   1. Given you’re using open-source models for initialization of all weights, and the training is ‘light-weight’, can you share the training hyper parameters/recipe? Learning rate? Weight decay? That would allow others to reproduce it.

• The main weakness is that the main claim in the paper is not convincingly demonstrated. That the resulting LLM2CLIP model has conclusively take advantage of an LLM replacement for its text encoder without losing the original capabilities of CLIP. The closest result that could be used to demonstrate this would be only in Table 7 as it is the only set of results that test the proposed model on downstream tasks e.g. VQAv2, GQA, MMBench, etc. However, here the results are mixed at best, the proposed method only surpasses the baseline model in some cases, and the results are likely not significant e.g. 79.04 vs 79.68 on VQAv2. I am not insisting on state of the art results here since LLaVA1.5 is not the state of the art for most of these benchmarks but improvement from the proposed LLM2CLIP model is really marginal here.
• A key missing experiment is Imagenet top-1 zero-shot accuracy. One of the biggest challenges of replacing the CLIP text encoders is to do so without affecting zero-shot top-1 accuracy on Imagenet, as this is one of the tasks where CLIP excels the most, as it has a large pre-training on 400M image-text pairs. I would also argue that top-1 accuracy on Imagenet and recognition of large number of categories more generally is the biggest reason for adoption of CLIP as a vision encoder.
• I also consider the paper has some presentation issues although I rank this as less important. e.g. page 9 says in Table X. Also the tables are not in the order they are discussed in the paper and Table 7 which has the most important results in the paper uses a really tiny font. The organization of the paper seems to be in the order of how discoveries were made along the way instead of explaining more concisely how the proposed method works and perhaps provide justifications along the way, or later. It makes it really hard to understand what is being proposed until later in the paper or what are the key results. I also take some issue with wordings such as miracle and magical power in the last paragraph of page 9 as it reads a bit as overclaiming, especially the claim that Chinese is a completely unfamiliar language, as Llama clearly used other languages in training but a casual reader might miss this fact and be mislead by these statements. Instead I would encourage the authors to explain how is this being accomplished rather than just attribute it to magic.