Contrastive Localized Language-Image Pre-Training

Thank you for taking the time to review our paper and provide your feedback.

We appreciate your comments, but we believe there may be a significant misunderstanding. Please note that we have made every effort to ensure that every word and claim in our manuscript is both ethical and true to the principles of our research community. Please find our response below:

• “This paper would have been strong if other works did not explore better pretraining over CLIP for open vocabulary computer vision tasks. However, that is not true anymore.”

We would like to clarify that our work is fundamentally different from open-vocabulary computer vision tasks. Our primary contribution lies in enhancing the localization capability of the CLIP encoder, transforming it into a robust representation for downstream tasks, particularly MLLM tasks. Additionally, to support large-scale pre-training, we developed a visually enriched and spatially localized captioning framework to effectively generate region-text pseudo-labels at scale. We demonstrate that CLOC enables the creation of high-quality regional embeddings for image region recognition and retrieval tasks. Furthermore, it serves as a drop-in replacement for CLIP, significantly enhancing MLLMs, especially in referring and grounding tasks.

We believe the distinction between our work and open vocabulary CV tasks should be evident. We also want to emphasize that novelty does not necessarily mean creating something entirely new. Introducing a new idea, examining existing approaches from a fresh perspective, or developing a method that advances the current state of knowledge can all qualify as novel contributions. We proposes a new encoder pre-training goal, promptable embedding, that is more suitable for the downstream MLLM. We design a prompter architecture on top of CLIP that enforces the image encoder to generate image embedding that is easy to be transformed into a region representation given a visual prompt, thus more suitable for MLLM to leverage. This perspective for improving CLIP pre-training for MLLM is novel.

• “This work ignores all the advancement in the field of pretraining image-text representation models.“

We respectfully disagree with the assertion that we are ignoring others' related work. We discuss the related work in the related work section (Section 2). To summarize, we appreciate the reviewer to list some work here, some of them are indeed related and we are happy to discuss more in the future version of the paper. However, we believe the reviewer has a serious misunderstanding of the concepts and our work. Our goal is very different than the listed work above and unique. As stated in the paper, we first improve MLLM by enhancing image encoder pre-training with better localization (Lines 49-52). Second, we explore how effectively pseudo-labeling images can scale up to region-level contrastive learning (Lines 53-86). We thank you for mentioning other works in the review. We will include a more comprehensive discussions on the additional related works in the next revision.

• “Several works in combining SSL with ITC exist and show better localization performance e.g. MaskCLIP and SILC“

We are unsure why the reviewer has linked our work to SSL (assuming it is self-supervised learning), as the two are entirely orthogonal. We acknowledge that combining SSL with ITC can potentially also improve localization performance; however, our work is not about combining SSL with ITC. As noted in Line 53: “we explore a data-driven approach that complements the original CLIP image-text pre-training objective with explicit region-text supervision.” Our method does not prevent adding another SSL objective with ITC, but in this work, we mainly focus on combining CLIP with region-text supervision with a scalable re-captioning framework.

• “Moreover, BRAVE at ECCV2024 showed that these works also improve MMLLM performance.”

We believe the reviewer may have misunderstood the concept of BRAVE. The BRAVE paper does not make an argument that these encoders alone improve MLLM performance. In their Table 1, they only “benchmarked” existing public checkpoints, which “differ in terms of objective, training data, and model size” as noted in the caption of their Table 1 as well. BRAVE demonstrates that consolidating features from multiple existing frozen encoders into a more versatile representation can achieve superior performance. Their results show that MLLM performance can be significantly enhanced when better representations contain signals from diverse datasets and training objectives. This insight strongly aligns with our work, which focuses on incorporating region-level signals into image-level contrastive learning to further improve MLLM performance.

• “Moreover, the primary idea of this work has also been explored in "LocCa: Visual Pretraining with Location-aware Captioners" recently."

Thank you for pointing out the LocCA paper. While it is indeed related, we do not understand why the reviewer believes that the primary idea has already been explored in LocCA. The two models are fundamentally different. LocCA incorporates region-level (location-aware) information into a captioning objective, whereas CLOC primarily focuses on utilizing it within a contrastive objective. Additionally, the methods for constructing training data and the evaluation strategies differ significantly between the two approaches.

• “While I commend the authors for doing an extensive study of their method vs CLIP, we have a different landscape today. It is not fair to claim large improvements without putting other improvements from the field in context.”

We respectfully disagree with this statement. We did not claim large improvements over all other works from the field but only a fair comparison to CLIP, with the same images and training setup. Previous works have mainly explored either SSL objectives due to lack of large-scale datasets (Lines 75-80) or different models than CLIP. However, as for the MLLM community, CLIP is still arguably the default encoder choice (Line 36-49, Lines 978-981). We humbly believe our work actually fills in a missing piece and can be complementary to the current pre-training landscape.

• “Moreover, while the authors improve upon CLIP, their performance metrics on imagenet or LVIS fall short compared to other works from the field.”

Again, we respectfully disagree. We believe our performance is competitive in the same setup. For instance, regarding ImageNet accuracy with B/16, OpenAI’s CLIP reports 68.3%, SigLIP [ICCV 2023] reports 76.0%, MetaCLIP [ICLR 2024] reports 70.8%, and our model achieves 75.5%. As for LVIS, we did not find zero-shot region classification accuracy explicitly reported in other works, and our focus is not on LVIS detection. Please refer to the paragraph starting at Line 806 for additional details. But more importantly, the primary goal of our method is to enhance region-level metrics rather than image-level metrics, as we stated in the beginning.

We thank the follow-up response from the reviewer. After the fruitful discussion, we respect the reviewer’s arguments. However, we still respectfully disagree with the unreasonably negative rating on our paper. We believe a rigorous ablation against CLIP to prove the effectiveness of a simple approach is valuable to be presented to the research community. This is not against the larger scope mentioned by the reviewer to consider either comparing to or combining with another encoder pre-training method from a broader family, which we will keep investigating in the future.

We have some points to further clarify. We do have the budget to train more models but for a fair comparison, we need to re-implement or reproduce the baseline methods since our implementation is in JAX (as mentioned in section 5.1). Also, for many works including MaskCLIP and SILC, we have not found their training implementation to be released. For the numbers of LVIS on our paper and OWL-v2 paper, again as we stressed, they are both zero-shot but for completely different tasks and metrics. Ours in Table 2 is zero-shot region classification, while OWL-v2 paper reports open-vocabulary detection average precision. The numbers cannot be compared.

We thank you for your positive review and questions!

For UMG-CLIP, we did have mentioned it in footnote 1 (Line 269). At the time of the submission (9/28), it is shown as a withdrawn arXiv preprint and was later updated on 10/29. We are more than happy to add UMG-CLIP as an official reference and include more discussions about related works in the final version. UMG-CLIP uses a region-level contrastive loss similar to our Equation 2. Also, their model extracts region embedding by a RoI Align layer, which we actually have compared as the baseline for our prompter design (discussion in the paragraph “Extracting region embedding with visual prompts” in Line 252). We have ablation experiments to support it — the second item of observations of Table 2 around Line 421 and in row 3&4 in Table 3. Besides, UMG-CLIP is fine-tuned from CLIP on its 41M image datasets, which our work considers scaling up to 2B images (see Table 1) and pre-training from scratch.

About the region-text annotations, as mentioned in the reproducibility statement, we are working hard on releasing the annotations with internal legal approvals for the public.

[W1]

We agree that the labeler is always important for pseudo-labeled data. The goal of our work is indeed to explore how to effectively scale up the training data (the paragraph from line 82). The focus of the proposed visually-enriched and spatially-localized pipeline in section 4 is how to best leverage off-the-shelf models for region-text pre-training data. We think our report is a positive motivation for both the grounding, captioning (also will be discussed in [W2]), and CLIP pre-training community, as we show that a proper combination of them can benefit our CLOC pre-training.

[W2]

We want to clarify that we do not claim the contributions of using better image captions to improve multimodal tasks like Thao et al. and the VeCap paper we followed (line 322, line 451). In fact, our report embraces the advance of better captioning for MLLMs. As we noted in the future directions (line 958), we think our work provides another motivation for researchers to develop better re-captioning, which we think should be considered a strength rather than a weakness of our paper.

For X-VLM, we thank you for providing the reference and we will update our paper. X-VLM is an early pioneer in studying combining several losses like contrastive learning, matching, and masked language modeling by training on only 16M images. In contrast, we scale up pseudo-labeled data up to 2B images. In terms of integrating regional information into training, we want to point out that CLOC’s design does make a significant difference. X-VLM extracts regions from the image feature by the corresponding spatial patches similar to RoI pooling, which we have compared as a baseline approach of our prompter design discussed in the paragraph “Extracting region embedding with visual prompts” (line 252). As discussed in the second item of observations of Table 2 around Line 421, our prompter design is more effective in the scale we considered, observing 12-27% region task accuracy difference compared to the RoI baseline. In the Ferret experiments (row 3&4 in Table 3), pre-training with RoI-Align is worse than CLOC prompter by 4.6% on Ferret VQA tasks.

[Q1]

Good question. CLOC improves on CLIP with region embedding training thus region classification/retrieval (Table 2) and referring and grounding VQA tasks (Table 3, 4) are more direct evaluations. For general VQA, the performance gains are somewhat indirect and depend on many factors. We hypothesize LLaVAW and MME are relatively new benchmarks that are more challenging in targeting out-of-domain generalization and more diverse in terms of visual concepts, while GQA is constructed based on Visual Genome that was widely studied and may be better considered already in the SFT data of the LLaVA models.

Thank you for the detailed feedback and questions!

• Related works.
  • We thank you for providing many related works and we will incorporate them into the next version. We want to emphasize that all these works [^1 ^2 ^3 ^4 ^5 ^6] are improving for open-vocabulary (OV) object detection. Our goal is not to build a detection or grounding model but to pre-train a better CLIP encoder for MLLM tasks. As said in our discussions of RegionCLIP you have also pointed out (the paragraph from line 183 and the “Improving localization of CLIP” paragraph in section 2), that OV detection focuses on recognizing many objects and labeling them from a large vocabulary set. VQA focuses on holistic reasoning across the whole image via an MLLM. Also, as we discussed in the appendix “Why not use a local-enhanced encoder?” (line 972), these works mainly leverage CLIP for the OV detection task only. Our goal is to increase CLIP pre-training with region-level understanding without sacrificing the CLIP original image-language capability, which is crucial for MLLM tasks.
  • We believe these works are quite different and complementary from ours in terms of their methodology. We provide a concise summary here. DenseCLIP [^1] fine-tunes CLIP on a supervised pixel loss (e.g., with a segmentation dataset) but not changing CLIP pre-training. [^2 ^3 ^4 ^5] are mainly leverage CLIP as an external (frozen) classifier for expanding standard detection into open-vocabulary detection.
• We leave the answers for the implementation details and scalability to the same questions below
• About the the leaf entities in NER, we mean the noun phrases detected by the NER parser as shown in the example in the lower boxes in Figure 3. We provide the codes in Listing 1 in the Appendix. We will make this more clear in the final version.
• Questions about the region sampling strategy
  • The 4 regions are sampled randomly during training.
  • We adopt the OWLv2 detector that was trained with DETR-style loss [1], which naturally suppresses overlapping regions with the same object. Therefore, we do not specifically add assumptions on such cases.
  • We ablate sampling with 2 regions per image in item 12 in Table 2 and found sampling more (M=4) is slightly better.
• Questions about training Implementation
  • The intuition of \lambda we set is to make the additional CLOC loss (weighted by \lambda in Equation 4) proportional to the ratio of images that have region labels. For instance, if there are 80% of the images in a mini-batch have region labels, \lambda should be 0.8.
  • The images without region labels will only contribute to the CLIP loss but not the CLOC loss and grounding loss.
  • We did lightly tune the threshold in [0.5, 0.8, 0.9 0.95].
• Questions about scalability and efficiency
  • The region tasks in Table 1 are constructed mainly for ablation and development purposes (line 412-413). We think the zero-shot region tasks performance saturates since the tasks are relatively simple in terms (e.g., classifying from some fixed set of classes from COCO and LVIS). MLLM tasks require a higher level of understanding which might benefit from a larger scale of data.
  • Compared to RoI pooling, the Prompter is a lightweight single-layer self-attention layer. The extra computation cost is negligible compared to the whole forward-backward.
  • As discussed in the paragraph of Line 962, the extra overhead of the region embeddings is mainly for the contrastive loss as M times more examples, while this is an acceptable constant especially when training with a larger ViT. Note that, the main overhead is the VIT encoding and each image is only encoded once but not M times, which is the reason we design the prompter on top of the ViT (Line 245-247).

[1] End-to-End Object Detection with Transformers, 2020