Contrastive Localized Language-Image Pre-Training

We thank you for the positive review and constructive comments.

UMG-CLIP

We thank you for pointing out the reference that we briefly compared in L363 left. We agree that there is some technical similarity between our CLOC and UMG-CLIP, but the goals and positioning of our work and theirs are quite different from the following perspectives.

UMG-CLIP is designed primarily for vision-centric tasks such as detection and segmentation, whereas our focus is on MLLMs (L33, right in the Introduction). Unlike open-vocabulary vision tasks, which involve a relatively small set of classes, MLLM VQA tasks require more extensive language understanding and thus demand large-scale pre-training data. This distinction also influences the design of our prompter architecture, which incorporates attention layers tailored to the downstream use cases of the LLM decoder (further discussed in L256, left).

Regarding the annotation pipeline, the primary difference lies in scale. UMG-CLIP fine-tunes a pre-trained CLIP model for dense vision tasks on a 41M-image dataset, whereas our approach pre-trains from scratch on up to 2B images. Additionally, unlike UMG-CLIP and RegionCLIP, which first predict bounding boxes and then run a captioner on each box, our VESL pipeline (Section 4) does not scale with the number of boxes, making it significantly more efficient for data annotation. Notably, annotating billions of images for our experiments required over 500+ GPUs for more than a week (L757, right).

Last but not the least, we see the UMG-CLIP fine-tuning approach as complementary to our CLOC pre-training rather than conflicting with it.

Open-world detection

We thank you for the advice on evaluating our encoder on open-vocab dense vision tasks. First, we want to emphasize that our original motivation for this work is for MLLM tasks with localization use cases such as conversational referring and grounding (e.g., Table 4), but not for dense vision tasks. And in Table 2, we have reported competitive results of zero-shot object recognition and retrieval given bounding boxes.

To further address your concern, for open-vocabulary detection, we provide additional zero-shot evaluation results on COCO Detection (minival), ODinW (test-dev), and LVIS-Det (minival). When comparing GLIP [1] and CLOC, we observe that CLOC consistently achieves better results than GLIP across all backbone categories (T / B / L), suggesting that CLOC offers advantages in localization and object detection performance. Notably, GLIP employs DyHead—a strong decoder/head module—on top of the encoder, whereas our ablation study uses only two simple heads for classification and regression. This further supports that the encoder representation in CLOC is indeed superior. See the table below for detailed results.

[1] Grounded Language-Image Pre-training, CVPR 2022.

We thank you for the positive review and constructive comments.

Unclear whether the performance gains stem from the use of more or cleaner data or from the pretraining schema itself.

In Table 2, we provide detailed ablations of our proposed ingredients on top of the CLIP we trained by ourselves, including the prompter design, the training labels generated by our pipeline. Note that, the CLIP model (row 2) we compared in the experiments was trained on the same image data as CLOC for a fair comparison.

Can this work be seamlessly extended to zero-shot object detection?

Yes, CLOC can be extended to zero-shot object detection. We provide additional evaluation results on COCO Detection (minival), ODinW (test-dev), and LVIS-Det (minival). When comparing GLIP [1] and CLOC, we observe that CLOC consistently achieves better results than GLIP across all backbone categories (T / B / L), suggesting that CLOC offers advantages in localization and object detection performance. Notably, GLIP employs DyHead—a strong decoder/head module—on top of the encoder, whereas our ablation study uses only two simple heads for classification and regression. This further supports that the encoder representation in CLOC is indeed superior. See the table below for detailed results.

A broader comparison with additional methods

Thank you for the suggestion. We agree that a broader comparison to more encoders would certainly be great. However, many previous models (LocCa, RegionCLIP) are trained on quite different data, labels, training cost, architectures etc. that might make it hard to draw a fair comparison directly, and some of them are not open-sourced. Therefore, we have limited the scope of our paper to the CLIP method and carefully ablated it in the same setting (e.g., training images, number of steps) as closely as possible.

More discussion and comparisons on training efficiency (e.g., with CLIP, LocCa)

We provide discussion in the “Training cost” paragraph (L789 left). Compared to CLIP, the extra cost is small for the object-level contrastive loss and the prompter — we observe about 10% more GPU time. Notably, the lightweight prompter operates on the image embedding that is shared across all the prompts within an image. The main overhead is to compute the image embedding through the ViT, which does not scale with the number of prompts. Compared to LocCa, our CLOC is much more lightweight since LocCa needs a full encoder-decoder transformer for autoregressive next-token prediction.

Attribute binding issue of CLIP

In this paper, indeed, we do not tailor to address the attribute binding shortcoming of the original CLIP directly. However, with our promptable embedding design (Figure 2 & Section 3.2), we think CLOC provides an alternative approach that allows the users to interact with our encoder to obtain a fine-grained embedding given a prompt of their interest by specifying a box location or an object description.

Minor fixes for Eq2 and Eq4

Thanks for pointing them out. We will revise them accordingly.

Will you release the pre-trained model (including the Prompter) to the public?

Yes, we aim to release the pre-trained model, and are actively working on that.

[1] Grounded Language-Image Pre-training, CVPR 2022.

We thank you for the positive review and constructive comments.

The main weakness of the work is the superficial ablation of the VESL captioning pipeline introduced.

Thank you for your constructive comments, and we will consider better examples in our figures. We agree with the reviewer that image captioning could suffer from hallucination, though it enriches the visual description. The reviewer is also correct that our pipeline filters such cases by relying on the object detector, which is quite effective as it was pre-trained on over thousands of common objects. A central design challenge for our pipeline is balancing the benefits of richer captions with the potential risk of hallucination. We believe our design is robust, considering the effectiveness of current state-of-the-art open-vocabulary object detectors and the inherent resilience of contrastive learning objectives to noisy text annotations in large-scale training.

In Table 2 (rows 3-5 vs. 13-15), we compare with the baseline that does not use the image captioner but the AltText (e.g., left of Figure 3). It could be hard to extract useful object phrases from AltText for the open-vocab detector.

For the hallucination issue, we have investigated our captioning pipeline and compared it with other recent works to show its superiority in terms of the hallucination. We compared it to public models: LLaVA-1.5, Shikra, MiniGPT-4, and InstructBLIP on CHAIR scores [1] (lower values refer to less hallucination). CHAIR_i measures the fraction of hallucinated object instances and CHAIR_s calculates the fraction of sentences containing at least one hallucinated object. The results are summarized below:

While the long captions from our captioning pipeline may still unavoidably introduce hallucinations even though our captioning pipeline has less hallucinations compared to other models, it can be further mitigated since we only consider confident objects agreed by the detector (L296 left). We also remove very generic words and stopwords, as noted in the code in Listing 1 in the appendix. We believe having more accurate object labels is the key to the improvements in Table 2 for our pipeline, as evidenced by the statistics 11.6 regions per image identified by the pipeline (only 5.1 for the baseline) we report in Table 1 (L275 right).

lines 260-262 (right column): are the pairs ignored or are the gradients on $f_T$ ignored?

For filtering region-text conflicts in Section 3.4, the region-text pairs are ignored in Equation 2. That is, these elements are “masked” and will not be considered in the contrastive loss matrix. We will make this clearer in the final version.

Remarks on CLIP

We apologize for any confusion regarding this statement. Our reference to “CLIP” in the context of MLLMs was intended to denote the broader family of language-supervised methods, including both CLIP (as a representative model) and SigLIP. Specifically, we were citing the first row block in Table 12 of Section D (Tong, 2024), which demonstrates that these methods outperform others, such as self-supervised approaches, in MLLMs. We will make this clearer and revise both the introduction and related works as you suggested.

Typos

Thank you for pointing out the typos, and we have fixed them in the revised manuscript.

[1] Rohrbach, A., Hendricks, L.A., Burns, K., Darrell, T. and Saenko, K., 2018. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156.

We thank you for the positive review and constructive comments.

The quality and diversity of region-text pairs depend heavily on the open-vocabulary detector and captioning pipeline.

Thank you for pointing out this important aspect. We agree that the quality of the open-vocabulary detector and captioner is important. Our pipeline is built upon the recent advances in these models (L275, right). We think the proposed framework is promising since, with better and better detectors and captioners introduced, the pipeline can seemingly enjoy the improvements, such as less bias and errors. We briefly discussed it in L782 (left column) of the Appendix, and will further emphasize the pipeline’s dependency on these models for pseudo-labeling in the final version.

To further address the reviewer’s concern, we use the hallucination metric (CHAIR score, where lower is better) to assess the quality of the synthetic captions. As shown below, our captioner demonstrates high quality compared to other models.

Text-based region prompts

In our experiments, we observed a reasonably low L1 distance of 0.02 between the predicted boxes and ground-truth boxes when the Prompter receives a text region description as input, indicating that text-based region prompts performed well out of the box. In the revision, we will include qualitative visualization examples to illustrate this. A more in-depth investigation is left for future work.

Region caption is short

The reviewer is correct that the region-level captions are much shorter than the image-level captions. However, we want to clarify that in Equation 4, we still retain the original image-level CLIP loss, ensuring that the image embedding quality remains on par with the original CLIP, as evidenced by the “Image tasks” results in Table 2.

Comparison to open-vocab detection models

First, in our paper, we included a comparison in Footnote 1 (L379, left), demonstrating that on the region classification task (predicting class names given a bounding box), our approach achieves over 70% mAcc on COCO, significantly outperforming the 47% reported in previous work.

For comparison with open-vocabulary detection models, we also provide zero-shot evaluation results on COCO Detection (minival), ODinW (test-dev), and LVIS-Det (minival). When comparing GLIP [1] and CLOC, we observe that CLOC consistently achieves better results than GLIP across all backbone categories (T / B / L), suggesting that CLOC offers advantages in localization and object detection performance. Notably, GLIP employs DyHead—a strong decoder/head module—on top of the encoder, whereas our ablation study uses only two simple heads for classification and regression. This further supports that the encoder representation in CLOC is indeed superior. See the table below for detailed results.

How bounding boxes are sampled during training

During training, we simply sample 4 boxes randomly (padded if fewer) per image.

Designs of the prompter

In our experiments, we mainly consider the prompter takes a bounding box or a single text embedding as the prompt. We compared with a baseline implementation of RoIAlign in Table 2 (row 4, 9, 14), and confirm that the proposed prompter is a better design (discussion in L360 right and also Section 3.4). For other designs, we consider the following as promising future work: (1) different types of prompts, such as points, masks, etc.; (2) multi-prompts or compositional prompts for higher-level prompting. We included more discussions in L770 “Future directions” paragraph.

LocCa reference

We thank the reviewer for this suggestion. LocCa is indeed a relevant work which we have cited and discussed in Section 2 (L100, right). However, LocCa differs significantly from our approach in two important ways: (1) it employs a full encoder-decoder transformer architecture, thus being substantially less efficient, especially for large-scale training; (2) LocCa embeddings do not directly facilitate zero-shot retrieval or classification tasks as our embeddings do (Table 2). The focus of our method remains specifically on improving CLIP-based localization capabilities, and we will further clarify this distinction in our final revision.

[1] Grounded Language-Image Pre-training, CVPR 2022.