FG-CLIP: Fine-Grained Visual and Textual Alignment

1. Response to Question 1 in Essential References Not Discussed

Thanks for your comments. We discuss the difference between FG-CLIP and FineCLIP and provide experimental results in the first response to reviewer 6zz8. We refer you to that response for more details.

2. Response to Question 2 in Essential References Not Discussed and Question 1 in Questions For Authors

Thanks for your suggestion to include a comparison with Alpha-CLIP on region level classification. We test Alpha-CLIP using its "clip_b16_grit1m+mim_fultune_4xe" weight across three testing configurations: Alpha Map, RoIAlign, and Crop&Resize. The Alpha Map method involves creating a mask based on the label box information and then combining this mask with the entire image, which aligns with Alpha-CLIP 's training objective. The RoIAlign method is a testing approach in our FG-CLIP. Meanwhile, the Crop&Resize method refers to cropping the image region corresponding to the box and resizing it to the resolution required by ViT. The table in https://anonymous.4open.science/r/ICML_RE-3CF6/compare_with_alphaclip.md shows FG-CLIP consistently outperforms Alpha-CLIP across different configurations. We will add these results to our manuscript.

3. Response to Major weakness

Thank you for your insightful comment regarding the methods of regional contrastive learning and hard negative sampling, as well as the novelty and contribution of our dataset.

3.1 Regional Contrastive Learning and Hard Negative Sampling

We have discussed the difference between FG-CLIP and FineCLIP in an earlier response. Here, we highlight distinctions between hard negative sampling of FG-CLIP and ALBEF.

ALBEF proposes a strategy to sample hard negatives for its ITM task. Specifically, ALBEF samples hard negatives using softmax-normalized image-to-text and text-to-image similarity to find in-batch negatives, selecting one negative text/image per mini-batch. In contrast, FG-CLIP conducts a novel pipeline to create challenging fine-grained negative samples. As shown in lines 236 to 251 of the right column of the manuscript, we modify the attributes of bounding box descriptions while keeping the object names unchanged. We generate 10 negative samples for each positive sample. This process generates subtle variations where objects may appear similar but differ in specific details.

3.2 Contribution of the Dataset

Our work leverages two high-quality datasets. In the first stage, we utilize an extensive dataset of 1.6 billion long caption-image pairs to capture global-level semantic details. In the second stage, we employ a carefully curated dataset of 12 million images with 40 million corresponding bounding boxes and captions, which are specifically designed to provide fine-grained annotations. We also generate 10 million challenging fine-grained negative samples, improving the model's ability to distinguish subtle differences. These datasets enable the model to achieve superior performance on various benchmarks. In summary, we have validated the entire synthesis pipeline used to create these datasets, including innovative techniques such as generating challenging fine-grained negative samples, which provide reference for others. We plan to make this dataset public to support further research in visual grounding and fine-grained understanding.

4. Response to Question 2 in Questions For Authors

To quantitatively discuss the diversity of our dataset, we compare it with other fine-grained datasets such as LVIS and V3Det. We extract and aggregate category labels from captions generated through steps involving CogVLM and YOLO-world. The following table compares the number of images and unique category labels across different datasets. Notably, even when sampling an equivalent number of images (243k), our dataset yields more unique category labels than V3Det, indicating higher diversity. We visualize the category labels of sampled data (equivalent to 243k images) using t-SNE plots in https://anonymous.4open.science/r/ICML_RE-3CF6/data_tsne_pic.png The visualization also shows that our dataset has a more diverse set of category labels at the same image scale. As the dataset scales up to 12M images, the diversity in category labels and captions increases significantly.

To defend the claim of our paper, we evaluate our model on LVIS. The results in the following table show FG-CLIP achieves SOTA performance.

5. Response to Question 3 in Questions For Authors

We conduct experiments on other LVLM benchmarks at https://anonymous.4open.science/r/ICML_RE-3CF6/add_result_mm_compare.md . The experimental results show that LLaVA with FG-CLIP achieves better prefomance.

1. Response to Weakness 1

Thanks for pointing out this problem. We perform ablation studies on the number of hard negative samples. Specifically, we test configurations with 1, 5, and 10 hard negative samples per positive sample. Our experiments show that 10 hard negative samples per image-text pair yield the best performance. We agree that this ablation study adds significant value to our manuscript and will include it in the final version. Thank you for pointing out this important aspect.

2. Response to Weakness 2

Thank you for your comment regarding the need for more visualizations of hard negative samples. We appreciate the opportunity to provide additional insights into how our method, FG-CLIP, benefits from hard negative sampling. We provide more visualization in https://anonymous.4open.science/r/ICML_RE-3CF6/fgshow.png . Specifically, we extract the dense image feature and visualize the similarity matrix to qualitatively analyze the impact of hard negative sampling. As illustrated in the figures, our FG-CLIP can capture the regions more accurately after performing hard negative sampling. For example, in the first row, the phrase "Man in red clothes" is accurately identified with hard negative loss, whereas without it, the model struggles to capture the correct region.

3. Response to questions in Other Comments Or Suggestions

Thank you for your valuable suggestion regarding the comparison of our dataset with previous ones such as LAION, COCO, and others. We agree that a detailed discussion in the form of a table would better highlight the unique strengths and contributions of our dataset.

In addition to our visual grounding dataset of 12 million images with corresponding bounding boxes and captions, we have also incorporated an additional 1.6 billion image-text pairs in the first stage of training. The dataset in the first stage is generated using a large multimodal model to produce higher-quality fine-grained long captions compared for capturing global-level semantic details.

We have compared our dataset with several related datasets, as shown in the table below. Overall, our dataset stands out in terms of scale and quality, particularly in its fine-grained annotations and challenging negative samples. Here are the key points of comparison:

• Scale: Our dataset contains the largest number of images, bounding boxes, and captions among all the datasets except LAION. While LAION has the largest number of captions (2B), the quality of these captions is often noisy and inconsistent. Our supplementary dataset adds 1.6B high-quality, fine-grained long captions, enhancing the overall utility of our data.
• Bounding Boxes: Among the widely used datasets, only COCO provides bounding box annotations. However, our dataset surpasses COCO by an order of magnitude, with 40M bounding boxes compared to COCO's 1.5M.
• Hard Fine-Grained Negative Samples: A distinctive feature of our dataset is the inclusion of 10M hard fine-grained negative samples. These samples help the model differentiate subtle differences in semantically similar pairs, thereby improving its performance across various downstream tasks.

We sincerely thank the reviewer for your constructive comments. We address your concerns below.

1. Response to Questions 1-3 in Experimental Designs Or Analyses and Weakness 2

Thank you for your comments regarding the improvement factor and fair comparison. The improvement of FG-CLIP stems from both the model architecture and the proposed new dataset. To support this, we provide the differences between FineCLIP and FG-CLIP. Moreover, we train our proposed dataset on FineCLIP to conduct a fair comparison.

1.1 Differences between FineCLIP and our FG-CLIP.

• A large-scale dataset with diverse captions can enhance FG-CLIP's fine-grained understanding capabilities. Specifically, our FG-CLIP integrates short captions with long captions, while FineCLIP only utilizes short captions during global contrastive learning. Moreover, we use a large-scale dataset (i.e., 1.6B+12M) rather than the small dataset of FineCLIP (i.e., 2.5M).
• FG-CLIP discards the self-distillation strategy used in FineCLIP, which introduces significant additional computational overhead. For example, when using a single GPU with a batch size of 32 samples, incorporating the self-distillation strategy increases memory usage from 25GB to 75GB during training and reduces FPS from 25 to 9.8. This is mainly due to additional feature extraction for each bounding box in the input images.
• Hard fine-grained negative sample learning helps FG-CLIP distinguish subtle differences in semantically similar pairs.

1.2 We then train our proposed dataset on FineCLIP. Due to time constraints, we perform this experiment on the 12M dataset, instead of the larger 1.6B+12M setup. From the table, the substantial improvements (Row 1 -> Row 2 & Row 2 -> Row 3) highlight that both our proposed dataset and model architecture are significant for FG-CLIP.

2. Response to Question 4 in Experimental Designs Or Analyses

Thank you for this insightful question. The qualitative results in https://anonymous.4open.science/r/ICML_RE-3CF6/partshow_pic.png show that FG-CLIP is able to capture different parts of images that are strongly related to the input text, demonstrating its fine-grained understanding capabilities. We follow the same experimental settings as Appendix C.1.

3. Response to Question 5 in Experimental Designs Or Analyses

Thank you for pointing this out. In the "hard" fine-grained task, only one attribute within the caption is replaced. Similarly, there are two replaced attributes within the caption in the "medium" task. However, regional contrastive learning primarily enhances region-text alignment in FG-CLIP, which may struggle to distinguish these fewer replaced attributes. To this end, we propose hard negative sample learning to further improve the overall performance of FG-CLIP. We will make this clearer in the manuscript.

4. Response to Weakness 1

Thank you for raising this point. We follow the same experimental settings as Appendix C.1 and further provide the qualitative results in https://anonymous.4open.science/r/ICML_RE-3CF6/fgshow.png . After performing hard negative sampling, our FG-CLIP can capture the regions more accurately. For example, the highlighted region of "Man in red clothes" with hard negative loss in 1st row shows significantly better than that without hard negative loss.

5. Response to Weakness 3

Thank you for this suggestion. We compare several baselines and FG-CLIP on the validation set of OpenImages. We conduct bounding box classification and follow the settings of COCO. The results further demonstrate the effectiveness of FG-CLIP’s fine-grained capabilities.

6. Response to Questions For Authors

Thank you for these insightful questions. For the first four questions, we refer you to our earlier responses. Here, we focus on addressing the 5th question regarding computational resources.

For researchers with fewer computing resources, we suggest discarding the first stage and directly training FG-CLIP on the dataset of 12 million images in the second stage. We conduct this experiment using 4×NVIDIA A100 GPUs, and the training process takes approximately 14 hours. Our experimental results in https://anonymous.4open.science/r/ICML_RE-3CF6/result_different_data.md indicate that this approach achieves slightly lower performance compared to the original settings but remains effective.

Another possible method is to distill a model based on our pre-trained weights, which significantly reduces the computational burden. We plan to release detailed guidelines and tools for implementing this distillation process in future work, making our approach more accessible to researchers with limited resources.

Thank you very much for your positive feedback and recognizing the non-trivial contributions of our work. In response to your specific questions, we provide detailed explanations below, aiming to clarify any concerns.

1. Response to questions in Experimental Designs Or Analyses

Thanks for your insightful comments and suggestions. Your points about considering long and short captions at both global image level and region-level boxes, as well as the introduction of hard negatives during global image level training, are indeed critical aspects that deserve further elaboration.

As mentioned in our manuscript (line 264), the captions used for region-level training are derived from the global long caption using the SpaCy tool. Specifically, we extract region-specific descriptions from the global long caption to generate a single type of text description for each region-level box. This method is efficient and already contains detailed information about the objects within those regions. Given that this approach adequately describes the objects, it provides sufficient context for regional contrastive learning without the need for additional caption types. Therefore, we only utilized this single type of caption for Regional Contrastive Learning. This streamlined approach not only enhances computational efficiency but also ensures that the model focuses on high-quality, relevant textual descriptions, leading to improved performance.

Regarding the consideration of hard negatives during global image level training, we acknowledge that this could potentially enhance the robustness of the model. However, there are practical challenges associated with implementing this approach:

• Limitations with Long Captions: We introduce the process of creating challenging fine-grained negative samples in Section 3.2. Specifically, we modify attributes of bounding box descriptions while keeping the object names unchanged. For global long captions, which describe multiple regions within an image, generating meaningful hard negatives by modifying attribute words for each region becomes complex. The resulting negative samples might deviate too far from the original caption, losing their effectiveness as "hard" negatives. Consequently, they may no longer serve as hard negative examples for training.
• Limitations with Short Captions: As illustrated in Figure 1 of our manuscript, global short captions often lack detailed descriptions of individual objects or regions. This makes it difficult to create meaningful hard negatives through attribute modification, as the short captions may not contain enough fine-grained information to make such modifications impactful.

2. Response to questions in Supplementary Material

This is indeed an interesting phenomenon and can be attributed to several factors.

The fine-grained hard negatives we created involve modifying only a few attribute words in the captions. In some cases, these modifications may still result in captions that are quite similar to the original descriptions of the corresponding image regions. This similarity can cause FG-CLIP to give relatively high scores to these hard negatives because the model perceives them as still being relevant to the image regions. Additionally, in certain scenarios, the modified attributes might correspond to minor changes within the image region, which do not significantly alter the overall visual content. Consequently, the model continues to assign high scores due to the minimal perceptual difference between the original and modified captions.

3. Response to Question 1 in Questions For Authors

Thank you for your question regarding how we sample K regions in a batch (Line 216). To clarify:

K is not a fixed number but rather represents the total number of valid bounding boxes (bbox) across all images within a batch. This means that K dynamically adjusts based on the actual number of available regions in the batch. This approach ensures flexibility and adaptability in handling batches with varying numbers of regions without introducing artificial constraints.

We acknowledge that this explanation may have been unclear in the initial submission. To avoid any potential misunderstandings, we will provide a more detailed clarification in the final version of our manuscript.

4. Response to Question 2 in Questions For Authors

We introduce this process in lines 263 to 272 of our manuscript. Specifically, we utilize SpaCy to parse the captions and extract referring expressions. These extracted expressions are then fed into a detection model to generate corresponding bounding boxes. Non-maximum suppression is applied to eliminate overlapping bounding boxes, retaining only those with predicted confidence scores higher than 0.4. This method allows us to effectively link textual referring expressions with their corresponding visual elements, facilitating more accurate and contextually relevant training for our model.