FineCLIP: Self-distilled Region-based CLIP for Better Fine-grained Understanding

Q1: The performance of FineCLIP on both OV-COCO and OV-LVIS benchmarks is significantly lower than the original baseline of CLIPSelf. Why not directly increase the input resolution of FineCLIP instead of increasing the scale of the dataset?

A1: Thanks for your suggestion. We appreciate the opportunity to clarify the performance of FineCLIP in open-vocabulary detection tasks when adopting settings similar to CLIPSelf.

Following CLIPSelf, we train FineCLIP on COCO Train2017 split, using the region proposals from the trainset and region captions generated by BLIP2. After fine-tuning, FineCLIP's ViT parameters are used to initialize the F-ViT for downstream OV-COCO training. As FineCLIP is essentially a contrastive learning method, it needs a larger batch size than distillation methods for effective training. Due to the GPU memory limitations, we could only increase the training image resolution of FineCLIP to 640, which is much lower than 1024 used by CLIPSelf.

The results in Table 6 of the attached PDF show that despite FineCLIP's lower training image resolution, F-ViT+FineCLIP still outperforms F-ViT+CLIPSelf in $AP_{50}^{base}$ (57.8 vs. 54.9) and $AP_{50}$ (51.4 vs. 50.4) metrics. Due to the time limitation, we could only make this validation on OV-COCO benchmark.

Moreover, we believe that the settings for downstream tasks in our paper are adequately impartial. Even if the scores of different methods might be lower than those reported in original papers, these scores remain fair and reliable for comparison. We understand your concern that the current settings may not fully exploit the potential of distillation methods. However, we also have to avoid the risk that high training resolution adversely affects the training stability and effectiveness of contrastive learning methods. In this paper, we thus choose to increase the scale of the dataset instead of increasing the input resolution of FineCLIP.

Q2: According to Table 5, all the fine-tuning strategies cause a performance drop to some extent compared to the original pre-trained CLIP on all the image-level benchmarks, which cannot demonstrate FineCLIP's ability to preserve the power of image-level inference.

A2: Good question. We believe the performance decline on image-level benchmarks is caused by both training data and method factors. Therefore, we cannot ignore the impact of the training data, nor can we claim that the method is ineffective based sorely on the performance decline. This is crucial for us to determine whether FineCLIP has a positive or negative effect on image-level task. Considering that different methods used the same training data during fine-tuning, comparing the performance of methods after fine-tuning is sufficient to determine which method is better method. According to Table 5, FineCLIP comprehensively outperforms CLIP after fine-tuning, indicating that FineCLIP maintains stronger global representation capabilities than CLIP. Therefore, we can also reasonably infer that FineCLIP has the potential to exhibit better image-level representation capabilities after pre-training compared to pre-trained CLIP.

Q3: Considering that the supervision of self-distillation heavily relies on the quality of the image-level representation of the updated visual encoder of FineCLIP, the image-level performance suffers from a significant drop when compared to the frozen encoder. It's hard to conclude that the updated visual encoder can benefit the self-distillation process.

A3: Good question. To begin with, it is important to note that the evaluation on zero-shot retrieval tasks has biases, as it focus more on measuring generalization ability, rather than the representation capability in a specific data domain. For instance, as shown in Table 8 in the appendix of our paper, FineCLIP fine-tuned on COCO Train2017 greatly outperforms the pre-trained CLIP. Consequently, we believe the image-level performance of FineCLIP's updated encoder is significantly improved in the relevant data domain during the training.

A significant issue of the frozen pre-trained encoder is that it cannot learn from training data. In small-scale training scenarios, such as distillation, the frozen pre-trained encoder works well. However, as the training data scale expands, these methods would quickly hit bottlenecks, as indicated by the green line corresponding to CLIPSelf in Figure 2. In contrast, the updated visual encoder in our FineCLIP could be consistently improved during the training, thereby better surpporting the self-distillation process.

Furthermore, based on extensive experiments presented in our paper, FineCLIP with an updated encoder surpasses CLIPSelf with a frozen encoder in most evaluation metrics, clearly demonstrating the effectiveness of FineCLIP.

Q4: Perhaps the proposed strategy could potentially be beneficial to large-scale vision-language pre-training from scratch given the observations of fine-tuning on CC2.5M, but the current results of this paper are not very convincing to the reviewer.

A4: We believe the distinction between per-training and fine-tuning primarily lies in the scale of training data. Even when FineCLIP is initialized with pre-trained CLIP parameters, its training stage can also be considered as pre-training, if the data scale is sufficiently large. Therefore, there is no fundamental difference between pre-training and fine-tuning. In Section 4.2, by continuously expanding the data scale, Figure 2 shows that FineCLIP demonstrates excellent scalability in both global and local representation capabilities, indicating that our method has great potential to build large-scale vision-language pre-training models.

Q5: For Cat-Seg+CLIPSelf and Cat-Seg+FineCLIP, is the region defined by the generated proposals or patches (grids)?

A5: To ensure fairness, all models trained on CC2.5M utilize generated region proposals.

We greatly appreciate your constructive comments and suggestions. We'd like to address your concerns regrading the image-level performance of FineCLIP with further clarification.

It should be noted that Table 5 presents the performance of models on zero-shot retrieval benchmarks. These zero-shot benchmarks (i.e., Flickr30k and MSCOCO) involve the evaluation data that is somewhat out-of-distribution w.r.t. the trainset. As you have pointed out, we had also noticed that the domain gap between the trainset and evaluation data might degrade the model's zero-shot performance when we were preparing this paper submission.

Importantly, as shown in Figure 2, we find that data scale is crucial for zero-shot capabilities, and both CLIP and FineCLIP demonstrate excellent scalability. In this paper, we thus chose to use the largest dataset within our capacity, CC2.5M, as the trainset to preserve the model's zero-shot ability as much as possible. According to Table 5, the performance drop for CLIP or FineCLIP is not that significant. Given the scalability of CLIP and FineCLIP demonstrated in Figure 2, we can reasonably infer that their zero-shot image-level performance could be further improved and even surpasses that of the pre-trained CLIP by increasing the trainset scale.

Moreover, it should also be noted that Table 5 does not show the model's performance on in-domain data. According to Table 8 of the Appendix, when models are trained and evaluated on COCO, FineCLIP's performance on retrieval tasks significantly surpasses that of pre-trained CLIP. This result indicates that FineCLIP greatly enhances image-level representation on in-domain data, enabling it to better benefit the self-distillation process than pre-trained CLIP.

We hope that the above clarification could address your concerns. We are looking forward to your feedback if there is still any confusion. Thank you very much again.

Q1: With the same COCO validation set, training with CC2.5M shows worse performance on retrieval task but better top1 box classification accuracy. Any idea about this result?

A1: Good question. We believe this result can be explained by the differences between retrieval and box classification tasks, as well as the effectiveness of FineCLIP in utilizing region-text pairs to enhance local feature extraction capabilities.

i) Task Complexity Differences. From the perspective of text descriptions alone, there is a significant difference in class complexity between retrieval and box classification tasks. The retrieval task involves diverse global text descriptions that are flexible combinations of many objects, making it highly sensitive to the overall image content and corresponding textual content and style. This data distribution gap for retrieval task between trainset and testset is hard to eliminate. In contrast, the box classification task focuses on identifying a relatively smaller number of local objects with clearer concepts. Therefore, it is less affected by the zero-shot setting, which means that if the trainset includes objects from testset and the model has excellent local content learning capabilities, the model can still perform well on the testset.

ii) Effectiveness of FineCLIP. According to Figure 2(a) in our paper, FineCLIP greatly outperforms RegionCLIP and CLIPSelf, both of which also utilize region information. This result indicates that the region-text pairs generated using YOLO and BLIP2 effectively cover objects in testset, and FineCLIP has superior local feature learning capabilities compared to competing methods.

iii) Impact of Trainset Size. Additionally, we observe that when using the same scale of 100K training samples, FineCLIP under zero-shot setting does not surpass that of using in-domain COCO train2017 split in either task. However, as the data scale increases, FineCLIP achieves superior performance in box classification task, demonstrating FineCLIP's outstanding scalability.

Q2: Will different region proposal methods have a big influence on the final results?

A2: Thanks. Based on COCO Train2017 split, we evaluate four different region proposal methods: manual annotation $^{[1]}$, FastSAM $^{[2]}$, RPN $^{[3]}$ and YOLOv9 $^{[4]}$, to show their impact on the performance of FineCLIP. The results are detailed in Table 5 of the attached PDF document. From the obtained results, we find three key insights below.

i) Automated vs. Manual Proposals. Automated region proposals yield results comparable to manually annotated high-quality regions. They slightly perform better in box classification and worse in retrieval tasks, proving the feasibility of automated methods.

ii) Box Quantity: More boxes do not necessarily mean better performance. FastSAM generates the most boxes, but they appear too cluttered upon manual inspection, leading to poor model performance.

iii) RPN offers a moderate number of region proposals, resulting in balanced performance. YOLOv9, focusing more on specific object categories, produces fewer but more precise boxes, achieving the best box classification performance.

Overall, different region proposal methods indeed have a big influence on the final results. In this paper, we select YOLOv9 because of its efficiency and its significant enhancement of local feature extraction.

References:

[1] Microsoft COCO: Common Objects in Context.

[2] Fast Segment Anything.

[3] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.

[4] YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information.

Q1: As a common practice, the contribution of self-distillation scheme is not novel. What are significance and advantages of "real-time" capability of the self-distillation scheme? Why does using only $L_{SD}$ for supervision lead to model collapse?

A1: We fully understand your concerns. To better illustrate the novelty of FineCLIP, we will answer the above questions together.

Self-distillation is indeed a common practice, typically involving the distillation of some capabilities from a frozen teacher model to a trainable student model, as done in CLIPSelf. However, this implementation is only suitable for further fine-tuning of CLIP, and fundamentally conflicts with multi-modal pre-training scenarios. This conflict mainly arises from two aspects: i) This implementation requires a pre-trained teacher model as a prerequisite. ii) The frozen teacher model limits the performance ceiling of the student model, whereas the goal of pre-training is to continuously improve the model. Given that self-distillation has shown great potential in enhancing local feature extraction abilities, the value of our real-time self-distillation scheme lies in effectively integrating the self-distillation strategy into the pre-training process, fully leveraging its advantages to build a more powerful pre-trained model.

In terms of implementation, our real-time self-distillation scheme enables a trainable model to act both as the teacher and the student, achieving true "self-distillation". The novelty of this scheme is that it resolves the conflicts mentioned earlier: i) This scheme does not require a frozen teacher model, allowing the model to realize self-guidance. ii) During the pre-training process, the model continuously improves its global representation capabilities, thus more effectively guiding local feature extraction and breaking through performance ceilings, embodying the value of "real-time". We have demonstrated the effectiveness, stability, and scalability of FineCLIP through extensive experiments.

It is important to note that this scheme is tailored for multi-modal pre-training and relies on global contrastive learning (also denoted as "real-time" in our paper) to work effectively. When only $L_{SD}$ is used for supervision, the lack of global contrastive learning to maintain the model's global representation capabilities leads to the degradation of semantic meaning in model's global embeddings and local features. Consequently, the uni-modal alignment enforced by $L_{SD}$ becomes meaningless, ultimately causing the model collapse.

Q2: Sending all segmented local images into VLLM to generate text descriptions does not seem very efficient, especially when scaled to larger datasets.

A2: Good question. We answer it from the following three aspects:

i) Demand for data scale and quality. In the era of large models, the scale and quality of data have been proven to be critical factors affecting the performance of pre-trained models$^{[1]}$. Therefore, expanding the data scale and improving the data quality is an inevitable trend. Compared to manual annotation, using LLM or VLLM for automated data generation is much more efficient.

ii) Successful cases of large-scale data annotation with large models. In the field of LLM, using LLMs for large-scale pre-training data synthesis and cleaning has become a standard practice$^{[2,3]}$. In the contrastive learning domain, LaCLIP$^{[4]}$ utilizes LLM to augment the textual content of LAION-400M dataset$^{[5]}$, processing data on a scale larger than that involved in our work.

iii) Efficiency of data annotation with large models. In our response to common questions, we show the time cost of data preparation. Given the relatively small amount of data used in our experiment, further optimizations on the inference speed are not performed. However, deploying models using the vLLM$^{[6]}$ framework could significantly increase the throughput of data generation, potentially improving processing speed by at least 20 times.

In summary, using LVLM for large-scale data annotation is not only feasible but also advantageous in efficiency.

Q3: As shown in Table 4, CatSeg+FineCLIP shows only a minor improvement compared to CatSeg+CLIPSelf, which is not sufficiently support the author's claim of fine-grained semantic alignment.

A3: We appreciate the opportunity to clarify the performance of FineCLIP. We believe that the improvements brought by FineCLIP to CatSeg are comprehensive and significant, adequately demonstrating FineCLIP's superiority in fine-grained semantic understanding. Please see our detailed explanations blow.

On one hand, according to Table 4 in our paper, CatSeg+FineCLIP surpasses CatSeg+CLIPSelf on all metrics. On the other hand, we calculate the average improvements brought by CLIPSelf and FineCLIP to CatSeg on mIoU and mAcc across three benchmarks for straightforward comparisons, and show the results in Table 7 of the attached PDF document. We can find that the improvements provided by FineCLIP significantly exceed those brought by CLIPSelf, particularly when using the ViT-B/16 backbone.

Q4: The effect of using $L_{RC}$ alone should be supplemented.

A4: Thanks for your suggestion. We present the results of using $L_{RC}$ alone in line #4 of Table 4 in the attached PDF document. We can observe that regional contrastive learning benefits the model's local feature extraction abilities and can effectively combine with global contrastive learning and self-distillation scheme to obtain the optimal fine-grained understanding performance.

References:

[1] Scaling Laws for Neural Language Models.

[2] The Llama 3 Herd of Models.

[3] Qwen2 Technical Report.

[4] Improving CLIP Training with Language Rewrites.

[5] Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs.

[6] Efficient Memory Management for Large Language Model Serving with Paged Attention.

Q1: The authors could provide more semantic analysis to better explain the performance of FineCLIP on downstream tasks.

A1: Following your suggestion, we add semantic distribution statistics in Table 1 of the attached PDF document. The results show that the training data in CC2.5M related to the labels in PC-59 has the lowest proportion, which explains why training on CC2.5M does not significantly improve model performance on PC-59. Furthermore, ADE-847 has more than half of all 847 categories being irrelevant or only weakly relevant to CC2.5M, making it the most challenging benchmark.

Q2: Under what conditions does add $L_{SD}$ help? It's confusing why all three losses in row 5 of Table 1 don't give the best retrieval performance.

A2: According to the results shown in Table 1 (line #2 vs #3, line #4 vs #5) of our paper, $L_{SD}$ improves local feature extraction but slightly reduces retrieval performance.

Importantly, we conduct further validation by using trainset in different scales, with results shown in Table 3 of the attached PDF document. We observe an interesting trend: as the scale of trainset increases, the negative impact of $L_{SD}$ on retrieval tasks gradually diminishes. This means that $L_{SD}$ tends to have a positive impact on retrieval performance when FineCLIP is fine-tuned with larger-scale data.

Q3: Why did the authors choose to scale the dataset down rather than up and see how it affects performance?

A3: Due to the limitations in computational resources, we could only afford to experiment with a total of 2.5M training samples. With our current experiment setup, using a single server with 8xA800 GPUs, the experiments in Section 4.2 alone would take at least two weeks. If we increase the training samples from {100K, 500K, 2.5M} to {1M, 5M, 25M}, the experimental duration will extend to at least two months, which is far beyond our capacity.

Q4: Can the authors explain the performance discrepancy between CLIPSelf in Table 3 in the original paper versus this work?

A4: The training image resolution is the key factor of this performance discrepancy. Specifically, in our paper, the training image resolution was standardized to 224 for ViT/B and 336 for ViT/L, whereas the original CLIPSelf used a resolution of 1024. CLIPSelf, an efficient distillation method based on pre-trained CLIP, allows the input image resolution to be increased to 1024 due to its low computational costs. According to Table 1(c) in CLIPSelf paper $^{[1]}$, increasing resolution from 320 to 1024 boosts Top 1 Mean Accuracy by 25.6%. However, contrastive per-training methods have much higher GPU memory costs, requiring a trade-off between batch size and input image size, making it challenging to achieve such high resolutions. To ensure a fair comparison, we used the default image size employed by CLIP, resulting in a performance decline for CLIPSelf in this work compared to the original paper.

Q5: What are the domains specifically in the "domain shift" in Line 26?

A5: The term "domain shift" in Line 26 follows the usage in RegionCLIP paper $^{[2]}$. It means that CLIP aligns global image contents with corresponding global text descriptions, but cannot realize well fine-grained alignment between local image regions and corresponding local textual concepts. This lack of fine-grained alignment results in suboptimal performance on downstream tasks requiring fine-grained abilities. We understand your concern that "domain shift" might be too abstract in this context. "Task shift" might be a more accurate expression.

Q6: What losses are happening during the pre-training stage in Line 48?

A6: The "pre-training" mentioned in Line 48 refers to the training process of FineCLIP, not the pre-training of CLIP. Therefore, the loss function at this stage is the complete loss function of FineCLIP as defined in Equation 5. We describe it in this way because FineCLIP is essentially a pre-training method based on contrastive learning combined with self-distillation scheme.

Q7: Does the "instance" in Line 108-109 refer to a single image or an instance of object?

A7: The "instance" in Line 108-109 refers to the image and corresponding text.

References:

[1] CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Prediction.

[2] RegionCLIP: Region-based Language-Image Pretraining.