Zero-Shot Visual Classification with Guided Cropping

Thank you for your valuable comments as well as suggestions. We address your questions and concerns as follows.

could we report more results from more common image classification benchmarks?

Response: Thank you for your suggestion. ImageNetS919 and CUB datasets are chosen in our work as these datasets are classification datasets with annotations (e.g., bounding boxes) from which object sizes can be retrieved. This allows us to perform detailed analysis under different object size conditions. These datasets are also commonly used in weakly supervised object localization task (e.g., [1]) as it needs similar annotations during evaluation.

Nevertheless, for the completeness of our paper, as suggested by the reviewer, we perform additional evaluation on more common datasets (ImageNet, ImageNetV2, Stanford Dogs). The results can be found in the appendix A.9 in the updated version of our paper. The results still demonstrate that integration of Guided Cropping can still be beneficial even in datasets with unconstrained object sizes (similar to the unconstrained variants of ImageNetS919 and CUB).

[1] "Weakly Supervised Object Localization as Domain Adaption", CVPR, 2022.

could we report the additional inference cost?

Response: Additional parameters are required for OWL-ViT that attributes to a total of 153M model parameters. When performing a single sample inference, additional inference time required on top of CLIP is 0.15 and 0.16 seconds without and with box augmentation on a single Tesla V100-SXM2 GPU respectively. We will include these details in the final version of the paper.

In Table 2, the comparison between CALIP and GC-CLIP is not solid: we may be able to also say “CALIP can improve performance over vanilla GC-CLIP” (if this is the truth). The result in Table 2 is also hard to compare with Table 1. So I would suggest reusing the same setup as Table 1 to compare CALIP and GC-CLIP for a clearer result.

Response: Thank you for your constructive feedback. We have adapted Table 2 in the updated version of our paper to have the same setup as Table 1 accordingly so that both tables are now comparable. We can see that, even without box augmentation, vanilla GC-CLIP gives better performance compared to CALIP. Additionally, Guided Cropping can also be integrated into CALIP to further improve the performance. We have included this discussion in the updated version of our paper.

In 5.4, could we also show the scores of the examples? So that we could understand a bit more possibly the confidence of the model (e.g. whether a smaller score could indicate that the model needs more context for classification?).

Response: Thank you for your suggestion. Visualizing the scores of these examples can indeed improve understanding of the model behaviors. We will incorporate this suggestion in the final version of our paper.

Thank you for your valuable comments as well as suggestions. We address your questions and concerns as follows.

The observations of current limitations of CLIP and OWL-ViT models are somewhat surface-level and I believe well-known (although possibly not written down in a publication)

Response: In our work, we address two limitations of CLIP, its vulnerability to extraneous image regions and its sensitivity to non-semantic crops. I agree with the reviewer that some can have assumptions similar to these limitations. However, validating the assumptions and systematically quantifying impacts of these limitations are also crucial but missing from the community. For the vulnerability to extraneous image regions, we show that performance gaps between scenarios with unconstrained objects and scenarios with small objects can be up to 15-20 percents (see Figure 6 in section 5.3). The performance drops can also be observed in various supervision strategies, i.e., in zero-shot, few-shot and fully supervised models (see Table 3, 4 and 5 in the appendix). For the sensitivity to non-semantic crops, its effect is quantitatively evaluated. From Figure 4, we demonstrate that around 40% of test samples in our setup have changes in their final predictions just from small removal around image boders. These empirical studies show that impacts of these limitations are not negligible. We believe that these studies can be helpful for the research community.

Although the focus of the proposed approach is to correct failure cases of CLIP and OWL-ViT (i.e. small object classification), and the benchmarks were chosen to assess this specifically, it would be very useful to see the proposed method benchmarked on widely used zero-shot datasets like ImageNet.

Response: Thank you for your suggestion. We conducted further experiments on additional datasets (ImageNet, ImageNetV2 and Stanford Dogs). The results can be found in the appendix A.9 of the updated version of our paper. The results still demonstrate that GC-CLIP improves baseline performance even in datasets with unconstrained object sizes.

It would be interesting to discuss the limitations of the proposed method.

Response: Guided Cropping can be viewed as a strategy to refine image features conditioned to classes of interest. Therefore, it is ideal to be employed in the setting of classification task since all classes of interest are known in advance. For some generic tasks (image-text retrieval, image-conditioned detection) that all classes of interest are not defined, it would not be straigtforward to employ Guided Cropping in general.

However, on the specific scenarios that domains of interest are known in advance, Guided Cropping can also be beneficial. For example, in case of image-conditioned detection, if we already know that the domain of interest is animal, we can use Guided Cropping to refine image features using generic animal prompts (e.g., the word "animal" itself). In this case, information of unrelated contexts of query images can be discarded by Guided Cropping which could lead to better detection performance. We will include this discussion in the final version of the paper.

Thank you for your valuable comments as well as suggestions. We address your questions and concerns as follows.

The novelty is limited. Many papers [1,2,...] have discussed the impact of cropping in image classification.

Response: Thank you for your feedback. According to the mentioned papers, we agree that the paper [1] is closely related to ours. However, our work provides some additional insights which have not discussed in [1]. Firstly, while [1] provides the evidence that optimal crops can be helpful for classifiers, [1] has not discussed the properties which the optimal crops should have. In our work, we demonstrate that the optimal crops should be close to target objects but not too close. We empirically provide evidence to support this argument by varying multiple margin ratios as presented in section 5.2. Additionally, we show that, even though emphasizing on object information is preferable, we should not ignore the context information completely. Some strategies to balance object and context information must be developed. This is the main motivation for us to propose Multi-Margin box augmentation. Lastly, the aggregation-based approach proposed in [1] is also similar to our Random Cropping variants in the sense that they determine crops from static strategies without considering where target objects are. Our Guided Cropping takes locations of the target objects into account leading to better performance. We will include this discussion in the final version of our paper.

The potential applicability is limited. This method is only suitable for classifying images whose labels associating to a small object it contains. The authors did provide some qualitative evaluation on these cases but I think it could benefit the paper to frame it more aggressively into this direction since it seems to me this is the only scenario where it might prove a significant advantage.

Response: We understand your concern. To strengthen our motivation, we would like to give an example of potential scenarios in which recognition of small objects is beneficial. One example scenario is under automated driving application. In this scenario, recognizing objects which are far away is crucial. It allows automated vehicles to have better decision making resulting in safer maneuver. As the far away objects are mostly small in image frames, our Guided Cropping can be employed in this case to improve recognition performance for those objects. We will include this motivation in the final version of the paper.

The method is very specific to CLIP and the core method doesn't work right off the bat but requires post-processing steps on top of the initial boxes.

Response: Our Guided Cropping aims to improve models whose performances are likely to be degraded by extraneous, non-discriminative image regions. In our work, we show that CLIP is one of the models which has this mentioned behavior so that its integration with Guided Cropping can be beneficial. In principle, Guided Cropping can be applied not only to CLIP but also other models which are vulnerable to extraneous non-discriminative image regions. For example, we demonstrate that Guided Cropping integration with CALIP and Tip-Adapter can be helpful (see Table 2 and Table 4). Other potential relevant models are few-shot models as observed by [1] that they suffer from shortcut learning due to background information.

Regarding to the requirement of post-processing steps on top of the initial boxes, we do not see this as a disanvantage of our work but a contribution. In fact, using the initial boxes directly can already improve performance. The post-processing only aims to exploit the availability of the boxes as much as possible.

[1] "Rectifying the Shortcut Learning of Background for Few-Shot Learning", NeurIPS, 2021.

Thank you for your valuable comments as well as suggestions. We address your questions and concerns as follows.

I suggest the authors report the computational cost of GC-CLIP in the paper.

Response: Additional parameters are required for OWL-ViT that attributes to a total of 153M model parameters. When performing a single sample inference, additional inference time required on top of CLIP is 0.15 and 0.16 seconds without and with box augmentation on a single Tesla V100-SXM2 GPU respectively. We will include these details in the final version of the paper.

In the experimental section show that the difference between introducing OWL-ViT for guided cropping and using random cropping is slight.

Response: We understand that your concern about slight improvement mainly refers to results on datasets with unconstrained object sizes (ImageNetS919 and CUB). In this case, this observation is expectable since many images of these datasets already have large enough object sizes (images have small extraneous regions) and thus do not greatly benefit from our Guided Cropping component. Note that our work is mainly motivated and designed to address the challenging cases of input images with small object sizes and large extraneous regions. Henceforth, our improvements are more evident on ImageNetS919-SM and CUB-SM (curated with objects with small size), where the improvements from random cropping can be high up to almost 3 percent.

The authors did not report results with random cropping alone when guided cropping was used on CALIP.

Response: Thank you for your suggestion. We have integrated random cropping box augmentation with CALIP and updated results in Table 2 to be more detailed. The results still show the benefit of combining Guided Cropping with CALIP.

An essential prerequisite for the successful application of OWL-ViT in the GC-CLIP is that OWL-ViT can provide a detection box for every target in the image. The authors have yet to carry out validation on more datasets to verify the impact and constraints of the detectors on their method, so it cannot judge the processing performance of GC-CLIP for other more complex datasets, and more analysis is needed.

Response: GC-CLIP can be viewed as a framework utilizing available object detectors to improve classification performance. In our work, we use OWL-ViT as a candidate of object detectors. We agree that the performance of OWL-ViT can be limited but we would like to show that it is good enough for improving classification performance under our framework. To demonstrate benefits of our setup in other scenarios, as suggested by the reviewer, we have conducted further experiments on additional common classification datasets (ImageNet, ImageNetV2 and Stanford Dogs) with their original test splits. The results are presented in the appendix A.9 in the updated version of our paper. According to the results, our Guided Cropping strategy can still consistently improve performance of the baselines.

Authors should report the performance comparison of GC-CLIP with current popular methods.

Response: CALIP is actually one of the recent models from AAAI-2023 we consider in our work. Anyway, we also agree that including more baselines would make our results statistically more meaningful. In this regard, we incorporate a CLIP(ViT-L/14) variation trained with DataComp from NeurIPS-2023 [1] as one of our baselines. Its performance is reported in Table 3 of the appendix A.2 in the updated version of our paper. It must be noted that performance of DataComp can still be further improved by integrating with our Guided Cropping. This demonstrates consistent observations among various baselines.

[1] "DATACOMP: In search of the next generation of multimodal datasets", NeurIPS, 2023.

Authors also need to check for grammatical problems.

Response: Thank you for your comments. We have updated our text accordingly.