Enhancing Instance-Level Image Classification with Set-Level Labels

Weakness (1): I have some questions about the method part, Sec. 2.1. In Fig.2 (a), are input samples all belongs to the same set-level labels? I am confused by this figure and Fig.1(a) CIFAR images. What I believe is correct is that each batch contains samples belong to different set-level labels, and the coarse label is assigned to each sample for pre-training.

A: In the revision, we have added set-level label and fine-grained labels in Figure 2. This highlights that the input to the model is a set of images from CIFAR-100 and the corresponding set-level label is the most frequent superclass, i.e., fruit and vegetables. In Figure 2(a), we only show one input set (not a batch of input sets) due to limited space and the corresponding most frequent superclass of the input set, i.e., 'fruit and vegetables'. Figure 2(b) shows the fine-grained labels of the support set are apple, pear, orange, and mushroom.

To clarify, during pretraining in our FACILE framework, each batch indeed consists of multiple input sets. Each of these sets is associated with one set-level label. For the FSP (Fully Supervised Pretraining) baseline model used for comparison, we assign the corresponding coarse-grained label to each sample within an input set.

Weakness (2): How is SupCon trained? It is superised that supervised contrastive learning performs a lot worse than basic CE approach in most setups. There is not much information about training details.

A: Please find the training details of SupCon in Appendix A and G.2. When we use set-input data with SupCon, the conventional way of augmenting each instance inside a set could create challenges for SupCon training. It is typically trained to maximize agreement between differently augmented views of the same data point using labeled data. In our application, when using set-input data, the conventional data augmentation methods applied to each instance within a set and the set-input models that extract permutation invariant representations for all instances inside a set may present unique challenges for the SupCon model especially when the model is trained from scratch. However, you can see from Tables 1, 2, 4, 5, 7, and 10 that FACILE-FSP and FACILE-SupCon performance similarly. It's important to note that our framework is flexible and allows for any supervised learning algorithm to be used during the pretraining stage.

Question (1): There should be more training details about the framework in Sec.2.1.

A: We have documented these details to facilitate full reproducibility. The primary experimental setups, including configurations and parameters, are described in the Experiment section of the paper. For more in-depth information, including additional configurations and nuanced methodology, we direct readers to Appendix sections A, D, and G.

Question (2): Fig. 3 is referenced in text before Fig. 2

A: Thanks for pointing this out! We have reorganized the figures in the revised version.

Question (3): Most of the reference hyperlinks other than page 1 is not working.

A: Thank you for bringing this up. We have fixed the reference issue in the revised version.

We appreciate your feedback and recognize that the score may indicate some reservations about our work. We're committed to improvement and would greatly value any additional detailed feedback or specific suggestions you can provide!

We thank the reviewer very much for the questions and comments. We would like to kindly follow up with the reviewer, and respectfully ask the reviewer if they can reconsider their score if our response helps address their concerns.

Weakness (1): More baselines for strong self-supervised pretraining methods (e.g., BYOL, DION are needed to demonstrate the effectiveness. As proposed method is essentially a pretraining strategy, that bears a lot of similarity with exisiting self-supervised learning method

A: Thank you for raising this important point. The primary aim of FACILE is not to engage in self-supervised learning or "foundation" model training that require extensive data and computational resources. Instead, our focus is on leveraging coarse-grained labels to improve performance in downstream tasks. In this context, foundation models such as BYOL or DINO V2 serve as robust starting points. We can enhance these models by finetuning them with domain-specific data, effectively tailoring their capabilities to the nuances of the target domain.

To this end, we have added two additonal experiments.

First, we sought to enhance the model performance with coarse-grained labels of anomaly detection datasets (similar to Deep Set [5] and Set Transformer [6]). A total of 11,788 input sets of size 10 are constructed from the CUB200 training dataset by including one example that lacks an attribute common to the other examples in the input set. This setup creates a challenging scenario for models, as they must identify the outlier among otherwise similar instances. In downstream tasks, we evaluate the finetuned feature encoder composed of the CLIP [3] image encoder ViT-B/16 and appended fully-connected layer on the classification of species of the CUB200 test dataset. The batch normalization and ReLU are applied to the fully-connected layer.

The model training approach in this experiment centered around the CLIP image encoder, enhanced with an additional fully-connected layer. FACILE-FSP and FACILE-SupCon incorporate this setup, utilizing the CLIP-based feature encoder and focusing on finetuning the fully-connected layer through the FACILE pretraining step. In contrast, the SimSiam approach leverages the CLIP image encoder as a backbone while finetuning the projector and predictor components. Similarly, the SimCLR method also uses the CLIP encoder as its foundation but focuses solely on finetuning the projector. These varied strategies reflect our efforts to optimize the feature encoder for accurately identifying anomalies and improving classification performance in related tasks. The training details will be added to revised appendix.

Pretraining on input sets from CUB200. Testing with 5-shot 20-way meta-test sets; average F1 and CI are reported.

Second, similar to first experiment, we finetune a fully-connected layer that is appended after DINO V2 [2] ViT-B/14. This methodology is applied across various models to assess their performance on histopathology image datasets. By adopting the DINO V2 architecture, known for its robustness and effectiveness in visual representation learning, we aim to harness its potential for the specialized domain of histopathology. We refer interested readers to the revised appendix for details of pretraining.

1-shot 5-way test on LC dataset

Weakness (1) The use of coarse-grained labels like TCGA or NCT is an interesting choice. These are indeed umbrella terms for various subcollections, and traditionally they may not provide a strong learning signal. It could be beneficial to delve into why these particular labels were chosen and what advantages they offer in this context.

A: Thank you very much for this suggestion. We have included these explanation and advantages in each experiment in the revision.

Our theorem suggested that the relatedness, quantified by the relative Lipschitzness relative to $\mathcal{E}$, is an important factor for excess risk bound. It is intuitive that tasks in different domains do not share the same embedding space (and thus a large Lipschitz constant). Therefore, we deliberately choose pretraining tasks in source domain that are closely related to the target downstream task.

In response to the use of coarse-grained labels like those in TCGA, please note that the TCGA consortium has categorized an extensive array of cancer types and subregions, covering a diverse range of tissues. The strategic use of TCGA's whole slide level labels is rooted in their potential to enrich tissue-level classification. While these labels may appear broad, they encapsulate a wealth of underlying heterogeneity inherent to different cancer regions and tissue types and are routinely used in the cancer research community.

Weakness (2) Your team's approach to pretraining with coarse labels and then fine-tuning on a support set is a solid and proven method. However, it would enrich the work to articulate what sets this particular application or implementation apart in terms of novelty.

A: Thank you for the opportunity to clarify the novelty of our work. We have edited the introduction to highlight the novelty accordingly. We summarized novelty (and contribution) in the last three paragraphs of the introduction of our paper. Our work introduces FACILE, a novel generic learning framework that innovatively uses coarse-grained labels for pretraining, before fine-tuning on fine-grained tasks. This method diverges from standard fine-tuned supervised learning (FSL) and transfer learning (TL) by leveraging coarse labels from the source domain rather than heavily relying on detailed annotations. Our theoretical analysis underpins the empirical benefits of FACILE, providing evidence of how coarse-grained labels expedite the learning process with a significant increase in the convergence rate for fine-grained tasks. Furthermore, our experiments across different data domains---ranging from natural images to complex histopathology images---demonstrate FACILE's superior performance and versatility. Notably, we record a remarkable 13% performance improvement over existing baselines in a challenging histopathology benchmark. These elements collectively distinguish our work's novelty and significance in advancing the state of the art in representation learning (more specifically, in few-shot learning and model fine-tuning).

Weakness (3): The comparison with SimCLR and simSIAM provides useful insights, but considering the advancements in the field, benchmarking against more recent self-supervised learning methods like DINO or DINOv2 might offer a more comprehensive evaluation.

A: Thank you for raising this important point. The primary aim of FACILE is not to engage in self-supervised learning that requires extensive data and computational resources. Instead, our focus is on leveraging coarse-grained labels to improve performance in downstream tasks. In this context, foundation models such as BYOL or DINO V2 serve as robust starting points. We can enhance these models by fine-tuning them with domain-specific data, effectively tailoring their capabilities to the nuances of the target domain.

To this end, we conducted two additional experiments. First, we sought to enhance model performance with coarse-grained labels of anomaly detection datasets (similar to Deep Set [5] and Set Transformer [6]). A total of 11,788 input sets of size 10 are constructed from the CUB200 training dataset by including one example that lacks an attribute common to the other examples in the input set. This setup creates a challenging scenario for models, as they must identify the outlier among otherwise similar instances. In downstream tasks, we evaluate the finetuned feature encoder composed of the CLIP [3] image encoder ViT-B/16 and appended fully-connected layer on the classification of species of the CUB200 test dataset. The batch normalization and ReLU are applied to the fully-connected layer.

We thank the reviewer very much for the questions and comments. We would like to kindly follow up with the reviewer, and respectfully ask the reviewer if they can reconsider their score if our response helps address their concerns.

Thank you for reviewing our additional experiments and for your insightful feedback. Your engagement with our work and the detailed attention to our responses are greatly appreciated. We respect your decision. Should you have any more questions or require further clarification on any aspect of our research, please do not hesitate to reach out. We are fully committed to addressing any additional concerns you may have and value your input highly.

Thank you to the authors for their detailed responses and for providing additional experimental results. While I appreciate these efforts, I believe that further clarification on the use of coarse-grained labels, particularly in how they influence downstream performance, would significantly enhance the understanding of this method. The current rationale, especially regarding the choice of a coarse-grained label like TCGA, seems to lack sufficient detail to fully grasp its potential impact. It reminds me of generalizing all images in the ImageNet dataset under a single, broad category. I am open to and would welcome a more compelling explanation, as it would provide a clearer insight into your approach.

A: Thank you for your feedback and for highlighting areas in our manuscript where further clarification is needed. We appreciate your interest in the nuances of our approach, especially concerning the use of coarse-grained labels.

The idea of generalizing all images in the ImageNet dataset under a single, broad category has been explored in prior lines of research [1][2]. Building upon this, our work extends this framework to a more general setting. In our setting, coarse-grained labels are associated with a set of instances, rather than being limited to individual instances (explained in section 2). This expansion allows for a more nuanced understanding and application of coarse-grained labels in complex learning environments.

The rationale for selecting coarse-grained labels in the TCGA is detailed in the third paragraph of the introduction and in Section 3.4. As previously discussed, both in this section and in our related work, this approach differs from "the generalization of all images in the ImageNet dataset into a single, broad category". First, unlike the methods used in [1] and in Experiment 4.2 of [2], we do not rely on a class taxonomy to differentiate between coarse-grained and fine-grained labels. Second, the inputs provided to the models for predicting these labels vary: we use set input for coarse-grained labels and instance input for fine-grained labels.

In response to your concerns, we would like to direct your attention to several key sections of our paper that address these issues:

• Rigorous Definitions and Theoretical Analysis (Section 2): We provide detailed definitions of coarse-grained set-level labels, fine-grained labels, and related key conditions. The distinction between coarse-grained set-level labels and fine-grained labels is crucial for understanding the subsequent theoretical analysis. Our theoretical framework explores how learning with coarse-grained set-level labels can enhance downstream task performance. This is not just a speculative statement but is backed by rigorous analysis. The results presented in Figures 4 and 5 are not just empirical findings; they are in alignment with our theoretical predictions, demonstrating the practical effectiveness of our approach.
• Intuitive Illustrations (Introduction, third Paragraph): To aid in understanding, we provide two illustrative examples that demonstrate why coarse-grained labels can be beneficial. These examples are designed to be intuitive and accessible, providing a clear rationale for our approach.
• Experiment-Specific Explanations (Empirical Study Section and Appendix Section B): In each experiment, we do not merely apply coarse-grained labels. Instead, we provide specific reasons why certain coarse-grained labels are chosen and how they are expected to contribute to the learning of the particular downstream tasks.

We believe that these sections and figures collectively offer a comprehensive explanation of our methodology and its advantages. Our aim is to demonstrate not only the theoretical soundness of using coarse-grained labels but also their practical efficacy in enhancing downstream task performance. We trust that this clarification has addressed your concerns and offered a more transparent understanding of our approach. We are eager to answer any further questions you may have. Thank you for your time and effort.

References:

[1] Phoo, Cheng Perng, and Bharath Hariharan. "Coarsely-labeled data for better few-shot transfer." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9052-9061. 2021.

[2] Robinson, Joshua, Stefanie Jegelka, and Suvrit Sra. "Strength from weakness: Fast learning using weak supervision." In International Conference on Machine Learning, pp. 8127-8136. PMLR, 2020.