FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification

$**Q1:**$ This paper is overall rough and sub-par in writing, requiring substantial improvements in clarity.

$**R1: **$ Thanks very much for pointing out the problem. We have revised the above errors as follows: “slice-label” has been corrected to “slide-label,”, “a efficient annotation strategy” has been corrected to “an efficient annotation strategy,”, the symbol “V” has been corrected to “L,” and a “workshops” flag has been added to the citation information for reference [20]. We will conduct sentence-by-sentence proofreading of the paper before the official publication and will also have professionals polish the language.

$**Q2 and Q3:**$ The technical novelty is quite limited. The authors are encouraged to rephrase this sentence. Moreover, could the authors explain to readers why their methods can avoid collapsed training?

$**R2 and R3:**$ Thanks for your great suggestion on improving the quality of our manuscript. I believe that the second and third comments in the Weaknesses section are closely related, so we have combined them in our response. I consider the proposed method in this paper to be innovative for the following reasons.

1. New Paradigm: For the problem of few-shot classification in pathology images, we propose a novel Dual-tier Few-Shot Learning Paradigm, which not only improves classification accuracy but also reduces annotation costs.

2. Annotation Strategies and Classification Framework: We have introduced a dual-level WSI annotation strategy and a dual-branch classification framework. These two components must work together to achieve excellent WSI classification performance.

3. Addressing Training Issues: As noted by the reviewer, using learnable keys and values in Tip-Adapter can lead to training instability. Therefore, directly applying a learnable Tip-Adapter to WSI classification is not feasible. Our method ensures that the cache model does not suffer from instability due to its overall design, including the classification paradigm, annotation strategy, and classification framework. First, we obtain precise labels for some patches through the new annotation strategy. Second, in the cache model we construct, the labels of annotated patches are not optimized; only the labels of unannotated patches are optimized. The labels of annotated patches provide correct guidance for learning the labels of other patches, ensuring that the proposed cache model remains stable.

We accept the reviewer’s suggestion “The authors are encouraged to rephrase this sentence.” To clearly state our contributions, we revised “To fully utilize all patches in WSIs, we build a cache model where key and value are learnable, enabling the CLIP to adapt well to the WSI classification task” to “To fully utilize all patches in WSIs, we built a cache model where keys and values are learnable. This effectively facilitates the learning of correct label information for a large number of unlabeled patches, significantly enhancing the performance of CLIP for WSI classification.” At the same time, we rephrased the extent of the differences between our method and Tip-Adapter, revising “Our work is inspired by Tip-Adapter, but it differs significantly from Tip-Adapter.” to “Our work is inspired by Tip-Adapter, but it differs from Tip-Adapter.”

$**Q4:**$Some important vision-language-based models in computational pathology are not cited and compared to the proposed methods, such as PLIP and CONCH.

$**R4:**$ Thanks for your great suggestion on improving the quality of our manuscript. We have added comparison experiments with PLIP and CONCH. We used the same experimental setting as Zero-shot CLIP, then employed the PLIP image encoder and text encoder to obtain Zero-shot PLIP, and used the CONCH image encoder and text encoder to obtain Zero-shot CONCH. The experimental results are shown in the following two tables: Table A1 presents the results of the comparison methods with FAST on the CAMELYON16 dataset, and Table A2 shows the results on the TCGA-RENAL dataset.

First, from Table A2, we can see that the bag classification AUC of Zero-shot CONCH that we reproduced on the TCGA-RENAL dataset (referred to as TCGA RCC dataset in CONCH) is 91.94%, which significantly exceeds the 90.2% reported in the CONCH original paper. This indicates that our constructed prompt is very effective and that our Zero-shot CLIP serves as an excellent baseline model. This result also demonstrates that our comparative results across multiple datasets are very valid. From Table A1, it can be observed that our method outperforms PLIP and CONCH in all metrics on the CAMELYON16 dataset. From Table A2, it is evident that our method surpasses PLIP in all metrics on the TCGA-RENAL dataset. Compared to CONCH, FAST significantly outperforms CONCH in the average bag-level classification AUC. It is worth noting that our method uses a maximum of only 16 WSIs, while CONCH relies on 1.17 million pairs of pathology images based on CoCa, resulting in a significantly high training cost.

Last but not least, our method is orthogonal to studies such as PLIP and CONCH, and they do not conflict with each other. For example, in the “Few-shot classification with task-specific supervised learning” section of CONCH, it is stated: “However, it may still be desirable to specialize the model with labeled training examples to maximize performance for a given task, ideally using as few labels as possible. In this section, we investigate the label efficiency when using the pretrained representation of the image encoder backbone of the visual-language foundation models for task-specific supervised classification.”. This indicates that PLIP, CONCH, and similar methods will also need to be combined with few-shot learning methods like FAST proposed in this paper in the future. Therefore, our method can be effectively combined with methods such as PLIP and CONCH to further enhance WSI classification performance.

$**Q5:**$ It is suggested to rewrite Section 3.2 to make sure that all notations (those in texts and figures) and the implementation of components are clear and well-explained.

$**R5:**$ Thanks for your great suggestion on improving the quality of our manuscript. Firstly, we have revised the corresponding errors according to the suggestions in $**Q1**$. Secondly, we have corrected line 180 from “$\tilde{y}^{\text{cache}} = f_{\text{train}} F_{\text{train}}^T Y_{\text{train}}^I$” to “$\tilde{y}^{\text{cache}} = f_{\text{train}} F_{\text{train}}^T {Y_{\text{train}}^I}^ {{\prime}{\prime}}$”. We have also added the definition of $P_{\text{train}}$ as follows. $P_{\text{train}} = [\{ p_{(1, L+1)}, p_{(1, L+2)}, \ldots, p_{(1, K_1)}\}, \{ p_{(2, L+1)}, p_{(2, L+2)}, \ldots, p_{(2, K_2)} \}, \ldots, \{ p_{(i, L+1)}, p_{(i, L+2)}, \ldots, p_{(i, K_i)} \} ]$ represents the pseudo-labels of all unannotated instances in ${{X}_{\text{train}}}^{\prime}$, where $p$ is a learnable high-dimensional vector.

$**Q6:**$ The original Tip-Adapter is proposed for traditional single-instance settings. Its implementation for few-shot WSI classification is not clear.

$**R6:**$ We apologize for any inconvenience brought to you. We will add the implementation details of the Tip-adapter for few-shot WSI classification tasks to the supplementary materials. Additionally, we will open-source the code to facilitate further research by other researchers. The implementation details are as follows. We conducted experiments according to the settings of the optimal model in the original Tip-Adapter paper. For aspects that cannot be adapted to the few-shot WSI classification task, we used the following approach.

1. We designed a set of text prompts specifically for pathology images, which has been proven superior in the CONCH comparison experiments we conducted.
2. We used all annotated patches to build the cache model.

$**Q7:**$ Few-shot instances are randomly selected from the core set. Since only a few instances, e.g. 16, are selected from a very large instance pool and pathological patches (instances) often present heterogeneity, the final selected instance set could have a large variety and thus lead to unstable performances, calling into question the usability of the proposed method. This could also be observed from Fig 3 in the paper. So, the authors are encouraged to analyze and discuss the impact of randomly selected few-shot instances.

$**R7:**$ We apologize for any inconvenience brought to you. In our constructed cache model, not only does it include a few labeled patches, but it also contains a large number of unlabeled patches, making our model relatively stable. Additionally, as shown in Figure 3, when the instance shot reaches 16, the variance becomes relatively small.

$**Q8:**$ The original TCGA-RCC is not annotated at the pixel level. This work does a good job in terms of annotating the fine-grained region-of-interest of WSIs. However, if the annotation is not made public, this work would be difficult to follow and be very limited in research impact. The authors are encouraged to make their annotated dataset public.

$**R8:**$ Thank you very much for your suggestions on our work. To promote progress and development in the community, we will release the relevant datasets available for academic research.

Thanks for the authors' efforts and responses. After carefully reading the replies, I still have the following concerns:

• R2: I am more concerned with the technical novelty since the proposed framework does incremental work to the existing Tip-Adapter. By the way, I do acknowledge the novelty of the two-level few-shot learning paradigm, as I mentioned in the Strengths.
• R4: Thanks for the experimental results. These results could be helpful to justify the value of this work. However, I cannot agree with the authors' claims made in R4.
  • "Compared to CONCH, FAST significantly outperforms CONCH in the average bag-level classification AUC". Is there any statistical test to verify the significance difference between the AUC of 0.9235 and 0.9141? Or, does this conclusion just come from a personal sense? I think in scientific research the conclusion given by the authors must be rigorous enough.
  • "while CONCH relies on 1.17 million pairs of pathology images based on CoCa, resulting in a significantly high training cost.". CONCH is a foundational model for pathology, just like CLIP. Here, discussing its efficacy and comparing it with the proposed FAST is not appropriate, because I) foundation models generally rely on massive data and large-scale pretraining, and ii) FAST also stands on the shoulder of such foundation models like GPT and CLIP, right?
  • Additionally, I mentioned a comparison with CONCH. It intends to encourage the authors to justify the significance and value of this work, not to question the validity of the experiments. Concretely, for example, if the foundational model, CONCH, could achieve an accuracy of 90% in zero-shot settings yet the FAST framework with CONCH only obtains 90.5% in few-shot settings, the improvement would be too marginal to demonstrate the value of the proposed few-shot FAST.
  • "Therefore, our method can be effectively combined with methods such as PLIP and CONCH to further enhance WSI classification performance.". I failed to find the experiments on FAST + CONCH and see the improvements, so I think this claim, i.e., it can further enhance WSI classification performance, may not be valid from my point of view.
• R6: I cannot figure out the authors' implementation for Tip-Adapter in few-shot WSI classification, after carefully reading the authors' instructions. Could the authors please explain more? Thanks.
• R7: The authors mention that the model is relatively stable because the cache model includes a few labeled patches and a large number of unlabeled patches. It seems not obvious to me, since I just don't understand the logic behind the cause and consequence provided by the authors.

I would like to thank the authors for their efforts. Most of the time during the rebuttal, I feel inefficient communication in reading the author's responses, as most of these responses often fail to capture the true meaning of my questions and lead to undesirable QA.

Yet, overall, most of my concerns have been resolved. Thanks for the authors' experiments provided in the rebuttal. I believe the proposed framework could facilitate the study of few-shot WSI analysis. In view of these, I am happy to increase my score. The authors are encouraged to include the important suggestions & questions into the final version of the paper.

$**Q1:**$ The function $\phi(\cdot)$ in Figure 2 is not mentioned or explained in the paper, which may confuse readers.

$**R1:**$ We apologize for any inconvenience brought to you. We have added a description of the function \phi(\cdot) in Section 3.2, revising “The retrieval result is $(\dot{Q}\dot{K}^T)\dot{V}$ ” to “ The retrieval result is $\phi(\dot{Q} \dot{K}^T)\dot{V}$ , where $\phi(\cdot) = \text{softmax}(\cdot)$ . ”

$**Q2:**$ The authors conducted extensive comparisons in terms of accuracy and annotation cost but lacked analysis of time and memory consumption.

$**R2:**$ Thanks for your great suggestion on improving the quality of our manuscript. We conducted experiments on training time and memory consumption for the scenario with an instance shot of 16 and a bag shot of 16 using an NVIDIA RTX 3090. The results are shown in the Table A3. Our method achieves performance close to fully supervised methods with a training time of only 0.21 hours and a memory usage of just 5.12 GB. In comparison, training pathology large models like CONCH requires 8 NVIDIA A100 GPUs, highlighting a significant advantage of our method.

$**Table A3: analysis of time and memory consumption**$

$**Q3:**$ In section 3.2, the authors mention obtaining the optimal fusion weight α through grid search but lack specific details in the paper.

$**R3:**$ Thanks for your great suggestion on improving the quality of our manuscript. We have added the following description in Section 3.2. We divide the fusion weight \alpha into equal intervals with a step size of 100, then sequentially calculate the classification accuracy for each fusion ratio, and finally select the fusion weight that yields the highest classification accuracy as the fusion weight \alpha for this task.

$**Q4:**$ Will the authors open source the relevant code and all model weights for this project?

$**R4:**$ Thank you very much for your recognition of our work. We will open-source all related code and model weights to promote further development in WSI classification research.

$**Q1:**$ The study lacks experiments with V-L models specific to the pathology field. Since CLIP is not originally based on pathology images, the authors should include comparisons using PLIP [1] and CONCH [2].

$**R1:**$ Thanks for your great suggestion on improving the quality of our manuscript. Due to the huge size of WSIs, extracting features for the entire dataset using foundational models such as CLIP, PLIP, and CONCH requires a significant amount of time. We have not yet completed the extraction of features encoded using PLIP and CONCH. Additionally, many function wrappers in PLIP and CONCH are different from those in CLIP. Therefore, we need more time to build models using PLIP and CONCH. Once we obtain the latest experimental results, we will include these results in our subsequent replies. We have now completed the extraction of all features for the test set and have conducted some zero-shot classification experiments. Please refer to Table A1 and Table A2 in the PDF file for the experimental results.

$**Q2:**$ The few-shot learning method proposed by the authors operates at both the instance-level and bag-level. Therefore, the comparative methods should include both instance-based methods and bag-level methods (multi-instance learning). However, the fully supervised methods chosen for comparison are only instance-based. The authors should supplement their comparisons with bag-level methods based on multi-instance learning, such as R2T [3].

$**R2:**$ Thank you very much for your valuable suggestion. We did not compare bag-level methods based on multi-instance learning for the following reasons. First, instance-level fully supervised methods represent the upper bound of supervised learning classification results. Bag-level methods generally perform worse than instance-level fully supervised methods due to the lack of precise fine-grained labels. Second, we found it challenging to accurately reproduce the results of R2T within the limited time available. For these reasons, we did not include this comparison experiment. However, we have added an explanation in Section 4.2, “Comparing Methods and Evaluation Metrics,” about why we did not compare with “bag-level methods,” and we have also included references to relevant methods in the related work section. The specific content added is as follows. “Instance-level fully supervised methods represent the upper bound of supervised learning. Bag-level weakly supervised multi-instance learning methods, such as R2T, generally perform worse than instance-level fully supervised methods due to the lack of fine-grained labels. Therefore, this paper does not directly compare with multi-instance learning methods.”

$**Q3:**$ Comparing the third and fourth rows in Table 3 of the paper reveals that adding the Prior Branch on top of existing components brings almost no improvement. However, it requires first processing through GPT and then the Text-encoder, significantly increasing the cost without enhancing performance.

$**R3:**$ We apologize for any inconvenience brought to you. If we only compare the third and fourth rows of Table 3, it might indeed seem that way. However, we found that these results in Table 3 are due to the experiments being conducted under the 16-bag shot setting. To fully demonstrate the role of the prior branch, we conducted further experiments, and the related results and analysis can be found in the main text, Figure 4, and line 288. For convenience, we have included some key conclusions here: “When there are only 1 or 2 bags, the instance classification results of the prior branch are significantly higher than those of the cache branch. The instance and bag classification results that combine both the cache and prior branches also surpass those of using each branch separately, indicating that the prior branch performs better in extreme samples, and the information learned by the prior branch and the cache branch is complementary. Therefore, in extreme few-shot scenarios, FAST is dominated by the prior branch, but as the sample size gradually increases, FAST is dominated by the image branch.”

$**Q4:**$ What the difference between the proposed work and previous few-shot WSI classification methods like [4]? [4] The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. NeurIPS 2023

$**R4:**$ Our method differs significantly from Top in the following ways.

1. Different Scenarios: Top is a few-shot learning method under slide-level labels. While it also reduces annotation costs, it lacks precise patch-level label information. In contrast, we propose a dual few-shot learning scenario tailored for WSIs, which not only provides patch-level label information but also significantly reduces annotation costs.
2. Different Technical Approaches: The focus of Top’s research is on designing better text prompt strategies to serve CLIP. In contrast, our method introduces new approaches from both the image cache branch and the text prior branch to fully utilize existing annotation information.

$**Q5:**$ The authors only tested up to 16 instances and bags, but there is still a significant performance improvement from 8 to 16. I am curious at what data ratio in few-shot learning the model will begin to overfit.

$**R5:**$ We conducted experiments with 64 instances in the supplementary materials, and the results are shown in Figure 6. From Figure 6, it can be observed that as the instance shot increases, the classification accuracy also gradually improves. When the number of shots reaches 64, the rate of accuracy increase slows down. Therefore, considering both training costs and performance benefits, we recommend using an instance shot of 16 or 64 in practical applications.

Thank you very much for your valuable suggestions. We have implemented FAST using the vision-language model CONCH from the pathology field and conducted experiments on the CAMELYON16 dataset. The experimental results are shown in Table A4. We define the method using CLIP as the feature extractor as FAST-CLIP, and the method using CONCH as the feature extractor as FAST-CONCH. As shown in Table A4, compared to using CLIP as the feature extractor, using CONCH significantly improves the classification performance of FAST. Notably, the bag-level classification AUC can reach 0.957. This indicates that our method not only integrates well with V-L models like CLIP in natural images for WSI classification but also enhances the classification performance of V-L models in pathology.

$**Table A4: Results of using CONCH on CAMELYON16 dataset.**$

$**Q1:**$ This paper lacks some important related work. The proposed method is based on the Tip-Adapter. While, there are many improvements based on Tip-Adapter, such as [1-4]. I think the experiments should include comparisons with these related methods, or at the very least, mention and briefly analyze them.

$**R1:**$ Thanks for your great suggestion on improving the quality of our manuscript. our method is orthogonal to these four studies, and they do not conflict with each other. Additionally, the four studies mentioned above mainly focus on natural images and would face similar challenges as Tip-Adapter when directly applied to WSIs. Combining our method with these four methods has great potential to further improve the WSIs classification accuracy. For these reasons, we did not compare them directly through experiments, but we analyzed and summarized these four important studies in the related work section. The specific content added in the related work section is as follows. In the field of natural images, many subsequent works based on Tip-Adapter have also made significant contributions to the development of foundation model adaptation. For example, CaFo [2] effectively combines the different prior knowledge of various pre-trained models by cascading multiple foundation models. CO3 [1] goes a step further by considering both general and open-world scenarios, designing a text-guided fusion adapter to reduce the impact of noisy labels. Similarly, for open-world few-shot learning, DeIL [3] proposes filtering out less probable categories through inverse probability prediction, significantly improving performance. APE [4] proposes an adaptive prior refinement method that significantly enhances computational efficiency while ensuring high-precision classification performance. Due to the huge size and the lack of pixel-level annotations, these methods cannot effectively solve the classification problem of WSIs.

$**Q2:**$ In Figure 1, the blue and red boxes should be explained. Additionally, if space permits, I suggest that in future work, the authors could add more detailed descriptions in the caption. By this way, readers can understand the general idea of the method just by looking at the figure and caption, without having to spend effort finding the corresponding description in the main text.

$**R2:**$ Thanks very much for pointing out the problem. We have added the following description below Figure 1. Figure 1: Different few-shot learning paradigms for WSI classification. (a) The instance few-shot method divides all WSIs into a series of patches, then selects a few samples at the patch level and annotates them at the patch level. The red box represents positive samples, and the blue box represents negative samples. (b) The bag few-shot method directly selects a few WSIs at the slide level and annotates them weakly at the slide level. (c) Our method first selects a few WSIs at the slide level, then annotates a few patches for each selected WSI. Compared to (a) and (b), our method significantly reduces annotation costs while providing patch-level supervision information.

$**Q3:**$ For instance-shot, only the results of 16-shot are shown, without the results of 1-shot, 2-shot, etc.

$**R3:**$ We apologize for any inconvenience brought to you. Due to space limitations in the main text, we did not present the experimental results for different shot settings. Instead, we included these results on page 22 of the supplementary materials. In Figure 6, we show the results for 4-shot, 16-shot, and 64-shot settings. From Figure 6, it can be observed that the classification accuracy gradually increases with the number of shots. When the number of shots reaches 64, the accuracy increase nearly converges. Therefore, we recommend using 16-shot or 64-shot in practical applications. We have also added experimental results for the 1-shot and 2-shot settings. The results are shown in Figure A1, which we have uploaded in the PDF file. When the number of shots decreases to 1-shot or 2-shot, the accuracy decreases, with the lower limit being the result of Zero-shot CLIP.

$**Q4:**$ Line 160 seems to have a typo; it should be $y_{1,L}$ instead of $y_{1,V}$.

$**R4:**$ Thanks very much for pointing out the problem. The correct term here should indeed be $y_{1,L}$ . We have corrected this typo in the manuscript.