Prototype antithesis for biological few-shot class-incremental learning

We sincerely appreciate your valuable review comments and the effort you have put into them. We have addressed each of the questions you raised in detail. If anything is unclear, please inform us, and we will respond promptly.

W1: Questions about cosine similarity calculation in Eq. 2.

Response: The cosine similarity is computed by evaluating the similarity between the feature $F$ and each prototype individually, resulting in a set of scores that serves as the standard output of the classification model. Specifically, for the term $\cos\left( F; \boldsymbol{\theta}_c, \boldsymbol{\theta}_r \right)$, we calculate the similarity score between $F$ and each prototype in $\boldsymbol{\theta}_c \cup \boldsymbol{\theta}_r$. These similarity scores collectively form the output of the model. To ensure clarity, we have revised the term from $\cos\left( F, \boldsymbol{\theta}_c, \boldsymbol{\theta}_r \right)$ to $\cos\left( F; \boldsymbol{\theta}_c, \boldsymbol{\theta}_r \right)$ and further explained this calculation in Section 4.2 of the revised paper.

W2: Is this label provided by the dataset?

Response: Yes, the family-level labels are obtained directly from the dataset. For example, in CUB-200, species such as "Black-footed Albatross," "Laysan Albatross," and "Sooty Albatross" are assigned the family-level label "Albatross".

W3: What is the number of features for $\boldsymbol{\theta}_r$, and does this number change during training?

Response: The number of features for $\boldsymbol{\theta}_r$ is equal to the number of families of base species, and it is fixed during training.

W4: How RPM encourage the model to capture the unique characteristics of the new species?

Response: As described in Lines 256–296, this paragraph explains how RPM helps the model focus on "species-unique" features in new categories based on the classification model's optimization process. To clarify, both features and prototypes are decomposed into species-related and species-unrelated components. For features, the species-unrelated part is further divided into family-shared and background components, while for prototypes, the species-unrelated part consists solely of the family-shared component.

When the new and old species belong to the same family, RPM enhances the family-shared part of the new species by mixing the old species' family-shared features with the new species' features. This makes the classifier likely to produce a high response to the family-shared part, potentially causing the model to misclassify it as the family label. To avoid this, the model enhances the response of the species-unique components, which is achieved through the optimization of Eq. 3, as mathematically detailed in Lines 285–291 of the original paper.

W5: Does the method require integrating residual features during the inference phase?

Response: No, during the inference stage, the features of new species are directly classified by the prototypes without incorporating residual features. This is because, through RPM, the model has established well-separated decision boundaries, allowing it to achieve high accuracy when directly classifying new categories. We have included this clarification in the revised paper (Section 4.1).

We sincerely appreciate your valuable review comments and the effort you have put into them. We have addressed each of the questions you raised in detail. If anything is unclear, please inform us, and we will respond promptly.

W1: Questions about Table 2.

Response: The feature decomposition strategy directly impacts the learning of seen and unseen species, as measured by the classification accuracy of old and new species presented in Table 2. To make the comparison more intuitive, we introduced a metric to quantify the similarity between "unique" and "common" features after decoupling. A lower similarity indicates a better decoupling effect. The experimental results are as follows, and RPL achieves the lowest similarity. We have updated this metric in the revised paper (Table 2).

W2: Issues about bolded accuracies.

Response: Thank you for your advice. In the revised paper, we have bolded only the highest performance results. As shown in Table 3, our method achieves the highest performance in the final session and exhibits the lowest performance drop (PD), indicating superior incremental learning ability.

W3: Experiments should be run multiple times to ensure the robustness.

Response: In fact, for the CUB-200, PlantVillage, and Tree-of-Life datasets, we conducted 5 experiments with different random seeds and reported the averaged results to minimize the impact of randomness. We have included this information in the revised paper (Section 5.1).

Dear Reviewer KRan,

We deeply appreciate the time and effort you have invested in evaluating our work. As the author-reviewer discussion period nears its conclusion, we kindly ask for your feedback on whether our responses have sufficiently addressed your concerns. If you have any additional suggestions or comments, please feel free to share them with us. We are more than willing to engage in further discussion and remain committed to making any necessary improvements.

Once again, thank you for your thoughtful insights and invaluable suggestions. We look forward to your response!

Thank you for showing the generalizability across the tree-of-life dataset! I have updated my rating to accept, good paper.

Q3: Is the delta of cosine similarity optimized implicitly when we optimize as in Equation 3?

Response: Yes, this occurs during optimization as described in Eq. 3. Specifically, in Eq. 3, the cross-entropy loss function takes the labels and the cosine similarity scores between the features and all prototypes of $\boldsymbol{\theta}_c$ and $\boldsymbol{\theta}_r$ as input. To minimize the loss, the cosine similarity score for the target class must be the highest, which effectively maximizes the delta of cosine similarity.

Q4: Does “without PA” mean the baseline model from Table 1?

Response: Yes, it refers to the baseline method in Table 1, and we have fixed this in the revised paper.

Q5: Questions about the calculation of cosine similarity.

Response: Actually, the cosine similarity is computed by calculating the similarity between the feature and each prototype individually, resulting in a set of scores that serves as the standard output of a classification model. We have clarified this point in the revised paper (Section 4.2) to prevent any potential misunderstandings.

Q6: Issues about the bolded performance.

Response: Thanks for your advice. We have bolded only the highest performance results in the revised paper to fix this issue.

We sincerely appreciate your valuable review comments and the effort you have put into them. We have addressed each of the questions you raised in detail. If anything is unclear, please inform us, and we will respond promptly.

W1: Missing several baselines on PlantVillage and Tree-of-Life datasets.

Response: Here are the reasons why not every baseline is applied to all datasets: In our experiments, we reproduced publicly available code and methods suitable for biological recognition applications on the PlantVillage and Tree-of-Life datasets. However, NC-FSCIL requires pre-calculating classifier heads for all categories, which means it assumes prior knowledge of the number of new categories that will appear. This constraint makes NC-FSCIL a closed-set classification algorithm, whereas in biological recognition scenarios, it is often impossible to predict the emergence of new categories. As a result, we did not apply this method to biological datasets in our primary evaluation.

Nevertheless, we have tested the NC-FSCIL algorithm on the PlantVillage and Tree-of-Life datasets. The results are as follows:

PlantVillage:

Tree-of-Life:

The results show that our method outperforms NC-FSCIL in the final session and achieves the smallest performance drop on the PlantVillage dataset, while delivering comparable performance (slightly lower than NC-FSCIL) on the Tree-of-Life dataset. Unlike NC-FSCIL, our method does not require prior knowledge of the total number of unknown classes, making it more scalable and practical. These results have been included in the revised paper (appendix).

W2: Possible limitation of RPM?

Response: In fact, regardless of whether the new class's family appeared in the initial learning stage, RPM can work properly. Species confusion is more likely to occur when the new and old species belong to the same family. In contrast, when they do not share a family, the model only needs to perform standard few-shot incremental learning for the new species. Therefore, in Section 4.3, we focus on analyzing the case where the new and old species belong to the same family. Furthermore, when the family of a new class was not present during the initial learning stage, RPM generates additional feature samples by combining the new species features with the residual prototypes. This process helps mitigate overfitting caused by the limited number of training samples, as described in Lines 294–296 of the original paper.

Q1: How is the assumption that prototypes do not contain background information made?

Response: The reasoning is as follows: As established in prior works, such as ProtoNet [1], learned prototypes represent the feature centers of their respective classes. Consequently, prototypes are expected to converge on category-specific features, while background features, being category-independent, are naturally excluded from the prototypes.

[1] Prototypical Networks for Few-shot Learning, NeurIPS 2017.

Q2: Why do species-level prototypes represent different meanings in the RPL and RPM stages?

Response: Here are the reasons: With sufficient training data for the base species, the model can learn more representative prototypes (Lines 231–232 of the original paper), and the "species-unique" and "family-shared" traits are further decoupled through the bi-directional optimization process defined in Eq. 2. However, during the incremental learning phase (Lines 275–276 of the original paper), the limited training data for new species makes it challenging for the model to learn accurate prototypes. As a result, the prototypes for these new species often contain coarse semantic information, incorporating both the most and second most discriminative features. This is why we employ RPM to guide the prototypes toward better convergence on species-unique features. We have emphasized this point in the revised paper (Section 4.3).

Q1: Questions about Table 2.

Response: The feature decomposition strategies include RPL, CAM and Grad-CAM. The details are stated in Lines 402–404 of the original paper: "Specifically, we set a threshold to extract the highest-response regions from the feature maps as unique features, while the secondary-high-response regions represent common features." The feature decomposition strategy plays a crucial role in the learning process for seen and unseen species, as reflected in the classification accuracy reported in Table 2.

To make the comparison more intuitive, we introduced a new metric to quantify the similarity between "unique" and "common" features after decoupling. A lower similarity score indicates a more effective decoupling process. The experimental results (shown below) demonstrate that RPL achieves the lowest similarity score, highlighting its superior performance. This similarity metric is updated in the revised version of Table 2.

Q2: Questions about BioCLIP.

Response: According to the official data provided by BioCLIP, despite being pre-trained on a large number of animal and plant datasets, BioCLIP's accuracy on the PlantVillage dataset remains low in both zero-shot and 5-shot settings, achieving accuracies of 24.4% and 80.9%, respectively. These results are significantly below the average performance of PA's incremental learning (94.51%), suggesting that BioCLIP struggles with substantial class confusion when dealing with closely related species, such as leaf lesions.

Additionally, we conducted continual learning experiments using the official BioCLIP source code on both the CUB200 and PlantVillage datasets. The results demonstrate a notable decline in BioCLIP's average classification performance, as detailed below.

PlantVillage:

CUB200:

Reference

[1] Fewshot class-incremental learning, CVPR 2020.

[2] Few-shot incremental learning with continually evolved classifiers, CVPR 2021.

[3] Few-shot class-incremental learning via class-aware bilateral distillation, CVPR 2023.

[4] Few-shot classincremental learning via training-free prototype calibration, NeurIPS 2023.

[5] Orco: Towards better generalization via orthogonality and contrast for few-shot class-incremental learning, CVPR 2024.

[6] Prototypical Networks for Few-shot Learning, NeurIPS 2017.

[7] Attribute Group Editing for Reliable Few-shot Image Generation, CVPR 2022.

[8] Cognitive computational neuroscience, Nature neuroscience, 2018.

[9] Generalized zero-and few-shot learning via aligned variational autoencoders, CVPR 2019.

We sincerely appreciate your valuable review comments and the effort you have put into them. We have addressed each of the questions you raised in detail. If anything is unclear, please inform us, and we will respond promptly.

W1: Why species similarity leads to forgetting.

Response: The forgetting is caused by species confusion. As noted in lines 15–16, "... often faces species confusion, leading to 'catastrophic forgetting' of old species...", Fig. 1 provides a detailed illustration of this phenomenon. The confusion between old and new species during incremental learning results in a significant drop in the performance of old species, a process known as 'old class forgetting'.

W2-first: choice of backbone.

Response: We address this question from three key aspects:

1. As stated in Line 353, we selected ResNet18 as the backbone to "maintain consistency with prior works." ResNet18 is the standard backbone adopted by most mainstream and state-of-the-art methods for few-shot incremental learning [1][2][3][4][5], ensuring fair comparisons and alignment with established benchmarks.
2. While "large foundation models" exhibit strong representational capabilities and have been widely applied across various scenarios, their use in few-shot incremental learning introduces the issue of data leakage. These models are pretrained on extensive datasets that often overlap significantly with the datasets used in mainstream few-shot incremental learning benchmarks. For instance, categories such as animals and plants commonly appear in the pretraining datasets of models like DINO and BioCLIP. As a result, the "new classes" in incremental learning are not genuinely unseen, which undermines the objectivity and validity of algorithm evaluations.
3. Our primary contribution, the prototype antithesis strategy, is not confined to ResNet18 and can be seamlessly integrated with other backbones, including ResNet-50, ResNet-101, and transformer-based architectures. To demonstrate its generalizability, we conducted experiments on the CUB200 dataset using ResNet-50, with the results showing below, from which we can see that PA (ours) further improves against the baseline method with significant margins.

W2-second: Is the backbone pretrained?

Response: Following prior works [1][2][3][4][5], we utilize ResNet-18 pretrained on ImageNet for experiments on the CUB200 dataset, while a randomly initialized ResNet-18 is used for the PlantVillage and Tree-of-Life datasets. We have included this information in the revised paper (Section 5.1).

W2-third: Why old classes will be forgotten?

Response: The old classes could be forgotten even if the backbone and old class prototypes are fixed, here are the reasons: Even with the feature backbone and old class classifiers frozen, newly added class classifiers can disrupt the classification probability distribution, leading to misclassifications and performance degradation of old classes, termed as "old class forgetting". For example, when a new class is highly similar to an old class, both old and new classifiers may produce high classification scores, increasing the likelihood of misclassifying an old class as a new one. This issue is also discussed in CEC [2] (Introduction, Paragraph 3), where both the feature backbone and old class classifiers are frozen during incremental learning.

W2-fourth: Do the authors impose an orthogonal constraint in Line 270?

Response: It is not an imposed constraint, any feature vector can inherently be decomposed into two orthogonal components due to the properties of vector spaces. Specifically, given a vector a and an auxiliary vector b, a can be expressed as:

1. A component parallel to b:
   u = $Projection_{to-b}$(a),
2. A component orthogonal to b:
   v = a - u.

These components satisfy a = u + v and u $\cdot$ v = 0. The existence of projections guarantees that this decomposition is always possible. For ResNet-18, the feature vectors for each image, which reside in $\mathbb{R}^{512}$, can similarly be orthogonally decomposed along a species-unique direction.

W3-first: Why computing the residual between the prototypes and feature vectors makes sense? And why residuals represent family traits?

Response: The computation of the residual is well-justified. As demonstrated in prior few-shot learning works like ProtoNet [6], prototypes naturally converge to the centers of their respective feature distributions. In our approach, prototypes are used as classifiers by directly calculating the feature distances between prototypes and features without scaling (refer to Lines 207–208 of the original paper). This alignment ensures that both features and prototypes reside in the same space, making the residual operation meaningful. Additionally, GAN-based methods, such as AGE [7], also adopt this approach to remove category-related attributes effectively.

Notably, during the bi-directional optimization defined in Eq. 2 under "family-level" label supervision, residual prototypes—not residual features—converge to "secondary discriminative features," representing family-shared traits.

W3-second: Why adding the residual to the new species features make sense? What do the new feature vectors encode?

Response: This is because both the residual and new species features are encoded from input images by the same feature extractor. Since they exist in the same aligned feature space, adding the residual to the feature vectors of the new species represents a linear feature fusion strategy. And the newly obtained feature vectors represent the new species features enhanced with family-shared traits (as described in Lines 271-274 of the original paper).

W3-third: The meanings of the decompositions and terms introduced in Lines 270 - 290.

Response: Lines 270-290 argue why RPM helps the model focus more on "species-unique" features in new categories, based on the classification model optimization process. To simplify, we decompose both features and prototypes into species-related and species-unrelated parts. For features, the species-unrelated part is further divided into family-shared and background components, while for prototypes, the species-unrelated part is simply the family-shared component. When the new and old species belong to the same family, RPM enhances the family-shared part of the new species by mixing the old species' family-shared features with the new species' features. This makes the classifier likely to produce a high response to the family-shared part, potentially causing the model to misclassify it as the family label. To avoid this, the model enhances the response of the species-unique components. This is achieved through the optimization of Eq. 3, as mathematically detailed in Lines 285–291 of the original paper.

In the revised paper, we have polished this section to make it more intuitive and easier to understand.

W3-fourth: Details about the inference stage.

Response: During inference stage, the features of the new species are classified by the prototypes without mixing with the residual. This is because, through RPM, the model has established well-separated decision boundaries, allowing it to achieve high accuracy when directly classifying new categories. And we have added this information in the revised paper (Section 4.1).

W4: Questions about the experiment details.

Response:

1. Dataset statistics: In the Experiment Settings section, we focus on describing the categories of the dataset and the way it is partitioned, and the family information are directly obtained from their classification labels. Additional statistics, including the number of images, species, and families, are provided in the revised paper (appendix).

2. Although the PlantVillage dataset contains a relatively small number of categories, it is a classification dataset focused on leaf diseases caused by pests and pathogens, which holds significant importance in biological research. Furthermore, as the dataset consists of various lesions on different types of leaves, the similarity between categories is very high, making it prone to category confusion during incremental learning. This makes it a highly challenging dataset.

3. The Tree-of-Life and iNaturalist datasets have a significant overlap, which is why we did not choose the iNaturalist dataset as the benchmark for this paper. Nonetheless, we randomly sampled 500 species from the iNaturalist 2021 dataset for exploratory experiments (using ResNet-50 as the backbone). These 500 species span 136 families and contain a total of 103,582 images. Among them, 340 species were designated as base species, while the remaining 160 species were evenly divided into 8 incremental stages, with 20 new species added per stage. The experimental results (shown below) demonstrate that, compared to the baseline, our method maintains a clear performance advantage in incremental learning on the iNaturalist 2021 dataset.

4. As described in Line 350, the 100 species are randomly sampled from the Tree-of-Life dataset, and we have the included the detailed statistics in the revised paper (appendix).

5. We do not need to ensure that all families appear during the base training phase. If the family of a new class does not appear during the base training phase, it indicates that the new class differs to some extent from the old classes, and in this case, we can directly proceed with classification. Our method is designed to handle scenarios where new and old classes are extremely similar, which is clearly a more challenging situation.

6. The mainstream datasets for few-shot incremental learning typically have fewer than (or equal to) 200 categories. For example, CUB-200 has 200 categories, mini-ImageNet has 100 categories, and CIFAR100 has 100 categories. The challenge of this task lies in the forgetting and overfitting that occurs during the incremental learning process. As the amount of data decreases in each incremental stage, the model's learning becomes more difficult. Especially in biological recognition scenarios, the number of new or mutated categories and samples is often very small, so the setup of this paper is closer to real-world application scenarios. However, we appreciate your suggestion, and we will explore experiments in scenarios with more categories in the future.

W5 and W6: Missing reference.

Response: We cited all datasets in Lines 342–343 of the original paper. To enhance clarity, we have also cited them in the introduction of the revised paper. Additionally, we have included a citation for "Inspired by human’s reference-based learning mechanism" [8] in the revised paper. We have also included the discussion about GFSL [9] in the revised paper (section 2): "GFSL addresses this (forgetting) by aligning old and new class distributions to mitigate forgetting, but it struggles when old and new classes distribution gaps are large, and risks class confusion when old and new classes are highly similar".

Dear Reviewer qswS,

We deeply appreciate the time and effort you have invested in evaluating our work. As the author-reviewer discussion period nears its conclusion, we kindly ask for your feedback on whether our responses have sufficiently addressed your concerns. If you have any additional suggestions or comments, please feel free to share them with us. We are more than willing to engage in further discussion and remain committed to making any necessary improvements.

Once again, thank you for your thoughtful insights and invaluable suggestions. We look forward to your response!

Thank you for your reply. We would like to provide further clarification on the concerns you raised:

Response to W1:
We analyze the two scenarios you mentioned separately:

1. When the new classes are significantly different from the old ones, the parameter drift caused by model updates during the learning of new classes results in a performance drop for old ones, leading to "forgetting". This issue has been the focus of most prior studies and can be mitigated by freezing network parameters.
2. However, when the new classes are highly similar to the old ones, the classifier tends to confuse them. For instance, the model might misclassify an old class as a similar new class, as we explain the reasons and mechanisms behind this phenomenon in Figure 1 of our paper. This also causes a drop in the classification accuracy of the old classes, leading to "forgetting". Such a scenario, where new and old classes are similar, is common in biological recognition tasks and remains a significant challenge that has not been well-addressed in the few-shot incremental learning domain. In this paper, we specifically focus on addressing this issue.

Response to W2:
Feasibility of using DINO: DINO adopts a self-supervised pretraining approach, which indeed does not require explicit labels. However, during the pretraining phase, it is exposed to a large number of categories and learns their corresponding data distributions. Many of these categories are likely to appear in few-shot learning benchmarks, such as various plants and animals. This exposure could influence the model's incremental learning process; for instance, the model might leverage the learned data distributions to classify these categories by measuring the similarity between distributions. Consequently, from a rigorous academic perspective, this scenario constitutes data leakage, as the new categories are not genuinely unseen by the model. Nonetheless, if practical applications take precedence over strict academic rigor, DINO could be considered a viable backbone.

Gain on ResNet-50: With ResNet50 as the backbone, our method achieves an improvement of 2.67% in the final classification accuracy across all species and a reduction of 3.04% in 'PD' compared to the baseline, which is a considerable improvement on the CUB200 dataset. For example, in current SOTA algorithms like NC-FSCIL [a] and TEEN [b], the final classification accuracy differs by merely 0.13%. Due to time constraints, we only conducted additional experiments using NC-FSCIL [a] with ResNet50 as the backbone for comparison. The results demonstrate that our method achieves a notable improvement of 1.19% in final classification accuracy (64.75% vs. 63.56%) and a reduction of 1.37% in 'PD' (18.32% vs. 19.69%) compared to NC-FSCIL [a]. This further validates that our method offers consistent advantages across different backbones.

Response to W3:
If you could kindly share your concerns regarding the description of our method in more detail and with greater clarity, it would greatly assist us in addressing your questions more effectively. In addition, we would like to outline the logical structure of the methodology section in our paper for your reference:

1. We first introduce the overall optimization process of the model. (Section 4.1)
2. Next, we detail the step-by-step optimization objectives and technical implementation details, progressively refining the description. (Section 4.2 and 4.3)
3. Finally, we provide theoretical justification for the rationale and effectiveness of our method. (Section 4.4)

We hope our responses adequately address your concerns. If so, we would sincerely appreciate your consideration in revising the scores accordingly. Thank you for your time!

References

[a] Neural collapse inspired feature-classifier alignment for few-shot class-incremental learning.

[b] Few-shot class-incremental learning via training-free prototype calibration.