Overcoming bias towards base sessions in few-shot class-incremental learning (FSCIL)

We would like to express our sincere gratitude for the time and effort you dedicated to reviewing our paper. Our response to the weakness and questions is present below.

Weakness_1.

A: Although aggregating predictions falls under the category of ensemble operations, we claim that our method differs significantly from conventional ones. Unlike typical ensemble methods that do not specify particular types of classification modules, our pipeline employs two distinct types: the base-favored and incremental-favored modules (Sec 3.4). It is crucial to note that simply combining a large number of classification modules does not necessarily lead to improvement. This is because many current methods excel only in the base, and combining such methods does not address the persisting issue of performance bias. As we have proposed in Sec 5.1, incorporating a classification module favoring the incremental session is essential for achieving overall balanced performance.

Weakness_2.

A: When training LN, we do not replay samples from the previous sessions. As we have described in the supplementary material (A), we trained a Gaussian generator for each class in each session (which is inspired by the setting of FeSSSS) and subsequently sampled data from the generator for the respective class.

Weakness_3.

A: While SAVC and BiDistFSCIL are the latest approaches, the high IMB values in Table 3 indicate that their performance exhibits a pronounced bias toward the base session. This results in an inflated value of the biased conventional metric, misleadingly suggesting proficiency as FSCIL methods. Conversely, ALICE achieved the most balanced performance according to our proposed metrics. Therefore, we decided to use ALICE as our baseline.

Weakness_4.

A: The discrepancy arises from our evaluation using the newly proposed metrics. While previous approaches assess the performance for all classes encountered up to the current session after training at each session, we assess session-level adaptability by exclusively evaluating the classes relevant to a specific session only after training on the last session (Sec 4.2). Hence, our reported values differ from those provided in the original papers.

Weakness_5.

A: We acknowledge that the improvement in performance compared to existing methods can be attributed to the larger backbone. We would appreciate it if the focus could shift from such a perspective to ‘exploring how to effectively utilize backbones with a larger number of parameters (especially in the size constraint of ResNet50) in FSCIL.’ Tables 1 and 2 illustrate that when using a larger backbone, simply employing LN, GAT, or NCM individually results in a biased performance—either exhibiting high accuracy only in the base session (LN and GAT) or relatively higher incremental performance with a notable decline in base performance (NCM). We emphasize that integrating the two specific types of classification modules we proposed can effectively address this issue by achieving an optimal balance in performance compared to the naive use of a large backbone.

Weakness_6.

A: We assumed that fine-tuning the backbone with base classes induces an inevitable bias towards the base classes, a notion supported by the biased performance of current methods observed in our experiments. Such biased features may not only be unhelpful but could potentially be detrimental in adapting to the incremental sessions. Consequently, we posited the frozen pretrained backbone would be better rather than fine-tuned ones even if the pretrained knowledge is not highly correlated with the incremental sessions. While there may be instances where fine-tuning with base classes can be highly beneficial for adaptation to incremental sessions if there is a significant similarity between the distributions of base and incremental sessions, we believe such instances are generally rare. Therefore, we did not fine-tune the pretrained backbone with the base classes. However, as you suggested, using a fine-tuned backbone as a baseline could provide a more in-depth understanding of the impact of fine-tuning with base classes. We will carefully consider this advice.

Weakness_7.

A: From the perspective of the number of images per base class, (as we have described in sec 4.1,) the proposed datasets have more images—Flowers102 has 40, DCC has 80 images—than CUB200 where there are around 30 training images per base class. Furthermore, they can serve as valuable tools for assessing the robustness against the target distribution shifts. However, we agree that more test images would have been beneficial.

Q1.

A: Table 8 indicates the results evaluated with our proposed metrics where both baseline and pretrained backbones are used. The setup is the same as that of Figure 4.

Q2.

A: As you mentioned, since only the first 100 classes of Flowers102 are utilized, renaming it to Flowers100 seems appropriate to distinguish it from the original Flowers102.

We would like to express our sincere gratitude for the time and effort you dedicated to reviewing our paper. Our response to the weakness and questions is present below.

1. In the incremental sessions, whether there have two results for one sample from the NCM classifier and Lightweight Network, and how to deal with this situation.

A: We summarize the detailed process of prediction aggregation below. The aggregation process primarily entails summing the two output distributions from LN and NCM. However, we noticed LN's output distribution exhibits excessive sharpness compared to NCM, leading to LN’s dominance when simply summed, making the prediction identical to using only LN's output distribution. To address the issue, we smooth the LN’s output (softmax) distribution with an appropriate temperature [ref.1] before adding it to that of NCM. Through multiple experiments, we determine the optimal temperature value for this adjustment. With this modification, we observed that when evaluating test data for the base session, LN's smoothened output distribution has a slightly stronger influence than that of NCM. Conversely, when evaluating test data for the incremental sessions, the influence of NCM's output distribution is slightly stronger than that of LN. Consequently, this aggregation process allows for achieving a more balanced overall performance. [ref.1] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733–3742, 2018.

2. In the experiments, I'm concerned about the accuracy of each session and the accuracy of each session compared to other methods.

A: We summarize the details about our proposed ‘accuracy of each session’ (Section 4.2) below. As we have described in Sec 4.2, unlike previous methods, we have conducted evaluations only after completing training on the last session. While previous approaches assess the performance for all classes encountered up to the current session after training at each session, we assess session-level adaptability by exclusively evaluating the classes relevant to a specific session only after training on the last session.

If we have misunderstood your question, please feel free to clarify, and we appreciate your understanding.

3. Whether the paper makes the comparison without using a pre-trained model.

A: We conducted experiments in our pipeline without utilizing a pretrained backbone (the baseline is still ALICE). The results of the experiments are as follows.

The results show that using only the baseline backbone exhibits biased performance towards the base session. This reaffirms the significant assistance that the pretrained backbone provides in adapting to incremental sessions.

[ref.1] Zhirong Wu et al. "Unsupervised feature learning via non-parametric instance discrimination." CVPR 2018.

We would like to express our sincere gratitude for the time and effort you dedicated to reviewing our paper. Our response to the weakness and questions is present below.

Weakness_1,4.

A: Using only LN or NCM individually does not yield optimal outcomes. LN excels in the base session but falters in the incremental sessions, while NCM performs well in the incremental sessions but poorly in the base. Given their contrasting tendencies, we hypothesized that the two classification modules can potentially complement each other. Indeed, we observed that aggregating their output distributions significantly improved overall performance (Section 5.1).

• The detailed process of the aggregation The aggregation process entails summing the output distributions from LN and NCM. However, we noticed LN's output distribution exhibits excessive sharpness compared to NCM, leading to LN’s dominance when simply summed. To address this, we smooth LN’s distribution with an optimal temperature [ref.1] before adding it to NCM's. Through multiple experiments, we find the optimal temperature, observing that LN's smoothened distribution slightly dominates when evaluating test data for the base session while NCM's has a slightly stronger influence when evaluation for the incremental sessions. Consequently, this aggregation enhances overall performance balance.

Weakness_2,6.

A: As we have described in Sec 4.1, we addressed the concern of the violation by setting the target distribution that diverges significantly from the pretraining dataset (ImageNet). Our criterion is as follows: if a self-supervised pretrained backbone yields better transferability to a target dataset than the supervised pretrained one, the target dataset has a considerable distribution shift from the pretraining dataset. This is primarily grounded in the fact that the supervised training aligns more closely with the distribution it trained with, due to the availability of label information.

According to [ref.2], when the pretraining dataset is ImageNet, and the target dataset is Flowers102, DTD, or CD-FSL benchmarks (including CropDiseases and ChestX), self-supervised pretrained backbones outperform. Based on this, we chose Flowers102, DTD, and the CD-FSL benchmarks for the new datasets. However, due to limited data in DTD and CD-FSL benchmarks, we decided to combine them. Among the four datasets in CD-FSL, since EuroSAT has a much smaller resolution than the others, and including ISIC makes training challenging, we propose the new dataset DCC, which consists of DTD, CropDiseases, and ChestX. Unlike previous benchmarks similar to ImageNet, these datasets can introduce greater diversity to the benchmarks.

Weakness_3.

A: We appreciate your feedback on our oversight.

Weakness_5.

A: The reported a_0 from the previous methods are measured after the training on the base session. However, as we have described in Sec 4.2, we evaluated performance after training on the last session.

We acknowledge that the improvement in performance can be attributed to the larger backbone. We would appreciate it if the focus could shift from such a perspective to ‘exploring how to effectively utilize larger backbones (especially in the size constraint of ResNet50) in FSCIL’. (Further details are provided in the 5th response to R3)

Weakness_7.

A: Unlike the conventional metric, we assess session-level accuracy by exclusively evaluating the classes in a specific session only after training on the last session, i.e., the evaluation occurs solely for the classes in the i-th session to obtain a_i. Therefore, the difference(IMB) captures the extent of the disproportionately high performance of the base session compared to the average incremental performance, effectively measuring the degree of bias. Additionally, given the different composition of test set classes, our proposed 'session average' represents the average session-level accuracy, differing fundamentally from the previous one.

Weakness_8.

A: Here, the robustness is related to the selection of the pretrained backbone in our pipeline. As seen in Figure 4, in the case of CNNs, we notice that the supervised pretrained backbone outperforms the self-supervised ones on CUB200, close to ImageNet's distribution (compared to the other two). However, the trend reverses for Flowers102 and DCC, indicating sensitivity to the target dataset. In contrast, for ViTs, DINO consistently outperforms others across all datasets. Without the proposed datasets, choosing CNN backbones might favor the supervised pretrained one. When selecting ViT backbones, doubts may arise about DINO's performance in distributions different from ImageNet. Therefore, with our proposed datasets, we can identify backbones robust against target distribution shifts.

[ref.1] Zhirong Wu et al. "Unsupervised feature learning via non-parametric instance discrimination." CVPR 2018.

[ref.2] Linus Ericsson et al. "How well do self-supervised models transfer?" CVPR 2021.