MetaAdapter: Leveraging Meta-Learning for Expandable Representation in Few-Shot Class Incremental Learning

Thanks for your constructive comments! We hope our responses address your concerns, and we kindly invite you to reassess our submission. Please let us know if you have any further questions or suggestions.

Q1: The primary motivation of this paper is that previous methods do not update the learned representation during incremental sessions, thereby compromising model plasticity. However, there is a considerable body of research that focuses on balancing model stability and plasticity during incremental sessions while also updating the backbone. This broader body of work is not sufficiently discussed.

A1: In the FSCIL setting, continual updates often result in significant performance drops for the base session due to the limited data available for new classes. This is why many existing methods in FSCIL prioritize stability over continuous backbone adaptation [1]. Our approach focuses on addressing this challenge by using lightweight meta-initialized adapters to enhance flexibility while preserving stability, without overly compromising the performance on base classes.

Q2: MetaFSCIL, a method that updates the backbone during incremental sessions using meta-learning, aligns conceptually with the approach proposed in this paper. However, this method is not adequately discussed, and its relevance is under explored.

A2: Thank you for your valuable suggestion. We agree that MetaFSCIL is an important related method, and we will include a more detailed discussion in the revised manuscript to highlight its relevance and differentiate it from our approach. To clarify, MetaFSCIL samples a sequence of sessions to mimic the evaluation protocol during the base phase and evaluates the model using a meta-objective. In contrast, our MetaAdapter approach is designed with a different focus. Instead of mimicking meta-testing during the base phase, we focus on using meta-learning to obtain meta-initialized adapters that provide a generalizable starting point for expanding and refining feature representations. During the online incremental learning stage, MetaFSCIL uses Bi-directional Guided Modulation (BGM) to generate activation masks to mitigate forgetting. In comparison, our MetaAdapter framework keeps the backbone frozen and utilize it as a teacher model for knowledge distillation to guide the adaptation of lightweight adapters.

Q3: The concept of feature compactness loss (FCL) seems to overlap with FACT. A comparison with FACT would strengthen the contribution.

A3: FACT uses manifold mixup during the base session to generate virtual classes, which creates space for new categories in subsequent incremental learning stages. Our feature compactness loss (FCL) takes a different approach by compacting both inter-class and intra-class distances during the base session. This design prevents the embedding space from becoming overly dispersed, effectively enabling the model to better adapt to new tasks in future incremental sessions.

To further substantiate the advantages of FCL, we conducted additional experiments in the General Response to analyze the relationship between inter-class feature representations and embedding space reservation. Specifically, we evaluated the relative angular disparity $T(f_\theta)$ between new-class samples and base-class prototypes (Table 3) and the inter-class angular distance among base-class prototypes (Table 4). The results in Table 4 show that Inter-class angular distance among base-class prototypes decreases as $w_{fcl}$ increases, which shows that FCL effectively compacts the embedding space. Meanwhile, The $T(f_\theta)$ metric in Table 3 improves when $w_{fcl}$ is set to a moderate value, showing that reducing the angular separation among base-class features to an appropriate extent supports better adaptation to new classes.

Q4: The training sequence appears non-intuitive. Adapters are trained before fine-tuning the backbone, which may create inconsistencies.

A4: Thank you for this observation. The reason we train adapters before fine-tuning the backbone is due to the nature of the meta-learning task. In our approach, the meta-learning phase focuses on tasks involving unseen categories. If we fine-tune the backbone first, it would have already seen all categories, which would defeat the purpose of meta-learning by reducing its ability to generalize to new, unseen classes.

Thanks for your constructive comments! We hope our responses address your concerns, and we kindly invite you to reassess our submission. Please let us know if you have any further questions or suggestions.

Q1: The meaning of 'c_pseudo,' mentioned in lines L223-L224, is difficult to understand, and a formal definition would be helpful. Additionally, in Equation 4, the dimension of 'p_concat' is unclear. It appears that 'c_batch' has a shape of $B \times C \times d$, while 'p' has a shape of $B \times d$. If this is the case, concatenation would be impossible; if not, further clarification on the structure of 'p_concat' is necessary.

A1: We appreciate your suggestion and will provide a formal definition of 'c_pseudo' and clarify the dimensions of 'p_concat' in the revised version of our manuscript. To clarify, 'c_pseudo' refers to prototypes for categories not present in the current batch, derived from the mean feature vectors of these unseen classes from the previous epoch. The batch means $\mathbf{c}_{\text{batch}}$ are prototypes for each class in the current batch, while the original feature vectors $\mathbf{p}$ represent the features of all samples in the current batch. We concatenate these three components to form 'p_concat'. For example, if 'c_batch' has a shape of (N, d), 'c_pseudo' has a shape of (K, d), and $\mathbf{p}$ has a shape of (M, d), then 'p_concat' will have a shape of ((N + K + M), d) after concatenation.

Q2: Figure 2 is difficult to follow, particularly regarding the adapter’s structure. The adapter appears to share convolutional layers with the backbone model, as suggested by the gray and sky-blue colors. However, the gray coloring makes this unclear.

A2: To clarify, the gray components in Figure 2 represent the convolutional layers in the backbone where no adapter is applied. These layers are shared across all tasks and remain unchanged during incremental sessions. The sky-blue components indicate the convolutional layers within the blocks that contain parallel adapters. These adapters form residual connections with the corresponding convolutional layers in the backbone, enabling task-specific adaptation without modifying the structure of the backbone.

Q3: The channel dimensions between the adapter and backbone layers seems to differ, but Equation 13 suggests a straightforward addition, which seems inconsistent if the channel numbers do not match.

A3: The addition of the adapter and backbone layers is done by merging the final $3 \times 3$ kernel with the $1 \times 1$ kernels. We achieve this by zero-padding the $1 \times 1$ kernels and aligning them at the center of the $3 \times 3$ kernel. This transformation requires both layers to have the same stride, with the $1 \times 1$ layer having one pixel less padding. For example, if the $3 \times 3$ layer uses padding = 1 (commonly used), the $1 \times 1$ layer should have padding = 0, ensuring consistent channel dimensions for addition.

Q4: The motivation for Feature Compactness Loss (FCL) is to compact the feature space after the base session to reserve room for new classes. However, existing FSCIL methods with similar motivations focus on maximizing inter-class margins instead. Given this contrast, additional analysis is needed to explain why compacting features is more effective than maximizing margins.

A4: Existing FSCIL methods that focus on maximizing inter-class margins, such as ALICE [1], aim to enhance feature discrimination by increasing the separation between base-class prototypes. However, as CLOM [2] suggests, this strategy can lead to overfitting on base classes, which may result in degraded generalization to few-shot new classes. Moreover, such methods ignores the issue of Minority Collapse [3] , where features of few-shot classes cluster excessively tightly in imbalanced scenarios, further reducing the model's adaptability to new classes. In contrast, our Feature Compactness Loss (FCL) is specifically designed to compact both inter-class and intra-class distances during the base session. This design prevents the embedding space from becoming overly dispersed, effectively enabling the model to better adapt to new tasks in future incremental sessions. As demonstrated by the experimental results in our manuscript, our method outperforms ALICE.

To further substantiate the advantages of FCL, we conducted additional experiments in the General Response to analyze the relationship between inter-class feature representations and embedding space reservation. Specifically, we evaluated the relative angular disparity $T(f_\theta)$ between new-class samples and base-class prototypes (Table 3) and the inter-class angular distance among base-class prototypes (Table 4). The results in Table 4 show that Inter-class angular distance among base-class prototypes decreases as $w_{fcl}$ increases, which shows that FCL effectively compacts the embedding space. Meanwhile, The $T(f_\theta)$ metric in Table 3 improves when $w_{fcl}$ is set to a moderate value, showing that reducing the angular separation among base-class features to an appropriate extent supports better adaptation to new classes.

Q5: Fairness issue: In Appendix A, you use ResNet-12 for both mini-ImageNet and CIFAR-100 experiments, while other methods like FACT and ALICE adopt ResNet-18 for mini-ImageNet. This discrepancy may result in unfair comparisons.

A5: In our experiments, we observed that prior studies use different backbone architectures for mini-ImageNet and CIFAR100, as highlighted in Table 8 of our paper. We followed C-FSCIL [4] and selected the shallowest architecture, ResNet-12, for both mini-ImageNet and CIFAR100. For CUB200, all methods consistently use the same architecture, ResNet-18 pretrained on ImageNet.

Additionally, the parameter differences between ResNet-12 and ResNet-18 are relatively small. Specifically:

• ResNet-18: 11.57M total parameters, with 0.29M trainable parameters for adapters.
• ResNet-12: 12.81M total parameters, with 0.32M trainable parameters for adapters.

[1] Few-Shot Class-Incremental Learning from an Open-Set Perspective. ECCV2022.

[2] Margin-based few-shot classincremental learning with class-level overfitting mitigation. NeurIPS2022.

[3] Neural collapse inspired attraction--repulsion-balanced loss for imbalanced learning. Neurocomputing2023.

[4] Constrained Few-shot Class-incremental Learning. CVPR2022.

Thanks for your constructive comments! Below, we address your concerns point by point.

Q1: While the paper presents reproducible results across three datasets, some existing techniques implemented in the program are not mentioned in the manuscript. For example, the rotation technique appears to draw from previous work [1].

A1: Due to space constraints, we did not include the detailed augmentation strategies in the manuscript, but we will add them in the revised version. The augmentation strategies used in our work are consistent with those in prior studies.

Q2: A more detailed discussion of the novelty in combining existing techniques is needed. The paper should elaborate on how these techniques contribute to improved performance and reduced forgetting, highlighting any unique modifications or interactions.

A2: Our MetaAdapter framework introduces a novel combination of techniques specifically designed to balance plasticity and stability in few-shot class-incremental learning (FSCIL). To enhance plasticity, we leverage Meta-Initialized Adapters, which provide task-agnostic initialization for adapting to new tasks, and Feature Compactness Loss (FCL), which compacts both inter-class and intra-class distances during the base session to reserve embedding space for future incremental tasks. To improve stability, we incorporate Sharpness-Aware Minimization (SAM) to locate flatter local minima during backbone training. This improves the generalization of the learned representation and reduces forgetting, particularly when adapter parameters are merged back into the backbone for final inference.

Q3: The benchmark comparisons should include detailed network architecture specifications for all compared methods. Additionally, model parameter sizes should be listed, as MetaAdapter introduces additional parameters.

A3: Detailed network architecture specifications for all compared methods have been included in Appendix (Table 7). Additionally, we provide the parameter sizes for our method to illustrate its efficiency:

• ResNet-18: 11.57M total parameters, 0.29M trainable parameters for adapters
• ResNet-12: 12.81M total parameters, 0.32M trainable parameters for adapters

The additional parameters introduced by MetaAdapter constitute less than 3% of the total parameter count.

[1] Learning with Fantasy: Semantic-Aware Virtual Contrastive Constraint for Few-Shot Class-Incremental Learning. CVPR2023.

[2] Few-Shot Class-Incremental Learning via Training-Free Prototype Calibration. NeurIPS2023.

[3] Few-Shot Incremental Learning with Continually Evolved Classifiers. CVPR2021.

Thanks for your constructive comments! Below, we address your concerns point by point.

Q1: Two highly related works [1, 2] with the same task setting are not compared.

A1: We have updated our experimental results to include a detailed comparison with [1] in our manuscript. However, we are unable to directly compare with [2] as it primarily utilizes a ViT-based architecture, which differs from the ResNet backbones used in our work. Nevertheless, our method demonstrates superior performance on both CIFAR100 and mini-ImageNet datasets.

On mini-ImageNet

On CIFAR100

Q2: Why don't more similar inter-class feature representations affect classification?

A2: Excessive feature compactness, caused by larger $w_{fcl}$ values, can negatively impact base session performance by reducing inter-class separation. By contrast, an appropriately $w_{fcl}$ balances compactness and separation, preserving space for new-class adaptation while maintaining stability in the feature space. In the General Response, we conducted experiments on both base and incremental accuracy across varying $w_{fcl}$ values. From Table 1, base accuracy remains stable across different $w_{fcl}$ values, but excessively high values (e.g., $w_{fcl} = 3.0$) reduce performance due to over-compactness, which impacts base-class feature representation. From Table 2, incremental accuracy improves with moderate $w_{fcl}$ values (e.g., $w_{fcl}=1.0$), as this balances compactness and separation. Extremely high or low $w_{fcl}$ values reduce incremental accuracy due to excessive compactness or dispersion in the feature space.

Q3: Need further proof or experiments to demonstrate that more similar inter-class feature representations reserve embedding space. Additionally, will more similar inter-class representations possibly lead to the overall embedding space shrinking, similar to collapse in contrastive learning?

A3: To address this concern, we conducted additional experiments in the General Response to analyze the relationship between inter-class feature representations and embedding space reservation. Specifically, we conducted experiments on both base and incremental accuracy across varying $w_{fcl}$ values (Table 1 and Table 2) and evaluated the relative angular disparity $T(f_\theta)$ between new-class samples and base-class prototypes (Table 3) and the inter-class angular distance among base-class prototypes (Table 4). The results in Table 4 indicates that Inter-class angular distance among base-class prototypes decreases as $w_{fcl}$ increases, which shows that FCL effectively compacts the embedding space. As shown in Table 1 and Table 2, this compactness does not result in an overall collapse of the embedding space. Additionally, the $T(f_\theta)$ metric in Table 3 improves with moderate $w_{fcl}$, showing that reducing the angular separation among base-class features to an appropriate extent supports better adaptation to new classes.

Q4: Need to prove that it is the reservation of space that helps future tasks, not other factors. For example, reducing the number of novel categories in the few-shot adaptation task can also reserve space. Will this also improve performance?

A4: To address this, the ablation study has been updated to include results using only FCL in Table 2 of the manuscript, which demonstrate that the reservation of embedding space during the base session contributes to the observed performance improvements. And we would also like to clarify that this space reservation occurs mainly during the base session learning process and is irrelevant to the number of novel categories in subsequent tasks.

Q5: The results using only FCL are missing in Table 2.

A5: Thank you for catching this. We have updated Table 2 to include the results using only FCL. The updated results demonstrate that FCL alone improves the model’s performance.

Q6: It would be better to state how to expand $W_{t-1}$, as Figure 2(b) only shows $W_t$.

A6: Thank you for your suggestion. Figure 2(b) illustrates that $W_t$, the weight matrix for the current task $t$, consists of two components: (1) weights from all previous classes ($W^{t-1}$), which are retained as the learned representations from previous tasks, and (2) new weights for the current task ($w^t_1,w^t_2,w^t_3,\dots$), which are derived from the feature means of the samples for each new class.

Q7: Since the ViT backbones are receiving more attention, I wonder if MetaAdapter and FCL could be applied to methods with ViT backbones?

A7: Your suggestion provides a valuable direction for future research. Our MetaAdapter framework is general and can be extended to Vision Transformer (ViT) backbones. Specifically, adapting MetaAdapter would involve inserting the lightweight neural modules to work in parallel with self-attention layers. These modules can be implemented with residual connections and structured with a down-projection matrix, a nonlinear activation function, and an up-projection matrix to enable efficient representation learning. In addition, FCL is flexible and can be directly applied to ViT features, as it focuses on regulating feature dispersion. We believe that combining our MetaAdapter framework and FCL with ViT backbones has the potential to achieve superior performance on more complex tasks.

[1] Improved Continually Evolved Classifiers for Few-Shot Class-Incremental Learning. TCSVT2023.

[2] Rethinking Few-shot Class-incremental Learning: Learning from Yourself. ECCV2024.

Thank you to the authors for their efforts in the rebuttal. While some of my concerns have been addressed, several issues still remain:

1. Regarding the $W_{t-1}$ and $W_t$ mentioned in Q6, I am actually curious that although their dimensions are different, both $z^{t-1}$ and $z^t$ have $c^t$ weights in Eq.10. Is there perhaps some intermediate steps missing here?
2. According to the newly added results in Table 2 of the article, using only FCL yields better performance than SAM+FCL and is comparable to MIS+FCL. Could the authors please provide further explanation on this?
3. In Table 1 of the General Response, the results don't seem very stable. Together with Table 2, they show that under reasonable parameter values, FCL can enhance the model's generalization ability with minimal harm to its classification performance. This is a somewhat clever method, but I'm not quite sure about its stability and robustness. In addition, Could the authors provide the baseline performance in Tables 1 and 2 of the General Response, so that we can observe the range of $w_{fcl}$ where FCL brings improvements?

Thank you to the authors for their detailed responses. Most of my concerns have been addressed, and I have raised my score to 6: marginally above the acceptance threshold. However, considering the performance of the method in the ablation study and its potential to harm classification, I still believe that the stability and generalizability of the method could be improved.

Thanks for your constructive comments! Below, we address your concerns point by point.

Q1: Limited Novelty in Meta-Learning Approach. The application of meta-learning for adapter initialization resembles existing methods. Clarification on how this approach differs from established frameworks would strengthen the contribution.

A1: Existing methods based on meta-learning, such as MetaFSCIL [1], focus on meta-testing simulation by sampling sequences of incremental tasks from base classes during the base phase. However, the inherent discrepancies in data distributions make it difficult to use base session data to accurately simulate real incremental sessions. Instead of mimicking the incremental evaluation process, we use meta-learning to obtain meta-initialized adapters that serve as a generalizable starting point for expanding and refining feature representations. This design helps reduce overfitting and enhances the model's plasticity for adapting to new tasks efficiently.

Q2: Training Complexity. The three-phase training process, including feature compactness loss and sharpness-aware minimization, adds complexity. This may pose challenges for implementation in resource-constrained environments.

A2: Our training process is divided into three phases:

1. Phase 1 (Adapter Meta-Training): During the base session, we first construct few-shot tasks by randomly sampling instances from each base class and then train the adapters using meta-learning algorithms to obtain generalizable initial parameters.

2. Phase 2 (Backbone Pretraining): In this phase, we focus on training the backbone. We introduce the feature compactness loss (FCL) to bring feature representations closer together and apply sharpness-aware minimization (SAM) to find flatter minima.

3. Phase 3 (Few-Shot Adaptation): We only fine-tune the lightweight adapters for each new task to expand the current representations to encompass new class features.

Phase 1 is computationally efficient because it involves only the limited size of few-shot tasks. And the feature compactness loss and sharpness-aware minimization are applied solely during Phase 2 to train the backbone. In Phase 3, we fine-tune the adapters with just 1-5 iterations, which enables rapid adaptation for few-shot new classes.

Q3: Base Task Performance. The framework appears to underperform in base classification tasks compared to other methods using the same backbone. Understanding the reasons for this discrepancy is crucial, as base task accuracy is vital for incremental learning stability.

A3: We would like to clarify that our framework achieves comparable performance to NC-FSCIL [2] and outperforms other compared methods on base classification tasks. In the context of FSCIL, while base task performance is important, the most critical challenge is mitigating catastrophic forgetting. Simply optimizing for base session performance may not lead to better results in the final session, as overfitting to base classes can reduce adaptability to new tasks.

Furthermore, the improved base task performance can be attributed to several factors. We utilize a prototype-based classifier with cosine similarity, where we re-scale the normalized features using a preset scale factor $\tau$. The choice of $\tau$ controls the separation between classes, which improves feature discrimination. For CIFAR100 and miniImageNet, we set $\tau = 64$, while for CUB200, we use $\tau = 32$ to better adapt to the characteristics of each dataset. Furthermore, we incorporate sharpness-aware minimization (SAM) to find flatter local minima by adding gradient-based perturbations to the parameters, which enhances the model’s generalization in base classification tasks.

Table 1: Final Base Accuracy Across Datasets for Varying $w_{fcl}$ Values ($w_{kd} = 1.0$)

Table 1 shows that base accuracy is relatively stable across varying $w_{fcl}$ values for all datasets. However, excessively high values (e.g., $w_{fcl}= 3.0$) lead to a slight drop in performance due to over-compactness in the embedding space, which can harm the representation of base classes.

Table 2: Final Incremental Accuracy Across Datasets for Varying $w_{fcl}$ Values ($w_{kd} = 1.0$)

As presented in Table 2, incremental accuracy benefits from moderate $w_{fcl}$ values (e.g., $w_{fcl}=1.0$), as this balances compactness and separation. Extremely high or low $w_{fcl}$ values reduce performance due to excessive compactness or dispersion in the feature space.

As presented in Table 2, incremental accuracy benefits from moderate $w_{fcl}$ values (e.g., $w_{fcl}=1.0$), as this balances compactness and separation. Extremely high or low $w_{fcl}$ values reduce performance due to excessive compactness or dispersion in the feature space.

Table 3: Relative Angular Disparity $T(f_\theta)$ for New-Class Adaptation Across Datasets for Varying $w_{fcl}$ Values ($w_{kd} = 1.0$)

As presented in Table 3, the $T(f_\theta)$ metric improves with moderate $w_{fcl}$, indicating that reducing the angular separation among base-class features appropriately supports better adaptation to new classes. However, excessively high $w_{fcl}$ compresses base-class features too much and reduces the model's ability to adapt to new-class representations effectively.

Table 4: Inter-class Angular Distance Among Base-Class Prototypes Across Datasets for Varying $w_{fcl}$ Values ($w_{kd} = 1.0$)

From Table 4, Inter-class angular distance among base-class prototypes decreases as $w_{fcl}$ increases, which shows that FCL effectively compacts the embedding space.

We hope that our response could relieve your concerns. Please let us know if you have any further questions or suggestions.

[1] Forward compatible few-shot class-incremental learning. CVPR2022.

[2] Few-Shot Class-Incremental Learning from an Open-Set Perspective. ECCV2022.

[3] Learning with Fantasy: Semantic-Aware Virtual Contrastive Constraint for Few-Shot Class-Incremental Learning. CVPR2023.

[4] Momentum contrast for unsupervised visual representation learning. CVPR2020.

[5] Margin-based few-shot classincremental learning with class-level overfitting mitigation. NeurIPS2022.

[6] Neural collapse inspired attraction--repulsion-balanced loss for imbalanced learning. Neurocomputing2023.

[7] CLOSER: Towards Better Representation Learning for Few-Shot Class-Incremental Learning. ECCV2024.

[8] Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class-Incremental Learning. ICLR2023.