Learning Expandable and Adaptable Representations for Continual Learning

We sincerely appreciate your positive feedback regarding the clarity of our writing and the validity of our collaborative backbone design motivation.

For weaknesses and questions, we provide our feedback as follows.

Q1: Implement of KLDBFA.

A1: We clarify that we approximate the feature distributions from both the Local backbone and Frozen Local backbone as multivariate Gaussian distributions, for which we derive closed-form solutions of their KL divergence. Actually, many modern generative model evaluation metrics (FID, Kernel MMD) assume that real and generated data can be approximated by Gaussian distributions in a high-dimensional feature space. Therefore, we can model the feature distributions from both backbones for a data batch as Gaussian distributions. The rationale for choosing KL divergence over alternative distributional metrics is because: (1) Unlike symmetric metrics such as Jensen-Shannon divergence, KL divergence's directional nature allows us to specifically constrain the current feature distribution to remain close to the historical distribution; (2) The computational overhead of distances like MMD and Wasserstein distance is relatively high.

In code implementation, we separately compute the mean vectors and covariance matrices of the feature vectors output by the Local Backbone and the Frozen Local Backbone, and then calculate the KL divergence between these two distributions $P({\bf Z}^l)=\mathcal{N}(\boldsymbol{\mu}_1, \boldsymbol{\Sigma}_1)$ and $P({{\hat{\bf Z}}}^l)=\mathcal{N}(\boldsymbol{\mu}_2, \boldsymbol{\Sigma}_2)$ ($\boldsymbol{\mu}$ and $\boldsymbol{\Sigma}$ denotes mean vectors and covariance matrices, respectively) using the following equation:

$D_{\mathrm{KL}}(P({\bf Z}^l) \,||\, P({\hat{\bf Z}}^l) ) = \frac{1}{2} \left[\log \left( \frac{\det \boldsymbol{\Sigma}_2}{\det \boldsymbol{\Sigma}_1} \right)- d + \mathrm{tr}(\boldsymbol{\Sigma}_2^{-1} \boldsymbol{\Sigma}_1)+ (\boldsymbol{\mu}_2 - \boldsymbol{\mu}_1)^\top \boldsymbol{\Sigma}_2^{-1} (\boldsymbol{\mu}_2 - \boldsymbol{\mu}_1)\right]$

where 'det', 'd', and 'tr' denote the determinant, dimension, and trace of a matrix, respectively.

Q2: The source of LEAR performance improvement and collaborative backbone design independent performance.

A2: We clarify that we did not employ larger pre-trained backbones compared to baseline methods to enhance performance. Specifically, the performance improvement stems from three synergistic components: (1) the collaborative global-local backbone structure, (2) the task-specific dynamically expandable expert networks, and (3) the proposed three regularizations terms.

We note that all four reviewers raise concerns in the individual contributions of LEAR's components and have provided consistent suggestions for improving our ablation studies. Recognizing the importance of this issue, we have conducted extensive additional experiments on ETI and TRC sequences during the rebuttal phase and reorganized the ablation study tables accordingly:

Table 1: Results of revised ablation studies

Where ''CB'' denotes using only the collaborative backbone with a single shared expert network across all datasets; ''CBE'' extends CB with task-specific expert network expansion and ESM expert selection; ''SBE'' denotes the configuration where ''CBE'''s dual backbone architecture is replaced with a single backbone; and "CBE+MI/KL/HSIC" represents CBE augmented with individual components or combination of components.

This table demonstrates that the collaborative backbone design (''SBE''->''CBE'') and the dynamically expanded expert network (''CB''->''CBE'') significantly enhances model performance on both ETI and TRC. In addition, each regularization term and their combinations contribute to varying degrees of performance improvement over ''CBE'' on these two sequences.

Q3: The use of pre-trained ViT.

A3: We initialize both the global backbone and the local backbone using a ViT-B/16 model pre-trained on ImageNet-21K. All comparison methods used the same pre-trained model, except for the MoEAdapter which adopted the CLIP ViT-B/16.

Q4: The relation between the number training parameters and the GPU usage.

A4: We would like to clarify that GPU memory consumption in our experiment setting is not solely determined by the number of trainable parameters; it can stem from multiple sources, including the size of the pre-trained model used, additional components such as adapters or prompts introduced for continual learning, and the use of memory buffers. For instance, although MoE Adapter has a relatively small number of trainable parameters, it employs the CLIP model as its backbone. Furthermore, the majority of LEAR's trainable parameters originate from fine-tuned layers in the ViT backbone, resulting in substantial GPU usage overlap with the original frozen ViT. Additionally, gradient information is not preserved during training. Therefore, the low GPU memory usage of LEAR is both reasonable and expected.

Q5: Why LEAR has the almost comparable number of training parameters with DER++ and CLS-ER.

A5: We clarify that both DER++(Rresh) [1] and CLS-ER [2] were originally implemented using a single ResNet-18 backbone without frozen parameters. To ensure fair comparison when migrating these baselines to ViT, we adopt the consistent backbone configurations between LEAR (keep last three layers of both backbones trainable) and other baselines (DER++(Rresh) and CLS-ER). This results in comparable scales of trainable parameters across all three methods.

In practice, we have also conducted comparative experiments by activating the last three layers of ViT backbones for other baselines, yielding the following results:

where ''FTX'' denotes that we activate the final ''X'' layers of the ViT backbone. These results indicate that merely scaling up model parameters cannot guarantee performance gains in MDCL scenario, which necessitates the incorporation of regularization constraints during backbone adaptation to mitigate adverse parameter optimization caused by domain shift.

In addition, we further considered how to reduce the number of training parameters. We tried fine-tuning only the MHSA and MLP sublayers in the last three layers of the VIT backbone:

These results demonstrate that fine-tuning only the MHSA achieves comparable or even better performance than fine-tuning the entire block. This move reduced the total number of parameters to 14.27M, a significant reduction of 28.27M compared to the original.

Q6: Typo: concentration operation.

A6: Thank you for highlighting this oversight. It is indeed a concatenation operation, we have corrected the word and symbol in the revised paper.

References:

[1]A unified and general framework for continual learning//ICLR24

[2]Learning fast, learning slow: A general continual learning method based on complementary learning system//ICLR22

We are grateful for your valuable feedback on our work. We have reorganized the ablation studies below to systematically evaluate component contributions:

Table 1: Effectiveness of different component combinations in LEAR

Table 1 clearly demonstrates the contribution of each module in LEAR:

(1) ''UpperBound'' denotes the results in a single-domain setting, which closely approximates LEAR's performance upper bound in MDCL;

(2) ''CB'' denotes using only the collaborative backbone with a single shared expert network across all datasets;

(3) ''CBE'' extends ''CB'' with task-specific expert network expansion and ESM expert selection. The results demonstrate that the dynamic expert network expansion and selection (''CB''->''CBE'') significantly enhance model performance on both ETI and TRC;

(4) ''SBE'' denotes the configuration where ''CBE'''s dual-backbone architecture is replaced with a single backbone. The dual-backbone design yields about a 4% performance gain over the single-backbone counterpart (''SBE''->''CBE'');

(5) "CBE+MI/KL/HSIC" represents ''CBE'' augmented with individual components or combination of components. Each regularization component and its respective combinations yield effective performance enhancements relative to ''CBE'' on both sequences;

(6) "LEAR" denotes the complete framework, while "LEAR w/o ESM" represents the variant where experts are randomly selected during both the initialization and testing phases of each task. This configuration explains the observed significant performance degradation ("LEAR"->"LEAR w/o ESM")

Table 2: Performance comparison of different backbone fine-tuning configurations in LEAR​

We conducted comparative experiments on different fine-tuning configurations for the collaborative backbones of LEAR, where "FTX" denotes fine-tuning the last X layers while freezing preceding backbone layers. As shown in Table 2, the configuration utilizing the last three trainable layers ("FT3") of the collaborative backbones achieves robust performance with limited parameter growth compared to alternative configurations.

Dear Reviewer tTuR,

We sincerely appreciate that you may be occupied with other commitments and thus unable to provide a response to our rebuttal at this time. In light of this, we provide further explanations for your previous comments. We hope these supplemental analyses meet with your approval.

Regarding KLDBFA, While KL divergence traditionally applied to prediction probabilities, it can effectively regularize feature distributions under a parametric distributional assumption, such as Gaussian distributions, as implemented in Variational Autoencoders (VAEs). In the implement of KLDBFA, We derive closed-form solutions for the KL divergence between the estimated feature distributions of the frozen local backbone and those of the local backbone to align historical and current representation distributions at the feature level. In addition, the t-SNE visualization results presented in Figure 1 of the Appendix from Supplementary Materials demonstrate unimodal patterns in the feature representations extracted from two MDCL sequences, thereby providing additional support for modeling these representations as Gaussian distributions.

Concerning analyses of performance gain, following all four reviewers' collective suggestions, we have reorganized the ablation studies to systematically evaluate component contributions in LEAR. The results have demonstrated that our proposed:

(1) collaborative backbone architecture,

(2) task-specific expert networks,

(3) three regularization terms (MIBPA, KLDBFA, HSICBCO), and

(4) Expert Selection Mechanism (ESM)

collectively ensure LEAR’s superior performance in MDCL scenarios, achieving optimal balance between plasticity, stability and efficiency.

Regarding parameter count and GPU usage, We provide a more detailed technical elaboration:

While baseline methods including prompt-based approaches (e.g., DualPrompt) and adapter-based methods (e.g., MoE-Adapters) indeed contain fewer trainable parameters (0.5%-5% of ViT-B/16 backbone), they inevitably require complete backpropagation through all ViT layers, necessitating the storage of intermediate activations throughout the entire vit backbone due to chain rule dependencies, which maintain extensive computation graphs;

conversely, LEAR's innovative design strategically fine-tunes only the last three ViT layers while terminating backpropagation after the third-last layer, thereby significantly reducing computation graph and GPU usage.

We sincerely look forward to receiving any additional suggestions you may have. We are fully committed to addressing all of your concerns, which will undoubtedly enhance the quality and impact of our work.

We appreciate the valuable comments by Reviewer NZTA.

Q1: The use of dual-branch architecture in CL.

A1: We clarify that our method differs fundamentally from dual-branch approaches in both architecture and learning dynamics. Specifically, dual-branch methods like DualNet and CLS-ER rely on buffer to preserve historical knowledge, while LEAR is Rehearsal-Free. Additionally, these methods fully fine-tune the entire pretrained models, whereas LEAR only fine-tunes the last three layers of the backbone. Furthermore, existing approaches usually lack or even omit the feature interaction between the two branches, while LEAR employs MIBPA, KLDBFA, and HSICBCO to establish robust global-local knowledge consolidation and adaptation.

Q2: The differences from the existing dynamic expandable models.

A2: Conventional approaches typically utilize a single pre-trained ViT as the foundational backbone to facilitate adapter integration. In contrast, the proposed LEAR framework optimizes a collaborative backbone architecture, incorporating both global and local backbones to deliver task-agnostic and adaptive feature representations. Leveraging this collaborative structure, LEAR executes expert construction via a comprehensive feature integration process. This architecture effectively accommodates heterogeneous data domains, representing a novel direction within the continual learning landscape. Furthermore, while baseline methods rely on routers or dynamic weighting schemes for expert aggregation, LEAR introduces an innovative Expert Selection Mechanism (ESM) that autonomously identifies the most suitable expert and leverages its parameter information for constructing new experts. This mechanism enhances the learning of novel tasks and constitutes an unexplored strategy in current continual learning research. To further demonstrate LEAR's effectiveness compared to other expansion-based approaches in MDCL,We have supplemented the experimental comparison between LEAR and three baseline methods: EASE[1], SEMA[2], and the LoRA-based approach InfLoRA [3], on both ETI and TRC sequences. Experimental results in Table 1 confirm LEAR's significant performance advantage over these baselines.

Table1: Supplementary performance comparison between relevant methods and LEAR

Q3: The novelty of MIBPA and the motivation of MI rather than CE.

A3: Ensuring consistency with historical predictions has been extensively adopted in regularization-based approaches. Typically, these methods formulate the current active model (classifier) as a student module and duplicate it as a teacher module following each task transition. Catastrophic forgetting is mitigated by minimizing the divergence between the outputs of the student and teacher modules. While this framework offers robust stability, it often compromises adaptability when confronted with complex datasets. In contrast to conventional regularization-based techniques, the proposed MIBPA introduces two key innovations:

(1) Rather than duplicating the entire classifier as a teacher module after each task switch, MIBPA incorporates a parameter-sharing auxiliary model that retains select parameters from the global backbone (specifically, the final three layers) and leverages this auxiliary model to guide the optimization of the global backbone. This strategy effectively reduces the overall parameter count;

(2) Whereas existing methods enforce output alignment between the current classifier and the teacher module during training, MIBPA focuses on preserving the prediction consistency of each historical expert during global backbone optimization, while simultaneously encouraging the current expert to assimilate data from the ongoing task (MI is symmetric). This approach utilizes supervised signals from all previously acquired experts to control the optimization process of the global backbone and is entirely different from the existing regularization-based methodologies.

Exceptionally, SLCA [4] models the class-wise distributions of historical tasks and aligning the classification layers in a post-hoc fashion. However, the experimental results in Table 1 demonstrate that this modeling and alignment approach struggles to adapt to the complex task prediction distributions in MDCL.

Additionally, we performed ablation studies substituting the mutual information component in MIBPA with cross-entropy (CE) loss. The experimental results ('CBE+CE' in Table 3) demonstrate a substantial performance degradation when employing asymmetric CE. This is because the model over-emphasizes the historical prediction distributions while neglecting to learn the current task. This empirical evidence motivated our choice of mutual information.

Q4: Truly novel components in LEAR.

A4: We claim that the expert construction process is a known technology that was adopted in this study. In contrast, the proposed collaborative backbone structure, Mutual Information-Based Prediction Alignment, and Kullback–Leibler (KL) Divergence-Based Feature Alignment are novel technologies, which are not explored in the existing continual learning studies.

Q5: The distribution modeling in Eq7.

A5: We clarify that no probabilistic models were employed in this process. We sincerely invite you to refer to our response to Q1 in the rebuttal to reviewer tTuR, as we are constrained by word limits and unable to answer this question.

Q6: The task definition.

A6: We clarify that in the MDCL setting, each individual task corresponds to a complete, undivided benchmark dataset. Regarding the second setting you mentioned, we designed three distinct data sequences, each comprising two datasets and further partitioned into 10 tasks per dataset (20 tasks total). Each task is trained only 1 epoch. The experimental results demonstrate that LEAR still shows robust adaptability to domain shift and outperforms all three compared methods in terms of average task performance.

Table2

Q7: Improvement of ablation study.

A7: we have conducted extensive additional experiments on ETI and TRC sequences during the rebuttal phase and reorganized the ablation study tables accordingly:

Table3: Results of revised ablation studies

Where "CBE" denotes the configuration using solely the collaborative backbone with task-specific network expansion and ESM expert selection, and "CBE+MIBPA/KLDBFA/HSICBCO" represents CBE augmented with individual components or combination of components. The results demonstrate that each regularization term contributes to varying degrees of performance improvement over CBE across both ETI and TRC.

Q8: Code implement.

A8: Should our paper be accepted, we will definitely release the full codebase with comprehensive documentation.

References:

[1]Expandable subspace ensemble for pre-trained model-based class-incremental learning//CVPR24

[2]Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning//CVPR25

[3]Inflora: Interference-free low-rank adaptation for continual learning//CVPR24

[4]SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model//ICCV23

Thank you for your thoughtful feedback and for taking the time to review our paper. The proposed LEAR diverges fundamentally from existing techniques as follows.

Distinctiveness from Dual-Branch Methods

Compared to existing dual-branch methods like DualNet [1],TwF [2] and CLS-ER [3], LEAR diverges significantly in three aspects:

• Rehearsal-Free: Conventional dual-branch approaches often rely on memorized buffers​​ to retain historical knowledge. In contrast, the dual-backbone structure in the proposed LEAR does not require accessing any past examples.

• Optimization Scope: Unlike the existing methods that optimizes all parameters of the backbone, resulting in huge computational costs. In contrast, the proposed LEAR aims to optimize a few high-level representation layers of the backbones, which not only reduce computational cost but also improve the model's plasticity.

• Knowledge Interaction: Current dual-branch methods often lack feature interaction mechanisms, whereas LEAR incorporates three novel mechanisms (MIBPA, KLDBFA, HSICBCO) to enable robust global-local adaptation.

Differentiation from Expansion-based Paradigms

Existing expansion approaches fall into three categories:

• Backbone-wise: DER [4], Foster [5], MEMO [6], BEEF [7] train separate backbones per task
• Token-wise: Dytox [8], L2P [9], DualPrompt [10], CODA-Prompt [11] utilize dynamic tokens
• Expert-wise: MoE-Adapters [12], EASE [13], SEMA [14] grow task-specific subnetworks

LEAR belongs to the third category and introduces two fundamental innovations:

• Collaborative Backbone Architecture: Unlike adapter-based expansion approaches with a single, fully frozen backbone, LEAR fine-tunes the last three layers of both global and local backbones to deliver task-agnostic and adaptive feature representations to heterogeneous data domains.

• Autonomous Expert Expansion and Selection: While existing methods use routers for expert aggregation, this study introduces a novel Expert Selection Mechanism (ESM) that automatically determines an optimal expert and transfers parametric knowledge of the selected expert into the new expert construction process, which is an unexplored strategy in continual learning.

These architectural and algorithmic innovations position LEAR as a novel direction in domain-incremental and rehearsal-free continual learning, advancing beyond incremental adaptations of existing techniques.

References:

[1] DualNet: Continual Learning, Fast and Slow//NeurIPS21

[2] Transfer without forgetting//ECCV22

[3] Learning fast, learning slow: A general continual learning method based on complementary learning system//ICLR22

[4] Der: Dynamically expandable representation for class incremental learning//CVPR21

[5] Foster: Feature boosting and compression for class-incremental learning//ECCV22

[6] A model or 603 exemplars: Towards memory-efficient class-incremental learning//ICLR23

[7] Beef: Bi-compatible class-incremental learning via energy-based expansion and fusion/ICLR22

[8] Dytox: Transformers for continual learning with dynamic token expansion//CVPR22

[9] Learning to prompt for continual learning//CVPR22

[10] Dualprompt: Complementary prompting for rehearsal-free continual learning//ECCV22

[11] Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning//CVPR23

[12] Boosting continual learning of vision-language models via mixture-of-experts adapters//CVPR24

[13] Expandable subspace ensemble for pre-trained model-based class-incremental learning//CVPR24

[14] Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning//CVPR25

Current Continual Learning (CL) research primarily focuses on Class-Incremental Learning (CIL) within a single domain, neglecting the scenario of learning across multiple domains, known as Domain-Incremental Learning (DIL). Although studies [1-3] have investigated DIL, their evaluated domains (e.g., Aircraft, MNIST) have achieved near-perfect accuracy with pre-trained ViTs, making these benchmarks inadequate for assessing genuine continual learning capabilities. Therefore, we establish a more challenging and more realistic Multi-domain Continual learning (MDCL) scenario in which the discrepancy among tasks remains large. In this study, we aim to improve the model's performance in MDCL by considering three aspects including plasticity, stability and efficiency. To implement this goal, we propose a novel approach called LEAR and its core idea is to fully explore the stable and dynamic representations extracted by the pre-trained ViT backbones to achieve fast adaptation while adaptively optimizing the backbones to maintain all previously learned information.

(1) Plasticity. Existing dual-branch approaches [4-6] and Expansion-Based Methods (EBMs) [7-10] improve downstream task performance by integrating task-specific prompts or adapters into a fixed pretrained backbone. However, these methods focus on exploring representations from a single pre-trained backbone, which fails to address more challenging data domains such as CUB200. Thus, to improve plasticity in a challenging MDCL scenario, we introduce a novel collaborative backbone architecture for LEAR, comprising a global and a local backbone, designed to capture general and task-specific information across all tasks. Leveraging this collaborative backbone structure, the proposed LEAR framework dynamically generates a lightweight expert to learn the decision boundary for each new task, thereby achieving commendable performance. The results presented in Tab. 1 and 2 of the paper demonstrate that our method achieves superior performance on most individual datasets in the MDCL scenario, which also validates that EBMs with frozen pretrained backbone cannot provide sufficient plasticity in MDCL.

(2) Stability. Many Pre-Trained Models (PTMs) based methods have shown to achieve excellent stability in continual learning by dynamically creating new sub-models. However, the excellent stability is usually achieved by freezing all parameters of the PTMs during the training, which prevents from learning new tasks effectively, especially when facing the severe domain shifts (ChestX→ImageNet-R) in long task sequences. To address the limitation of the existing PTMs-based methods, we propose a unified optimization function to regulate the optimization behaviour of the LEAR framework. This function consists of a MIBPA loss and a KLDBFA loss. The former dynamically optimizes the global backbone while preventing negative knowledge transfer at the prediction level, and the latter aligns historical and current representation distributions at the feature level. Such a design enables LEAR to achieve rehearsal-free continual learning by actively consolidating historical knowledge at both the prediction and feature levels when fine-tuning the collaborative backbones with new task data, rather than simply freezing parameters passively. Such a design has not been explored in the existing CL field. Furthermore, to mitigate optimization interference and information redundancy between the collaborative backbones, we propose a novel Hilbert-Schmidt Independence Criterion (HSIC)-Based Collaborative Optimization (HSICBCO) strategy to encourage two backbones to capture different semantic information, thus promoting effective complementary learning of MDCL tasks. The experimental results demonstrate that LEAR significantly outperforms all baseline methods in terms of overall average accuracy in MDCL scenarios.

(3) Efficiency. Many existing expansion-based methods usually ignore the task relevance and do not explore the previously learned parameter information to accelerate the new task learning. As a result, these methods optimize each new expert from scratch, resulting in leading to considerable computational costs and parameter redundancy. To address this issue, we aim to promote the efficient learning process of LEAR by proposing a novel Expert Selection Mechanism (ESM) that selectively transfers the parameter information learned by a selected expert into the new expert construction process. Specifically, the proposed ESM models each expert's knowledge as a Gaussian memory distribution and only preserve its critical statistical information. For each new task, the proposed ESM selects the most relevant expert by minimizing the Mahalanobis distance between stored distributions and incoming data, and reuses its parameters to facilitate new task learning. During testing phase, ESM autonomously routing testing samples to the most suitable expert in a task-agnostic manner.

We sincerely appreciate your time and valuable comments again.

Incorporating feedback from all reviewers, we have restructured the Introduction section to emphasize LEAR's novel contributions in the next comment.

Specifically, through systematic analysis of three key challenges (plasticity, stability and efficiency) in the proposed MDCL scenario (in contrast to CIL and conventional DIL), we demonstrate:

(1) The fundamental reason for existing continual learning methods' inability to address MDCL,

(2) Present our corresponding solutions for each challenge,

and ultimately forming the comprehensive LEAR framework to address the MDCL.

Meanwhile, we have incorporated the enhanced ablation studies from the rebuttal and their analyses into the main text to explicitly demonstrate the contributions of each module in LEAR and their various combinations.

We sincerely hope our revisions meet your expectations.

References

[1] S-prompts learning with pre-trained transformers//neurips22

[2] Preventing zero-shot transfer degradation in continual learning of vision-language models//ICCV23

[3] Coleclip: Open-domain continual learning via joint task prompt and vocabulary learning//TNNLS25

[4] DualNet: Continual Learning, Fast and Slow//NeurIPS21

[5] Transfer without forgetting//ECCV22

[6] Learning fast, learning slow: A general continual learning method based on complementary learning system//ICLR22

[7] Dualprompt: Complementary prompting for rehearsal-free continual learning//ECCV22

[8] Boosting continual learning of vision-language models via mixture-of-experts adapters//CVPR24

[9] Expandable subspace ensemble for pre-trained model-based class-incremental learning//CVPR24

[10] Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning//CVPR25

We sincerely appreciate the reviewer's constructive comments and recognition of the (1) the practical significance of our proposed Multi-Domain Continual Learning (MDCL) compared to conventional CIL learning settings, and (2) the demonstrated superior performance of LEAR over prior methods in this challenging scenario.

For weaknesses and questions, we provide our feedback as follows.

Q1: The discussion of motivation before introduction of components.

A1: We sincerely appreciate your invaluable suggestions and guidance. Due to space constraints in the main text, it's challenging to provide a detailed elaboration on the motivation of each module. We have thoroughly revised the introduction of the KLDBFA module to provide clearer motivation and methodological justification. The revised content is presented as follows:

The iterative updating of the pre-trained backbones facilitates the temporal capture of local representations, thereby potentially enhancing the acquisition of novel tasks. However, this process risks inducing adverse knowledge transfer and performance degradation across historical experts. Regularization methods like EWC [1] and MAS [2], which typically impose constraints on parameter updates, are not desirable to capture the complex distributional shifts across domains, while knowledge distillation methods like LWF [3] and iCaRL [4] require maintaining additional teacher networks that become computationally prohibitive as the number of tasks grows. To address these limitations, we propose Kullback-Leibler Divergence-Based Feature Alignment (KLDBFA), designed to preserve crucial historical parameters during the optimization of the local backbone.

Our design rationale for selecting KL divergence stems from two key considerations: Firstly, modern generative evaluation metrics (e.g., FID, Kernel MMD) operate on the Gaussian distribution assumption in high-dimensional feature spaces. This motivates us to model backbone features as Gaussian distributions. Fig. 1 in Appendix-C from the SM also provides additional empirical validation for the Gaussian distribution assumption. Secondly, KL divergence offers unique advantages over alternative distributional metrics: (1) Its directional property enables targeted constraint of current features toward historical distributions, unlike symmetric metrics (e.g., Jensen-Shannon divergence); (2) It maintains computational efficiency compared to expensive metrics like MMD or Wasserstein distance. These characteristics make KL divergence ideally suited for continual learning scenarios requiring efficient knowledge preservation.

Q2: Performance comparison on CIL settings.

A2: We sincerely agree with your comment. As MDCL is indeed significantly more challenging than standard CIL, comparing with existing methods under the CIL setting would further demonstrate and strengthen the superiority of LEAR. Therefore, we have further supplemented the comparison of LEAR with relevant methods on CIL performance over CIFAR-100, CUB-200, ImageNet-R, and TinyImageNet, each partitioned into 10 tasks:

Table 1: Results of CIL

The results demonstrate LEAR's excellent adaptability to CIL settings, where it achieves competitive performance. We have incorporated this comparative analysis, along with detailed discussions, prior to presenting the MDCL results in the revised paper.

Q3: Suggestions of ablation studies and performance upper bound.

A3: We note that all four reviewers raise concerns in the individual contributions of LEAR's components and have provided consistent suggestions for improving our ablation studies. Recognizing the importance of this issue, we have conducted extensive additional experiments on ETI and TRC sequences during the rebuttal phase and reorganized the ablation study tables accordingly:

Table 2: Results of ablation studies

Where ''CB'' denotes using only the collaborative backbone with a single shared expert network across all datasets; ''CBE'' extends CB with task-specific expert network expansion and ESM expert selection; and "CBE+MI/KL/HSIC" represents CBE augmented with individual components or a combination of components. ''UpperBound'' denotes the results in a non-continual setting.

This table demonstrates that the dynamically expanded expert network ('CB'->'CBE') significantly enhances model performance on both ETI and TRC. Furthermore, each regularization component and their respective combinations yield distinct enhancements in performance relative to CBE across both sequences.

Furthermore, the ablation study for ESM was initially omitted because ESM serves as the core component of LEAR. Without ESM, LEAR would lose its capability to dynamically select appropriate experts for domain-specific evaluation under task-agnostic conditions. To address this omission, we further conduct experiments with randomly selected experts during the beginning and testing phase of each task, which is denoted as ''LEAR w/o ESM''. The results demonstrate that the model's performance will significantly drop when inappropriate experts are chosen, thereby demonstrating the necessity of our ESM.

Q4: Storage at the end of the task.

A4: Following the network's input-output pipeline, we preserve the following components per task: (1) The last three layers of both global and local backbones, which are saved and frozen as Frozen-Global and Frozen-Local backbones (overwrite their corresponding components saved from the previous task); (2) Task-specific mean vectors and covariance matrices which computed from the features extracted by the Static backbone for a random subset of training samples.

Q5: Total parameter count.

A5: The total parameter count reaches approximately 241.7M, with the following detailed composition: (1) Two complete ViT backbones (Global and Local) comprise 172M (86M*2) parameters; (2) Three backbone variants (Frozen-Global, Frozen-Local, and Static) collectively contribute 63.6M (21.2M*3) parameters; (3) 10 task-specific expert networks account for about 4.8M (0.38M*10 for 10 fc layers + 1M for all classifiers) parameters; and (4) Mean vectors and covariance matrices for 10 tasks sum to 1.3M (0.13M* 10 for 10 mu & sigma) parameters.

Q6: Performance of solely collaborative architecture.

A6: Please refer to the results of ''CBE'' in Table 2. However, if you are referring to the results after excluding the task-specific expert network expansion, please refer to those associated with ''CB''.

Q7: Performance on CIL setting.

A7: Please refer to ''A2''.

References:

[1] Overcoming catastrophic forgetting in neural networks//PNAS17

[2] Task-free continual learning//CVPR19

[3] Learning without forgetting//TPAMI17

[4] iCaRL: Incremental Classifier and Representation Learning//CVPR17

I thank the authors for their detailed rebuttal. It helped clarify several points, and in particular, addressed my concerns regarding the ablation study. I appreciate the effort in responding comprehensively.

One suggestion I would like to share is that the paper currently includes a number of components—each contributing incrementally to the final performance. While these additions are reasonable, the overall structure feels a bit too complex for a conference paper. As a result, it was difficult to identify a single core idea or takeaway that clearly distinguishes this work from prior approaches.

In general, successful conference papers tend to center around a focused contribution that addresses a well-defined limitation of existing methods. In this case, although the individual components have been applied in a slightly different way, many of them build on existing techniques and the combined architecture may come across as a collection of ideas rather than a unified, novel contribution. This makes it harder to articulate the main innovation when reading the paper.

Of course, this is a matter of personal preference, and I do not hold it against the authors in my final assessment. I have updated my score accordingly, as I still believe the paper presents valuable insights and shows strong empirical results. I hope this feedback is helpful, and I encourage the authors to consider a more streamlined presentation.

Current Continual Learning (CL) research primarily focuses on Class-Incremental Learning (CIL) within a single domain, neglecting the scenario of learning across multiple domains, known as Domain-Incremental Learning (DIL). Although studies [1-3] have investigated DIL, their evaluated domains (e.g., Aircraft, MNIST) have achieved near-perfect accuracy with pre-trained ViTs, making these benchmarks inadequate for assessing genuine continual learning capabilities. Therefore, we establish a more challenging and more realistic Multi-domain Continual learning (MDCL) scenario in which the discrepancy among tasks remains large. In this study, we aim to improve the model's performance in MDCL by considering three aspects including plasticity, stability and efficiency. To implement this goal, we propose a novel approach called LEAR and its core idea is to fully explore the stable and dynamic representations extracted by the pre-trained ViT backbones to achieve fast adaptation while adaptively optimizing the backbones to maintain all previously learned information.

(1) Plasticity. Existing dual-branch approaches [4-6] and Expansion-Based Methods (EBMs) [7-10] improve downstream task performance by integrating task-specific prompts or adapters into a fixed pretrained backbone. However, these methods focus on exploring representations from a single pre-trained backbone, which fails to address more challenging data domains such as CUB200. Thus, to improve plasticity in a challenging MDCL scenario, we introduce a novel collaborative backbone architecture for LEAR, comprising a global and a local backbone, designed to capture general and task-specific information across all tasks. Leveraging this collaborative backbone structure, the proposed LEAR framework dynamically generates a lightweight expert to learn the decision boundary for each new task, thereby achieving commendable performance. The results presented in Tab. 1 and 2 of the paper demonstrate that our method achieves superior performance on most individual datasets in the MDCL scenario, which also validates that EBMs with frozen pretrained backbone cannot provide sufficient plasticity in MDCL.

(2) Stability. Many Pre-Trained Models (PTMs) based methods have shown to achieve excellent stability in continual learning by dynamically creating new sub-models. However, the excellent stability is usually achieved by freezing all parameters of the PTMs during the training, which prevents from learning new tasks effectively, especially when facing the severe domain shifts (ChestX→ImageNet-R) in long task sequences. To address the limitation of the existing PTMs-based methods, we propose a unified optimization function to regulate the optimization behaviour of the LEAR framework. This function consists of a MIBPA loss and a KLDBFA loss. The former dynamically optimizes the global backbone while preventing negative knowledge transfer at the prediction level, and the latter aligns historical and current representation distributions at the feature level. Such a design enables LEAR to achieve rehearsal-free continual learning by actively consolidating historical knowledge at both the prediction and feature levels when fine-tuning the collaborative backbones with new task data, rather than simply freezing parameters passively. Such a design has not been explored in the existing CL field. Furthermore, to mitigate optimization interference and information redundancy between the collaborative backbones, we propose a novel Hilbert-Schmidt Independence Criterion (HSIC)-Based Collaborative Optimization (HSICBCO) strategy to encourage two backbones to capture different semantic information, thus promoting effective complementary learning of MDCL tasks. The experimental results demonstrate that LEAR significantly outperforms all baseline methods in terms of overall average accuracy in MDCL scenarios.

(3) Efficiency. Many existing expansion-based methods usually ignore the task relevance and do not explore the previously learned parameter information to accelerate the new task learning. As a result, these methods optimize each new expert from scratch, resulting in leading to considerable computational costs and parameter redundancy. To address this issue, we aim to promote the efficient learning process of LEAR by proposing a novel Expert Selection Mechanism (ESM) that selectively transfers the parameter information learned by a selected expert into the new expert construction process. Specifically, the proposed ESM models each expert's knowledge as a Gaussian memory distribution and only preserve its critical statistical information. For each new task, the proposed ESM selects the most relevant expert by minimizing the Mahalanobis distance between stored distributions and incoming data, and reuses its parameters to facilitate new task learning. During testing phase, ESM autonomously routing testing samples to the most suitable expert in a task-agnostic manner.

We sincerely appreciate the reviewer's insightful feedback and recognition of our work's value. We are honored that our rebuttal addressed the concerns raised.

Incorporating feedback from all reviewers, in the next comment we have restructured the Introduction section to better articulate the core contributions while explicitly differentiating our approach from existing methods, thereby enhancing readability.

Specifically, through systematic analysis of three key challenges (plasticity, stability and efficiency) in the proposed MDCL scenario (in contrast to CIL and conventional DIL), we demonstrate:

(1) The fundamental reason for existing continual learning methods' inability to address MDCL,

(2) Present our corresponding solutions for each challenge,

and ultimately forming the comprehensive LEAR framework to address the MDCL.

Meanwhile, we have incorporated the enhanced ablation studies from the rebuttal and their analyses into the main text to explicitly demonstrate the contributions of each module in LEAR and their various combinations.

We sincerely hope our revisions meet your expectations.

References

[1] S-prompts learning with pre-trained transformers//neurips22

[2] Preventing zero-shot transfer degradation in continual learning of vision-language models//ICCV23

[3] Coleclip: Open-domain continual learning via joint task prompt and vocabulary learning//TNNLS25

[4] DualNet: Continual Learning, Fast and Slow//NeurIPS21

[5] Transfer without forgetting//ECCV22

[6] Learning fast, learning slow: A general continual learning method based on complementary learning system//ICLR22

[7] Dualprompt: Complementary prompting for rehearsal-free continual learning//ECCV22

[8] Boosting continual learning of vision-language models via mixture-of-experts adapters//CVPR24

[9] Expandable subspace ensemble for pre-trained model-based class-incremental learning//CVPR24

[10] Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning//CVPR25

We appreciate the valuable comments by Reviewer a1tB.

Q1: Concerns regarding writing quality.

A1: We have enhanced the paper's readability and improved Fig.1 to more clearly depict the main process of LEAR.

Q2: Reason for significant accuracy improvements.

A2: The proposed LEAR have two primary advantages over the existing continual learning methods:

(1) By emulating the brain's memory structure through CLS theory, the proposed LEAR manages a collaborative backbone architecture to provide more adaptive and semantically rich representation information when compared to the existing continual learning methods, including L2P, DAP, DualPrompt, MoEAdapter that usually employ a single fixed backbone network.

(2) Compared to the existing continual learning methods, the proposed LEAR integrates three mechanisms, including the Mutual Information-Based Prediction Alignment (MIBPA), Kullback–Leibler (KL) Divergence-Based Feature Alignment (KLDBFA) and HSIC-Based Collaborative Optimization (HSICBCO), aiming to maintain a good plasticity-stability trade-off and reduce information redundancy. Such a design can deal with a long sequence of complex tasks and achieve excellent performance. We evaluate the performance of the proposed approach by introducing a more challenging Multi-Domain Continual Learning (MDCL) setting that incorporates not only highly difficult datasets (such as ImageNet-R, TinyImageNet, and ChestX) but also features scenarios with both similar feature distributions between datasets (e.g., EuroSAT and RESISC45) and severe domain shifts (e.g., CIFAR100→CropDisease), as well as fine-grained datasets (CUB200) with extended task sequences. Consequently, as evidenced in the forgetting curves in Fig. 2 of Appendix-D.1 from the Supplementary Materials (SM), many baseline methods exhibit significant performance degradation in later stages of the task sequence, demonstrating their inability to learn and capture complex datasets.

Q3: Individual contributions of three regularization terms, particularly HSICBCO.

A3: To better demonstrate the individual contributions of each module in LEAR, we conducted additional ablation studies on two dataset sequences (ETI and TRC), as presented in the following tables:

Table 1: Results of ablation studies on ETI and TRC

Where "CBE" denotes the configuration using solely the collaborative backbone with task-specific network expansion and ESM expert selection, and "CBE+MIBPA/KLDBFA/HSICBCO" represents CBE augmented with individual components. The results demonstrate that each regularization term contributes to varying degrees of performance improvement over CBE across both ETI and TRC.

Specifically, the contribution of HSICBCO can be further reflected through the variation of accuracy. As shown in the "Var" columns of Table 1, "CBE+HSICBCO" achieves significantly reduced accuracy variance compared to "CBE" while maintaining improved average accuracy (Avg). This demonstrates that HSICBCO narrows the performance gaps across datasets, and enhances the performance on challenging datasets (e.g., ChestX, ImgR), which collectively validates HSICBCO's contribution to LEAR.

In addition, we present a performance comparison of the model after learning the first three domains of these two sequences:

Table 2: Performance after learning the 3th dataset of ETI and TRC

This table shows that HSICBCO effectively mitigates severe domain shifts (e.g., RESISC->CUB). This is because HSICBCO effectively encourages the global and local backbones to capture distinct knowledge, as further illustrated in Figure 3 of Appendix D.4 from SM, where it balances the model's generalization capability across dissimilar domains with its plasticity toward the current task.

Q4: The effectiveness of the collaborative backbone.

A4: To validate the necessity of the collaborative backbone for adapting MDCL, we conducted a comparative analysis between "SBE" (Single Backbone with task-adaptive network expansion) and "CBE":

As evidenced by the experimental results, the collaborative design yields significantly better performance than the single-backbone. Numerous current dynamic expansion frameworks utilize a single pre-trained ViT to enhance continual learning performance. Nevertheless, this approach yields only a uniform feature representation for each novel task, thereby constraining the model’s capacity to accommodate extended sequences of heterogeneous tasks. To overcome the inherent limitations of a single pre-trained ViT, we introduce a collaborative backbone architecture comprising both local and global backbones, designed to extract task-invariant and task-adaptive representations, respectively. To further refine this architecture, we propose a novel Hilbert-Schmidt Independence Criterion-Based Collaborative Optimization (HSICBCO) method, which incentivizes the local and global backbones to learn complementary semantic features. This strategy is particularly effective for handling complex and diverse datasets. In the revised version, we have provided a more comprehensive rationale and detailed exposition of the collaborative backbone architecture.

Q5: Why and how MIBPA mitigates catastrophic forgetting.

A5: The central concept of the proposed MIBPA is to preserve the established predictive patterns of each historical expert when optimizing the parameters of the global backbone. This is accomplished by generating both the prior and current prediction distributions for each expert, utilizing the previously updated and currently optimized backbones, respectively. MIBPA incorporates a mutual information-based regularization loss to align these historical and current prediction distributions, thereby mitigating catastrophic forgetting within the network.

Q7: Typo: concentration operation.

A7: That is a typo and we have corrected it to ''concatenation''.

Q8: The reason for freezing three Layers and auxiliary models.

A8: The auxiliary model is designed to retain the parameter knowledge acquired by the global backbone following each task transition. Consequently, all layers of the auxiliary model are frozen, and its fixed representations are leveraged to guide the optimization behaviour of the global backbone. To minimize parameter overhead, the auxiliary model shares the majority of its parameters with the global backbone across all but the final three layers. We have enhanced the clarity and coherence of this explanation in the revised version.

The rationale for selecting the final three layers lies in the fact that high-level representation layers capture semantically enriched features, which are advantageous for a variety of downstream applications [1]. [2] also find that deeper layers are disproportionately the source of forgetting. Empirical evidence presented below further substantiates that utilizing the last three trainable layers of the global backbone yields robust performance with limited parameter growth compared to other choices (FTX denotes fine-tune X layers of backbones).

Q9: Arch of expert.

A9: Each expert is composed of two distinct modules: the first is a fully connected layer comprising 500 hidden units while the second is a linear classifier realized through a fully connected layer with 1,268 hidden units.

References:

[1] Action bank: A high-level representation of activity in video. ICCV 2022.

[2] Anatomy of catastrophic forgetting: Hidden representations and task semantics. arXiv 2020.

Current Continual Learning (CL) research primarily focuses on Class-Incremental Learning (CIL) within a single domain, neglecting the scenario of learning across multiple domains, known as Domain-Incremental Learning (DIL). Although studies [1-3] have investigated DIL, their evaluated domains (e.g., Aircraft, MNIST) have achieved near-perfect accuracy with pre-trained ViTs, making these benchmarks inadequate for assessing genuine continual learning capabilities. Therefore, we establish a more challenging and more realistic Multi-domain Continual learning (MDCL) scenario in which the discrepancy among tasks remains large. In this study, we aim to improve the model's performance in MDCL by considering three aspects including plasticity, stability and efficiency. To implement this goal, we propose a novel approach called LEAR and its core idea is to fully explore the stable and dynamic representations extracted by the pre-trained ViT backbones to achieve fast adaptation while adaptively optimizing the backbones to maintain all previously learned information.

(1) Plasticity. Existing dual-branch approaches [4-6] and Expansion-Based Methods (EBMs) [7-10] improve downstream task performance by integrating task-specific prompts or adapters into a fixed pretrained backbone. However, these methods focus on exploring representations from a single pre-trained backbone, which fails to address more challenging data domains such as CUB200. Thus, to improve plasticity in a challenging MDCL scenario, we introduce a novel collaborative backbone architecture for LEAR, comprising a global and a local backbone, designed to capture general and task-specific information across all tasks. Leveraging this collaborative backbone structure, the proposed LEAR framework dynamically generates a lightweight expert to learn the decision boundary for each new task, thereby achieving commendable performance. The results presented in Tab. 1 and 2 of the paper demonstrate that our method achieves superior performance on most individual datasets in the MDCL scenario, which also validates that EBMs with frozen pretrained backbone cannot provide sufficient plasticity in MDCL.

(2) Stability. Many Pre-Trained Models (PTMs) based methods have shown to achieve excellent stability in continual learning by dynamically creating new sub-models. However, the excellent stability is usually achieved by freezing all parameters of the PTMs during the training, which prevents from learning new tasks effectively, especially when facing the severe domain shifts (ChestX→ImageNet-R) in long task sequences. To address the limitation of the existing PTMs-based methods, we propose a unified optimization function to regulate the optimization behaviour of the LEAR framework. This function consists of a MIBPA loss and a KLDBFA loss. The former dynamically optimizes the global backbone while preventing negative knowledge transfer at the prediction level, and the latter aligns historical and current representation distributions at the feature level. Such a design enables LEAR to achieve rehearsal-free continual learning by actively consolidating historical knowledge at both the prediction and feature levels when fine-tuning the collaborative backbones with new task data, rather than simply freezing parameters passively. Such a design has not been explored in the existing CL field. Furthermore, to mitigate optimization interference and information redundancy between the collaborative backbones, we propose a novel Hilbert-Schmidt Independence Criterion (HSIC)-Based Collaborative Optimization (HSICBCO) strategy to encourage two backbones to capture different semantic information, thus promoting effective complementary learning of MDCL tasks. The experimental results demonstrate that LEAR significantly outperforms all baseline methods in terms of overall average accuracy in MDCL scenarios.

(3) Efficiency. Many existing expansion-based methods usually ignore the task relevance and do not explore the previously learned parameter information to accelerate the new task learning. As a result, these methods optimize each new expert from scratch, resulting in leading to considerable computational costs and parameter redundancy. To address this issue, we aim to promote the efficient learning process of LEAR by proposing a novel Expert Selection Mechanism (ESM) that selectively transfers the parameter information learned by a selected expert into the new expert construction process. Specifically, the proposed ESM models each expert's knowledge as a Gaussian memory distribution and only preserve its critical statistical information. For each new task, the proposed ESM selects the most relevant expert by minimizing the Mahalanobis distance between stored distributions and incoming data, and reuses its parameters to facilitate new task learning. During testing phase, ESM autonomously routing testing samples to the most suitable expert in a task-agnostic manner.

We sincerely appreciate your time and valuable comments. We are honored that our rebuttal addressed the concerns raised.

Incorporating feedback from all reviewers, in the next comment we have restructured the Introduction section to clarify the differences from prior works better and to demonstrate the effectiveness of each proposed component more clearly.

Specifically, through systematic analysis of three key challenges (plasticity, stability and efficiency) in the proposed MDCL scenario (in contrast to CIL and conventional DIL), we demonstrate:

(1) The fundamental reason for existing continual learning methods' inability to address MDCL,

(2) Present our corresponding solutions for each challenge,

and ultimately forming the comprehensive LEAR framework to address the MDCL.

Meanwhile, we have incorporated the enhanced ablation studies from the rebuttal and their analyses into the main text to explicitly demonstrate the contributions of each module in LEAR and their various combinations.

We sincerely hope our revisions meet your expectations.

References

[1] S-prompts learning with pre-trained transformers//neurips22

[2] Preventing zero-shot transfer degradation in continual learning of vision-language models//ICCV23

[3] Coleclip: Open-domain continual learning via joint task prompt and vocabulary learning//TNNLS25

[4] DualNet: Continual Learning, Fast and Slow//NeurIPS21

[5] Transfer without forgetting//ECCV22

[6] Learning fast, learning slow: A general continual learning method based on complementary learning system//ICLR22

[7] Dualprompt: Complementary prompting for rehearsal-free continual learning//ECCV22

[8] Boosting continual learning of vision-language models via mixture-of-experts adapters//CVPR24

[9] Expandable subspace ensemble for pre-trained model-based class-incremental learning//CVPR24

[10] Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning//CVPR25

We are grateful for your valuable feedback on our work. We have reorganized the ablation studies below to demonstrate the effectiveness of each proposed component more clearly:

Table 1: Effectiveness of different component combinations in LEAR

Table 1 clearly demonstrates the contribution of each module in LEAR:

(1) ''UpperBound'' denotes the results in a single-domain setting, which closely approximates LEAR's performance upper bound in MDCL;

(2) ''CB'' denotes using only the collaborative backbone with a single shared expert network across all datasets;

(3) ''CBE'' extends ''CB'' with task-specific expert network expansion and ESM expert selection. The results demonstrate that the dynamic expert network expansion and selection (''CB''->''CBE'') significantly enhance model performance on both ETI and TRC;

(4) ''SBE'' denotes the configuration where ''CBE'''s dual-backbone architecture is replaced with a single backbone. The dual-backbone design yields about a 4% performance gain over the single-backbone counterpart (''SBE''->''CBE'');

(5) "CBE+MI/KL/HSIC" represents ''CBE'' augmented with individual components or combination of components. Each regularization component and its respective combinations yield effective performance enhancements relative to ''CBE'' on both sequences;

(6) "LEAR" denotes the complete framework, while "LEAR w/o ESM" represents the variant where experts are randomly selected during both the initialization and testing phases of each task. This configuration explains the observed significant performance degradation ("LEAR"->"LEAR w/o ESM")

Table 2: Performance comparison of different backbone fine-tuning configurations in LEAR​

We conducted comparative experiments on different fine-tuning configurations for the collaborative backbones of LEAR, where "FTX" denotes fine-tuning the last X layers while freezing preceding backbone layers. As shown in Table 2, the configuration utilizing the last three trainable layers ("FT3") of the collaborative backbones achieves robust performance with limited parameter growth compared to alternative configurations.