Q1:Fairness of performance comparison, architectural differences, and pre-training strategy

We sincerely thank the reviewer for their professional comments regarding the categorization of our method and the fairness of our experimental comparisons. We provide a unified response to the three sub-issues raised:

1. Key Distinction from Architecture-Expansion Methods

Although LMSRR utilizes multiple pre-trained ViT models as feature extractors, our method does not expand model structures per task (e.g., by adding branches or task-specific heads). Instead, our goal is to construct a semantically aligned multi-source representation space within a unified architecture.

Specifically, feature representations from multiple pre-trained models are fused via our proposed MSIDF (Multi-Source Interactive Dynamic Fusion) module. This fused representation is then progressively optimized through MLRO (Multi-Level Representation Optimization) and ARO (Auxiliary Representation Optimization) modules. This design maintains feature continuity and robustness across tasks, and is fundamentally different from task-isolated routing strategies like those used in RPSNet [1] and HAT [2], which suffer from parameter growth as the number of tasks increases.

In essence, LMSRR achieves parameter sharing + multi-source alignment + localized optimization, offering a unified alternative to architecture expansion-based methods.

---

2. Pre-training Strategy Alignment and Experimental Fairness

To ensure a fair and comparable experimental setting, we adopted a consistent backbone configuration across methods, which is reflected in both our main paper’s ablation study (Fig. 2(a)) and the appendix (Fig. 1):

• For existing methods that do not alter backbone architectures (e.g., ER, DER, DER++), we used the same set of multiple pre-trained ViT models as in LMSRR.
• All ViT backbones were trained under the same protocol: early layers were frozen, and only the final few layers were fine-tuned to ensure equivalent trainable capacity.
• These baseline methods fused features from multiple ViTs via concatenation and used a shared linear classifier, serving as a strong and fair multi-source fusion baseline.

---

3. Comparison with Architecture-Based and Prompt-Based Methods

To further address the reviewer’s concern, we additionally compared LMSRR with both architecture-based and prompt-based recent SOTA methods, including:

• RPSNet [1]: A classical architecture-expansion method that employs task-specific path selection masks over a ResNet-18 backbone to dynamically activate sub-networks. While this design reduces forgetting, it introduces structural growth and high memory costs with increasing tasks.
• DualPrompt [3] and L2P [4]: Prompt-based continual learning methods built on frozen ViT-1K backbones, injecting learned task-specific prompts. These are closely related to DAP and represent the most recent line of pre-trained prompt-injection approaches.

We faithfully reproduced these methods following their original settings and training protocols. We conducted comparisons on CIFAR-100 and Tiny ImageNet, and, to ensure fairness, we adopted the same buffer size of 2000 used in the RPSNet paper. The evaluation metric was Average Accuracy:

---

The results demonstrate that LMSRR significantly outperforms RPSNet by more than 20%, and also shows better generalization than recent prompt-based methods. These results confirm the robustness of LMSRR’s unified representation strategy, even when compared to strong architecture-specific or prompt-injection baselines.

We have thus made every effort to ensure fairness in design, backbone pre-training protocols, and evaluation settings, and we supplemented our main experiments with comparisons to recent competitive approaches.

Q2:Performance variation with different numbers of experts

We appreciate the reviewer’s insightful question. We would like to first clarify a potential misunderstanding regarding the term “expert” in our method.

In our framework, the term expert does not refer to a Mixture-of-Experts (MoE) architecture, where independent expert networks are assigned to different tasks or classes. Instead, our method consistently employs a single unified linear classifier $f_\phi$, which takes as input the fused multi-source representations generated from different pre-trained ViTs and outputs the final prediction.

Thus, in our architecture, the number of experts remains one, and increasing the number of ViT backbones only affects the diversity and richness of the input features to be fused, rather than increasing the number of prediction heads or models.

That said, we fully understand that the reviewer’s intention was likely to ask:

Does the performance improve as more ViT models are involved in the fusion process?

This question was already addressed in our response to Reviewer ABxL Q4, where we conducted a systematic analysis. In that experiment, we used combinations of one, two, and three different pre-trained ViTs and evaluated the performance on three datasets. The results clearly showed that adding more ViT models consistently led to improved performance, validating our motivation for designing a multi-model fusion strategy.

In summary, while our architecture contains only one final classifier (i.e., one expert), the performance gains obtained by increasing the number of fused ViTs demonstrate the effectiveness of leveraging feature diversity, and confirm the value of our fusion mechanism.

We thank the reviewer again for helping us further clarify this design choice.

Q3:Effectiveness under a first task with more classes

We thank the reviewer for raising this insightful question. To evaluate whether our proposed method maintains its effectiveness under varying task configurations—particularly when the first task contains more classes—we conducted a dedicated experiment on the CIFAR-100 dataset.

Experimental Setup

We compared two task-splitting scenarios, keeping the total number of classes at 100:

• Scenario A (First20): [20,10,10,10,10,10,10,10,10] — First task contains 20 classes
• Scenario B (First50): [50,10,10,10,10,10] — First task contains 50 classes

For all methods, the memory buffer size was 500 samples to ensure fairness.

We also compare with the default continual learning scenario (i.e., uniform splits such as 10 tasks × 10 classes), where LMSRR achieves:

• LMSRR: 95.76 ± 0.08
• DAP: 90.11 ± 0.33
• DERPP: 75.64
• DERPPRE: 77.71

Findings and Insights

• LMSRR consistently outperforms all baselines across both settings.
• The performance of baseline methods drops more significantly when the first task becomes larger (e.g., DERPP drops by 5.0 points).
• In contrast, LMSRR remains highly stable, with only a minor drop of 0.7 points from 95.32 (First20) to 94.62 (First50).

This stability is attributed to LMSRR's multi-source representation fusion, which allows the model to dynamically integrate knowledge from multiple pre-trained ViTs. This fusion provides a richer, more flexible feature space, enabling better adaptation even when the first task involves a larger or imbalanced set of classes. Unlike methods like DERPP and DAP, which rely on single-model structures or fixed mechanisms, LMSRR's dynamic fusion ensures that the model remains robust to the variability in task distribution, thus reducing the impact of initial task bias.

References:

[1] Rajasegaran, J., et al. Random Path Selection for Continual Learning. NeurIPS, 2019.

[2] Serra, J., et al. Overcoming Catastrophic Forgetting with Hard Attention to the Task. ICML, 2018.

[3] Wang, Z., et al. DualPrompt: Complementary Prompting for Rehearsal-Free Continual Learning. ECCV, 2022.

[4] Wang, Z., et al. Learning to Prompt for Continual Learning. CVPR, 2022.

Q1: Lack of Originality, The method mainly adapts existing techniques, reducing originality.

This paper presents two significant and previously unexplored contributions to the field. First, we introduce a novel Multi-Scale Interaction and Dynamic Fusion (MSIDF) approach that adaptively synthesizes features from multiple representation networks. In contrast to conventional feature fusion techniques that merely aggregate features into a single representation, the MSIDF framework incorporates a suite of adaptive attention modules, each characterized by distinct attention window sizes to capture diverse local information within the representations. Consequently, each representation is partitioned into several segments, each processed by different attention modules, and subsequently integrated through a trainable adaptive mechanism. This process yields a robust and less redundant representation, marking a novel direction in continual learning research. Second, we propose an innovative Adaptive Regularization Optimization (ARO) method that selectively penalizes parameter updates at each layer of the representation network. Unlike traditional regularization approaches that uniformly penalize changes across all network layers, the ARO method employs a learnable penalization strategy that dynamically assesses the significance of each layer during training. This strategy assigns adaptive weights to modulate the regularization strength per layer, effectively mitigating over-regularization and enhancing network plasticity. Both the MSIDF and ARO approaches represent original contributions that have not been previously addressed in the continual learning literature.

Q2: Plasticity Dependence, Performance may depend on pre-trained ViT backbones, with unclear performance on unseen domains.

We thank the reviewer for raising this important question. To assess LMSRR’s performance on unseen domains, we conducted experiments on two datasets with substantial distributional shifts from ImageNet:

• RESISC45[1]: A remote sensing dataset with 45 scene categories.
• ISIC[2]: A medical imaging dataset for skin lesion classification.

We ensured consistent pre-training configurations across methods, using two ViT-B/16 models for LMSRR (In21k and In21k-ft-In1k) and a single ViT-B/16 for baselines.

LMSRR outperforms all baselines on both datasets, with +2.49% gain on RESISC45 and +14.24% on ISIC. This is attributed to LMSRR’s multi-backbone fusion and learnable attention-based fusion mechanism (MSIDF), which enables domain-adaptive and robust representations. These results show LMSRR’s effectiveness in generalizing to out-of-distribution tasks, essential for real-world continual learning.

Q3: Missing Ablation Studies: No comprehensive ablation studies on MSIDF, MLRO, and ARO to clarify their individual impacts.

We thank the reviewer for raising the valuable suggestion to analyze the individual contributions of each module within LMSRR. To address this concern, we conducted a comprehensive module-wise ablation study to quantify the impact of the three core components—MSIDF, MLRO, and ARO—across different datasets.

We performed experiments on CIFAR-100, CUB-200, and Cars196, using Average Accuracy as the evaluation metric. Starting from the complete LMSRR framework, we systematically removed each module while keeping all other settings and architectures fixed:

• LMSRR: The full proposed model, including MSIDF, MLRO, and ARO.
• w/o MSIDF: Replaces the multi-source dynamic fusion with simple feature concatenation (static combination of ViT outputs).
• w/o MLRO: Disables multi-level layer selection regularization (no guidance for stable feature selection).
• w/o ARO: Removes the auxiliary representation alignment using the frozen teacher network.

---

Analysis:

• The MSIDF (Multi-Scale Interaction and Dynamic Fusion) module proves to be the most critical component. Its removal results in the most significant performance drop (up to –3.57%), particularly on more challenging datasets like Cars196. This underscores the importance of dynamic fusion in enhancing the semantic expressiveness of multi-source features.
• The MLRO (Multi-Level Representation Optimization) module contributes 0.7%–1.3% performance gain, demonstrating its role in guiding the model to prioritize stable and transferable features across tasks, thus enhancing robustness during task transitions.
• While the ARO (Auxiliary Representation Optimization) module leads to a relatively smaller drop in average accuracy, its primary role lies in enhancing stability during task shifts rather than boosting peak performance. This auxiliary alignment contributes to smoother representation transitions over time.

In summary, all three modules contribute meaningfully to the final performance and stability of LMSRR, with MSIDF being the most impactful, and MLRO/ARO supporting continual adaptation and resilience to forgetting.

Q4: Unfair Comparisons & Stability: The comparison may be unfair due to architecture/backbone differences, and LMSRR shows greater performance drops, raising stability concerns.

We sincerely appreciate the reviewer’s concern regarding the fairness of our experimental setup. To ensure a fair and consistent comparison across all baselines, we have adopted a unified experimental protocol in both the main paper’s ablation studies (Fig. 2(a)) and Appendix (Fig. 1):

• For baseline methods that do not alter backbone architectures (e.g., ER, DER, DER++), we use the same set of multiple pre-trained ViT models as in LMSRR.
• All ViT models are trained using a “frozen early layers, fine-tune last layers” strategy to maintain similar learnable capacity across methods.
• Feature outputs from multiple ViTs are simply concatenated and passed through a shared linear classifier, forming a generic and fair multi-source fusion baseline for comparison.

For DAP, we clarify that it is fundamentally prompt-based, with a distinct training mechanism. Specifically:

• DAP freezes the entire ViT backbone and introduces an adaptive prompt generator to guide task-specific adaptation.
• The prompt generator produces instance-level prompts that are injected into every layer of the ViT, enabling the frozen network to generate task-aware representations.
• In our experiments, DAP is implemented using a single pre-trained ViT (ViT-1K), strictly following the configuration and freezing strategies described in its original paper.

Hence, from backbone choices to optimization protocols, all methods are trained under standardized and comparable settings. The performance improvements of LMSRR are therefore attributed to our proposed architectural design and optimization strategies, rather than any unfair experimental bias.

---

2.Regarding the "Average" vs. "Last" Task Accuracy Gap:

We acknowledge the reviewer’s observation that LMSRR sometimes exhibits a larger gap between the average and last task accuracy, especially compared to methods like DAP or PNN. We believe this behavior arises naturally from differences in method design philosophies, as detailed below:

• DAP and PNN belong to architecture- or prompt-expansion categories, where task-specific modules or prompts are introduced per task. This design helps preserve prior knowledge and reduces forgetting but comes at the cost of linearly growing parameters and limited scalability.
• In contrast, LMSRR leverages a unified and shared representation space, enhanced by MLRO and ARO, which dynamically adjust representations across tasks. This allows LMSRR to efficiently balance plasticity and stability, achieving strong last-task performance (reflecting adaptability) while maintaining competitive average accuracy (reflecting retention).
• Furthermore, as shown in Fig. 2(b) of the main paper and Fig. 2 in the Appendix, we provide forgetting curve analysis on ImageNet-R and other benchmarks. The results demonstrate that LMSRR not only achieves the highest overall accuracy, but also exhibits the smoothest and most stable forgetting curves, indicating strong robustness against catastrophic forgetting.

In summary, while LMSRR may show a relatively larger gap between last and average accuracy in some cases, this does not indicate weaker stability. Rather, it reflects a trade-off rooted in efficient shared representation learning, which achieves superior scalability, performance, and generalization compared to architecture-expanding methods. We will clarify this point further in the revised manuscript.

References:

[1] Cheng, G., et al. Remote Sensing Image Scene Classification: Benchmark and State of the Art. IEEE Transactions on Geoscience and Remote Sensing.

[2] Codella, Noel, et al. "Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).".

Q1: It should compare with existing multi model strategies.

We thank the reviewer for raising this concern. While multi-model fusion methods like [1] and [3] are important in multi-model learning, our work targets Class-Incremental Continual Learning (Class-IL), which involves sequential tasks and catastrophic forgetting. Therefore, our design and comparisons are situated within continual learning literature.

However, we agree that the relationship with [1] and [3] needs clarification. Below is a comparison:

Comparison with Model Soup [3]: Model Soup fuses models via parameter-space averaging, assuming independent training on static tasks, which contradicts the online adaptation required in continual learning. It also does not address catastrophic forgetting. In contrast, our method uses dynamic feature-space fusion via multi-scale attention, adapting to tasks without independent model training or post-hoc merging.

Comparison with Weighted Ensemble [1]: [1] uses a fixed-weight logit-level ensemble, which is static and task-specific. Our approach dynamically learns interactions among multiple pretrained backbones during training, resulting in more adaptive representations. Unlike [1], which trains models from scratch, we use multiple pretrained ViT backbones, offering better resource efficiency and deployment.

Additional Comparisons: To address the lack of direct comparison with multi-model strategies, we performed experiments on CIFAR-100, CUB-200, Cars196, and ImageNet-R using the protocol from [1], comparing with CoMA and CoFiMA.

Here, LMSRR (200) and LMSRR (500) refer to our method evaluated with replay buffer sizes of 200 and 500, respectively. As shown, LMSRR significantly outperforms both CoMA and CoFiMA across all datasets, confirming the effectiveness and adaptability of our learnable feature-level fusion mechanism in continual learning scenarios.

---

Q2: Visualization results.

We thank the reviewer for raising this important point. We understand that improved performance alone does not validate the MSIDF module’s claims. To address this, we designed two additional experiments to provide interpretable evidence:

1. Feature Discriminability Analysis (Structured Class Cluster Separation) We compared Concatenation (baseline) with MSIDF on the CIFAR-10 dataset, using t-SNE for visualization. Due to rebuttal policy, the visual results are included in the supplementary material. The results show that MSIDF provides better inter-class separation, indicating that MSIDF learns discriminative and structured representations.
2. Visualization of MSIDF’s Dynamic Fusion Behavior We analyzed the activation distribution across MSIDF’s two attention modules, using the L2 norm of the output features. A heatmap was plotted showing the normalized activation strength for each class. Results indicate that different classes activate different attention modules, confirming that MSIDF performs dynamic, task-aware fusion, not a fixed strategy.

---

Q3: The computational costs.

We thank the reviewer for highlighting the concern about computational overhead. We now provide a detailed analysis of LMSRR's computational resource consumption compared to several baseline methods, including replay-based, regularization-based, prompt-based, and multi-model ensemble approaches. We evaluated the following metrics:

• Params: Trainable parameters (millions)
• GPU Avg/Max: Average and peak GPU memory usage (MiB)
• CPU Avg/Max: Average and peak CPU memory usage (MiB)
• Iteration Time: Average time per training iteration (sec/iter)

Pretraining and Fine-Tuning Strategy: To ensure fair comparison, we adopted a consistent protocol across all baseline methods (e.g., ER, DERPP, DERPPRE), using the same ViT backbone and freezing most layers, fine-tuning only the last three layers. This protocol is also applied to LMSRR. Despite fusing multiple ViTs, LMSRR maintains a moderate number of trainable parameters (42.53M), significantly lower than CoFiMA’s 85.80M.

Comparison with CoFiMA [1]: CoFiMA unfreezes all ViT parameters, leading to 85.80M parameters and higher memory usage. In contrast, LMSRR maintains moderate resource usage while achieving dynamic fusion and task adaptation, providing a better efficiency-performance tradeoff.

Comparison with DAP: DAP, as a prompt-based method, uses 0.51M parameters, but is sensitive to prompt quality and task shifts. LMSRR increases parameters modestly but shows better performance and stability.

Inference and Deployment Efficiency: LMSRR uses a single fused representation for inference, avoiding multi-model inference overhead and making inference cost similar to a single model.

Q4: The demonstraction of the limitation of a single pre-trained model.

We thank the reviewer for raising this point. To address the lack of empirical evidence for our claim about the suboptimal performance of single pre-trained models, we conducted a comparative study on CIFAR-100, CUB-200, and Cars196.

Experimental Setup:

We selected three ViT backbones:

1. ViT1: ViT-B/16, pre-trained on ImageNet-21k and fine-tuned on ImageNet-1k
2. ViT2: ViT-B/16, pre-trained only on ImageNet-21k
3. ViT3: ViT-L/14, a larger ViT model

We evaluated the following configurations:

• Single models: ViT1, ViT2, ViT3
• Two-model fusion: ViT1 + ViT2
• Three-model fusion: ViT1 + ViT2 + ViT3

Results (Average Accuracy):

All experiments were conducted under the Class-IL setting using LMSRR's feature fusion pipeline. The results are reported in terms of Average Accuracy (%):

Single-model performance varies across datasets: ViT2 underperforms on Cars196 compared to ViT1 and ViT3, while ViT1 excels on CIFAR-100, but not on fine-grained datasets. Multi-model fusion consistently improves performance, with the three-model fusion outperforming all single-model configurations. These results confirm that multi-model fusion enhances robustness and adaptability, supporting our claim.

Q5: The novelty.

Compared to [1], the proposed approach has three differences: (1) [1] only employs a single pre-trained ViT, which has limited learning ability when addressing unknown data domains. In contrast, the proposed MSIDF optimizes several pre-trained representation networks to provide semantically rich features. A novel Multi-Scale Interaction and Dynamic Fusion approach is proposed to integrate features from all representation networks to further enhance the representation capacity; (2) [1] adopts a mixture of experts framework, while the proposed MSIDF adopts a single classifier and does not need to perform the expert selection process. As a result, [1] requires knowing the task information in order to select the suitable expert for each testing sample, while the proposed MSIDF can efficiently predict the data without requiring access to the task information.

Compared to [2], the proposed MSIDF has three differences: (1) [2] trains the parameters of the network from scratch while the proposed MSIDF adopts pre-trained ViTs to provide diverse and robust representations; (2) [2] employs the mixture of experts framework, which requires knowing the task information during the inference process. In contrast, the proposed MSIDF does not incrementally create new experts when learning new tasks and can be efficiently performed in the inference process without knowing task information. (3) [2] simply freezes all previously learned parameters to relieve network forgetting, which can decrease plasticity. In contrast, the proposed MSIDF optimizes most parts of the representation networks using the proposed Multi-Level Representation Optimization, which can significantly improve the model's plasticity.

References:

[1] Marouf, Imad Eddine, et al. "Weighted ensemble models are strong continual learners.".

[2] Douillard, Arthur, et al. "Dytox: Transformers for continual learning with dynamic token expansion.".

[3] Wortsman, Mitchell, et al. "Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.".

I thank the authors for their thorough response to my comments.

Q1: It should compare with existing multi-model strategies.

I appreciate the comparison with [1], however, I am still troubled by the answer from the authors.

• " While multi-model fusion methods like [1] and [3] are important in multi-model learning, our work targets Class-Incremental Continual Learning (Class-IL), which involves sequential tasks and catastrophic forgetting.". Well, it turns out [1] operates in the exact same setup, which is why I am mentioning this work.
• " Unlike [1], which trains models from scratch, we use multiple pretrained ViT backbones, offering better resource efficiency and deployment. ". This is simply not true, as the objective of [1] is literally to start from pre-trained models.

Both points, combined with the fact that the displayed performances of CoFiMA are lower than those presented in the original work, raise questions about the fairness of the comparison. As mentioned by reviewer 3Jvb, the compared methods are old, and the fact that the authors struggle to correctly acknowledge them is concerning. Eventually, the method is memory-based, and comparing its performance to memory-free or outdated memory-based methods is questionable.

Other points

• I deeply appreciate the additional insights. The point regarding task information requirements is extremely valuable, in my opinion
• I understand the visualization is not possible due to policies from NeurIPS2025; however, as many points raised, this should have been considered in the first stage of the submission
• I would like to know, for the computation section, if this is during training or inference. I believe the main issue here is inference, as multi-model inference must be done

Final Remark

Overall, I believe the work to be valuable. There are indeed strong points made by the authors during the rebuttal phase, and even though the novelty might not be the strongest, I believe that with careful writing regarding the difference with concurrent methods, updated experiments with latest state-of-the-art approaches, ablation study and interpretation (as mentioned in Q2) this work has its place in such a venue. However, in its current stage, I cannot confidently recommend it for acceptance. Therefore, I will maintain my score.