Dear Reviewer,

Thank you for your thorough review of our paper and for acknowledging the advantage of our method not requiring a replay buffer. We appreciate your constructive feedback, which has been invaluable in improving our work. Below, we address each of your concerns and questions in detail.

1. The methods, like UCIR, PASS, DyTox, and DER, are not initiated with CLIP because their specialized network designs are not easily compatible with CLIP. To ensure a fairer comparison, we also included the LwF and iCaRL methods based on CLIP initialization. Additionally, we present results based on ViT-B/16, with results for ViT-L/14 specifically marked in the tables. This distinction should not cause any ambiguity. We agree that comparing SimE with methods that also leverage pre-trained models provides a fairer assessment of our approach. In response to your suggestion, we have extended our experiments to include several state-of-the-art parameter-efficient continual learning methods and evaluations on multiple datasets, including ImageNet 1000(IN-1K)The updated results are summarized in Table10 in paper.

   As the results demonstrate, SimE consistently outperforms these methods across all datasets, both in average and last task accuracy. This confirms the effectiveness of our approach in comparison to relevant baselines that also leverage pre-trained models without a memory buffer.

2. Thank you for this valuable suggestion. We have expanded our evaluation to include larger and more challenging benchmarks to thoroughly assess the scalability and robustness of SimE, as shown in Table10 in paper. The extended experiments confirm that SimE scales effectively to larger datasets and maintains its advantages over existing methods.

3. We appreciate the opportunity to clarify the unique contributions of our work: Our key contribution lies in the discovery of a non-monotonic relationship between the number of adapter connections and incremental learning performance. Specifically, we found that: Between Transformer Blocks: Increasing adapter connections enhances IL performance. Within Transformer Blocks: Adding more adapters does not always improve performance and can even degrade it during early incremental stages. This nuanced understanding of adapter placement's impact on IL performance is, to our knowledge, a novel contribution that has not been explored in prior work, including AdapterFormer. As another reviewer says:

   “The introduction of Multi-Adapters in an IL context is an underexplored perspective, with good insights about the adapter's role and capabilities in retaining old information while keeping learning.”

   While it's expected that larger models may perform better, our contribution lies in demonstrating how to effectively and efficiently harness these models for IL. Prior works like $6$ focus on parameter-efficient fine-tuning for general downstream tasks, but they do not specifically address the unique challenges of IL, such as catastrophic forgetting. Our work bridges this gap by adapting and analyzing these concepts within the IL framework. While $7$ discusses the importance of pretraining data for latent replay in continual learning, our work extends this by providing specific guidelines on how to effectively adapt these models for incremental learning without a memory buffer. Additionally, we used t-SNE to visualize CLIP models trained on different datasets, as shown in Fig.8 in paper. It can be seen that the CLIP pre-trained on the larger dataset (LAION2B) clusters data points of the same class more tightly, indicating that it has a better ability to distinguish between different classes of data. Therefore, based on the t-SNE visualization results, we selected CLIP models pre-trained on different datasets.

4. Thank you for highlighting this concern. Although it slightly increases the inference cost, this approach eliminates the need for a memory buffer, reduces memory usage, and improves accuracy. In scenarios where memory constraints or data privacy are critical, SimE offers a favourable trade-off, which we believe is a reasonable choice.

Questions

1. We have updated Figure 2 to accurately reflect the placement of LayerNorms before the attention and MLP blocks, consistent with the CLIP ViT architecture.
2. We have corrected Equation 3 to reflect that the attention block precedes the MLP block, aligning with standard transformer architectures.
3. The depiction of AdaptFormer now correctly includes the residual MLP in addition to the standard residual connection. We also have clarified in the figure that the standard residual connections are retained in SimE, and the diagram now accurately represents this.

We appreciate your attention to detail, and these corrections have been incorporated into the revised manuscript to prevent any confusion.

Dear Reviewer,

Thank you for your thoughtful review and for acknowledging the strengths of our paper, including its clarity, identification of important issues in existing IL models, and the comprehensive ablation studies. We appreciate your constructive feedback, which has been invaluable in improving our work. Below, we address each of your concerns and questions in detail.

Weaknesses

1. Thank you for bringing this to our attention. We apologize for the oversight in Figure 4 and any confusion it may have caused. We have thoroughly revised the figure to correct the mistakes and added detailed labels and explanations to enhance its clarity.

2. We appreciate your perspective on the novelty of our work and would like to clarify how our contributions differ from and build upon prior research. Our key contribution lies in the discovery of a non-linear relationship between the number and placement of adapter connections and the model's IL capabilities. Specifically, we found that: Between Transformer Blocks: Increasing adapter connections improves IL performance. Within Transformer Blocks: Adding more adapters during early incremental stages does not enhance, and may even degrade performance, with improvements manifesting only in later stages.

   This nuanced understanding of adapter placement's impact on IL is, to our knowledge, a novel insight not explored in previous works, including [1][2]. As another reviewer says:

   "The introduction of Multi-Adapters in an IL context is an underexplored perspective, with good insights about the adapter's role and capabilities in retaining old information while keeping learning."

   Wang et al. [1] propose a training-free prototype calibration method for few-shot class-incremental learning. Their focus is on calibrating prototypes without additional training, which differs fundamentally from our approach that integrates adapters into VLMs to enhance IL capabilities.

   Zhou et al. [2] investigate the generalizability and adaptability of pre-trained models in class-incremental learning, emphasizing the use of different adaptation structures. While they explore adaptation structures, our work specifically introduces a Multi-Adapter framework with strategic adapter placements within VLMs, and provides detailed analyses on how these placements affect IL performance at different stages. The specific phenomenon we report regarding adapter connections within and between transformer blocks is not addressed in their work.

   Additionally, we also conduct a systematic study on enhancing the utilization of CLIP's zero-shot capabilities, providing practical guidelines for selecting backbones and pre-trained models for IL tasks.

Questions

1. Thank you for bringing up this important suggestion. We conducted additional experiments using different backbones (ResNet) on CUB200 to evaluate the generality of this finding, and the results are shown as follows:

The results indicate that the observation that larger model sizes lead to better performance still holds true. Moreover, the pre-training dataset definitely influences this phenomenon, so we carried out the experiments presented in Table 3 of the paper. The experiments demonstrate that when there is a significant gap between the pre-training dataset and the downstream dataset, the model's performance changes considerably. 2. Additionally, we have presented visualization results of CLIP under different pre-training datasets at Fig.8 in paper. It can be seen that CLIP pre-trained on a larger dataset (LAION2B) can cluster data points of the same class more tightly, indicating a better ability to distinguish between different classes of data. Therefore, we selected CLIP models pre-trained on different datasets based on the t-SNE visualization results.

We are grateful for your constructive feedback, which has helped us enhance the quality and impact of our work.

Dear Reviewer, Thank you for your thorough review and for recognizing the strengths of our paper, including the introduction of Multi-Adapters in an IL context, the efficiency of the SimE framework, and the completeness of our experimental results. We appreciate your constructive feedback and have taken your suggestions to heart. Below, we address each of your concerns and questions in detail.

Weaknesses

1. We agree that demonstrating the generality of SimE across different Vision-Language Models (VLMs) is important. In response, we have extended our experiments to include ResNet, with the experimental results presented in the table (SimE(RN50×4)). It is worth noting that, due to time constraints, this represents only a straightforward application of ResNet within the SimE framework.
2. Thank you for highlighting this oversight. We have now included comprehensive comparisons with several state-of-the-art parameter-efficient CL methods, including prompt-based approaches like L2P (CVPR 2022), DualPrompt (ECCV 2022), CODA-Prompt (CVPR 2023), SLCA (ICCV 2023), and APER (IJCV 2024). The updated experimental results on multiple datasets, CIFAR100, CUB200, ImageNet-R(IN-R), and ImageNet100(IN-100) , are summarized in the table below:

As the results show, SimE consistently outperforms these methods, both in average and last task accuracy, demonstrating its effectiveness and efficiency in the IL setting. 3. Thank you for your suggestion. As you mentioned, this approach could further demonstrate the generalizability of SimE. Due to time constraints, we have currently only conducted the experiments shown in the table, but this will be part of the future work.

Questions

1. In addition to VLMs, we have extended our experiments to include standard convolutional backbones like ResNet, as shown in Table. The results demonstrate that SimE also enhances IL performance in Resnet, suggesting that our framework is versatile and beneficial across different architectures. For now, we have only conducted exploration on CLIP, and we will explore other vision-language models in the future.
2. Certainly. The phenomenon where adding more adapter connections within transformer blocks does not enhance IL performance in early stages can be attributed to the following intuitive explanation: In early incremental stages, the model may have sufficient capacity to learn new tasks without additional adapters. Introducing more adapters increases the model's capacity disproportionately to the complexity of the new tasks, leading to overfitting on recent data and forgetting previous tasks. Additionally, excessive adapter connections within blocks may cause interference between the learning of new and old tasks, as the fine-grained adjustments can disrupt the representations learned for previous tasks. As the number of tasks increases in later incremental stages, the additional capacity provided by more adapters becomes beneficial.

We are grateful for your insightful comments, which have helped us improve our work significantly.

Dear Reviewer,

Thank you for your thoughtful review and for acknowledging the strengths of our paper, including its clarity, simplicity, efficiency, and superior performance. We appreciate your constructive feedback and would like to address the concerns you've raised.

Weaknesses

Although adapters have been applied in incremental learning, there are still limitations within vision-language models:

1.1 There is not always a positive correlation between the number of adapter connections and the model's incremental learning (IL) capability. Adding adapter connections between transformer blocks can improve the model's performance, but adding adapter connections within transformer blocks does not enhance and may even degrade the model's performance. This means that an excessively large adapter parameter space may harm the model's performance when task increments are small. Misplacing adapters can lead to inefficient learning and a waste of computational resources.

1.2 Previous works generally simply stack adapters, whereas we adaptively increase them. This approach solves the interference caused by an excessively large adapter parameter space in the model's incremental learning. By adopting adaptive adapter connections, we achieve incremental learning without a memory buffer, with less computational overhead and higher efficiency.

1.3 Our work offers a novel perspective on how adapter connections affect IL performance in vision-language models. Specifically, our key contribution lies in the discovery of a non-linear relationship between the number of adaptive adapter connections and the IL capabilities of the model. We observed that: Between Transformer Blocks : Increasing adapter connections positively impacts performance. Within Transformer Blocks : Adding more adapter connections during early incremental steps does not enhance, and may even degrade IL ability. Improvements are observed only at more advanced incremental stages.

This counterintuitive finding provides new insights into the architectural design of adapters for IL tasks. To the best of our knowledge, this specific analysis and the resulting guidelines for adapter placement in vision-language models have not been explored in prior work. As another reviewer says:

"The introduction of Multi-Adapters in an IL context is an underexplored perspective, with good insights about the adapter's role and capabilities in retaining old information while keeping learning."

2. You raised a valid point regarding the use of CIFAR-100, given that CLIP's pre-training data might contain similar images, potentially inflating performance metrics. To address this concern, we have conducted experiments on other datasets less likely to overlap with CLIP's pre-training data, such as CUB200 , ImageNet-R (IN-R) , and ImageNet100(IN-100) . These datasets offer a more stringent test of SimE's generalization capabilities.

Moreover, we have found that the methods we compared in the paper have few results on other datasets. In light of other reviewers repeatedly mentioning our lack of comparison with well-known efficient fine-tuning methods (such as L2P , DualPrompt , etc.), we have included comparisons with these methods. We would greatly appreciate it if you could provide us with the materials of the methods compared in the paper on other datasets. Our experimental results are as follows:

As the results show, SimE consistently outperforms these methods, both in average and last task accuracy, demonstrating its effectiveness and efficiency in the IL setting.

Questions

See weakness.

Conclusion

We are grateful for your feedback, which has helped us identify areas for improvement. We are confident that the revisions will address your concerns and enhance the overall quality and impact of our work.

Dear Reviewer,

I hope this message finds you well. I wanted to follow up regarding our manuscript and the rebuttal we submitted in response to your valuable feedback. We haven't heard back since our last communication and wanted to check if you have any further questions or if there's any additional information we can provide to assist in your evaluation.

Your insights have been instrumental in improving our work, and we are eager to address any remaining concerns you might have.

Thank you for your time and consideration.