• As RTable. 4 (our rebuttal) stated, the LLM fine-tuning method such as Adapter and LoRA (6.91M) requires additional parameters in the pre-trained models to optimize end-tasks, including language and vision tasks.
• In our optimization process, as shown in RTable. 2, only the final 12 attention layers (Soft-TF-L[12] with 2.3M) were tuned to obtain performance close to the best in accuracy and training times. The trainable parameter 2.3M per task corresponds to 3% of the pre-trained model parameters 68.5M, which is negligible.
• Furthermore, the parameter-efficient fine-tuning of Soft-TF was experimentally proven to achieve equivalent or par performance with only WSN, c=90% parameters. This will significantly contribute to parameter-efficient CL models.

Thank you for your constructive feedback and concerns. We have carefully prepared the following responses to address all the points raised.

• We have conducted an additional experiment on Random Initialized Performances of Soft-TF to show the importance of well-initialized pre-trained parameters.

• Additionally, we have prepared LLM-based fine-tuning (Adaptor and LoRA) performances to demonstrate the effectiveness of Soft-TF fine-tuning, as shown in RTables 1, 2, 1-1, 2-1, 3, and 4.

• Furthermore, we have explained more about Algorithm 2 in detail for clearance of the Soft-TF algorithm.

We believe that the rebuttals have further strengthened our main contribution to the well-initialized parameters of Soft-TF. We hope we have addressed the reviewer's major concerns, and we are happy to answer any further questions or concerns.

Best regards,

Authors

On $\alpha_t$ in Algorithm 2

• The value of $\alpha_t$ is determined by 1/the number of all seen tasks $T$, ensuring that $\alpha_t$ assigns equal importance to all seen tasks.
• Subsequently, by applying the entropy loss and computing gradient w.r.t. $\alpha_t$, the most sensitive task_id ($t_x$) can be obtained.

Redundant Calculation of ($t_x$)

• In Algorithm 2, we select one of the task inference methods: (1) Prompt ID or (2) Gradient ID.

• During the inference step, if Gradient ID is selected, we use $t_x$ inferred by the Gradient ID algorithm to identify the corresponding e-prompt $e_{t_x}$ and soft-TF $m_{t_x}$. ​

• Extensive experiments prove that $e_{t_x}$ is a minor factor in determining Gradient ID, as trained e-prompts are not perfect solutions for specific tasks. Instead, $m_{t_x}$, the primary tuning parameter, plays a crucial role. All empirical results consistently support this conclusion, i.e., the contribution of task inference: $m_{t_x} \gg e_{t_x}$.

• In the Prompt ID method, we use $t_x$ estimated by Prompt ID as the index to retrieve both the e-prompt $e_{t_x}$ and soft-TF $m_{t_x}$.

Few-shot Gradient ID Inference

• The 3-shot Gradient ID inference of Soft-TF is the most efficient approach regarding performance and inference speed, as shown in RTable. 2-1. Please refer to RTables 1-1 and 2-1 for more information on the time complexity of few-shot Gradient ID inference.

• Random initialization of Soft-Transformer's weights plays a critical role when leveraging well-pretrained models like Vision Transformers (ViTs).
• The optimal training point is the parameters of a well-pretrained model. Among the initialization methods, Uniform initialization for Soft-TransFormer satisfies this requirement effectively.
• To validate our claims, we analyze the impact of common random initialization methods, including Xavier, Kaiming, Normal, and Uniform Initialization, as shown in RTable. 5.
• The results demonstrate that the same well-initialization point leads to independent optimal task performance, particularly with Gradient ID inference.
• Furthermore, this ablation study has strengthened our Soft-TF with state-of-the-art performances inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), showing better performances than LLMs (Adapter & LoRA) in RTable. 4.

• As outlined in the Uniform Initialized performances (RTable 5), Soft-TF independently learns each task under identical initialization conditions (using non-overlapped datasets) with task IDs provided during CIL training.

• By beginning with the same initialization point, Soft-TF can discover a unique solution $m_t$ for each task by learning from its corresponding independent category dataset.

• This capability of Soft-TF enables Gradient ID Inference to overcome the limitations of Prompt ID effectively.

• Extensive experiments on continual learning scenarios involving repeated or similar category datasets are beyond the scope of this study and will be addressed in future work. According to our expectations, Soft-TF's performance on overlapped dataset continual scenarios could range from [Prompt ID] to [Gradient ID with 100% inference accuracy].

We acknowledge reviewer d7YX’s observation that architecture-based continual learners increase model parameters as the number of tasks grows.

However, similar to Soft-TF, large language models (LLMs) such as LoRA and Adapter also increase the parameters when fine-tuning sequential tasks, as shown in RTable 4. Notably, Soft-TF outperforms all existing baselines (LoRA and Adapter) by a margin of 6% under the same number of tunable parameters (6.9M), demonstrating superior performance.

Additionally, as suggested by reviewer ygEK, we conducted an ablation study to further highlight the novelty of Soft-TF, with results presented in RTable 3 and 4 (Comparisons of Soft-TF with LLMs). Soft-TF exhibits two key distinctive features compared to conventional baselines like LoRA and Adapter:

(1) Direct-full fine-tuning at the sparse attention layer: Soft-TF is well-initialized at the last three layers or one single layer (see RTable. 6) and fine-tuned with pre-trained parameters directly for all end tasks. In contrast, LoRA fine-tunes a small set of low-rank parameters across all attention layers, while the Adapter introduces additional modules in the FeedForward (FF) Network layer to enable representation learning for sequential tasks.

(2) Efficiency in Fine-tuning: Our extensive experiments demonstrate that fine-tuning Soft-TF on three attention layers is sufficient for sequential tasks. By comparison, LoRA and Adapter require fine-tuning across all layers. From our observations, LoRA and Adapter failed to fully fine-tune all sequential tasks, as shown in RTable. 6.

In conclusion, as an innovative novel method (the first approach in CL), we have demonstrated the effectiveness of Soft-TF with sparse full fine-tuning, surpassing conventional fine-tuning methods, including LoRA and Adapter. Our work has introduced a novel full fine-tuning approach that preserves the integrity of the pre-trained model's parameters while achieving superior performance.

Sincerely,

The authors.

• To clarify the motivation of Soft-TF, we have revised the sentences as follows:

"The only prompt-tuning of the pre-trained model cannot capture all the nuances of uncorrelated sequential tasks even though leveraging the frozen well-initialized model pre-trained on large-scale datasets since the only frozen well-initialized model provides global solutions rather than task-specific solutions provided by fine-tuning."

• We have revised the motivation in our updated script.

• Clarifying the structural differences between DualPrompt and L2P is essential to interpreting the performance differences between DualPrompt-L2P and DualPrompt-Soft-TF.

• L2P employs a straightforward structure where prompt embeddings are directly fed into a pre-trained ViT model. In contrast, DualPrompt incorporates global prompt tuning (G-Prompt) in the initial layers (layers 1 and 2) to adapt to task-specific domains. Additionally, DualPrompt applies task-specific prompt tuning (E-Prompt) in specific attention layers to learn specialized sub-tasks. Prior studies have demonstrated that the combined G/E-Prompt tuning strategy in DualPrompt consistently outperforms the simple L2P approach.

• This study of Soft-TF builds upon DualPrompt's training philosophy, utilizing G/E-Prompt as baselines. However, only DualPrompt fails to achieve optimal sequential task learning in continual learning (CL) scenarios. To address these limitations, we propose Soft-TF tuning, which enhances performance by updating pre-trained model parameters through soft-learnable parameters alongside E-Prompt tuning. This approach achieves superior results since E-Prompt and Soft-TF provide more task-specific optimal solutions.

• As demonstrated above, in RTable. 4, Soft-TF outperforms baseline approaches such as LLM-based methods (Adapter and LoRA) by providing more optimal solutions at attention layers for specific sub-tasks.

• Soft-TF can exceed the upper-bound performance because joint learning (upper-bound) requires simultaneous training across the entire category space with fixed parameters (e.g., 100-way classification in CIFAR-100). In contrast, under a Class-Incremental Learning (CIL) scenario, Soft-TF focuses on task-specific training (e.g., 10-way classification in 10-Split-CIFAR-100 or 5-way classification in 20-Split-CIFAR-100). This focused approach allows Soft-TF to handle 5/10-way classification tasks more effectively than 100-way ones (joint training). The 5/10-way classification problem can also be interpreted in this context. From a Soft-TF perspective, the 5-way classification problem is more accessible than the 10-way classification problem.

• To demonstrate the effectiveness of Soft-TF, we compare Soft-TF against LLM fine-tuning methods such as Adapters and LoRA, as shown in RTables 3 and 4.

• Under identical experimental conditions — including trainable model parameters (~6.9M per task), layers (L[10,11,12]), and Prompt ID — Soft-TF outperformed other LLM-based fine-tuning approaches, as shown in Rtable. 4. The results highlight that directly updating well-pretrained model parameters and prompt-tuning via Soft-TF is more effective than combining representations through LoRA or learning representations with Adapters.

• Furthermore, we observed that Soft-TF and the other methods exhibited comparable training and testing time complexity for a similar number of trainable parameters. Notably, single-layer fine-tuning using Soft-TF (with L[12]) surpasses the baselines' performances.

• These findings firmly establish Soft-TF as the most competitive approach among strong LLM baselines (Adapters and LoRA) in the continual learning (CIL) scenario.

• We have reported more parameter-wise performances of Adapter and LoRA in the updated script.

Thank you for your constructive feedback and concerns. We have carefully prepared the following responses to address all the points raised as follows:

• We have compared Soft-TF with recommended LLMs (Adapter and LoRA) with the same trainable parameters (RTables 3 and 4), showing outperformance (the most parameter-efficient architecture, Soft-TF) in terms of accuracy and forgetting.

• We have added the upper bound of Soft-TF for fair comparisons.

• We explain the reasons for the performances of DualPrompt-Soft-TF v.s. L2P-Soft-TF.

• We have revised the Motivation of Soft-TF as the reviewer suggested.

We believe the rebuttals have strengthened our main contribution through extensive investigations. We hope we have addressed the reviewer's major concerns, and we are happy to answer any further questions or concerns.

Best regards,

Authors

• We use DualPrompt and DualPrompt-PGP as baselines to evaluate the parameter efficiency of Soft-TF.

• As the number of Soft-TF layers increases, as shown in RTable. 1, the number of trainable parameters increases slightly, and task performance improves correspondingly.

• Gradient ID's inference (mini-batch) complexity is reported to be approximately 1.6 times that of Prompt ID; however, Gradient ID boosts performance and reduces forgetting dramatically.

• The Prompt ID inference complexity of Soft-TF-L[10,11,12] with three layers is comparable to the baselines (PGP) while delivering superior performance.

• Notably, even single-layer fine-tuning using Soft-TF (with L[12], 2.3M) outperformed the baselines (PGP) with comparable training time complex, as shown in RTable. 2.

• Moreover, as outlined in our script, sparse learning techniques such as WSN (with c=90%) can be applied to Soft-TF-L[12], significantly reducing the number of tunable parameters (2.0M) while maintaining Soft-TF's high performances.

• To show Soft-TF's effectiveness further, we have compared it with LLM fine-tuning methods such as Adapter and LoRA, as demonstrated in RTables 3 and 4. We have included all ablation studies in our updated script.

• (Time Complexity of Gradient ID ): We investigate the most parameter-efficient and gradient-based task inference methods, as shown in RTables 1-1 and 2-1. - Our findings reveal that the 3-shot Gradient ID inference cost (using samples within a mini-batch) with the last layer (Soft-TF-L[12], 2.3M) is approximately 1.1 times that of Prompt ID, maintaining comparable efficiency while delivering superior performance. Note that m-batch denotes mini-batch.

Thank you for your constructive feedback and concerns. To address all the points raised, we have prepared the following responses. We have clarified the proposed parameter-efficient Soft-TF through extensive experiments and further emphasized its novel contributions as follows:

• We have investigated layer-wise training efficiency & inference of Soft-TF (RTables 1 and 2) with gradient and prompt ID: only prompt ID-based Soft-TF outperformed strong baselines such as PGP, Adapter, and LoRA.

• Furthermore, we have observed that the 3-shot Gradient ID inference cost (using samples within a mini-batch) with the last layer (Soft-TF-L[12]) is approximately 1.1 times that of Prompt ID, leading to the most efficient Soft-TF structure, as shown in RTables 1-1 and 2-1.

We hope we have addressed the reviewer's major concerns, and we are happy to answer any further questions or concerns.

Best regards,

Authors

Dear Reviewers,

We would like to express our heartfelt gratitude for your dedicated time and effort in reviewing our paper. Your constructive feedback has been instrumental in enhancing the quality and clarity of our work.

If you have any further concerns regarding our response, please do not hesitate to share them. We would be delighted to engage in follow-up discussions and address any additional comments you may have.

Sincerely, The Authors

Dear reviewers,

We hope our responses have thoroughly addressed the reviewer’s questions and concerns.

We would clarify or provide further details if you have additional feedback or questions.

If you think that all concerns have been adequately resolved, please make sure to update your official score correspondingly.

Thank you for your time and effort in reviewing our work.

Sincerely,

The Authors