Response to Reviewer J5Ty

Thanks for your detailed review and precious comments. Our responses to Reviewer J5Ty follow. We will enhance the paper with these invaluable suggestions.

1. W1 & Q2: Complexity analysis

Given a pretrained weight matrix $W \in \mathbb{R}^{d_O \times d_I}$, PaCA estimates causal effects via a second-order Taylor expansion with a diagonal Hessian approximation. Both the first-order and second-order terms have a time complexity of $\mathcal{O}(d_O \times d_I)$, making the overall complexity of PaCA linear and comparable to a standard backward pass. This computation can be efficiently parallelized and seamlessly integrated into the training process.

In a task sequence of $T$ tasks, for each new task, each gradient projection and affinity computation costs $\mathcal{O}(d_O \times d_I)$, leading to a total overhead of $\mathcal{O}((t-1) \times d_O \times d_I)$. Thus, the per-task complexity is approximately $\mathcal{O}(t \times d_O \times d_I)$. The total complexity for the entire sequence is $\mathcal{O}\left(\sum_{t=1}^{T} t \times d_O \times d_I\right)$.

Table 1 shows the time complexity and average per-epoch runtime across 10 tasks on ImageNet-R for our method compared to two LoRA-based baselines. Our method maintains training efficiency on par with projection-based LoRA fine-tuning (i.e., InfLoRA), with only marginal extra cost.

Table 1: Comparison of computational complexity and average per-epoch runtime across 10 tasks on ImageNet-R. $𝑟$ is the rank of LoRA.

2. W2 & Q3: Inter-task correlation

Ablation experiments have validated the effectiveness of task correlation in knowledge transfer, as shown in Table 5 of the paper. Positive task correlations facilitate knowledge transfer. To further demonstrate this, we conduct an additional experiment. Specifically, we select five tasks from the Long Sequence benchmark to show task correlation: BoolQA, IMDB, Yelp, DBpedia, and MultiRec. BoolQA and MultiRec are both question-answering tasks, IMDB and Yelp are sentiment analysis tasks, and DBpedia is a topic classification task.

Table 2 shows the average task correlation and task affinity for these five tasks under the order 1 setting at the initial training stage, while Table 3 shows the results at the middle training stage. The results show that correlated tasks, such as IMDB and Yelp, and BoolQA and MultiRec, exhibit higher task correlation and affinity. As gradient updates progress, the correlation between positively correlated tasks increases, while the correlation between negatively correlated tasks decreases, demonstrating the effectiveness of gradient adjustment.

Additionally, Table 4 presents the backward knowledge transfer results for BoolQA and IMDB. For BoolQA and IMDB, the correlated tasks, MultiRec and Yelp, respectively, transferred knowledge, improving their performance. This indicates that the task correlation and affinity can capture inter-task relationships and facilitate knowledge transfer.

Table 2: Task correlation/affinity on initial epoch

Table 3: Task correlation/affinity on the middle epoch

Table 4: Backward knowledge transfer for BoolQA and IMDB

3. W3: Results from 5 random seeds

To further validate the stability of our method, we conduct additional experiments with 2 random seeds on ImageNet-R (20 tasks). As shown in Table 5, experiments with 5 random seeds demonstrate that our method consistently outperforms the baselines while maintaining stability.

Table 5: Overall results (mean ± std over 5 random seeds) on ImageNet-R (20 tasks)

4. Q1: Efficient storage

In our framework, model parameters are stored using quantization methods (such as FP16), and the old task gradients are decomposed using SVD to obtain basis vectors. Only the basis vectors of the gradients are stored in memory to reduce storage overhead. In practice, similar to current LoRA-based methods (e.g., SD-LoRA, Inflora), the number of trainable parameters is very small. For example, in the T5-Large model (0.77B parameters), the trainable parameters account for approximately 0.33%. In the future, we plan to explore combining causal effects with parameter pruning to reduce storage overhead further.

5. Q4: Specific features for knowledge transfer

Correlated task data can help the model learn shared features. E.g., in the ImageNetR benchmark, correlated categories like goldfinch and junco, both bird species, share visual features such as beaks and feathers, contributing to knowledge transfer.

To study how specific features influence this process, we introduce a task-specific learnable mask on the output features of the ViT model. The mask consists of real values between 0 and 1 and acts as a filter over the features. We design a two-phase training scheme. In the first phase, each task is trained independently with an additional regularization term that encourages the masked features to predict task labels accurately. This enables the model to focus on task-specific features via the mask.

In the second phase, we incrementally train multiple tasks, applying either the learned mask (w/ mask) or its complement (w/o mask) to examine how task-specific features affect task correlation and knowledge transfer. Table 6 presents an ablation study on ImageNetR (20 tasks), comparing results with and without task-specific features. This demonstrates that task-specific features can enhance knowledge transfer.

Table 6: Ablation results on ImageNet-R (20 tasks) with mask vs without mask.

6. Q5: Case study

Using BoolQA as an example, we present 2 samples that are initially misclassified but are corrected after incorporating parameters from MultiRC. These samples are prone to semantic confusion, yet the model's reasoning ability improved after exposure to MultiRC, enabling it to resolve these more complex cases correctly.

Table 7: Case study samples

Reply to Reviewer NXah

Thanks very much for your in-depth review and thoughtful comments. Here are our replies to Reviewer NXah. We will improve the paper based on the constructive suggestions.

1. W1: Gradient projection and gradient similarity for capturing task relationships

Although the linear assumption in the gradient space may limit the ability to capture complex nonlinear relationships, numerous studies have demonstrated that gradient projection is effective in incremental learning and has become a representative method [1-9]. Since the gradient space reflects the learning patterns of task models and the solution space, gradient projection and gradient similarity can capture dynamic task relationships, serving as a local linear approximation. This approximation is often sufficient to describe the evolution between tasks [9-11]. Therefore, we use gradient projection and gradient similarity to characterize the correlation and affinity between tasks, enabling fine-grained adjustments that alleviate catastrophic forgetting and promote backward transfer.

The experiments further show that our method can capture task relationships and facilitate knowledge transfer (the experiments can be found in our responses to Review J5Ty's W2 & Q3). We acknowledge that the linear assumption in the gradient space may limit the method's performance in more complex scenarios. To address this limitation, we plan to explore nonlinear learning methods, such as function space modeling and causal learning, in future work to better capture the nonlinear relationships between tasks.

[1] Saha, G., et al. Gradient Projection Memory for Continual Learning. International Conference on Learning Representations, 2021.

[2] Deng, D., et al. Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning. Advances in Neural Information Processing Systems, 2021, 18710–18721.

[3] Lin, S., et al. TRGP: Trust Region Gradient Projection for Continual Learning. International Conference on Learning Representations, 2022.

[4] Qiu, B., et al. Geodesic-Aligned Gradient Projection for Continual Task Learning. IEEE Trans. Image Process., 34, 1995–2007, 2025.

[5] Liang, et al. InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, 23638–23647.

[6] Liang, et al. Adaptive Plasticity Improvement for Continual Learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, 7816–7825.

[7] Qiao, J., et al. Prompt Gradient Projection for Continual Learning. International Conference on Learning Representations, 2024.

[8] Yang, E., et al. Revisiting Flatness-Aware Optimization in Continual Learning With Orthogonal Gradient Projection. IEEE Trans. Pattern Anal. Mach. Intell., 47(5), 3895–3907, 2025.

[9] Lin, S., et al. Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer. Advances in Neural Information Processing Systems, 2022.

[10] Yu, T., et al. Gradient Surgery for Multi-Task Learning. Advances in Neural Information Processing Systems, 2020.

[11] Liu, B., et al. Conflict-Averse Gradient Descent for Multi-task Learning. Advances in Neural Information Processing Systems, 2021, 18878–18890.

2. W2: Experiments on causal effects

Ablation experiments have validated the effectiveness of the PaCA module, as shown in Table 5 of the paper. To further demonstrate the causal effects of the parameters, we conduct an additional experiment on 5 tasks from the Long Sequence benchmark. These tasks cover 3 types: sentiment analysis (Yelp, IMDB), question answering (BoolQA, MultiRec), and topic classification (DBpedia).

The T5-Large model consists of an encoder and a decoder, each containing 24 attention modules. Specifically, we divide the 24 attention modules in both the encoder and decoder into 3 groups (Down, Middle, and Up) based on their position, with each group containing 8 attention modules. For each task, the average causal effect is recorded 10 times throughout the training process.

Table 1 shows the average (Avg), maximum (Max), and minimum (Min) causal effects in each block. The results indicate that parameters in the middle and upper blocks generally exhibit larger causal effects, receiving more attention from the tasks. For simpler tasks, such as BoolQA and IMDB, the differences between the minimum and maximum values are large, suggesting that the model selects fewer parameters. In contrast, for more complex tasks, such as MultiRC, the difference between the minimum and maximum values is smaller, and the overall causal effect of parameters is larger, indicating that the model selects more parameters.

Table 1: Average causal effects of different blocks in the T5-Large model for different Tasks.

3. W3 & Q1: Task correlation and affinity

We would first like to clarify that Section 3.3 of the paper does not claim that task order affects performance. Furthermore, most comparison methods rarely consider different task order settings (except for SAPT). Following SAPT, experiments on two NLP benchmarks have validated the effectiveness of our method under different task orders, as shown in Tables 2 and 3 of the paper.

To further validate the impact of gradient updates on task correlation and affinity under different task orders, we add a reverse task order (Order 3) to the original Order 1 in the Long Sequence benchmark. Specifically, we select 5 tasks in the Long Sequence benchmark: Yelp, IMDB, BoolQA, MultiRec, and DBpedia. Tables 2-5 present the average task correlation and task affinity for the initial and middle stages of training under the two orders.

The results show that correlated tasks, such as IMDB and Yelp, BoolQA and MultiRec, exhibit higher task correlation and affinity. As gradient updates progress, the correlation between positively correlated tasks increases, while the correlation between negatively correlated tasks decreases, demonstrating the effectiveness of our method. Our gradient update facilitates backward knowledge transfer from new tasks to correlated old tasks (details can be found in our response to Reviewer 5jhA's W1 & Q4).

Table 2: Task correlation/affinity under Order 1 (Initial Epoch)

Table 3: Task correlation/affinity under Order 1 (Middle Epoch)

Table 4: Task correlation/affinity under Order 3 (Initial Epoch)

Table 5: Task correlation/affinity under Order 3 (Middle Epoch)

4. Q2: Experimental reproduction

The results are reproduced by us. Specifically, we adapt the baselines to the SAPT framework for NLP tasks, and to the InfLora framework for CV tasks.

5. Q3: Setup of the ablation experiment

In the ablation study, we set task correlation to 1 to eliminate its influence and use an all-one vector to remove the effect of task affinity.

Dear Reviewer NXah,

Thank you for your valuable and constructive reviews. We have made an extensive effort to address your questions and concerns by providing additional results for causal effect, task correlation, and task affinity. We hope our response can effectively address your concerns. If you have any further concerns or questions, please do not hesitate to let us know, and we will respond on time.

Best regards,

The Authors of Paper #16688

Response to Reviewer 5jhA

Thanks very much for providing a detailed review and insightful comments. Our responses to the comments of Reviewer 5jhA are given below. We will further improve our paper based on these valuable comments.

1. W1 & Q4: Quantitative evidence for backward knowledge transfer

Existing studies have already demonstrated that new tasks can facilitate backward knowledge transfer to correlated old tasks [1,2]. Extensive experiments have validated the effectiveness of our method in backward knowledge transfer from multiple perspectives. These include different task orders, different model scales, and different task lengths, as shown in Tables 2-4 and Figure 2 in the paper, as well as Figure 1 and Table 3 in the appendix. The backward knowledge transfer (BWT) metric of our method outperforms the comparison methods across all benchmarks.

To further demonstrate the effectiveness of our method, we conduct an ablation experiment on the Long Sequence benchmark under order 1. Table 1 shows some results of backward knowledge transfer using the T5-Large model. For example, the performance of the first task, MNLI, improves after loading parameters from CB and DBpedia, likely because CB and MNLI are both natural language inference tasks (correlated), and DBpedia's Wikipedia data aids MNLI inference.

Similarly, BoolQA and MultiRC are question-answering (QA) tasks, while Yelp, Amazon, SST-2, and IMDB are all sentiment analysis tasks. The model shows similar learning patterns for correlated tasks, enabling general knowledge transfer. Notably, BoolQA is binary-choice QA, while MultiRC is more complex multiple-choice QA. After learning MultiRC, the model better understands the simpler BoolQA task (The task correlation validation can be found in our responses to Reviewer NXah's W3 & Q1).

Table 1: Partial backward knowledge transfer under the Long Sequence Benchmark order1.

2. W2 & Q2: Differences between PaCA and influence function, and between CaGA and EWC

1). PaCA differs from the influence function. Although both PaCA and influence function [3] assess the importance of elements (parameters or samples) to a task through interventions, they differ significantly in the following two aspects:

(1) The intervention targets differ. PaCA directly intervenes on parameters and estimates their causal effect via counterfactual reasoning. In contrast, the influence function perturbs the sampling weights of the samples to assess the importance of the samples.

(2) The calculation methods differ. PaCA uses the second-order Taylor expansion of the loss function before and after intervention to approximate the causal effect of parameters, while the influence function approximates the effect of a sample by using the inverse of the Hessian matrix of parameters after perturbing the sample.

2). CaGA differs from EWC. CaGA and EWC (Elastic Weight Consolidation) [4] both alleviate catastrophic forgetting, but there are notable differences. CaGA mitigates catastrophic forgetting through task-correlation-based gradient correction, a novel adaptive gradient update method. EWC, on the other hand, is a regularization method that adds a regularization term based on the Fisher matrix to prevent the update of important parameters. It does not consider the potential positive impact of new tasks on old tasks, which differs from CaGA, which encourages beneficial parameter updates.

3. Q1: A scenario of backward knowledge transfer

In the task incremental learning, when new tasks are correlated to old tasks and the new task is more difficult or contains more knowledge, loading the new task parameters can facilitate knowledge transfer to the simpler old tasks [1,2]. For example, given a large language model and an article, the model needs to read the article to perform reading comprehension tasks (question-answering, QA).

The tasks are divided into two types: one is a binary-choice QA, where the model selects one answer from two options; the other is a multiple-choice QA, where the model must choose the correct answer from multiple options (at least four choices), and some options may be similar. The multiple-choice QA is more difficult than the binary-choice QA. Following the normal task sequence, the model first trains and tests on the binary-choice QA, then trains on the multiple-choice QA.

Since the multiple-choice QA is more challenging, requiring more complex reasoning and deeper understanding, after training on it, the model's comprehension of the article will be more profound, and its reasoning abilities will improve. This enhanced ability not only helps the model perform better on the multiple-choice QA but also improves its accuracy on the binary-choice QA, particularly in error correction and fine-grained judgment. In this way, the knowledge and skills learned by the model while handling more complex tasks can be effectively transferred to simpler tasks, ultimately optimizing its overall performance.

4. Q3: Generalization on the randomly initialized model

Most existing parameter-efficient fine-tuning methods for incremental learning(e.g., InfLoRA, SAPT, SD-LoRA, etc.), do not consider validation on randomly initialized models. This is because the randomly initialized model lacks pre-trained knowledge and remains frozen during training, with only a small number of parameters being trained. As a result, it fails to learn effective representations.

To demonstrate this, we evaluate the performance of our method and several baselines on the ImageNet-R benchmark under the 10-task setting using a randomly initialized ViT-B/16 model. In addition to the four metrics reported in the paper, we compute AVG (Top-1), the average of the best Top-1 accuracy across tasks. As shown in Table 2, both our method and the baselines perform poorly with randomly initialized models.

The average of the best task-wise Top-1 accuracy remains just above 10%. These results indicate that parameter-efficient incremental learning frameworks rely on pre-trained models to achieve competitive performance with minimal tunable parameters, supporting the core assumption of parameter-efficient fine-tuning.

Table 2: Overall results on ImageNet-R (10 tasks) with the randomly initialized ViT model.

5. Q5: Comparison with MoE-based methods

Following most existing parameter-efficient fine-tuning methods for incremental learning (e.g., InfLoRA, SAPT, SD-LoRA, HidePrompt, etc.), their comparison methods mainly include prompt-based learning and LoRA-based methods. Thus, our comparison methods are also primarily based on prompt-based learning and LoRA-based methods.

Mixture-of-Experts (MoE)-based continual learning methods introduce multiple expert modules, activating a task-specific subset to minimize interference with prior knowledge. E.g., MoE-P[5] incorporates a Non-linear Residual Gate (NoRGa) mechanism to enhance MoE performance in continual learning. However, compared to LoRA-based lightweight parameter fine-tuning, MoE requires training parameters for multiple experts, which incurs significantly higher overheads in incremental learning scenarios with a large number of tasks.

To highlight the superiority of our method, we compare it with MoE-P under the 10/20 task setting of ImageNetR. As shown in Tables 3 and 4, our method outperforms MoE-P.

Table 3: Comparison results on ImageNet-R (10 tasks)

Table 4: Comparison results on ImageNet-R (20 tasks)

[1] Ke, Y., et al. Continual learning of a mixed sequence of similar and dissimilar tasks. Proceedings of Advances in Neural Information Processing Systems, 2020, 18493–18504.

[2] Lin, S., et al. Beyond not-forgetting: Continual learning with backward knowledge transfer. Proceedings of Advances in Neural Information Processing Systems, 2022, 16165–16177.

[3] Gao, R., et al. Defying catastrophic forgetting via influence function. Artificial Intelligence, 339, 104261, 2025.

[4] Kirkpatrick, J., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 2017.

[5] Le, M., et al. Mixture of Experts Meets Prompt-Based Continual Learning. Advances in Neural Information Processing Systems, 2024.

Response to Reviewer TJb6

Thank you very much for your detailed review and insightful comments. Below are our responses to your valuable suggestions. We will refine the paper accordingly.

1. W1: Computational overhead analysis

Our method includes PaCA and CaGA modules. Given a pretrained weight matrix $W \in \mathbb{R}^{d_O \times d_I}$, PaCA estimates causal effects via a second-order Taylor expansion with a diagonal Hessian approximation. Both the first-order and second-order terms have a time complexity of $\mathcal{O}(d_O \times d_I)$, making the overall complexity of PaCA linear and comparable to a standard backward pass. This computation can be efficiently parallelized and seamlessly integrated into the training process.

In a task sequence of $T$ tasks, for the current task $T_t$, CaGA performs gradient projection and computes task affinity. The bases of old task gradients are derived through a single singular value decomposition (SVD) and subsequently stored in memory, thereby reducing the memory overhead. Each projection and affinity computation costs $\mathcal{O}(d_O \times d_I)$, leading to a total overhead of $\mathcal{O}((t-1) \times d_O \times d_I)$. Thus, the per-task complexity is approximately $\mathcal{O}(t \times d_O \times d_I)$. The total complexity for the entire sequence is $\mathcal{O}\left(\sum_{t=1}^{T} t \times d_O \times d_I\right)$.

Table 1 shows the time complexity and average per-epoch runtime across 10 tasks on ImageNet-R for our method compared to two LoRA-based baselines. Our method maintains training efficiency on par with projection-based LoRA fine-tuning (i.e., InfLoRA), with only marginal extra cost.

In addition, the previously submitted appendix provides a comparison of memory overhead, showing that our method does not significantly increase storage costs compared to the baselines. Moving forward, we will consider further reducing memory overhead in incremental learning by combining causal effects with techniques such as parameter pruning.

Table 1: Comparison of computational complexity and average per-epoch runtime across 10 tasks on ImageNet-R. $𝑟$ is the rank of LoRA.

2. W2: Task boundary

To adapt our task-incremental learning framework to online incremental learning, we will explore two future improvements. The first method adds an inference step. Specifically, during the training phase, we retain the prototype representations corresponding to each task. Then, in the inference phase, we determine which task a test sample belongs to by measuring representation similarity, and subsequently load the parameters of the corresponding task to complete the inference.

When the number of tasks is extremely large, to reduce storage and inference costs, we consider grouping correlated tasks, with only one prototype representation retained for each group. During the inference phase, we determine which group of tasks a test sample belongs to by measuring representation similarity and then load the parameters of the corresponding group of tasks to complete the inference.

The second approach constructs a task-agnostic online learning framework by integrating causal effects and parameter perturbation, enabling model parameters across tasks to converge to a shared parameter space via perturbation. Compared to the first method, it reduces storage costs but may compromise performance on certain tasks.

3. Q1: Causal effect in Algorithm 1

Algorithm 1 only outlines the flow across the main modules of the framework. Due to the paper's space constraints, the details of updates in different iterations are not elaborated. Only the initialization of the LoRA parameters is performed once at the beginning of training. Other operations, including causal effect estimation, computation of task correlation and affinity, calculation of the gradient correction term, and gradient update, are adaptively performed at each training step. We will provide a more detailed description of the algorithmic process.

4. Q2: Quantitative evidence of the role of gradient similarity in facilitating positive transfer

As discussed in the Task Affinity part, previous studies have shown that gradient similarity reflects task relationships, with positive transfer occurring between tasks with similar gradients [1-4]. In addition, ablation experiments, as shown in Table 5 of the paper, validate the effectiveness of gradient similarity (task affinity) in enabling positive transfer. To further demonstrate this, we conduct an additional experiment.

Specifically, we select five tasks in the Long Sequence benchmark, i.e., BoolQA, IMDB, Yelp, DBpedia, and MultiRec. BoolQA and MultiRec are both question-answering tasks, IMDB and Yelp are sentiment analysis tasks, and DBpedia is a topic classification task. Tables 2 and 3 present the average task correlation and task affinity for these 5 tasks in the initial and middle epochs under the order 1 setting. The results show that correlated tasks, such as IMDB and Yelp, and BoolQA and MultiRec, exhibit higher task correlation and affinity.

Moreover, as gradient updates progress, the correlation between positively correlated tasks increases, while that of negatively correlated tasks decreases, demonstrating the effectiveness of gradient correction. Additionally, Table 4 shows the results of backward knowledge transfer for BoolQA and IMDB. For BoolQA and IMDB, tasks that are positively correlated with them, i.e., MultiRec and Yelp, respectively, transfer knowledge that improves their performance. This suggests that our task correlation and affinity (similarity) can capture the task relationships and facilitate positive transfer.

Table 2: Average task correlation/affinity (Initial Epoch) under order 1

Table 3: Average task correlation/affinity (Middle Epoch) under order 1

Table 4: Backward knowledge transfer for BoolQA and IMDB

[1] Yu, T., et al. Gradient Surgery for Multi-Task Learning. Advances in Neural Information Processing Systems, 2020.

[2] Liu, B., et al. Conflict-Averse Gradient Descent for Multi-task Learning. Advances in Neural Information Processing Systems, 2021, 18878–18890.

[3] Chen, Z., et al. Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout. Advances in Neural Information Processing Systems, 2020, 2039–2050.

[4] Lin, S., et al. Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer. Advances in Neural Information Processing Systems, 2022.

5. Q3: Performance with noisy old task gradients

We conduct validation experiments on the ImageNet-R benchmark (10 tasks) by adding Gaussian noise to the old task gradients. Since LoRA-based parameter fine-tuning methods involve relatively few parameters, resulting in smaller gradient magnitudes, we test two noise ratios, 0.01 and 0.1.

Tables 5 and 6 present the performance of our method in comparison with InfLoRA, another gradient projection-based method, under the two noise settings. These results show that, even with noisy gradients, our method outperforms InfLoRA. This is likely due to our method's ability to identify effective parameters through causal effects, which helps mitigate the impact of parameter noise to some extent.

Table 5: Overall results on ImageNet-R (10 tasks) with a noise ratio of 0.01

Table 6: Overall results on ImageNet-R (10 tasks) with a noise ratio of 0.1