Learning without Isolation: Pathway Protection for Continual Learning

Response to Claims: Thanks for pointing out the issue. We would like to explain it as follows.

1. In continual learning, considering the lack of correspondence between neurons in Model 1 and Model 2, it is possible that the function of the p-th neuron in Model 1 is very similar to that of the (p+1)-th neuron in Model 2, despite their different positions. Therefore, when using the OT algorithm in neural networks, we aim to optimally (minimizing cost) transfer neurons from a specific layer in Model 1 to the corresponding layer in Model 2, achieving model alignment.
2. The advantages of using a soft matching algorithm.
   • Using entropy regularization and bidirectional relaxation ensures that the optimization function is smoother.
   • One-to-one matching hinders knowledge sharing. One weight in Model 1 might be similar to multiple weights in Model 2. In such cases, using one-to-one matching may set the corresponding weights with similar knowledge to zero, hindering knowledge sharing.
3. We compared direct fusion and fusion using hard matching, and the results show that soft matching is more advantageous. ResNet18-based model on the CIFAR-100 dataset, measured under task-aware scenarios.

Response to Essential References: Thanks for the meaningful suggestions. We indeed omitted the comparisons in our manuscript.

Comparison with [1]

1. The different approaches for protecting task-related knowledge. [1] employs a genetic algorithm. It selects the parameter subset for the next task while fixing the important parameters of the previous tasks, whereas our method achieves knowledge protection through a matching-based approach.
2. [1] does not train frozen parameters, whereas we train all parameters.

Comparison with [2]

1. [2] selectively retrains network parameters and adapts to different tasks by dynamically changing neurons. However, our method protects knowledge of different tasks by finding the optimal pathways for each task and using matching techniques.
2. The method proposed in [2] dynamically increases network capacity. In contrast, our method uses a fixed network capacity approach.

Comparison with [3]

1. [3] protects knowledge through parameter decomposition and masking techniques. However, our method protects important pathways through matching techniques.
2. [3] focuses on protecting network connections between adjacent layers, whereas we consider a complete pathway from input to output.

Response to Weakness: Thanks for the valuable question.

• Regarding the number of learnable tasks, our method employs soft protection, allowing for a greater number of learnable tasks. Compared to hard mask methods like SupSup, we use all network parameters and protect important task paths through misaligned fusion. Based on the reviewer's suggestions, we conducted experiments on Tiny-ImageNet with 100 tasks, each comprising two classes. The results show that our method performs very well.
• To verify the performance of our method when increasing the task number by an order of magnitude, we split Tiny-ImageNet into 100 tasks and conducted tests using different methods. The results show that our method achieves better performance.

Response to C1: Thank you for the reviewer's reminder. We sincerely apologize for the misunderstanding caused by our negligence. A precise description will be provided in future revisions. The concept of "Activation Level" refers to the average magnitude of the weights obtained after activation in the last layer of the feature extraction phase. We utilize activation levels to measure whether pathways associated with different tasks can be distinguished. Through the left side of Figure 2, we observed that among these channels, in our method, there consistently exists a channel specific to a task. In contrast, for the LwF method, the activation levels of its channels exhibit a phenomenon of intermixing.

Response to C2: Thanks for pointing out the issue. We would like to explain it as follows.

• We conducted a corresponding analysis, including the optimization of our proposed method in terms of time complexity. In the context of our hierarchical matching approach, we analyze its time complexity as follows. Given a deep network with $N_L$ layers, each containing $C$ channels, traditional graph matching incurs a time complexity of O(N^4), where $N$ denotes the total number of nodes in the graph. However, by employing a hierarchical matching strategy for deep networks, we can compute the time complexity separately for each layer and then aggregate the results. Consequently, the overall time complexity of our approach is:

\begin{split} & O(\sum_{1}^{N_L} C^4) = O(\sum_{1}^{N_L} (\frac{N}{N_L})^4) = O(\frac{1}{N_L^3} N^4). \end{split}

• Although graph matching is generally an NP-hard problem, using bilateral relaxation transforms it into a solvable problem by converting the general graph matching problem into a bipartite graph matching problem. The original graph matching problem may involve graphs with arbitrary topological structures, but bipartite graph matching is relatively easier to solve computationally. In this paper, we use the Sinkhorn algorithm, which is a polynomial-time algorithm.

Response to Essential References: Thank you for providing these papers. We indeed omitted the comparisons in our manuscript. We will incorporate references to these papers in our study and conduct comparative analyses accordingly.

Comparison with [1]:

1. This method, like ours, focuses on protecting connection pathways.
2. The specific connection points protected are different. [1] focuses on protecting network connections between adjacent layers.
3. The methods for preserving knowledge from previous tasks differ. [1] protects important connections for previous tasks by freezing them, whereas we use a soft protection method.
4. The number of trainable parameters for the next task is different. [1] does not train frozen parameters.
5. The protection of data privacy protection is different. [1] requires storing previous data, whereas our method does not require storage.

Comparison with [2]:

1. The different approaches for protecting task-related knowledge. [2] protects task-specific knowledge by compressing and safeguarding neurons that are crucial for particular tasks. In contrast, our method achieves task knowledge protection through matching-based approach.
2. [2] aims to reduce the interference between different tasks, whereas our method achieves knowledge sharing through interference.

Comparison with [3]:

1. The different approaches for protecting task-related knowledge. [3] employs dynamic masking (including for parameters and gradients) to protect task knowledge, whereas our approach utilizes a matching-based method.
2. Our method facilitates the propagation of knowledge, while [3] prohibits it. [3] employed in this paper utilizes hard masking, which is detrimental to the sharing of common knowledge across different tasks.
3. The protection of data privacy protection is different. [3] requires storing previous data.

We replicated two of the above methods and integrated them into our framework for comparative analysis.

ResNet32-based model on the CIFAR-100 dataset.

Response to Weaknesses: We conducted corresponding experimental tests on the ImageNet-R dataset and also performed tests using ResNet50.

(1) ResNet32-based model on the Imagenet-R.

(2) ResNet50-based model on the Cifar100

Response to C1: Thank you for the reviewer's reminder.

• We conducted a corresponding analysis, including the optimization of our proposed method in terms of time complexity. In the context of our hierarchical matching approach, we analyze its time complexity as follows. Given a deep network with $N_L$ layers, each containing $C$ channels, traditional graph matching incurs a time complexity of O(N^4), where $N$ denotes the total number of nodes in the graph (All neurons in the neural network). However, by employing a hierarchical matching strategy for deep networks, we can compute the time complexity separately for each layer and then aggregate the results. Consequently, the overall time complexity of our approach is: \begin{split} & O(\sum_{1}^{N_L} C^4) = O(\sum_{1}^{N_L} (\frac{N}{N_L})^4) = O(\frac{1}{N_L^3} N^4). \end{split}
• To address the reviewers' concerns, we tested the time cost and found that due to the addition of the matching fusion module, our method's runtime is second only to LwF.

Runtime of ResNet32 on CIFAR-100 (min)

Response to C2 and Experimental2,3: Thank you for the reviewer's reminder.

• During inference, the space and time complexity of [1] are both O(N), where N is the number of tasks. In contrast, our method has both time and space complexity of O(1). To ensure a fair comparison under O(1) complexity, we tested using the mask of the last task as well as the intersection of masks from different tasks, ensuring that the time complexity during testing remains O(1). The results of methods such as WSN and GPM were obtained through comparisons in task-incremental learning. ResNet18-based model on the Cifar100. |Method|10 splits|20 splits| |-|-|-| |Ours|50.9|47.7 |HAT + [1](Final)|10.2|12.9 |HAT + [1](Intersection)|13.5|15.4 |SupSup + [1](Final)|15.2|18.8 |SupSup+ [1](Intersection)|18.7|23.4

• Due to differences in the training environment, dataset splitting methods, and training details (e.g., we use 200 epochs, whereas [1] uses 700 and 1000 epochs), the final training results vary. Additionally, the baseline methods in our paper are implemented based on the code from [2], which aligns well with the corresponding results in [2]. We integrated the LwF method into the training process of [1], leading to the following results: ResNet18-based model on the Cifar100. |Method|10 splits|20 splits| |-|-|-| |Ours|50.9|47.7 |LwF|42.2|40.8

[2] Class-incremental learning: survey and performance evaluation on image classification. TPAMI,2022.

Response to C3 and Experimental2,3: Thank you for the reviewer's reminder. Thank you for the reviewer's reminder. The reviewer is indeed correct—it is possible to infer the task ID, just not using the approach in [1]. The SupSup paper presents an algorithm for task ID inference based on optimization rather than an O(N) complexity method. However, the inferred task ID may not always be accurate (especially when the number of tasks is large, such as 100 tasks). When we compared this approach with ours, we found that our method outperforms it in task-agnostic scenarios. Lastly, we will revise our wording and include additional experiments in task-agnostic settings.

Response to Theory: We greatly appreciate the reviewer for pointing out this issue. We have performed the corresponding theoretical derivation and will include it in the paper.

The detailed derivation process for 1 and 2 can be found in the response to Reviewer mxBD.

1. Forgetting Bound

$\mathcal{F}(T) \leq \eta\sqrt{2(1-\kappa_S)} \, \text{(Shallow)}+\lambda \epsilon \, \text{(Deep)}+O\left(\sqrt{\frac{\log T}{T}}\right)$

2. Task Number

$T_{\max} \leq \frac{C}{2\epsilon} \log(1 + \frac{C}{\epsilon \Delta^2})$ We divided Tiny-ImageNet into 100 tasks and compared different approaches.

3. Naive Avg Bound:

$\mathcal{F}_{\text{avg}} \geq \frac{1}{2}\sqrt{\sum \delta_c^2} + \lambda C + \mathcal{O}(1)$

• $\mathcal{F}_{\text{avg}}$: Forgetting measure (naive averaging)
• $\delta_c$: Channel misalignment distance for filter $c$ When channel alignment quality $\kappa_S > 0.5$ and overlap ratio $\epsilon < 1/C$, the forgetting ratio satisfies:

$$\frac{F_{\text{LwI}}}{F_{\text{avg}}} \approx \sqrt{2(1-\kappa_S)} \, \text{(alignment gain)} \cdot \epsilon \, \text{(sparsity gain)} + O(T^{-1/2})$$

1. Our method provides an upper bound on forgetting
2. Naive averaging only provides a lower bound
3. Experiments |Method|5 splits|10 splits|20 splits |-|-|-|- |Naive Avg|72.13 |74.80 |75.30 |Ours| 81.10 | 84.90 |86.49

Response to W1: Thank you for the reviewer's reminder. We have supplemented our theoretical derivations.

1. Core Theoretical Framework

1.1 Shallow Layers (Shared knowledge)

Core: Minimize weight differences through optimal transport alignment of similar channels

1. Parameter Update Merged shallow weights: $W_S^{\text{merged}} = (1-\eta)W_S^{\text{old}} + \eta P_S W_S^{\text{new}}$ where $P_S$ is the permutation matrix maximizing similarity.
2. Difference Bound

• Define channel similarity matrix $K_S$ where:

$$P_S = \arg\max_P \text{tr}(P^\top K_S) \quad \text{where} \quad K_S[i,j] = \frac{w_i^\top w_j}{\|w_i\|\|w_j\|}, w_i = W_S^{\text{old}}[:,i], w_j = W_S^{\text{new}}[:,j]$$

• Optimal transport guarantees existence of $\kappa_S = \max K_S[i,j]$: $\|P_S W_S^{\text{new}} - W_S^{\text{old}}\|_F^2 = 2(1 - \kappa_S)$
• Therefore: $\|W_S^{\text{merged}} - W_S^{\text{old}}\| \leq \eta \sqrt{2(1 - \kappa_S)}$

1.2 Deep Layers (Task-specific isolation):

Core: Limit inter-task interference through sparse channel separation

1. Channel Overlap Let tasks $t$ and $t'$ share $\epsilon C$ channels where: $|\mathcal{A}_t \cap \mathcal{A}_{t'}| \leq \epsilon C$
2. Probability Bound Using Chernoff bound:
   $\mathbb{P}(\text{Overlap} \geq \epsilon C) \leq e^{-\epsilon C/2}$
3. Worst-case Gradient Conflict With Lipschitz constant $\lambda$ and task separation $\Delta$: $\| \nabla L_t - \nabla L_{t'} \| \leq \lambda / (\epsilon \Delta^2)$ Simplified Form: When $\Delta \propto 1/\epsilon$ (implied by sparsity):

$$\text{Deep Term} = \lambda \epsilon$$

1.3 Dynamic Error Term Derivation

Core: Azuma-Hoeffding Inequality for martingales

1. Regret Definition For task sequence with losses $L_t(\theta_t)$: $R(T) = \sum_{t=1}^T L_t(\theta_t) - \min_\theta \sum_{t=1}^T L_t(\theta)$
2. Concentration Bound With probability $1-\delta$: $R(T) \leq \sqrt{2T \log(1/\delta)}$
3. Per-Task Error Setting $\delta = 1/T$:
   $\frac{R(T)}{T} = \mathcal{O}\left(\sqrt{\frac{\log T}{T}}\right)$

1.4 Forgetting Bound Theorem

$\mathcal{F}(T) \leq \eta\sqrt{2(1-\kappa_S)} \, \text{(Shallow)} + \lambda \epsilon \, \text{(Deep)} + O\left(\sqrt{\frac{\log T}{T}}\right)$ Where:

• $\kappa_S = \max \mathbf{K}_S[i,j]$ (Peak shallow similarity)
• $\epsilon$ = Channel overlap ratio
• $\lambda$ = Task conflict intensity (Lipschitz constant)

1.5 Hyperparameter Settings

2. Task Capacity Bound

1. Define regret $R(T) = \sum_{t=1}^T L_t(\theta_t) - L_t(\theta^*)$
2. Apply Azuma-Hoeffding: $\mathbb{P}(R(T) \geq \sqrt{2T \log T}) \leq 1/T$
3. Per-task average: $\frac{R(T)}{T} = \mathcal{O}(\sqrt{\log T/T})$
4. Expected overlap: $\mathbb{E}[X] = \epsilon^2 C$ Concentration: $\mathbb{P}(X \geq \epsilon C) \leq e^{-\epsilon^2 C/2}$
5. Conflict bound: $\lambda \epsilon C$
6. Channel combinations: $\binom{C}{\epsilon C} \geq T$
7. Stirling approximation: $\log T \leq C H(\epsilon)$ where $H(\epsilon)$ is binary entropy
8. Final bound: $T_{\max} \leq \frac{C}{2\epsilon} \log(1 + \frac{C}{\epsilon \Delta^2})$

3. Experiments

We divided Tiny-ImageNet into 100 tasks and compared different approaches.

Response to W2: Thank you for the reviewer's suggestions.

• At first, we considered that CL is designed for resource-constrained edge devices (e.g., mobile phones) where models must adapt quickly to new tasks with limited computation/memory.
• To address the reviewer's concern. We conducted corresponding experimental tests on the ImageNet-R dataset and also performed tests using ResNet50.

(1) ResNet32-based model on the Imagenet-R

(2) ResNet50-based model on the Cifar100

Response to Q1: Thanks for the comments. We would like to explain them as follows:

• When the number of tasks grows or data complexity rises, the model requires more channels to mitigate inter-task interference.
• ResNet18's deep channels (512) far exceed those of ResNet32 (64), enabling ResNet18 to better meet the demands of sparse allocation in complex task scenarios. We verified that, under the same scenario, the sparser the deeper layers of the network, the greater the performance improvement.