Probabilistic Group Mask Guided Discrete Optimization for Incremental Learning

Thank you for your valuable comments.

Generalization to Non-Vision tasks: To ensure fairness and comparability, we adopt the same task setup as in [1,2], which are widely used as baselines for evaluating incremental learning performance. To further assess the generalization capability of our method beyond the vision domain, we extend it to an audio classification task using the KineticsSounds dataset [3]. The dataset is divided into five incremental tasks, denoted as KS-5. As shown in Table 1, PGM outperforms WSN in both accuracy and parameter capacity when using the ResNet18 architecture.

Table 1. Performance evaluation on the KS dataset.

Analysis of the Plateau Effect at Larger Group Sizes: While the theoretical search space for mask combinations grows exponentially with group size $K$, our Gumbel-Softmax sampling reduces the practical complexity to linear time (i.e., $\mathcal{O}(K)$). As shown in Figure 4a of the original paper, while computational overhead increases only marginally with larger $K$, the average accuracy begins to plateau beyond a certain threshold. More detailed hyperparameter settings will be given in the final version.

Extended to Class Incremental Learning: Extending task-incremental learning to class-incremental learning requires accurate task identity recognition. Prior works [4,5] have shown that accurate task-ID prediction in the CIL setting relies on strong out-of-distribution (OOD) detection capabilities. To this end, we integrated the Energy-based OOD detection method [6] into our framework. As shown in Table 2, PGM achieves higher accuracy than WSN on both the Last and AIA metrics, suggesting that the subnetworks selected by PGM demonstrate stronger OOD detection capability.

Table 2. CIL performance comparison using DeiT architecture. AIA is the average incremental ACC. Last is the ACC after learning the final task.

Training Time on Large Scale Models: To ensure fairness and comparability, we adopt the same model architectures as in [1,2], which are widely recognized as baselines for incremental learning. To further assess the training time of PGM across different architectures, we follow the settings in [5,7] and evaluate it on both ResNet18 and DeiT, comparing against parameter-isolation methods that require no additional storage and exhibit no forgetting. As shown in Table 3, the joint evaluation and optimization of grouped parameters introduce noticeable computational overhead as model size increases. While this results in longer training time, it helps reduce the risk of suboptimal parameter selection by explicitly modeling parameter dependency.

Table 3. Performance evaluation across different model architectures.

References:
[1]. Haeyong Kang, et.al. Forget-free continual learning with winning subnetworks. ICML, 2022.
[2]. Yusong Hu, et.al. Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning. ICML, 2024.
[3]. Relja Arandjelovic, et.al. Look, listen and learn. ICCV, 2017.
[4]. Gyuhak Kim, et.al. A Theoretical Study on Solving Continual Learning. NeurIPS, 2022
[5]. Haowei Lin, et.al. Class incremental learning via likelihood ratio based task prediction. ICLR, 2024.
[6]. Weitang Liu, et.al. Energy-based out-of-distribution detection. NeurIPS, 2020.
[7]. Md. Sazzad Hossain, et.al. Rethinking Task-Incremental Learning Baselines. ICPR, 2022.

Thank you for your valuable comments.

Extending to Longer Task Sequences: To ensure fairness and comparability, we adopt the same task configurations as in [1,2] (i.e., 10, 20, and 40 tasks), which are widely recognized as baselines for evaluating incremental learning performance. To further assess performance under longer task sequences, we extend the evaluation on CIFAR-100 by increasing the number of tasks to 10, 20, and 50 under the DeiT architecture [4]. As shown in Table 1, PGM consistently achieves strong performance across all settings and demonstrates better scalability than WSN, particularly with respect to CAP.

Table 1. Performance evaluation on long task sequences using DeiT architecture.

Stability under Conflicting Dependency across Tasks and Layers: Parameter dependency are modeled at the intra-layer levels, reflecting the fact that the importance of a parameter can be influenced by other parameters within the same task. To facilitate knowledge transfer while reducing task interference, we employ a similarity-based mask initialization strategy that modulates the influence of prior tasks according to their similarity to the current task. Greater task discrepancy results in lower similarity scores, which suppress the transfer of conflicting knowledge. While the current framework focuses on modeling intra-layer dependency, incorporating cross-layer dependency may further improve performance and generalization. This remains a promising direction for future work, and we plan to explore it in subsequent research.

Different Performance Gains on TinyImageNet Compared to CIFAR-100: The difference in ACC improvement can be attributed to variations in dataset characteristics and model configurations. TinyImageNet uses images with a resolution of 64×64, includes 40 tasks, and is trained with a network composed of four convolutional layers and three fully connected layers. In contrast, CIFAR-100 uses images with a resolution of 32×32 and includes 10 and 20 tasks, with AlexNet used for the 10-task setting and LeNet for the 20-task setting. These differences result in performance variation across the two benchmarks.

Impact of Group Size K on Complexity and Training Time: While the theoretical search space for mask combinations grows exponentially with group size $K$, our Gumbel-Softmax sampling reduces the practical complexity to linear time (i.e., $\mathcal{O}(K)$). This is achieved by directly learning parameter activation distributions, thereby avoiding exhaustive combinatorial search. As shown in Figure 4a of the original paper, performance gains tend to plateau beyond K = 8, suggesting that further increasing the group size yields limited benefit while introducing additional computational overhead.

Extending to Class-Incremental Learning with Rehearsal Sample: The proposed method is developed under the task-incremental learning setting, where parameter isolation methods prevents forgetting via task-specific subnetworks. However, as noted in [3], forgetting may re-emerge in the more challenging class-incremental learning (CIL) setting due to the absence of task identity during inference. To address this, [3] introduces OOD detection for implicit task inference, and [4] further enhances it with rehearsal samples. Following the protocol in [4], we replace their mask selection strategy with ours, and compare against parameter-isolation baselines without additional memory. As shown in Table 2, our method achieves competitive performance in the CIL setting.

Table 2. CIL performance comparison using DeiT architecture. AIA is the average incremental ACC. Last is the ACC after learning the final task.

References:
[1]. Haeyong Kang, et.al. Forget-free continual learning with winning subnetworks. ICML, 2022.
[2]. Yusong Hu, et.al. Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning. ICML, 2024.
[3]. Gyuhak Kim, et.al. A Theoretical Study on Solving Continual Learning. NeurIPS, 2022
[4]. Haowei Lin, et.al. Class incremental learning via likelihood ratio based task prediction. ICLR, 2024.

Thank you for your valuable comments.

Scalability to Modern Architectures: To ensure the fairness and comparability of results, we adopt the same model architectures as in [1,2], which are widely recognized as baselines for evaluating incremental learning performance. To further assess the generalization capability of PGM across different architectures, we evaluate it on both DeiT and ResNet18 following the settings in [3,4], and compare it with parameter-isolation methods that incur no additional storage cost and exhibit no forgetting. As shown in Table 1, PGM consistently demonstrates robust performance across all evaluated configurations. Notably, the greater parameter reduction observed on ResNet architectures can be attributed to the larger number of convolutional layers, where modeling parameter dependency tends to be more effective. In contrast, Transformer-based architectures contain more linear layers, where parameter dependency are inherently weaker, leading to comparatively smaller gains.

Table 1. Performance evaluation across different model architectures.

On Correlation Quantification and Adaptive Grouping Strategies: Figure 6 of the original paper shows that parameter interactions are predominantly local, with each parameter mainly influenced by its immediate neighbors, while the impact of distant parameters is considerably weaker. This observation supports the assumption that grouped parameter blocks can be treated as approximately independent, suggesting that a simplified grouping strategy is sufficient to capture essential dependencies without significant loss in modeling capacity. Incorporating correlation-aware analysis and adaptive grouping mechanisms remains a promising direction for future research.

Huffman Encoding in CAP Calculation: To ensure fair comparison with prior work [1], we adopt Huffman encoding for consistency. The binary nature of the mask (i.e., 0/1 values) aligns well with Huffman's optimal prefix coding scheme, allowing for efficient compression and significant reduction in mask storage.

References:
[1]. Haeyong Kang, et.al. Forget-free continual learning with winning subnetworks. ICML, 2022.
[2]. Yusong Hu, et.al. Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning. ICML, 2024.
[3]. Haowei Lin, et.al. Class incremental learning via likelihood ratio based task prediction. ICLR, 2024.
[4]. Md. Sazzad Hossain, et.al. Rethinking Task-Incremental Learning Baselines. ICPR, 2022.

Thank you for your valuable comments.

Training Overhead and Parameter Sensitivity: While probabilistic sampling and Gumbel-Softmax reparameterization may increase implementation complexity, this design enables differentiable and learnable mask selection, which is essential for effective parameter grouping. Larger group sizes $K$ can incur longer training time, and we observe that the performance improvement tends to plateau as $K$ increases (see Figure 4a of the original paper). Regarding hyperparameter sensitivity, the optimal value of $K$ may vary across datasets due to differences in task difficulty and parameter dependency. More detailed hyperparameter settings will be given in the final version.

Generalization across Model Architectures.: To ensure the fairness and comparability of results, we adopt the same model architectures as in [1,2], which are widely recognized as baselines for evaluating incremental learning performance. To further assess the generalization capability of PGM across different architectures, we evaluate it on both DeiT and ResNet18 following the settings in [3,4], and compare it with parameter-isolation methods that incur no additional storage cost and exhibit no forgetting. As shown in Table 1, PGM consistently demonstrates robust performance across all evaluated configurations. Notably, the greater parameter reduction observed on ResNet architectures can be attributed to the larger number of convolutional layers, where modeling parameter dependency tends to be more effective. In contrast, Transformer-based architectures contain more linear layers, where parameter dependency are inherently weaker, leading to comparatively smaller gains.

Table 1. Comparative performance evaluation across different model architectures.

Generalization across Different Task Types: To further evaluate the generalization capability of our method beyond the vision domain, we extend it to an audio classification task using the KineticsSounds dataset [5]. The dataset is partitioned into five incremental tasks, referred to as KS-5. As shown in Table 2, PGM outperforms WSN in both accuracy and parameter capacity when using the ResNet18 architecture.

Table 2. Comparative performance evaluation on the KS dataset.

References:
[1]. Haeyong Kang, et.al. Forget-free continual learning with winning subnetworks. ICML, 2022.
[2]. Yusong Hu, et.al. Task-aware Orthogonal Sparse Network for Exploring Shared Knowledge in Continual Learning. ICML, 2024.
[3]. Haowei Lin, et.al. Class incremental learning via likelihood ratio based task prediction. ICLR, 2024.
[4]. Md. Sazzad Hossain, et.al. Rethinking Task-Incremental Learning Baselines. ICPR, 2022.
[5]. Relja Arandjelovic, et.al. Look, listen and learn. ICCV, 2017.

Thank you for your valuable comments.

Clarifying the Role of Dependency in Our Method: Modeling parameter dependency during subnetwork selection is the key contribution of this work, enabling more effective parameter allocation in incremental learning. Specifically, this involves two key aspects: (1) Dependency refers to the notion that the importance of a parameter should be evaluated in the context of its interactions with other parameters [1], in order to better reflect the collective contribution of parameter subsets. (2) To implement this idea, we partition parameters into groups and learn a categorical distribution over all possible selection combinations within each group, enabling joint evaluation of parameter subsets and thus capturing intra-group dependency.
In addition to dependency modeling, we introduce task-informed mask initialization to initialize task-specific mask distributions by leveraging similarities with prior task masks, thereby promoting efficient parameter reuse while preserving task-specific adaptability.

Clarifying the Novelty of Dependency-Aware Subnetwork Selection: Parameter-isolation based incremental learning methods aim to assign compact yet effective subnetworks to individual tasks, thereby reducing capacity overhead and mitigating forgetting. However, existing approaches typically assume parameter independence, selecting parameters based on weight magnitude [2] or learnable importance scores [3], without considering how the importance of one parameter may depend on others. This assumption can lead to inaccurate parameter importance estimation, ultimately resulting in suboptimal subnetworks. To address this limitation, we propose a dependency-aware approach, Probabilistic Group Masking (PGM), which explicitly models parameter dependency during subnetwork selection. Specifically, (1) At the methodological level, PGM partitions parameters into multiple groups and evaluates all possible combinations within each group to capture intra-group dependency. Based on this modeling, PGM performs probabilistic sampling within each group to generate task-specific masks, with the entire process made differentiable via Gumbel-Softmax reparameterization. This design enables more dependency-aware subnetwork construction by jointly evaluating parameter combinations, thereby facilitating the selection of higher-quality subnetworks and enhancing the reuse of previously activated parameters. (2) At the theoretical level, we provide a theoretical analysis showing that modeling parameter dependency significantly reduces the risk of selecting suboptimal parameters. (3) At the empirical level, we conduct extensive experiments, including comparisons with state-of-the-art baselines, ablation studies of key components, and analysis of group size. The results consistently demonstrate that our dependency-aware approach reduces overall parameter usage while maintaining or even improving task performance.

References:
[1] Denis Kuznedelev, et al. CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models. NeurIPS, 2023.
[2] Arun Mallya, et al. PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning. CVPR, 2020.
[3] Haeyong Kang, et al. Forget-free Continual Learning with Winning Subnetworks. ICML, 2022.