Learning with Preserving for Continual Multitask Learning

Response to Weakness 1:

We appreciate your suggestion to consider combining datasets, as it could represent more complex and realistic scenarios. However, our focus on individual datasets aligns with the assumptions of the CMTL setting, where the input data distribution remains consistent across tasks and the labels are independent.

Combining datasets like CelebA (image data) and PhysiQ (IMU sensory data) would introduce significant shifts not only in the input distribution but also in the data modality, which falls outside the scope of our current study. Our method is specifically designed to perform optimally under the condition of a consistent input distribution within a single modality, with allowances for minor deviations.

Addressing scenarios that combine multiple modalities, or heterogeneous data would require extending the current framework to handle such complexity. This represents an interesting direction for future work, where our approach could be adapted to more general continual learning scenarios involving multimodal datasets.

Response to Weakness 2:

Thank you for bringing this to our attention. Figure 2 is a conceptual diagram designed to illustrate the framework of our proposed LwP method. The images depicted are examples from the datasets (e.g., cancer tissues) and are included to convey the idea that the model is continuously learning different attributes (tasks) over time.

We acknowledge that the caption and explanation of Figure 2 could be clearer. In the revised manuscript, we will provide a more detailed description to clarify its purpose and ensure it effectively communicates the intended concept to the readers.

Dear Reviewer,

We deeply value the time and effort you have dedicated to reviewing our paper and providing insightful suggestions. As the discussion phase is coming to an end and no further author-reviewer interactions are planned, we would like to confirm if our responses from this and a few days ago have successfully addressed your concerns. We hope we have resolved the issues raised. However, if there are any points that require further clarification or additional concerns you would like us to address, please feel free to reach out. We remain fully committed to continuing our discussion with you.

Best regards.

Continue to our previous comment

Response to Weakness 4:

We appreciate your suggestion to explore the performance of the model when continuously learning additional tasks beyond the initial base tasks. This is an important scenario to evaluate the scalability and robustness of our approach. We will conduct further experiments in this direction and include the results in the revised paper to provide a more comprehensive validation of our method.

Response to Weakness 5:

Thank you for your valuable feedback. We agree that integrating the theoretical insights on the extension to learning problems more seamlessly into the main body would improve the paper's readability and coherence. In the revised manuscript, we will restructure the content to incorporate this section into the main text, ensuring a smoother narrative flow and better alignment with the rest of the paper.

References:

Buzzega, Pietro, et al. "Dark experience for general continual learning: a strong, simple baseline." Advances in Neural Information Processing Systems 33 (2020): 15920-15930.

Fostiropoulos, Iordanis, Jiaye Zhu, and Laurent Itti. "Batch model consolidation: A multi-task model consolidation framework." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Kim, Sanghwan, et al. "Achieving a better stability-plasticity trade-off via auxiliary networks in continual learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2L: Contrastive Continual Learning, June 2021. URL http://arxiv.org/abs/2106.14413. arXiv:2106.14413 [cs].

Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational Knowledge Distillation, May 2019. URL http://arxiv.org/abs/1904.05068. arXiv:1904.05068 [cs].

Shunjie Han, Cao Qubo, and Han Meng. Parameter selection in svm with rbf kernel function. In World Automation Congress 2012, pp. 1–4. IEEE, 2012.

Response to Weakness 1:

Our proposed method addresses catastrophic forgetting in the CMTL setting through the Dynamically Weighted Distance Preservation (DWDP) loss. This loss function is aimed to preserves approximate solutions for any problems that can be defined in z by leveraging the universality of kernel machines with the Gaussian kernel as approximators (see Sec. 3.2). By maintaining the integrity of the shared representation space, the DWDP loss enables the model to retain valuable knowledge from prior tasks while effectively learning new ones.

Moreover, our approach inherently balances the stability-plasticity dilemma. By employing dynamic weights, it prioritizes the acquisition of new information without destabilizing previously acquired knowledge. This ensures a harmonious trade-off between stability (preserving past knowledge) and plasticity (adapting to new knowledge), which is critical for success in continual learning scenarios.

Response to Weakness 2:

We agree that a deeper theoretical exploration of the DWDP loss would enhance the paper. While our work provides initial analysis and ablation studies (see Appendix E.5) comparing DWDP with other distance-preserving techniques, we found that using the unnormalized Euclidean distance effectively preserves the global structure of the representation space, resulting in improved performance. To briefly summarize, we evaluated the impact of our proposed loss function by selectively disabling the dynamic weighting feature and comparing it with other structure-preserving loss functions. The baselines included in our assessment are CO2L [Cha et al., 2021], RKD [Park et al., 2019], cosine similarity, and the RBF kernel [Han et al., 2012].

We hypothesize that this is due to the unnormalized distances capturing absolute relationships between representations more accurately. To strengthen the paper, we will expand the theoretical discussion in the revised version.

Response to Weakness 3:

Thank you for your feedback regarding the methods used for comparison. We understand the importance of evaluating our approach against current and relevant baselines, and we did our best to utilize well-established methods. In our study, we included Dark Experience Replay (DER) and its enhanced version, DER++, as part of our comparative analysis [Buzzega et al., 2020], as they are publicly available and well-documented in the literature. These methods are not only widely used in continual learning scenarios but have also been utilized as strong baselines in other settings, as demonstrated in recent works [Fostiropoulos et al., 2023; Kim et al., 2023]. This reinforces their relevance to our comparative analysis.

DER and DER++ are robust baselines in continual learning scenarios, offering strong performance and relevance to our research setting. By incorporating these methods, we ensured a comprehensive and up-to-date evaluation of our approach.

If you have specific methods in mind that could further strengthen our comparative analysis, we are open to including them to provide a more thorough and well-rounded evaluation.

Response to the concern of the performance of a model when continuously learning five tasks in the presence of five base tasks.

We have conducted additional experiments to assess the applicability of LwP in continual multitask learning scenarios where the model continuously learns additional tasks after initially learning five tasks. The model was tested on five base tasks and five additional tasks in a multitask learning manner on the CelebA dataset. The results are presented in the table below:

The table presents the performance of various methods when continuously learning five tasks in the presence of five base tasks using ResNet18 with a 64×64 resolution. We show that LwP outperforms the other methods, demonstrating its effectiveness in handling continual multitask learning scenarios. These results have been included in the revised manuscript to provide a more comprehensive evaluation of LwP's performance in continual multitask learning scenarios in Appendix D.7, along with a diagram illustrating the experiment in a different perspective. We thank the reviewer for highlighting this point.

Dear Reviewer,

We deeply value the time and effort you have dedicated to reviewing our paper and providing insightful suggestions. As the discussion phase is coming to an end and no further author-reviewer interactions are planned, we would like to confirm if our responses from this and a few days ago have successfully addressed your concerns. We hope we have resolved the issues raised. However, if there are any points that require further clarification or additional concerns you would like us to address, please feel free to reach out. We remain fully committed to continuing our discussion with you.

Best regards.

Response to Weakness 1:

In the CMTL setting, we focus on scenarios where the labels represent independent attributes of the same input domain shared across time. For example, using the CelebA dataset, one-third of the data may focus on learning one attribute (e.g., "smiling"), another third on a different attribute (e.g., "wearing glasses"), and the final third on yet another attribute (e.g., "gender"). Importantly, these new label tasks can also apply to data from previous tasks, making this setting distinct.

This introduces unique challenges that differ from earlier works like LwF. While LwF typically addresses scenarios involving mutually exclusive labels or data distribution shifts, CMTL operates under the stricter condition of consistent input distributions across tasks. This requires models to effectively utilize shared input distributions while handling independent labels, a setting where existing methods often struggle to outperform single-task learning baselines.

Our contribution lies in addressing this gap with a specialized approach tailored to the CMTL setting, enabling effective knowledge preservation and multitask learning under these more realistic and challenging conditions.

Response to Weakness 2:

Thank you for raising this important point. Our experiments currently cover datasets with varying scales—CelebA with 10 tasks at 32×32 resolution, FairFace at standard resolutions using ResNet-18, and PhysiQ as a time-series benchmark. We agree that testing our method on higher-resolution images and larger models would provide additional evidence of its effectiveness.

Since our approach is not constrained by the architecture, as long as there is a representation space before the output layer, we are confident it can generalize to other models like ViT . To strengthen our validation and demonstrate the scalability and robustness of our method in more demanding settings, we will conduct further experiments with larger models and higher-resolution datasets and include these results in the revised paper.

Response to Weakness 3:

Thank you for pointing out the importance of further validating the contribution of dynamic weighting. We have addressed this aspect in our ablation study (Appendix E.5). To briefly summarize, we evaluated the impact of our proposed loss function by selectively disabling the dynamic weighting feature and comparing it with other structure-preserving loss functions. The baselines included in our assessment are CO2L [Cha et al., 2021], RKD [Park et al., 2019], cosine similarity, and the RBF kernel [Han et al., 2012].

The results, presented in Table 3, show that the loss function with both dynamic weighting and Euclidean distance consistently outperforms these alternatives. This highlights the importance of each component in achieving optimal performance. We believe that the superior performance of Euclidean distance with dynamic weighting is due to its unnormalized nature across batches, unlike previously proposed methods.

We will ensure this point is emphasized more clearly in the revised manuscript to underline the contribution of dynamic weighting.

Reference:

Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2L: Contrastive Continual Learning, June 2021. URL http://arxiv.org/abs/2106.14413. arXiv:2106.14413 [cs].

Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational Knowledge Distillation, May 2019. URL http://arxiv.org/abs/1904.05068. arXiv:1904.05068 [cs].

Shunjie Han, Cao Qubo, and Han Meng. Parameter selection in svm with rbf kernel function. In World Automation Congress 2012, pp. 1–4. IEEE, 2012.

Response to the concern of the performance of model size and image resolution.

We have expanded our experiments to assess the applicability of LwP in bigger models and higher-resolution of the dataset, CelebA. We have conducted experiments using ResNet-50 and ResNet-101 models on 64x64 and 224x224 resolution datasets. But ViT was omitted as the architecture demands large samples for effective training and is relatively much more sensitive to optimizer hyperparameters compared to ResNets. The results are included in the appendix to provide a more comprehensive validation of our approach.

This demonstrates the competitiveness of our approach. We have included these results in the revised manuscript to provide a more comprehensive evaluation of LwP's performance in continual multitask learning scenarios in the Appendix D.6. We thank the reviewer again for highlighting this point.

Response to the Ablation Study

To strengthen our work and provide additional clarity, we have moved the ablation study from the appendix to the main paper. For further clarification, the results are also presented here for reference.

The ablation study demonstrates that LwP using $L^2$ with dynamic weighting outperforms other variations and baselines. We have included these results in the revised manuscript to provide a more comprehensive evaluation of LwP's performance in continual multitask learning scenarios in Section 4.6. We thank the reviewer for highlighting this point.

Dear Reviewer,

We deeply value the time and effort you have dedicated to reviewing our paper and providing insightful suggestions. As the discussion phase is coming to an end and no further author-reviewer interactions are planned, we would like to confirm if our responses from this and a few days ago have successfully addressed your concerns. We hope we have resolved the issues raised. However, if there are any points that require further clarification or additional concerns you would like us to address, please feel free to reach out. We remain fully committed to continuing our discussion with you.

Best regards.

Response to Weakness 1:

Thank you for your thoughtful feedback.

Our goal is to leverage the continual multitask learning (CMTL) framework to develop a generalized representation space that consistently outperforms single-task learning (STL) and other baseline methods.

In one of our evaluation settings, we treat each subset of the dataset, defined by a specific label, as a separate task. While there is no data distribution shift in this setup, the key challenge lies in effectively identifying the current feature embeddings and aligning them with the requirements of new tasks, ensuring seamless integration and preservation of prior knowledge.

Response to Weakness 2:

CMTL assumes that data for different tasks are sampled from the same overall distribution but are not necessarily the same data points. This reflects real-world scenarios where different tasks involve related data domains without requiring repeated labeling of the exact same data.

For example, consider datasets collected from U.S. roads for different purposes: one for pedestrian detection and another for lane marking detection, collected sequentially. While the specific images may differ, they share common characteristics due to being from the same environment. Similarly, in the medical domain, the same MRI scan could be labeled sequentially by different professionals. A primary care physician might annotate the scan from one perspective, while a radiologist provides a secondary annotation with a specialized focus. This process mirrors scenarios where tasks involve related but distinct labeling objectives, contributing to a more comprehensive understanding of the data [Freeman et al., 2021].

In our experiments, we ensured that the data for each task was unique to that task, reflecting practical scenarios like the examples above. We will revise the paper to better articulate this point and avoid potential misunderstandings.

Response to Weakness 3:

Thank you for your feedback regarding the methods used for comparison. We understand the importance of evaluating our approach against current and relevant baselines, and we did our best to utilize well-established methods. In our study, we included Dark Experience Replay (DER) and its enhanced version, DER++, as part of our comparative analysis [Buzzega et al., 2020], as they are publicly available and well-documented in the literature. These methods are not only widely used in continual learning scenarios but have also been utilized as strong baselines in other settings, as demonstrated in recent works [Fostiropoulos et al., 2023; Kim et al., 2023]. This reinforces their relevance to our comparative analysis.

DER and DER++ are robust baselines in continual learning scenarios, offering strong performance and relevance to our research setting. By incorporating these methods, we ensured a comprehensive and up-to-date evaluation of our approach. If you have specific methods in mind that could further strengthen our comparative analysis, we are open to including them to provide a more thorough and well-rounded evaluation

Response to Weakness 4:

We appreciate your concern about hyperparameter tuning. In our experiments, we used the same loss weights (λ_c, λ_o, λ_d) across all three datasets. This consistency demonstrates that our method is robust and does not require extensive hyperparameter adjustments for different tasks or datasets.

Additionally, we followed the parameter usage design outlined in the original papers, ensuring that our methodology is grounded in well-established practices. We will revise the paper to emphasize the stability, practicality, and reliability of our approach, further highlighting this important aspect of our work.

References:

Freeman, Beverly, et al. "Iterative quality control strategies for expert medical image labeling." Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. Vol. 9. 2021.

Buzzega, Pietro, et al. "Dark experience for general continual learning: a strong, simple baseline." Advances in Neural Information Processing Systems 33 (2020): 15920-15930.

Fostiropoulos, Iordanis, Jiaye Zhu, and Laurent Itti. "Batch model consolidation: A multi-task model consolidation framework." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Kim, Sanghwan, et al. "Achieving a better stability-plasticity trade-off via auxiliary networks in continual learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Dear Reviewer,

We deeply value the time and effort you have dedicated to reviewing our paper and providing insightful suggestions. As the discussion phase is coming to an end and no further author-reviewer interactions are planned, we would like to confirm if our responses from this and a few days ago have successfully addressed your concerns. We hope we have resolved the issues raised. However, if there are any points that require further clarification or additional concerns you would like us to address, please feel free to reach out. We remain fully committed to continuing our discussion with you.

Best regards.