Exploiting Hierarchical Taxonomies in Pretrained Continual Learning

1. While the method emphasizes separating classes within the same group to reduce overlap in the latent space, the connection to mitigating catastrophic forgetting is not well articulated. Improving class separation within groups does enhance performance, but it’s unclear how this specifically helps prevent forgetting across tasks. If separation is beneficial, why not enforce separation for all class samples rather than limiting it to within-group separation? Moreover, why isn’t separation applied between groups as well, which could further reduce potential class overlap and confusion?

2. Although the weight matrix \Gamma is computed via optimal transport to capture class distribution relationships, it’s unclear how \Gamma adjusts as new tasks and classes are introduced. In class incremental learning, the distribution of classes can change significantly with each new task. If the weight matrix is calculated only at the beginning, does it remain fixed throughout subsequent tasks, or does it get recalculated? Additionally, if the class distributions for new tasks differ markedly from previous ones, should \Gamma be adjusted to reflect these changes? The paper does not clearly address whether or how \Gamma dynamically adapts to accommodate shifting distributions in a continual learning setting.

3. The paper claims, “In particular, after Task 1, we have representations of ‘oak tree’, ‘mouse’, and ‘porcupine’ located in quite separate locations. However, when Task 2 and then Task 3 arrive, the appearance of ‘willow tree’ and ‘pine tree’ makes the latent space become fuller, and ‘oak tree’ no longer maintains the separation from the remaining classes as before and even its representation may be misassigned to other classes, leading to a remarkable drop in performance.” However, there is no visualization or data provided to substantiate this claim. Specifically, there is no clear evidence of how the feature distribution of “oak tree,” “mouse,” and “porcupine” changes before and after new classes are introduced. Including visualizations (such as t-SNE plots) or quantitative data showing this change in feature space would help validate the claim and illustrate the model’s behavior as new tasks are learned.

4. The paper claims that, “this strategy not only alleviates the forgetting of old knowledge—often caused by new classes that are difficult to distinguish from old ones within the same leaf group, but also enables active knowledge transfer between tasks.” However, there is no clear evidence or analysis in the paper demonstrating active knowledge transfer between tasks. Specifically, the paper lacks experimental results, metrics, or examples showing how knowledge learned from one task actively benefits performance on subsequent tasks. This aspect of “active knowledge transfer” should be substantiated with concrete experimental support to validate the claim.

1. In Section Introduction, there are also some methods using mutual learning/continual learning for CL, please add discussions. such as [1] Wang, M., Michel, N., Xiao, L. and Yamasaki, T., 2024. Improving Plasticity in Online Continual Learning via Collaborative Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 23460-23469). [2] Michel, N., Wang, M., Xiao, L. and Yamasaki, T., Rethinking Momentum Knowledge Distillation in Online Continual Learning. In Forty-first International Conference on Machine Learning.

2. Please provide some visualizations to show how the training strategy improves model performance.

3. Please provide some visualizations to show how the testing strategy enhances prediction performance.

4. Which GPT version is leveraged for providing expert knowledge?

5. Please provide more details on generating expert knowledge. Such as, how the prompt is designed.

• The paper is poorly written with unclear messages and equations. For instance, the problem is not well elaborated, especially about the concept taxonomy for each task. It was quite difficult to follow the objective and understand the continual learning problem with the hierarchical taxonomy prompts within a task. The equations are also not described well and ambiguous. For instance, in the denominator in Eq. 1, the set X has no clear description on the subscript and superscript. These unclear messages make the statements vague.
• As the problem statement is not clear, the continual learning task needs more description. Are the tasks having memory? What is the modality of memory to be stored in each task?
• The losses and predictions used for training and testing are not clearly explained. What forms the total loss during training? What energy-based function is used in the testing phase? Additionally, why is the "energy" function not applicable during training?
• The paper lacks motivation. Although the regularization loss is derived from the contrastive loss, no direct connection is established between addressing catastrophic forgetting and the use of contrastive loss. Also, there is no citation in the manuscript to support this connection.
• The proposed method depends on strong assumptions tied to a specific hierarchical tree structure that is custom-designed for each individual dataset. This reliance on a tailored structure, while effective for targeted datasets, restricts the model’s ability to generalize and adapt to different concepts or datasets that do not follow the same predefined hierarchy. As a result, the method lacks flexibility when applied to broader contexts or varying data structures.

The algorithm's description is a bit confusing, and the logical structure is not easy to understand.

-There is no problem with the motivation for improving plasticity and stability through a tree-like taxonomy of concepts/labels. This point is easy for everyone to understand and reasonable. However, the analysis of the motivation of the third paragraph does not seem relevant to the study's content.