Revisiting Prompt-based Methods in Class Incremental Learning

1. Potential test information leakage: There is a risk of information leakage during testing when using the threshold determination. According to Section 5.2, the proposed method requires updating the parameters of the $\beta$ distribution using all test samples.
2. The empirical analysis in Section 4 does not properly support the claim made in Section 5.2. Specifically, the paper states: "Our analysis in Section 4 reveals that even if the first-stage model correctly predicts the class label, the second-stage model may still misclassify the category, despite receiving the correct task identity." However, Section 4 does not provide any results regarding the proportion of correct class label predictions by the first-stage model. It only reports task-ID prediction results, and a correct task-ID prediction does not imply a correct class label prediction.
3. Observation 2 lacks meaning: The statement "a reliable task identity benefits the second stage without requiring identical architecture for both stages" is not a significant finding. It is obvious that better task identity predictions lead to improved second-stage predictions, irrespective of architecture. A more meaningful insight might be phrased as: "Self-supervised models can provide more reliable first-stage task identity predictions than supervised models."
4. The writting of this paper is very poor. It certainly requires careful revision and polishing. Some specific issues include:

• In section 5.1.2, the parameters $\theta_{s}$ and $\theta_{t}$ are not clearly defined. It is unclear whether they represent parameters in the whole first-stage model or only the newly added parameters. If they are the added parameters, Equation 4 is problematic, as the added component does not take the input image $x$ as input. If they represent the entire first-stage model, there is a conflict with the description in line 266, which states that $\theta_{s}$ is optimized using SGD, contrary to Figure 1’s illustration.
• Symbols $K$ and $i$ are not explained in Equation 4. Does "$K$ dimensions" refer to the number of classes or tasks?
• There are inconsistencies in symbol definitions. For example, $\theta$ represents network parameters in Equation 4, but is later used to represent a distribution in line 295, Equation 5, and Equation 6. Similarly, $\tau$ is used for temperature in Equation 4 and for filtering threshold in Equation 9.
• Punctuation is missing after all equations.
• Conflict results. In table 3, the reported result for the methods with Sup-21K is 47.47, but change to 47.37 in table 5, 6, and 7.

5. Negative effect on ID datasets. Although the method shows improvements on OOD scenarios, it brings negative effect on some ID datasets. For instance, -1.06% on CIFAR-100, -0.61% on ImageNet-R. No ablation studies were conducted on ID scenarios, making it unclear whether both the self-distillation loss and the threshold determination strategy are harmful in these cases.

1. It has been revealed by previous works [1,2] that the self-supervised technique can learn more generalizable and transferable features and thus benefit continual learning. Moreover, the self-distillation loss comes from DINO as mentioned in the Method section. Therefore, the self-supervised training for the first stage does not bring sufficiently novel insights or techniques for the community.

[1] F. Zhu, X.-Y. Zhang, C. Wang, F. Yin, and C.-L. Liu, “Prototype Augmentation and Self-Supervision for Incremental Learning,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition, Computer Vision Foundation / IEEE, 2021, pp. 5871–5880.

[2] Z. Li et al., “Steering Prototypes with Prompt-Tuning for Rehearsal-Free Continual Learning,” in IEEE/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 2523–2533.

2. The performance of the proposed method heavily depends on the pre-trained dataset and the domain/distribution of the dataset. Especially, the performance may degrade on non-OOD datasets. For example, as shown in Table 6, only the first-stage model pre-trained on DINOv2-LVD142M shows improvement on CIFAR-100, while those pre-trained on Sup-21K and DINO-1K underperform the baseline HiDe-prompt. It is essential to address the shortcomings in non-OOD scenario, since the ID data is more widely used in reality.

3. A deep analysis of the reason why the final accuracy declines (on non-OOD data) is required. In theory, the self-supervised training for the first-stage model can improve the inference accuracy of task identity. Is the threshold-filtering strategy in the second stage harmful to the overall performance?

4. In prompt-pool-based CL methods, the models of the two stages are originally decoupled and they are not enforced to be identical. Therefore, I think the first question in empirical analysis has marginal significance.

5. The threshold-filtering strategy is based on the distribution of confidence scores. The reason why the Beta distribution is adopted is not clear. Moreover, it is a straightforward and common technique, which does not provide novel ideas for a better filtering strategy.

1. The authors present results in non-OOD scenarios in the paper, where performance generally declines in most cases. However, they do not specifically analyze or explain the reasons for this performance drop. Why does performance improve significantly in OOD scenarios, while it declines in non-OOD scenarios?

2. The experiments in the paper demonstrate that the models in the first and second stages can use different architectures and pre-trained weights, supporting the authors' findings. However, there is a lack of theoretical proof, and it is hoped that the authors will provide a detailed theoretical analysis rather than just presenting experimental results.

3. In the paper, the authors mention, "the second-stage model may still misclassify the category, despite receiving the correct task identity. This situation occurs frequently in OOD scenarios, with an occurrence rate of around 10%." How did the authors calculate this 10% result, and is there any experimental evidence or theoretical analysis to support it?

• Some of prompt-based CIL methods dynamically generate prompts, and these important works are not discussed and compared in this paper. As described in this paper, the two-stage “select+prediction” prompt-based CIL methods usually meet the issues at the first stage. However, some “prompt-generation+prediction” prompt-based CIL methods (e.g. [a]-[b]) directly generate instance-wise prompts and do not need to select prompts at the first stage. It is necessary to give a detailed analysis about these methods. For instance, compare the performance and memory cost of these methods on both the ID and OOD CIL benchmarks.

[a] Generating Instance-level Prompts for Rehearsal-free Continual Learning, ICCV2023

[b] Open-World Dynamic Prompt and Continual Visual Representation Learning, ECCV2024

• The uniqueness of the summary on prompt-based methods is unclear. Previous works (e.g. [c]-[d]) have made a summary about prompt-based CIL methods and it makes the contribution of this paper limited. To better show the unique contribution of this paper, the key difference among these works should be highlighted. The authors can provide performance comparison experiments on ID and OOD scenarios to show the necessity of combining self-supervised learning to prompt-based CIL methods.

[c] Continual Learning with Pre-Trained Models: A Survey, IJCAI2024

[d] Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need, arxiv2023

• Missing the experimental analysis on some complex CIL scenarios (e.g. on some challenging CIL benchmarks). For example, the work[d] has proposed four new challenging ID CIL benchmarks without data overlapping. To better show the effectiveness of the proposed method, the experimental analysis on some complex CIL benchmarks are needed (e.g., evaluate these methods on the cross-domain CIL benchmark VTAB [d]).

• To my knowledge, the effectiveness of utilizing a self-supervised pre-trained model to provide task-ids depends on the gap between the pre-trained data and incrementally learned tasks. If the gap is large, a self-supervised pre-trained model can not provide reliable task identities. Can the authors make a discussion about this point? Compare the performance of the proposed method across CIL datasets with varying degrees of similarity to pre-trained data.