Q1. In Section 4.5, why uses distribution score first and then magnitude score later. Does the order of these two score matters? This is not discussed in Section 5.3 either.

As mentioned in lines 454–457, the magnitude score $\mathcal{M}(\cdot)$ focuses only on the size of the information, thereby emphasizing the quick learning of new tasks. In contrast, the distribution score $\mathcal{D}(\cdot)$ focuses on the examples whose information distribution is closest to the target Fisher information. In summary, employing $\mathcal{D}(\cdot)$ as the primary criterion allows for an AL algorithm that is more aligned with the motivation of our paper, which is to maintain a balance between the two learning properties of ACL: prevention of catastrophic forgetting and quick learning of new tasks. Ensuring the magnitude of informativeness after establishing the balance has shown effectiveness through empirical evaluation.

See Section 5.3 (lines 482-485) of our revised draft.

Q2. Does using regularization-based methods can further improve ACL performance? Maybe the authors can consider this idea to further improve ACL.

Thank you for the insightful question. As AccuACL leverages the Fisher information matrix, we share your interest in combining it with regularization-based CL methods, which also utilize the Fisher information matrix. If regularization-based methods can effectively preserve the values of important parameters from past tasks, this integration could potentially lead to a rehearsal-free version of AccuACL. We find this topic to be an exciting avenue for future work and appreciate your input in highlighting this possibility.

See Appendix I of our revised draft.

W1-1. The proposed method only works for memory-based continual learning methods. It can not be applied on naive continual learning or regularization-based continual learning methods. I understand that memory buffer is necessary for leveraging existing AL method, but I encourage authors to add this as limitations or propose future directions to handle this issue.

We appreciate the reviewer's insightful comment highlighting the dependency of our method on rehearsal-based CL. We have clarified in the revised draft that AccuACL requires at least minimal access to past distributions. Furthermore, we agree that developing a buffer-free method for estimating the target Fisher information matrix is a promising future research direction.

See Appendix I of our revised draft.

W1-2. In Figure 1, it seems like AccuACL is good for selecting new samples that related to the learned features. However, it is not clear what are those features, and Section 4 does not explain this part clearly either.

We appreciate the reviewer's insightful comment on clarifying the learned features in Figure 1. To clarify, the learned features in Figure 1 refer to the important parameters learned from past CL tasks. Intuitively, after identifying important parameters for the past tasks and the new task through the target Fisher information, AccuACL prioritizes sample selection to preserve the parameters important for both the past tasks and the new task.

See Section 4.5.1 (lines 356-359) of our revised draft.

W1-3. More background for Active Learning is needed for CL community readers.

Thank you for highlighting this point. We agree that additional background on AL would help bridge the gap for readers more familiar with CL. In response, we have refined the AL part of the related work section (Section 2) by including relevant survey papers [a,b] and providing a clear, straightforward definition of AL. This addition aims to make the paper more accessible to the CL community.

See Section 2 (lines 98-100) of our revised draft.

[a] A survey of deep active learning, CSUR, 2021 [b] A survey on active learning: State-of-the-art, practical challenges and research directions, Mathematics, 2023

W2-1. While theoretical space complexity shows AccuACL is more efficient than BAIT, no experiment results are support this claim. An additional experiment for memory usage is needed, like Figure 2c for time complexity.

Thank you for pointing this issue out. We have addressed space complexity concerns by conducting additional experiments analyzing the memory usage of AccuACL. As shown in Figure 2(b) of the revised paper, AccuACL maintains manageable memory requirements compared to AL baselines, underscoring its practicality for CL scenarios.

See Figure 2(b) of our revised draft.

W2-2. Compared with the Uniform strategy, some existing AL baselines have better forgetting score, but worse average accuracy. This phenomenon might indicates that existing AL baselines are not learning new tasks well either. Learning accuracy metric [1] may be able to support paper's claim. If the claim is true, I expect to see existing AL methods might have higher Learning accuracy. But I am not sure what will the final results be. I encourage authors to do quick additional experiments on CIFAR-10 to see their claims is true or not.

We appreciate the reviewer's suggestion to use learning accuracy as a metric for plasticity. We conducted experiments on SplitCIFAR100 with DER++, and the results in Table R5 show that AL baselines exhibit higher plasticity (learning accuracy) but lower average accuracy, aligning with our claim that these baselines focus more on learning new tasks.

In contrast, AccuACL explicitly balances the rapid learning of new tasks and the prevention of catastrophic forgetting under a labeling budget, leading to lower learning accuracy but superior performance on forgetting. AccuACL's superior average accuracy emphasizes the importance of maintaining this balance for effective ACL performance.

Table R5: Learning accuracy($LA_{10}$), forgetting($F_{10}$), and average accuracy($A_{10}$) comparison between AL baselines and AccuACL on SplitCIFAR100($M$=500) on DER++. Standard deviation is presented in parentheses.

See Appendix E.4 of our revised draft.

W3. The description of the greedy algorithm for subset selection is unclear.

W4. Additionally, detailed settings should be provided, such as the subset sampling method, whether subsets intersect, and the size of the subsets.

We appreciate the reviewer’s insightful feedback. Rather than randomly selecting subsets multiple times, our approach employs an example-wise scoring method using the distribution score and magnitude score to rank all samples. We then select the top-k examples based on these criteria. We have clarified the two-stage sampling process in Algorithm 1 in our revised draft.

When a new task(unlabeled data pool $U_t$) is introduced:

1. Randomly sample a subset from $U_t$, label them, and train the model from $\theta_{t-1}$ (the model checkpoint from task $t-1$), to obtain $\theta_t$.
2. Using $\theta_t$, we calculate the Fisher information embedding for $U_t$ according to Theorem 4.2 and calculate the target Fisher information embedding $\mathbf{F}_t$.
3. Over-sample(twice the budget $b$) examples whose Fisher information embedding's distribution is distributionally closest to $\mathbf{F}_t$.
4. Narrow down to $b$ samples that have the highest Fisher information magnitude.
5. Acquire labels for the $b$ samples, then train a better model estimate $\theta_t$ from $\theta_{t-1}$, and remove the labeled examples from $U_t$.
6. Repeat 2-5.

See Algorithm 1 of our revised draft.

W4. If my understanding is correct, how to guarantee the convergence of the greedy algorithm and ensure that it can choose the optimal subset?

Regarding convergence guarantees, we clarify that the objective function in Equation 8 is not submodular, meaning that the greedy algorithm does not offer a bounded approximation guarantee. To address this issue, we introduced two Fisher-optimality-preserving properties under complementary assumptions, as detailed in Section 4.4 and Theorem 4.3. These properties provide a theoretical foundation for the utility of our approach, even without convergence guarantees.

Moreover, we acknowledge that the intuition of the distribution score $\mathcal{D}(\cdot)$ aligns well with our motivation to balance quick adaptation to new tasks and prevention of catastrophic forgetting, as it prioritizes the samples whose information distribution matches the target Fisher information matrix, therefore leading to superior ACL performance.

W5. Regarding the budget experiment. There is no clear definition of budget and no experiments pointing out the budget limitation.

Thank you for pointing this presentation issue out. The budget $b$ in Algorithm 1 refers to the per-round labeling budget, set as {100, 200, 300} for SplitCIFAR10, SplitCIFAR100, and SplitTinyImageNet, respectively, resulting in total per-task labeling budgets of {1000, 2000, 3000}.

Regarding budget limitations, we conducted experiments analyzing the performance trends for varying per-task labeling budgets, as shown in Table R1. Overall, AccuACL consistently outperforms AL baselines with increasing query numbers, highlighting its robustness and adaptability.

Table R1: Average accuracy for SplitCIFAR100($M$=500) on ER for different labeling budgets per task. Standard deviation is presented in parentheses.

See Figure 2(a) of our revised draft.

W1. The method employs buffer data to evaluate the current parameter importance with respect to the historical task. However, a small buffer can not adequately represent the complete data distribution of historical tasks, which is easy to overfit. Then the network may memorize these buffer samples and exhibit low parameter sensitivity, leading to an incorrect Fisher information matrix.

We appreciate the reviewer's concern about the potential for incorrect Fisher information matrix estimation with a small memory buffer. To address this issue, we developed a variant, AccuACLFull, that uses all past data for Fisher matrix estimation. As shown in Table R4, AccuACL achieves competitive results compared to AccuACLFull, indicating that a minimal memory buffer(500 examples) suffices for adequate estimation. While AccuACLFull reduces forgetting due to more comprehensive parameter estimation, it results in lower average accuracy.

We hypothesize that in a practical scenario of rehearsal-based CL, estimating the Fisher information matrix using only the memory buffer strikes a better balance between retaining past knowledge and integrating new tasks by efficiently selecting new task samples without conflicts with the memory data, instead of attempting to preserve every knowledge of the past.

Table R4: Comparison of forgetting and average accuracy between AccuACL and AccuACLFull on SplitCIFAR100($M$=500) on DER++. Standard deviation is presented in parentheses.

See Appendix E.3 of our revised draft.

W2. The paper focuses solely on the Fisher information matrix of linear classifiers. However, linear classifiers in CL often exhibit task-recency bias, making them prone to favoring new classes [1,2,3]. This raises concerns about the accuracy of the Fisher information matrix calculation. How can an unbiased Fisher information matrix be derived from a biased linear classifier?

We appreciate the reviewer highlighting concerns about task-recency bias and its impact on Fisher information matrix estimation. We gently argue that AccuACL's query selection method helps alleviate this bias by prioritizing examples that preserve critical parameters of past tasks, thereby reducing the influence of recency bias in linear classifiers as well as leading to a more reliable Fisher information matrix calculation for the upcoming AL rounds.

As shown in Table R5, AccuACL achieves balanced performance between plasticity (measured by learning accuracy [a], $LA_{10}$) and stability (measured by forgetting, $F_{10}$). This balance leads to superior overall accuracy (measured by average accuracy, $A_{10}$), demonstrating AccuACL's ability to mitigate task-recency bias effectively.

Table R5: Learning accuracy($LA_{10}$), forgetting($F_{10}$), and average accuracy($A_{10}$) comparison between AL baselines and AccuACL on SplitCIFAR100($M$=500) on DER++. Standard deviation is presented in parentheses.

See Appendix E.4 of our revised draft.

[a] Wide neural networks forget less catastrophically, ICML, 2022

Minor W1. In Table 3 (lines 472-474), how to incorporate historical knowledge into the greedy algorithm? Why use the memory buffer as the initial subset?

We appreciate the reviewer's insightful question. As iterative algorithms, such as BADGE and KCenterGreedy, allow a custom initial subset, we initialize the subset with the memory buffer to incorporate past knowledge into these AL baselines. However, as shown in Table 3, adding past information to these AL baselines even deteriorates ACL performance. We hypothesize that these methods, which prioritize geometric diversity, shift focus to selecting new task samples significantly different from the memory samples, thereby further emphasizing quick learning of new tasks. This result highlights the importance of careful incorporation of past knowledge into AL baselies and verifies AccuACL's effectiveness in appropriately integrating past knowledge into the query strategy.

Q1. In active learning, why is unlabeled data not utilized in an unsupervised manner to further enhance the training process?

We appreciate the reviewer's suggestion to clarify the use of unlabeled data in an unsupervised manner. Active learning (AL) and semi-supervised learning (SSL) address annotation costs differently: AL focuses on selecting the most informative unlabeled data examples to maximize information gain, whereas SSL exploits partially labeled datasets to improve generalization. Thus, AL emphasizes querying unlabeled data rather than directly leveraging unlabeled data for optimization.

That said, the integration of AL and SSL has led to the emerging field of active semi-supervised learning [a], which combines the strengths of both to efficiently utilize labeled and unlabeled data. However, this direction is beyond the scope of this work.

[a] Semantic segmentation with active semi-supervised learning, WACV, 2023

Q2. In Figure 3, the choice of lambda depends on the amount of unlabeled data. Does this conclusion also hold in imbalanced datasets?

Thank you for raising this important point regarding the applicability of $\lambda$. We agree that evaluating our approach in the context of imbalanced datasets is essential for a more comprehensive analysis. Beyond our current formulation, $\lambda$ could potentially be estimated more effectively by considering the number of previously seen tasks or by adapting it based on the inter-distribution distance between past tasks and the new task. These adaptive strategies represent promising directions for future research, and we appreciate your suggestion for highlighting this important topic.

We greatly appreciate your insightful feedback and genuinely hope our response has resolved your concerns. Should any concerns persist, we are willing to engage in discussion with you throughout the remainder of the discussion period.

Thank you for the detailed and clear response. Most of my concerns have been solved and I am willing to raise my score to 8.

W1. The novelty of the paper is somewhat limited, as the use of the Fisher information matrix and replay mechanisms are already well-known methods in continual learning.

Thank you for your comment. We respectfully contend that the paper's novelty lies in its novel ACL algorithm, which effectively balances the prevention of catastrophic forgetting with the capacity to rapidly acquire new tasks, through the Fisher Information Matrix (FIM) and a replay mechanism. We are of the opinion that the novelty of our work would not be compromised just by the inclusion of the FIM and the replay mechanism, which are frequently employed in CL. In detail, our novelty is elucidated as follows:

1. New Usage of the FIM. While existing CL works use the FIM as a regularizer to retain information from past tasks when learning new tasks [a, b], we use it as a metric for sample selection to assess the informativeness of each unlabeled sample in the ACL scenario. Our proposed sample selection approach, AccuACL, precisely examines the FIM to preserve the quantity of the information in a selected subset, which is not considered in the existing CL works. This work demonstrates a new use case of the FIM in the context of ACL.
2. Theoretical Justification. We further provide theoretical justification of our sample selection approach in terms of utilizing the FIM effectively. Specifically, we introduce two Fisher-optimality-preserving properties under complementary assumptions, which lead to a novel scoring criterion that ensures the optimality of the selected subset, as detailed in Section 4.4 and Theorem 4.3.

Overall, we sincerely hope that our response has addressed your concern regarding novelty.

See lines 19-22 of our revised draft.

[a] Overcoming catastrophic forgetting in neural networks, PNAS, 2017

[b] Memory aware synapses: Learning what (not) to forget, ECCV, 2018

W2. The citations in the related work section are outdated, giving the impression that the authors may not be fully aware of the latest advancements in continual learning, particularly beyond replay and regularization-based methods. Please see the survey [1] and the related work section of [2] and [3].

Thank you for informing us of recent relevant studies. To ensure that our related work section reflects the latest advancements in CL, we have expanded it by including additional citations: [c, d, e] for rehearsal-based CL and [f, g, h] for dynamic structure-based CL. This update aims to provide a more comprehensive overview of the field while maintaining conciseness.

See Section 2, lines 110-112 and 115-116 of our revised draft.

[c] Loss Decoupling for Task-Agnostic Continual Learning, NIPS, 2023

[d] Learnability and algorithm for continual learning, ICML, 2023

[e] Class incremental learning via likelihood ratio based task prediction, ICLR, 2024

[f] Der: Dynamically expandable representation for class incremental learning, CVPR, 2021

[g] A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning, ICLR, 2023

[h] Beef: Bi-compatible class-incremental learning via energy-based expansion and fusion, ICLR, 2023

W3. It would be beneficial to examine how the system and baseline models perform as the number of queries increases.

We have conducted additional experiments on SplitCIFAR100, examining the average accuracy trend as the number of per-task queried examples increases (see Table R1 below). The results show that AccuACL consistently outperforms AL baselines across varying query numbers, demonstrating its robustness and adaptability.

Table R1: Average accuracy of SplitCIFAR100($M$=500) on ER for different labeling budgets per task. Standard deviation is presented in parentheses.

See Figure 2(a) of our revised draft.

Thank you for your response. It is unfortunate that our responses were not satisfactory to you. We would like to further clarify the following points:

Accuracy Results and State-of-the-Art Comparison: The observed accuracy is influenced by experimental settings, such as buffer size, backbone architecture, and optimization methods, which differ across studies. We are confident that the accuracy will reach the level of the referenced methods if we use the same experimental setting. Therefore, we will present the results where AccuACL uses the setting aligned with that in [2] by the end of the rebuttal period.

Focus of Our Work: Our primary goal is to prevent catastrophic forgetting in the active continual learning (ACL) environment, not in a typical continual learning (CL) environment. As far as we know, this is the first work that addresses catastrophic forgetting in the ACL problem.

Space Complexity and Memory Use: $m$ is the size of the memory buffer, and the memory buffer does not grow as CL progresses. Some old examples are replaced with new examples. $m$ was fixed to a value from 100 to 5000, depending on the dataset in the paper. In addition, $d$ is the dimensionality of an embedding and was fixed to 512 in the paper. Overall, $m$, $d$, and $K$ are constant throughout the entire CL process.

We are happy to discuss your remaining concerns. Importantly, we will get back to you with additional experimental results using the other setting. Thank you very much.

W6. Given that your method selects only a small number of samples, it would be essential to compare it with recent few-shot continual learning approaches [4], [5], [6], and [7].

This is a really great idea. Active continual learning (ACL) concentrates on selecting the most informative unlabeled data for labeling, whereas few-shot continual learning (FSCL) concentrates on exploiting partially labeled data, particularly when the number of labeled examples is extremely small. ACL typically allocates a greater labeling budget than FSCL. Thus, we believe that the utilization of FSCL for optimization during the initial AL rounds could enhance model estimation, thereby facilitating more effective subsequent querying. We will reserve this intriguing concept for future research.

W7. Catastrophic forgetting does not seem to be the main problem anymore. Inter-task class discrimination may pose a greater challenge. Please refer to [2] and [3].

While we do agree that inter-task class discrimination is an important challenge in recent CL research, we believe that mitigating catastrophic forgetting remains fundamental, as it is an inherent challenge in any CL setting. In the most recent top-tier conferences, there are approximately 12 (out of 17 CL) publications in ICLR 2024 and 9 (out of 18 CL) publications in ICML 2024 that address catastrophic forgetting in their abstract, indicating that it continues to be a fundamental challenge.

Additionally, the aforementioned methods in [2, 3] also face catastrophic forgetting when the model's capacity to accommodate new knowledge is exceeded. We believe that developing an ACL strategy capable of distinguishing unlabeled samples that help prevent forgetting can significantly contribute to the research community. Our theoretical foundation is expected to provide insights that may lead to further applications, particularly beyond class-incremental learning scenarios.

Nevertheless, we also acknowledge the importance of inter-task discrimination in class-incremental learning. To demonstrate the relevance of AccuACL in CL strategies that address the inter-task discrimation, we have conducted additional experiments using LODE [c], which decouples intra-task and inter-task discrimination with a hyperparameter $\rho$ to control inter-task discrimination strength. As shown in Table R5, AccuACL consistently outperforms other AL baselines across diverse $\rho$ values, demonstrating its robustness across varying levels of inter-task discrimination.

Table R5: Performance comparison of AccuACL and baseline methods on SplitCIFAR100($M$=500) using LODE across different values of $\rho$. Standard deviation is presented in parentheses.

See Appendix E.1 of our revised draft.

W4. The baseline methods used in this paper are also outdated. See those in [2] and [3].

Thank you again for your comment. First of all, we would like to clarify that AccuACL is agnostic to rehearsal-based CL methods. Consequently, we concentrated primarily on established CL methods rather than on emerging CL techniques. We concur that recent CL methods should be evaluated as well. Due to the short rebuttal period, we could incorporate only one more baseline. In conjunction with W7, we opted to incorporate a recent rehearsal-based CL method, LODE [c], due to its capability to manage inter-task class discrimination. As shown in Table R3, AccuACL consistently outperforms the AL baselines across the majority of settings together with a new CL method. Entropy and LeastConf exhibited low forgetting on SplitTinyImageNet, attributable to the exceedingly low average accuracy. In other words, there is less to forget when learning is minimal.

Table R3: Perforance comparison of AL baselines and AccuACL combined with LODE [c] on SplitCIFAR10, SplitCIFAR100, and SplitTinyImageNet.

Moreover, we will incorporate additional recent rehearsal-based CL baselines in the camera-ready version.

See Appendix E.1 of our revised draft.

W5. The current accuracy results are quite low and significantly fall short of existing state-of-the-art methods.

W8. The results on the full dataset are notably low too, and it is unclear what method was used in the "full" setting. Please see the papers mentioned above for comparison.

We greatly appreciate your meticulous review of the experimental results.

W5: A number of configurations for assessing CL methods are prevalent in the literature. That is, to the best of our knowledge, there is no definitive standard configuration for CL works. The experimental configurations in [2, 3] diverge from ours in terms of buffer size, network architecture, number of epochs and the optimization method. The distinction between [2] and ours can be summarized in Table R4.

Table R4: Comparison of the experimental setups for CL between BEEF[2] and AccuACL.

We follow the Avalanche codebase [i], with similar performance levels reported in [j]. Please refer to Table 2 of [j], which shows comparable ranges to the values presented in our paper.

W8: The full setting refers to the fully-supervised setting, identical to the conventional CL problem, where all training examples are labeled. W8 is addressed by the response to W5.

Overall, we have double-checked that our experiment results are correct. We sincerely hope that our response has addressed your concern regarding the validity of our evaluation.

See line 415 of our revised draft.

[i] Avalanche: an end-to-end library for continual learning, CVPR Workshop, 2021

[j] Cascaded Scaling Classifier: class incremental learning with probability scaling, ArXiv, 2024

W9. The paper is overly formal and not easy to follow. Some more descriptions of intuitions will be helpful.

Your feedback is greatly appreciated. We acknowledge that the underlying intuitions could be more clearly explained in certain sections. We will undoubtedly continue to refine the paper's presentation.

W10. The space completely is very high for continual learning.

AccuACL has a space complexity of $O((m+n)dK)$, where $m$ is the memory buffer size, $n$ is the data pool size, $d$ is the embedding dimensionality, and $K$ is the total number of classes. As $m$, $d$, and $K$ can be regarded as constant numbers, the space complexity is linear to the number of unlabeled examples.

To further verify our claim, we have conducted additional experiments analyzing the memory usage of AccuACL. As shown in Figure 2(b) of the revised paper, AccuACL maintains manageable memory requirements compared to AL baselines, due to its linear complexity concerning the number of unlabeled examples. In short, AccuACL has an average memory usage of 8.3MB, which is 85.5 times smaller than the full Fisher-based AL, making our algorithm easily deployable in CL environments.

See Figure 2(b) of our revised draft.

To validate the practicality of ACL in high-performing CL settings, we changed only the backbone to a ViT pretrained on ImageNet. As shown in Table A2, the average accuracy has increased significantly. Importantly, AccuACL maintains its effectiveness, achieving performance close to the fully-supervised setting. That is, the absolute accuracy values are heavily influenced by the experiment configuration. These results indicate that our framework retains its efficacy irrespective of the accuracy value range. We hope that the main issue you raised has been addressed.

Table A2: Comparison CL and ACL setting for SplitCIFAR10 and SplitCIFAR100 using ER($M$=2000) on a pre-trained ViT.

W1. What are the open problems or future directions related to your work? Adding this to the paper would improve the paper's discussion of broader impact and potential future work. This should be additional to the limitations section in the Appendix.

Thank you for highlighting the importance of discussing open problems and future directions. In response, we have added detailed discussions in our revised draft, addressing key avenues such as developing ACL methods for vision-language models (VLMs) [a,b], combining AccuACL with regularization-based CL methods to potentially achieve a rehearsal-free ACL, and designing adaptive ACL methods suitable for realistic scenarios, such as imbalanced CL [c]. We hope that these additions provide a strong foundation for future research.

See Appendix I of our revised draft.

[a] Active Prompt Learning in Vision Language Models, CVPR, 2024

[b] Boosting continual learning of vision-language models via mixture-of-experts adapters, CVPR, 2024

[c] Online continual learning from imbalanced data, ICML, 2020

W2. Typos: Please do a grammar check on your paper. There is a typo in the Abstract “informativeness”

Thank you for the feedback. We have corrected the typo in the abstract and revised the grammar. We will continue reviewing the draft for any remaining errors.

Q1. What are the characteristics of problems where the method works well? Additionally, what are the characteristics of problems where the method does not perform well?

We appreciate the reviewer for their insightful comment. We suggest that AccuACL performs well when rehearsal-based CL uses memory management strategies that summarize the overall data distribution, enabling accurate Fisher information matrix estimation. While smaller memory buffers pose challenges, our results (e.g., SplitCIFAR100 with $M$=500) show AccuACL remains effective with as few as 5 examples per class, demonstrating robustness under restrictive conditions.

We greatly appreciate your insightful feedback and genuinely hope our response has resolved your concerns. Should any concerns persist, we are willing to engage in discussion with you throughout the remainder of the discussion period.

W1. Some experiment details like annotation budgets and number of runs are not provided, which are crucial for reproducing the results.

Q1. ... how many runs are conducted in this experiment? Also, why is there no standard deviation for the BAIT rows?

We appreciate the reviewer bringing this to our attention. We query {1000, 2000, 3000} examples during ten rounds of AL for SplitCIFAR10, SplitCIFAR100, SplitTinyImageNet for each task respectively. Moreover, we have conducted the experiments over three runs with random seeds for each experiment. The values for BAIT lack standard deviation as performance was measured in a single experiment due to high computational costs.

See Appendix H of our revised draft.

W2. Further analysis of the specific data points chosen by each AL method across different CL settings could offer more insight into optimal query selection.

We appreciate the reviewer's suggestion to analyze query selection. In response, we conducted additional analysis on SplitCIFAR10, examining data points selected by various AL baselines, as shown in Figure 6 of the revised draft. While AccuACL does not explicitly utilize specific visual features for sampling, we present these findings to provide insights and allow potential future research.

See Figure 6 of our revised draft.

W2. ... examination of annotation budget sensitivity (as many methods peak around round 10 in Figure 2, suggesting diminishing returns) would add depth to the findings.

Thank you for your suggestion. We have conducted additional experiments on SplitCIFAR100 by examining the average accuracy trend according to the number of per-task queried examples (See Table R1 below). Overall, AccuACL consistently outperforms AL baselines with increasing query numbers, highlighting its robustness and adaptability.

Table R1: Average accuracy of SplitCIFAR100($M$=500) on ER for different labeling budgets per task. Standard deviation is presented in parentheses.

See Figure 2(a) of our revised draft.

W2. ... examination of the effect of task order (since CL can be highly sensitive to task sequencing) would add depth to the findings.

We thank the reviewer for suggesting an examination of task order effects. To address this comment, we have conducted additional experiments on SplitCIFAR100 with randomly permuted task orders, where the original dataset is divided into 10 tasks of 10 classes in each sequential order. As shown in Table R2, AccuACL consistently outperforms the second-best algorithm (Uniform) across all permutations, demonstrating its robustness and effectiveness regardless of the task order.

Table R2: Performance comparison of AccuACL and Uniform across multiple task order permutation on SplitCIFAR100($M$=500), trained with ER. Standard deviation is presented in parentheses.

See Appendix E.5 of our revised draft.

Q1. In the main experiments, it's intriguing to see ACL methods, especially AccuCL and random, outperform training on the full dataset.

We appreciate the reviewer's insightful question. The observed performance surpassing the full dataset aligns with analogous findings in core-set selection for CL [a], which enhances performance by selecting high-quality subsets that represent the dataset while avoiding noise or redundancy. Similarly, AccuACL evaluates instance quality in an unsupervised manner, constructing high-quality subsets that enable ACL to outperform CL in specific settings. In our experiments, the settings where ACL outperforms CL occur primarily with AccuACL, further highlighting the capability of our algorithm.

See Section 5.2 (lines 458-465) of our revised draft.

[a] Online Coreset Selection for Rehearsal-based Continual Learning, ICLR, 2022

Q2. Figure 2 shows an interesting trend: accuracy and forgetting metrics peak around AL round 10 and then decline for all AL methods. Could you provide an explanation for this phenomenon?

We appreciate the reviewer's insightful question. Figure 2 (Figure 5 in the revised draft) examines the effect of varying labeling budgets within a single round. For past (1st-4th) tasks, AL was conducted up to 10 rounds. Extending beyond 10 rounds for the new (5th) task results in an imbalanced data distribution, likely causing the observed performance drop. Recognizing that this approach may not be the most effective way to analyze budget sensitivity, we revised Figure 2(a) to explore budget variations across all tasks, providing a more balanced and comprehensive analysis.

See Figure 2(a) and Appendix E.2 lines(916-917) of our revised draft.

We greatly appreciate your insightful feedback and genuinely hope our response has resolved your concerns. Should any concerns persist, we are willing to engage in discussion with you throughout the remainder of the discussion period.