We thank the Reviewer for the time spent on our work and the suggestions for improving its quality. Below, we address the points raised by the Reviewer.

If we just care about the optimal final accuracy (the point at 100% inference cost), the amount of improvement provided by auxiliary classifiers seems limited for more recent methods, especially considering the additional training overhead of these auxiliary classifiers.

We do not consider ACs a standalone method, but rather a addon-method that enhances existing approaches. Our experiments demonstrate that ACs enable inference cost savings up to while providing moderate performance improvements, even for newer methods such as LODE, where we both improve the performance and save up to 20% computation without any performance degradation, or save 50% compute while keeping the same performance as the baseline (see Figure 1).

The observation that intermediate features are more stable in continual learning methods is not a novel one, as mentioned in the paper in lines 186-189.

As the Reviewer notes, we acknowledge prior work on the stability of intermediate features and we treat it as a motivation for our approach. To our knowledge, the idea of leveraging intermediate features to enhance both efficiency and performance in continual learning has not been explored before. Our work builds upon the known observations and introduces a novel approach to utilize it effectively.

The Riemer et al., 2018 paper is referred to as "ER" in the paper, which is a little bit confusing. (...)I would like the authors to confirm that the "ER" columns in the tables actually refer to the method proposed in the Riemer et al., 2018 paper.

We thank the reviewer for pointing out this inaccuracy. The "ER" in our work refers to Experience Replay as described in [1], and we will update the paper to make this clear.

To clarify the distinction between FT+Ex and ER, FT+Ex represents the most naive form of replay, where stored data is simply added to the training dataset for each task without modifying batching or sampling procedures. In contrast, ER ensures that each batch maintains a balanced number of samples from both old and new classes and involves oversampling from the memory. Therefore, ER is a more sophisticated replay method.

I have some difficulty understanding what figure 4a is plotting exactly ("we plot overthinking normalized by the accuracy of the final classifier" seems a bit unclear). (...)

We appreciate the Reviewer’s feedback. Given the oracle accuracy $Acc_{oracle}$ and final classifier accuracy $Acc_{final}$, normalized accuracy in Figure 4a is simply $\frac{Acc_{oracle} - Acc_{final}}{Acc_{final}}$. We will clarify this in the updated version of the paper.

In figure 2, the earlier layers (such as AC1) changes least during the process of continual learning, while AC6 changes most (closet to the final layer). However, in figure 3, we notice that AC6 provides most performance improvement for a majority of the tasks in non-trivial CL methods. Is there an intuition on why this is the case? (...)

Thank you for the question. Early feature stability for older tasks is a motivation behind our approach, but it does not conflict with the better discriminative abilities of the classifiers attached to the later layers, especially on the new data. Figure 2 demonstrates classifier performance in isolation, thus later classifiers will exhibit higher downstream accuracy. Our approach leverages both of those properties to enhance overall performance in CL. Note how with different ACs we learn to classify the samples based on diverse features (e.g. more primitive features in early layers, and more sophisticated features in the later ones) and this improves the robustness of our method (as evidenced by our analysis of overthinking).

We will elaborate this in more detail in the updated version of the paper.

Is there any justification on using the confidence score to choose which auxiliary classifier to use for the final prediction (...), especially given that neural networks are often not very well calibrated?

Our choice of the dynamic inference aggregation rule was primarily empirical. Methods that combine multiple predictions (e.g., voting, weighted ensembling) introduce additional complexity without yielding clear improvements (see our analysis of cascading and ensembling classifiers in Appendix C.2). Therefore, we opted for the simpler approach of using a single classifier’s output. We conducted an ablation study comparing the use of the final classifier prediction versus selecting the prediction with the highest confidence (Appendix E.5), and opted to use the best-performing method. See also response to the Reviewer FvJb about thresholding strategies.

References

[1] Chaudhry et al., "Continual learning with tiny episodic memories.", Workshop on Multi-Task and Lifelong Reinforcement Learning (2019).

We thank the Reviewer for the time spent on our work and the suggestions for improving its quality. Below, we address the points raised by the Reviewer.

Since the method focuses on forgetting, I expect to see a metric to measure it in the experimental results.

We appreciate the Reviewer’s suggestion and have now included forgetting metrics for CIFAR-100. Our method generally helps in mitigating forgetting. Even in cases where we see more forgetting with ACs, it still outperforms the baseline in terms of accuracy (see Table 1).

Assuming we train on total of $T$ tasks, we compute forgetting as $\frac{1}{T-1} \sum_{t=1}^{T-1} (Acc_{t}^{t} - Acc_{t}^{T})$, where $Acc_{i}^{j}$ refers to the accuracy on the i-th task after training on j-th task (we skip T-th task, as it has zero forgetting according to this definition). We will add these results to the paper.

One comparison that could make it stronger is the comparison against shallow architecture (with perhaps wider layers).

Thank you for the suggestion. To address this, we evaluated our method using WideResNet16 [1] on CIFAR100, which is shallower than the ResNet-32, but twice wider. Consistent with previous results, the AC-enhanced methods outperform the baselines.

Since the performance relies on earlier layers, reporting the achieved accuracy of task t at time t is useful.

Thank you for this suggestion. For the Reviewer’s convenience, we plotted such accuracies (link). We will add those plots to the paper appendix.

CKA is mentioned in Section 1 without definition. Figure 2 is not clear (...).

Thank you for the feedback. While CKA is a well-established metric in the literature, we acknowledge that adding a brief definition will improve clarity, and we will include it as suggested.

Do you think that these findings generalize to LLMs? And where to place ACs in that case?

Prior work, such as [2], explores the usae of early exits in LLMs, suggesting that our approach could, in principle, be applicable in that context. The placement and architecture of ACs in LLMs could follow a similar strategy to what we describe for ViTs (see Appendix A).

However, we are cautious about making direct claims regarding the transferability of our findings to LLMs. Our work explores class-incremental learning in vision, which differs fundamentally from language modeling, where tokenization constrains the class space. ACs might be applied to domain-incremental scenarios in LLMs, but investigating this requires further research beyond our current scope.

References:

[1] Zagoruyko et al., “Wide Residual Networks”, BMVC 2016

[2] Schuster et al., “Confident Adaptive Language Modeling”, NeurIPS 2022

We thank the Reviewer for the time spent on our work and the suggestions for improving its quality. We appreciate the Reviewer’s mention of Boschini et al., "Transfer without forgetting." ECCV 2022; we will include this paper in the updated related works. Below, we address the points raised by the Reviewer.

I found the use of a single seeded run for the results of the ViT-based models (lines 301-302) to be concerning. In my opinion, for a proper evaluation the tests should be repeated across multiple runs without seeds.

We appreciate the Reviewer’s concern. Due to computational constraints, we initially conducted these experiments with a single seed, as ViT-based models are particularly expensive to train. In response to the Reviewer’s suggestion, we have now rerun the ViT experiments from Figure 7 using two additional seeds (three in total). The updated figure, including confidence intervals, is available at link, demonstrating that our findings remain consistent across multiple runs. We will incorporate these results into the revised version of our paper.

While the manuscript investigates the change in the intermediate representations for a backbone trained from scratch, it could be interesting to analyze the effect of AC on pre-trained backbones, especially in light of [1] which conducts the same initial tests with CKA for pre-trained networks.

Thank you for the interesting suggestion. We conducted warm-start continual learning experiments (Appendix B.1) as a controlled alternative to pretraining, empirically demonstrating that our approach remains effective in such a setting. We consider the investigation of pre-trained models with ACs a promising future work direction.

How much is the overhead of computing all gradients for the losses of the ACs (e.g., in terms of required GPU memory and training time with respect to the baseline)?

We appreciate the Reviewer’s question. We measured training times and peak memory usage for training ResNet32 with different numbers of ACs on CIFAR-100, summarized in the tables below. "0 ACs" refers to a standard network. We report training times in hours (mean from three runs) and peak memory usage in GB. For memory, we include one table, as it does not depend on the data split.

Training times (in hours) - 5 tasks

Training times (in hours) - 10 tasks

Peak GPU memory usage (GB)

Our standard setup (6 ACs) results in approximately 50% training time and 10% memory overhead. The exact overheads will depend on the model, AC architecture, and training environment. However, we do not consider this overhead prohibitive for real-world use cases, particularly in offline class-incremental learning, where training resource constraints are not usually strict; therefore, we also did not optimize the code towards such constraints. As our primary concerns are efficiency and performance during the inference time, we consider the discussed overheads acceptable. We appreciate the Reviewer raising this important point, and we will incorporate this discussion in the paper.

We thank the Reviewer for all the suggestions. Below, we address the points mentioned by the Reviewer. We will include the clarifications and the Reviewer’s suggestions in the updated version of our paper.

Accuracy metrics and backward transfer (BWT)

In all our experiments, we report the average accuracy across all classes after the final task.

BWT can be seen as the negative of forgetting, which we discussed in the response to the Reviewer Y8XT. It is hard to observe BWT consistently in our setting across the whole training course and interpret this metric, so we do not report it like most class-incremental learning works (the metric is mostly used in task-incremental settings). However, BWT can be derived from the per-task accuracy plots we added in response to Reviewer Y8XT (link).

Early vs later layer representation stability

We agree with the reviewer that feature-sharing is a useful concept when explaining the results observed in our paper. Early-layer representations tend to capture shared, low-level features that are less task-specific, making them more stable across tasks and hence suffer less from forgetting. The more complex, later-layer features, are more task-specific and consequently suffer from more forgetting (since they might not be required for current task data). As a consequence of this phenomenon, during continual learning the ACs based on lower layers can potentially outperform those of the later layers.

Forgetting in the ACs

As noted by the Reviewer, our ACs are still classifiers and as such are also susceptible to forgetting. Our approach leverages the stability of earlier features shown in Figure 2 but also benefits from the diversity of features used by different ACs. Since we classify based on the information aggregated across multiple ACs, it is enough if just one classifier returns a confident prediction for a given sample to make the correct prediction. Empirical results indicate that AC-based networks generally exhibit lower forgetting on a per-task basis across continual learning (see the forgetting plots under the link and the response to Reviewer Y8XT), supporting our intuition that using multiple ACs results in more robust classification.

Thresholding strategy

The Reviewer correctly points out that our inference strategy is influenced by prediction confidence, which varies for each AC and the task. While confidence-aware inference strategies could improve performance, they are complex enough to warrant separate papers (e.g., Meronen et al., “Fixing Overconfidence in Dynamic Neural Networks”, WACV 2024) and often rely on calibration on a holdout dataset, which limits their applicability to exemplar-based methods. To maintain simplicity and generalizability, we use a shared threshold. The fact that our approach performs well without per-AC adaptation further highlights its robustness and broad applicability.

Overthinking definition

Our use of the term “overthinking” follows the SDN paper and is well-established in the literature, so we skipped it to keep the paper concise. Figures 4b and 4c are supposed to provide a more nuanced analysis of the phenomenon.

Unique overthinking

"Unique overthinking" indicates how many samples can be correctly classified by a unique AC. It highlights that each classifier specializes in different subsets of the data in a non-redundant way.

How are the ACs initialized?

We follow the same FACIL protocol for both the main classifier and the auxiliary classifiers (ACs) - we add a new head for the classes introduced in each task.

Gradient propagation in Section 3

We mention that gradients from ACs are detached from the backbone in Section 3.2 (L#201-204); in Section 3.4, we compare this with a setup where gradient propagation is enabled end-to-end to empirically evaluate which approach yields better results. Then in all following experiments, we used the setup with enabled gradient propagation.

Gradient propagation can sometimes negatively affect individual classifier performance. However, while some individual classifiers may degrade slightly, the overall benefit outweighs these effects. In most cases, gradient propagation training leads to better results (see Appendix C.3.). In Appendix E, we also show that findings from our analysis are consistent across with both detached and enabled gradients.

Training times with ACs

See the response to the Reviewer dDEJ.

Figures clarification

Figure 3 is intended to highlight which auxiliary classifiers outperform the final classifier.

The marks in Figure 5 are supposed to represent correct and incorrect predictions.

The accuracy reported in Figures 1 and 6 is the average accuracy after the final task.