REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning

Thank you for your review and constructive comments.

‘Q1: Could the authors provide a more detailed, quantitative exploration of this trade-off? Specifically, can they present tables or graphs that clearly map different levels of resource saving to their corresponding accuracy degradation on the tested datasets?’

[Response] Thank you. The tables below extend Table 4 of our main paper, showing how varying the intensity of AToM and ALD affects resource savings alongside accuracy drop and forgetting increase. REP provides flexible resource savings under different computational budgets while incurring only moderate accuracy loss. Nonetheless, there is a clear trade-off: more aggressive computation skipping leads to greater accuracy degradation. We will include these results in the final version.

‘Q2: The stated "curriculum limitation" is a technical detail that requires further clarification. The authors should elaborate on its meaning and explain why the inability to determine the learning curriculum constitutes a significant limitation.’

[Response] Thank you for asking for clarification. In Section 5, we intended to clarify the scope of our work: we focus on a standard continual learning setting where task arrivals are predefined as part of the problem formulation. In this setting, REP is designed to operate under the given task order.

Curriculum learning, which explicitly determines or optimizes the task sequence, addresses a related but distinct problem. Since our study does not include curriculum-based task scheduling, REP has not been evaluated in scenarios where the task order can be deliberately arranged. Investigating how REP could be combined with curriculum learning strategies is an interesting direction for future research.

‘Q3: Do they have insights into how their current prompt selection mechanism would perform when encountering truly novel tasks or highly divergent domains that might not align well with the existing prompts?’

[Response] Thank you for the comment. To clarify, the similarity-based prompt selection mechanism is part of the baseline CL methods rather than REP itself. To simulate more diverse domains, we conducted preliminary experiments with subsets of the 5-datasets benchmark, introduced in [3]. The results are reported below:

The average accuracy is lower due to the stronger domain drifts, and the optimal pool size shifted to larger values. Given the results, we agree that for tasks that cover more diverse domains, similarity-based prompt-selection mechanisms can become a limitation, as also discussed in L2P [3]. When testing L2P on a collection of 5 commonly used datasets, increasing the pool size leads to proportional improvements in accuracy. Nonetheless, as shown in the table above, the pool size is not a dominant factor for memory usage or computation time. For tasks that contain higher drifts or cover more domains, we can adopt a larger pool size without compromising the cost-efficiency that REP brings.

If we have misunderstood the reviewer’s comment, please advise us.

Reference

[2] Koh et al, WILDS: a benchmark of in-the-wild distribution shifts, ICML 2021.

[3] Wang et al, Learning to Prompt for Continual Learning, CVPR 2022.

Dear reviewer W2oY,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

Best, AC

Dear reviewer W2oY,

Thank you for your thoughtful follow-up questions. We address each point below.

‘Q1: Discrepancy between "no mem increase" and Appendix L’s "computation cost increase’

[Response] We apologize for the confusion caused by the statement in Appendix L (Figure 9). The primary factor driving computational costs is prompt length, not pool size. Increasing the prompt length (i.e., the number of trainable prompts per task) introduces more parameters to learn, which in turn increases both memory usage and computation time. In contrast, enlarging the prompt pool mainly requires additional memory to store candidate prompts; this cost is negligible compared with the backbone parameters and activations. Selecting prompts from the pool is essentially a lightweight lookup and has almost no impact on iteration time. To clarify this, we have updated the table to show training GPU wall-clock time.

We also measured how prompt length affects resource usage to support our claim. With a pool size of 10, increasing the prompt length from 5 to 10 adds roughly 500MB of memory usage and increases GPU Time by about 14% (from 2542.4s to 2896.1s). We will revise the caption and text for Figure 9 to emphasize this point and eliminate confusion.

‘Q2: Why Split ImageNet-R instead of Split PlantDisease’

[Response] We apologize for misjudging the relevance of Split PlantDisease to reviewer tV8e’s Q3. Thank you for bringing this to our attention. REP is designed to reduce computation cost and memory usage with minimal loss of accuracy. The accuracy gains observed in some cases of Split PlantDisease (e.g., L2P with ViT-L) and Split ImageNet-R (e.g., HiDe-Prompt with ViT-Ti) were not anticipated by design. Nonetheless, as suggested by reviewer tV8e, we conducted a deeper analysis to better understand these gains, assuming they might be related to reduced variance in token norms (a proxy of noisy features). We have updated the original table to include results from L2P with ViT-L on the Split PlantDisease dataset.

REP reduces normalized activation variances on both PlantDisease and ImageNet-R. However, this reduction in variance does not translate into improved accuracy for Split ImageNet-R, suggesting that variance reduction alone does not fully explain the accuracy gains. We agree with reviewer tV8e that this is an intriguing phenomenon that warrants further investigation and, with the reviewers’ approval, we propose to continue exploring the underlying causes through more extensive studies.

‘Q3: Why 10-task instead of 20-task for ImageNet-R’

[Response] L234-235 of our main paper specifies that ImageNet-R is divided into 10 tasks, each containing 20 classes. We have used this setup consistently across all evaluations in the main paper and supplementary experiments. Only in Appendix C do we alter the number of tasks---Split CIFAR-100 and Split ImageNet-R into 20 tasks and 5 tasks---to study REP under different task sequence lengths. For your reference, we copy the 20-task results of REP-L2P from Table 10 and 11 in Appendix C.

Thank you for your detailed review and highlighting both the strengths of REP and areas for improvement.

This raises the question on whether the computational speed and GPU memory saving is significant enough to justify the drop in performance.

[Response] We acknowledge that in specific setups (e.g., CODA-Prompt), REP might cause a 1–2% accuracy drop in exchange for noticeable efficiency gains. In many other cases, however, the drop is negligible; for instance, in Split PlantDisease, REP even outperformed the baseline. This shows that the performance impact varies across tasks and baseline methods. Importantly, in edge-centric contexts, a small accuracy gap is often outweighed by the substantial energy and memory benefits, making the trade-off justifiable.

‘Q1: What is the baseline method used in Table 4? Is it possible to show graph of ACC and forgetting of different n and theta for different CL method when combined with REP?’

[Response] In the main paper, our ablation study primarily uses L2P without REP as the baseline CL method on Split ImageNet-R with a ViT-L backbone (additional results on Split CIFAR-100 are provided in Appendix H). Below, we present extended results for Table 4, including other CL methods such as HiDe-Prompt (‘HiDeP’) and ConvPrompt (‘ConvP’). These results demonstrate REP’s effectiveness across diverse CL methods: accuracy drops by less than 1% for moderate settings (n=4-8, θ=0.5-0.75), while forgetting remains stable or improves thanks to adaptive merging and dropping that preserve task-specific features. See Section 4.3 of the main paper for further studies, including non-prompting and self-supervised CL methods for REP’s generalizability.

These results reveal REP's efficacy across diverse CL methods: Acc drops <1% for moderate n=4-8/theta=0.5-0.75, with Fgt stable or improved due to adaptive merging/dropping preserving task features. (Also, please refer to section 4.3 in our main paper for additional study, including non-prompting CL methods and self-supervised CL methods for REP’s generalizability.)

‘Q2: For figure 4. What happen if Random drop have different aggressiveness? Will it have a more closer performance compared to ALD?’

[Response] In Figure 4, Random Drop showed lower performance than ALD with a 25% drop. We further examined how varying aggressiveness (drop rate) affects the trade-off between accuracy and GPU time: We experimented with drop rates of 20%, 25%, 30%, and 50% on Split ImageNet-R using the ViT-L backbone, summarized in the table below. The results show that increasing the aggressiveness of Random Drop (30–50%) reduces GPU time below ALD but causes a significant accuracy drop (65–71%), making it less practical for CL. In contrast, ALD adaptively adjusts the drop ratio, maintaining resource efficiency with minimal accuracy loss.

‘Q3: For AToM, as shown in Figure 3, gradient w.r.t prompt is significant and contribute essential part to CF? Is it possible to improve AToM by involving Fisher Information Matrix like in EwC to have a gradient informed prompt merging?’

[Response] Yes, Fig. 3 shows that prompt gradients play a key role in mitigating CF, which motivates AToM’s prompt-preserving merging strategy. Incorporating the Fisher Information Matrix (FIM) for gradient-informed merging is indeed an interesting idea. We tested this approach using L2P-L (ViT-L backbone) on Split ImageNet-R. The performance improvement was limited (+0.1–0.2% in accuracy), while GPU time and memory usage increased by about 10% (e.g., baseline REP GPU time: 2542.4s → 2796.6s, memory: 4.5GB → 4.95GB). The additional computational cost of FIM reduced overall resource efficiency.

We do not consider this a fundamental limitation but note that it is nontrivial to achieve practical gains. With sufficient optimization, gradient-informed merging remains a promising direction for future work.

Dear reviewer jSzb,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

Best, AC

Thank you for your detailed review and constructive comments. Here we address your initial concerns below.

‘Q1: The core technical components, while cleverly adapted, are not fundamentally new concepts. The methodology relies on adaptive versions of existing techniques like Token Merging and Progressive Layer Dropping. While the paper's contribution lies in the intelligent, adaptive scheduling and combination of these techniques for the CL context…’

[Response] We appreciate the reviewer's observation that REP incorporates ideas related to Token Merging and Progressive Layer Dropping. However, REP introduces new algorithmic elements that go beyond simply reusing these techniques: REP introduces adaptive mechanisms for both token merging and layer dropping that are specifically designed for rehearsal-free continual learning. AToM selectively excludes prompt tokens and progressively merges tokens more aggressively in deeper layers to preserve task-specific features. Similarly, ALD dynamically adjusts layer-dropping probabilities based on merged token counts and temporal schedules. Naively applying prior static approaches results in significant accuracy drops (Table 3), whereas these tailored mechanisms maintain accuracy while improving efficiency.

In addition to these algorithmic components, REP offers a framework-level contribution by integrating them into a unified system applicable to diverse scenarios. It generalizes beyond prompting methods to non-prompting and self-supervised continual learning approaches (Sec. 4.3), consistently reducing computation and memory usage (up to 48% and 31%, respectively) with only minor accuracy degradation. REP is also instantiated on a real edge device (Appendix K), demonstrating its practical viability for on-device continual learning.

We believe that by combining these algorithmic-level components with a framework-level contribution that is both generalizable and practically validated, REP represents a novel and comprehensive solution for resource-efficient continual learning.

‘Q2: … introduce new hyperparameters …clear intuition or guidelines on how to set these parameters …without performing expensive, task-specific tuning… limit the "plug-and-play" applicability of the framework.

[Response] Thank you for raising this point. Following prior work (e.g., [1]), we tuned the hyperparameters through grid search with 5-fold cross-validation for each task. While this provides an automatic tuning procedure, we agree with the reviewer that it may limit the “plug-and-play” applicability of REP. This limitation is not unique to our method and is shared by many existing continual learning approaches.

Empirically, we observed that performance is relatively insensitive to variations in alpha, making it reasonable to fix alpha=0.9. For tau, the impact is more dataset- and model-dependent, but we found a consistent positive correlation between backbone size and the optimal tau value (see also our response to Reviewer mcMq, where we have a broader range of experiments). We will explicitly state this limitation in the final version and suggest a more systematic analysis of hyperparameter tuning as a future work direction.

‘Q3: The paper notes that in some cases (e.g., L2P with ViT-L), REP can even improve model accuracy. The authors hypothesize that this might be because ALD and ATOM help eliminate redundant computations, effectively acting as a form of regularization. This is a very interesting phenomenon, but the paper fails to provide a deeper analysis to support this hypothesis. It is unclear whether this effect is due to the suppression of noisy features or some other mechanism. Adding some visualizations or quantitative analysis to investigate this would have significantly enhanced the paper's depth.’

[Response] Thank you for noting this phenomenon. We agree that a deeper analysis would strengthen the paper. To this end, we conducted additional experiments analyzing activation variance as a proxy for noisy features in the backbone model's representations. Specifically, we measured the normalized variance of token norms in the last layer. Using Split ImageNet-R (10 tasks) with ViT-L as the backbone, we compared REP-L2P with the baseline L2P-L.

REP reduces average activation variance by 30%, suggesting that AToM and ALD suppress noisy features, which likely contributes to the observed accuracy improvements in certain settings. We propose to include this preliminary analysis, supporting the regularization effect introduced by REP, in the supplemental material.

Reference

[1] Veniat et al., Efficient continual learning with modular networks and task-driven priors, ICLR 2021.

Dear reviewer tV8e,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

Best, AC

I appreciate the authors’ thorough rebuttal and find that they have satisfactorily addressed all of my concerns. I therefore uphold my original positive evaluation.

We appreciate your detailed review and constructive comments, which are addressed below.

‘Q1: The hyperparameter comparison can be improved’

[Response] Thank you. We conducted experiments with a broader range of parameter values (alpha=0.6--0.95 and tau=6--20) and observed similar trends as before, as shown in the table below. The notation for alpha and tau was inadvertently swapped in Supplemental Figure 8 of our submission. We apologize for the mistake. As noted in line 682, hyperparameters were automatically tuned via five-fold cross-validation on the training set. The complete set of results will be included in the final version.

‘Q2: The presentation of ablation can be improved’

[Response] Thank you. The ablation results in Table 2 were reported additively from the baseline L2P-L, rather than subtractively from our final design. Our algorithm integrates three key components: (a) surrogate model f_efficient, (b) adaptive token merging (AToM), and (c) adaptive layer dropping (ALD). In Table 2, entry `(2) AToM' indicates that only AToM was used among the three components, which explains why configuration (6) is more computationally efficient. We will revise the table so that each row reflects performance with the corresponding component removed. The updated table will be as shown below.

‘Q3: About the Prompt Selection’

[Response] Thank you for the suggestion. We had not initially tested this approach, but your question prompted us to conduct experiments, with results shown in the table below. Specifically, we evaluated prompt selection using CLS tokens from ViT-L’s intermediate layers (l = 4, 8, 12). While using intermediate layers reduces the need for a separate surrogate, it increases GPU time since even a partial ViT-L forward pass (to l=8, covering ~67% of layers) is more computationally expensive than running the full ViT-Ti (5M vs. 307M params). Accuracy also drops for shallower layers (e.g., l=4: -1.3%) due to reduced global context. These findings demonstrate the efficiency advantage of our surrogate model. We will include this analysis in the final version.

Dear reviewer mcMq,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

Best, AC