Rethinking cross entropy for continual fine-tuning: policy gradient with entropy annealing

We thank the reviewer for taking the time to read and critically assess our paper.

We are encouraged by the reviewer’s positive feedback, particularly the comment that our work is “new and has the potential to influence future research.” We also appreciate the recognition that our paper effectively combines theoretical justification with empirical evidence. Below, we address the reviewer’s specific concerns:

1.Comparison with Related Continual Learning Works

The continual learning approaches mentioned by the reviewer—such as parameter regularization, knowledge distillation, and meta-learning—were originally developed for training-from-scratch settings, typically using full-model training with a global cross-entropy (CE) loss. These approaches are less applicable in the context of continual parameter-efficient fine-tuning (PEFT) with local cross-entropy loss.

Recent empirical studies (e.g., DualPrompt [1], L2P [2]) have shown that these traditional methods often underperform when compared to modern PEFT-specific methods such as L2P and DualPrompt. For this reason, our comparisons focus on state-of-the-art continual PEFT techniques, including DualPrompt, LAE, LoRA, and InferLoRA, which are more appropriate and competitive in the continual fine-tuning paradigm we target.

[1] "Dualprompt: Complementary prompting for rehearsal-free continual learning", ECCV 2022.

[2] "Learning to prompt for continual learning" (L2P) CVPR 2022

2. Benchmarks, and Scope of Application

Benchmarks: We carefully reviewed our benchmark setup in relation to prior works, including DualPrompt and LAE. Our evaluation spans a broader and more challenging set of benchmarks. While LAE and DualPrompt are evaluated on two datasets (CIFAR-100 and ImageNet-R), our study includes four more demanding datasets: ImageNet-R, Food101, CUB-200, and CLRS. These additions significantly increase both the data scale (e.g., Food101) and domain diversity (e.g., CLRS, which involves remote sensing data).

Robustness. We further evaluated the methods under label noise conditions, demonstrating that the proposed approach remains robust and continues to outperform CE-based training under noisy supervision.

Additional Result Table 1: Evaluation in noisy label setting

Scope (Class-incremental learning vs. Continual Detection/Segmentation). Our method is specifically tailored for class-incremental learning with a Markov Decision Process (MDP) formulation designed for classification tasks. In contrast, detection and segmentation require multi-step reward structures and pose fundamentally different challenges (e.g., background shift), which are typically studied as separate research problems. Our method is not directly relevant to those domains.

3. Orthogonality of Loss Investigation to Architecture Choice

Our work builds upon recent advances in continual PEFT, where CE loss is commonly adopted across various architectures such as LoRA, prompt tuning, and adapters.

Our investigation is orthogonal to architectural design choices. It focuses solely on the loss function and its effect on continual learning. For instance, we use LoRA with a rank of 4 in our main experiments and additionally tested with a rank of 8. The performance trends remain consistent, indicating that our findings generalize across LoRA configurations. Table 2 in the paper also shows that the performance gains are applicable across other PEFT architectures (prefix and adapters).

4. Clarification on Reported Performance Improvements

In Table 2, the observed performance improvements of our method range from 1.5% to 3.7%, rather than the 0.5% mentioned in the reviewer’s comment. These gains are statistically significant (p < 0.001, paired t-test) and are also supported visually in Figure 2, which presents 95% confidence intervals.

Once again, we thank the reviewer for their thoughtful feedback. We believe the clarifications and additional results we’ve provided strengthen the case for the relevance, robustness, and impact of our work.

C. On the Scalability of the Proposed Method

Reviewer cwyn previously noted that “only ViT-B is used for experiments, and there is no evidence on CNNs or larger ViT models.” Reviewer L5mY shares this concern. Despite access to high-end hardware (such as A6000 and A100 GPUs), the authors did not provide additional experiments involving larger architectures.

From this reviewer’s perspective, while it may not be strictly necessary to evaluate the method on object detection tasks (which typically rely on CE loss), an extension to semantic segmentation would have been valuable. This is because semantic segmentation can be framed as pixel-level classification, which is closely aligned with the authors’ current setup. Demonstrating performance in this setting could significantly enhance the paper’s impact and generalizability.

That said, as shown in the table below, this reviewer acknowledges that many works in this field have similarly limited benchmark and architecture coverage. However, in order to increase the influence of this work within the broader community, such scalability evaluations are viewed as an important next step.

In fact, recent papers in self-supervised learning (such as, MoCo, SimCLR, SimSiam), image classification, and model architecture research (including CNNs, ViTs, Transformers, Mamba, and sliding window attention) often include comprehensive and extensive benchmark evaluations as a standard practice.

---

Reviewer L5mY thanks the authors again for their thoughtful rebuttal. I would kindly encourage the authors to provide a concrete plan addressing points A, B, and C discussed above. It is understandable that the limited timeline may have made it difficult to conduct additional experiments related to point C, so this reviewer assures that the absence of such experiments will not be viewed negatively in the review score. Provided that a clear plan is presented, I remain open to further discussion with the reviewers and Area Chairs, and to revisiting my score.

Dear authors,

Thank you for providing a concrete revision plan and for including additional experimental results with ViT-L. I appreciate the author's the thoughtful direction in which you are planning to improve the paper. I am pleased to raise my score by one point in recognition of their efforts and the potential of thier method. That said, I still believe the initial manuscript requires notable restructuring and clarity improvements. This remains an important factor in my overall assessment. At present, I am continuing discussions with other reviewers and am sincerely taking all their feedback into account with an open mind. I commit to actively participating in the discussion and carefully considering all perspectives before finalizing my score.

Best regards, Reviewer L5mY

We thank the reviewer for taking the time to read and critically assess our paper.  

We are encouraged that the reviewer found the core idea of formulating 0-1 classification as a reinforcement learning problem interesting. We address the questions point by point as follows. 

1.The motivation of EPG/aEPG for continual learning

Thank you for raising the question regarding the motivation behind EPG. In retrospect, we agree that this aspect could have been communicated more clearly, and we will revise the paper accordingly to make this more explicit.

Our central motivation is that reinforcement learning enables direct optimization of the true classification objective: maximizing accuracy (i.e., minimizing the 0-1 loss). While the RL method (i.e. EPG) can directly minimize 0-1 loss, our analysis reveals that cross-entropy (CE) loss introduces an exploration bias, which can exacerbate forgetting.

aEPG mitigates this issue by leveraging CE’s exploratory behavior early in training, and gradually annealing toward 0–1 loss optimization as learning progresses. This enables aEPG to maintain CE’s adaptability (plasticity) while gaining the stability benefits of directly optimizing for accuracy, resulting in a more favorable stability-plasticity trade-off.

Concretely, in parameter space, CE’s exploration bias is evident in its gradient expression: $g_{ce}=-1/p_\theta(y|x)g_{epg}$. This means CE focuses on uncertain or hard examples, which yield high information gain (or "surprisal"), as the model must correct large prediction errors. While this drives adaptation to a new task, it can also disrupt existing knowledge, increasing forgetting. In contrast, EPG/aEPG focuses on easier samples that already align well with the model’s predictions, resulting in lower surprisal and less disruptive updates. In action space, CE’s exploration bias manifests as higher output entropy than 0-1 loss optimization.

We hope this clarifies the motivation behind aEPG and its observed benefits. We are happy to provide further elaboration if helpful.

2. Potential Applications in Semi-/Unsupervised Settings

Our current MDP formulation is specifically designed for supervised learning, where class labels are used to define the reward. Extending this framework to semi-supervised or unsupervised learning would require a new MDP formulation—a direction we agree is both interesting and promising for future work.

One possible extension would be to design RL tasks based on self-supervised objectives. For example, SimCLR can be reframed as a classification task: given an anchor sample and a batch of candidates, the model must select the correct augmented version of the anchor. A reward function can then assign 1 for correct selection and 0 otherwise, enabling a policy gradient formulation for contrastive learning.

3. Choice of hyperparameter $\tau$

The hyperparameter $\tau$ controls the annealing schedule for transitioning from CE loss ($\alpha=1$) to EPG loss ($\alpha=0$). We use a sigmoid-based schedule and $\tau$ determines the range of the transition: when $\tau=6$, we have $\alpha_{start}=sigmoid(6) =0.998$ and $\alpha_{end}=sigmoid(−6) = 0.002$. This provides a smooth and effective transition (approximately from 1 to 0) over the training period.

In response to the reviewer's question, we tested $\tau$ values of 4, 6, and 8 and found that they achieve similar performances.

Additional Result Table 1: The effect of $\tau$

Please note that the paper also contains an ablation study on different annealing strategies (sigmoid, cosine, linear) and found that performance remained robust across these schedules (see Section 4.2 and Figure 3).

4. Comparison with recent RL methods

Recent RL-based approaches to classification typically rely on sampling-based policy gradient methods (e.g., REINFORCE, PPO), which are more suited to multi-step MDPs with unknown reward structures. Our formulation, by contrast, is a one-step MDP with a fully known reward model.

This allows EPG to compute a closed-form policy gradient, avoiding the high variance typically associated with sampling-based approaches. As a result, we expect—and observe—better performance from EPG compared to methods like REINFORCE. To support this point, we added comparisons with REINFORCE. As shown below, EPG significantly outperforms REINFORCE (p<0.001).

Additional Result Table 2: Comparison with sampling-based policy gradient$

Once again, we thank the reviewer for their time and effort, which will help us improve the clarity and impact of the paper.

We thank the reviewer for taking the time to read and critically evaluate our work. We are encouraged by the reviewer’s appreciation of our clear motivation and recognition that the paper challenges the commonly held belief that “high-entropy helps.” We also value the acknowledgement that our work invites the community to reconsider loss function choices in the context of continual fine-tuning. Below, we respond to the reviewer’s concerns regarding novelty and experimental scope:

1. Novelty and Comparison with [1]

We appreciate the reviewer pointing us to [1] and commit to including a dedicated paragraph in the revised version to explicitly discuss its relationship to our work and add the experiment comparison result.

While [1] applies sampling-based policy gradient methods to detection and segmentation tasks, our work focuses on classification and proposes a tailored one-step MDP formulation with a fully observable reward model. This formulation allows us to derive a low-variance policy gradient estimator, eliminating the need for sampling actions and resulting in a more stable and efficient optimization process.

To highlight this distinction, we include an empirical comparison between REINFORCE (following Algorithm 2 in [1]) and our Expected Policy Gradient (EPG). As expected, EPG consistently achieves better performance across different benchmarks.

Additional Result Table 1: Comparison with sampling-based policy gradient

Conceptually, our proposed MDP formulation and EPG unify reinforcement learning (RL) and maximum likelihood estimation (MLE) within a single optimization framework, revealing a deeper theoretical connection between the two, as shown in Eq 10. In contrast, [1] uses separate optimization strategies for RL and MLE (as seen in their Algorithms 1 and 2).

Other differences between our work and [1] include different problem settings (continual learning vs. fine-tuning) and different tasks (classification vs. detection/segmentation).

2. Robustness to overfitting risk and label noise

Contrary to the reviewer's concern about increased overfitting, our findings indicate that EPG/aEPG is actually less prone to overfitting than cross-entropy. This is because EPG prioritizes easier samples during gradient updates, promoting the learning of general patterns in the data. In contrast, CE emphasizes hard or uncertain samples, which makes it more susceptible to overfitting to samples that contain noise.

To support this claim, we conducted additional experiments under label noise conditions (20% symmetric label noise). Our findings show that aEPG significantly outperforms CE in these settings, indicating greater robustness.

Additional Result Table 2: Performance under noisy label setting

 All reported results are averaged over three runs, and the results are statistically significant (paired t-test, p < 0.0001).

Theoretically, reinforcement learning is inherently designed for stochastic reward signals, making it naturally tolerant to noise. It can also be rigorously proved that the proposed EPG objective is noise-tolerant under symmetric or uniform noise, if the noise rate $n<1-1/K$, where K is the class number, i.e., that the global minimizer of EPG under label noise remains the same as the minimizer under clean labels.

3. Calibration performance

We agree that since aEPG produces lower output entropy compared to CE, aEPG can make the model more overconfident. However, this does not necessarily imply poorer calibration. As shown in "Do Not Be Afraid of Overconfidence"[1], overconfidence may not hurt final calibration performance if post-hoc calibration is allowed. And in some cases more overconfident training paradigm (e.g., "inverse focal loss") can even lead to more calibratable models after the post-doc calibration stage.

Following the insight in [1], we applied post-doc calibration (temperature scaling on a separate validation set) to both CE and aEPG. After calibration, aEPG achieves comparable or even better Expected Calibration Error (ECE) results than CE.

Additional Result Table 3: Calibration performance

[1] "Rethinking Calibration of Deep Neural Networks: Do Not Be Afraid of Overconfidence" NeurIPS 2021

4. Performance advantage persists for longer streams

To address the reviewer's concern about the performance over long CL tasks, we conducted additional experiments by splitting the original benchmarks into longer task sequences. The results confirm that aEPG's performance advantage significantly persists in extended continual learning settings (paired t test: p<0.001).

Additional Result Table 4: Evaluation with longer CL tasks

5. Entropy control: aEPG vs. Entropy Penalty

We appreciate the reviewer’s attention to the role of entropy. Indeed, a key message of this work is to highlight the impact of entropy reduction in continual fine-tuning. To our knowledge, this aspect has not been systematically explored before in continual fine-tuning. Our findings challenge the prevailing belief that high-entropy predictions are beneficial: Table 2 demonstrates that increasing entropy tends to hurt performance, while reducing entropy leads to better CL outcomes, opening a promising direction for future research on entropy-control strategies.

In terms of entropy control mechanisms, rather than applying explicit entropy penalties, aEPG provides a novel mechanism to reduce entropy based on sample weighting. We demonstrate that emphasis on harder examples (via importance sampling) in the gradient leads to effective entropy reduction. Empirically, aEPG outperforms standard entropy-penalty baselines in three out of four benchmarks, except for CUB-200. We hypothesize that this may be due to the low intra-class variance common in fine-grained classification tasks, where the entropy reduction achieved by importance sampling is less pronounced.

Once again, we thank the reviewer for their time and effort, which will help us strengthen the paper.

Thank you for your outstanding rebuttal :). It really addresses most of my concerns. I believe that including these discussions in the revision could immensely improve the manuscripts, and I will increase my final score accordingly.

For follow-up, I still believe that "CE+entropy penalties" (termed EP) and the proposed EPG conceptually are very close. EPG can be boiled down to "reverse-KL + entropy penalties", where "CE+entropy penalties" is equivalent to "forward-KL + entropy penalties". In Table 2, EP outperforms EPG across all benchmarks and is likely to further benefit from adaptive entropy annealing due to its similarity with EPG, which may outperform aEPG.

Personally, the paper is clearly motivated and technically sound, but positioning EPG/aEPG as a superior alternative may be a little overstated, since the main empirical lesson is the explicit entropy control. I would advice reframing the takeaway accordingly (e.g., “entropy control matters for continual fine-tuning; aEPG is one effective way to achieve it”) and tempering claims about RL-specific unless with more results that demonstrate the distinct benefit of aEPG over EP(+annealing).

We thank the reviewer for taking the time to read and critically assess our paper.  

We are encouraged that the reviewer found the proposed EPG method very sensible. We address the questions regarding the motivation and experiments as follows. 

1. The motivation of EPG/aEPG for continual learning

Thanks for your question about the motivation of EPG. On reflection, we could give more context about the motivation behind EPG. We can commit to adding more discussion on this.

Recent works in continual learning with pretrained models (e.g., L2P, DualPrompt, LAE, etc) all use local cross entropy (CE) as the training loss. Our analysis reveals that local CE exhibits an inherent exploration bias, which can exacerbate forgetting. 

Concretely, in the parameter space, CE’s exploration bias manifests as an emphasis on uncertain/hard samples in its gradient, i.e., $g_{ce}=-1/p_\theta(y|x)g_{epg}.$ Learning hard/uncertain samples leads to high information gain (or "surprisal"), as the model must correct large prediction errors. While this drives adaptation to a new task, it can also disrupt existing knowledge in order to accommodate the high information gain. In contrast, EPG/aEPG focuses on easier samples that already align well with the model’s predictions, resulting in lower surprisal and less disruptive updates. In action space, CE’s exploration bias manifests as higher output entropy than 0-1 loss optimization.

   aEPG addresses this issue by retaining CE’s exploratory behavior during the early phase of learning and gradually reducing exploration bias in the later training stage. This allows the model to benefit from CE’s plasticity while transitioning toward the stability of 0-1 loss optimization, thereby improving the overall stability-plasticity trade-off.

We hope this clarifies the motivation behind aEPG and its observed benefits. We are happy to provide further elaboration if helpful.

2. Comparison with Related Continual Learning Works

 Replay-based (e.g., ER, DER++) and regularization-based (e.g., LwF, EWC) methods were originally designed for continual learning in training-from-scratch scenarios, typically using global CE loss. These methods are less prevalent in the context of continual parameter-efficient fine-tuning (PEFT) with pretrained models. Empirical studies in early continual finetuning works (e.g., DualPrompt [1], L2P [2]) have established that these methods (e.g., LwF, EWC) often underperform compared to continual PEFT-specific approaches like L2P and DualPrompt.

Therefore, our comparisons focus on state-of-the-art continual PEFT techniques, including DualPrompt, LAE, LoRA, and InferLoRA.

[1] "Dualprompt: Complementary prompting for rehearsal-free continual learning", ECCV 2022.

[2] "Learning to prompt for continual learning" (L2P) CVPR 2022

3. Orthogonality of Loss Investigation​​

Our study (Table 2 in the paper) examines loss design independently of the architecture design of different continual PEFT techniques. Since methods like DualPrompt (with prompt-tuning module) and LAE (with old and new PEFT modules) all use CE loss, replacing their loss with aEPG is straightforward. Table 1 in the paper demonstrates that LAE (momentum-based update) combined with aEPG achieves significant gains for ImageNet-R. This finding is consistent with other benchmarks (see below).

Additional Result Table 1: Comparison with LAE on the other three benchmarks

 All reported results are averaged over three runs, and the improvements are statistically significant (paired t-test, p < 0.0001).

4. Clarifications​​ on the benchmarks

We carefully reviewed our benchmark setup and compared it to related works such as LAE. Contrary to the reviewer’s concern, our work does not use smaller benchmarks. In fact, we evaluate on a broader and more challenging set.

While LAE and DualPrompt evaluate on two datasets (CIFAR-100 and ImageNet-R), our study includes four more demanding benchmarks: ImageNet-R, Food101, CUB-200, and CLRS. Food101 introduces a significantly larger dataset size, while CLRS involves remote sensing data, further increasing diversity and difficulty.

If we have misunderstood the reviewer’s concern about the benchmarks, we would appreciate further clarification.

Once again, we thank the reviewer for their time and effort, which will help us improve the paper.