The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

We appreciate your positive feedback on our empirical observation, the effectiveness of our approach for various LLMs, and your acknowledgment of both the theoretical analysis and comprehensive evaluations. We will address each of your comments and concerns below and also in our revised manuscript.

---

Q1: Have the authors attempted to observe whether similar phenomenon happens to larger-sized models?

RQ1: We sincerely appreciate your valuable feedback. In response to your suggestion, we extended our experiments to larger models, specifically Qwen2.5-14B, to further validate our findings. The distribution of energy loss in the final layer of the LLM across various datasets is presented in Figure R7. As shown, the substantial increase in energy loss consistently correlates with reward hacking across all datasets. This observation demonstrates the existence of the energy loss phenomenon in larger-scale LLMs and further solidifies the contribution of our work.

Q2: What would the model performance change when the chain of thoughts strategy is used？

RQ2: Thank you for this insightful suggestion. To investigate the impact of the chain of thought (CoT) strategy, we construct a CoT RL dataset, where we append the phrase "Let’s think step by step." to each prompt. The comparison between PPO with CoT and PPO without CoT is shown in Figure R8. As demonstrated, incorporating CoT leads to a significant improvement in RLHF performance.

We hypothesize that this enhancement is due to the fact that the CoT strategy effectively constrains the output space, guiding the model to generate responses that better align with the user’s input.

Q3: Failure case of the proposed approach.

RQ3: Thank you for this comment. Based on our theoretical analysis, the proposed RL regularization term can also be interpreted as a contextual relevance constraint. This means that our method may not perform as well in cases where the model's responses are required to be highly divergent, but with less emphasis on contextual relevance. For instance, as shown in the case presented in Figure R9. In the revised version of our paper, we will explore and summarize additional failure cases to provide a more comprehensive and objective analysis.

Q4: It would be helpful to move one of the qualitative examples from the appendix to the main paper to better illustrate how the proposed approach compares to the baselines.

RQ4: Thank you for your suggestion. In the revised version, we will move one of the qualitative examples from the appendix to the main paper to better illustrate the advantage of our proposed approach.

Thank you for your comments. We would like to highlight that our main contribution is the empirical observation of the energy loss phenomenon and the corresponding RL regularization design, as acknowledged by all other reviewers (wKXn, mgdS, and mmxN). Theoretical analysis is included as an exploratory explanation, with its limitations explicitly discussed in the manuscript (Lines 198-206, Page 4). We acknowledge that the current analysis only applies under certain conditions, and a rigorous explanation is beyond the scope of this paper but remains a promising direction for future work.

---

Q1: Why does the regularization term penalize energy loss variation relative to the SFT model, rather than the loss itself?

RQ1: Our design treats the SFT model's energy loss as a state-dependent baseline—a common RL technique—to reduce variance from the regularization term. By penalizing deviations from this baseline, our approach effectively suppressing excessive increases, aligning well with our theoretical analysis. The effectiveness of this design is validated empirically in Figure R10.

Q2: Practical Relevance of Corollary 4.

RQ2: We would like to clarify that our main contribution lies in the empirical findings and the corresponding RL regularization. Corollary 4 serves as an exploratory explanation for our findings from the perspective of contextual mutual information. While it may not generalize to all settings, it offers a plausible interpretation under specific conditions.

Corollary 4 shows that if the energy loss increase stays below a threshold, the upper bound of contextual mutual information is reduced, suppressing response-context relevance, offering a theoretical explanation for the emergence of reward hacking in such scenarios. However, when increased energy loss exceeds the threshold, the upper bound may rise, but this does not necessarily improve contextual relevance. Why excessively increased energy loss continues to be associated with reward hacking in such scenarios remains an open question.

Thus, Corollary 4 offers a theoretical insight into reward hacking as energy loss increases, though it is limited to a specific regime. A more rigorous theoretical explanation is beyond the scope of this paper but is a promising area for future research.

Q3: Dynamic $\alpha$ in corollary 4 during fine-tuning.

RQ3: We would like to clarify that Corollary 4 is not a strict theoretical guarantee for the proposed RL regularization during fine-tuning, but rather a lens for interpreting empirical observations. While it is true that $\alpha$ may shift as training progresses, the underlying relationship it describes still holds.

Q4: Questions about the upper bound of mutual information derived in Theorem 3.

RQ4: We would like to clarify that the goal of Theorem 3 is to provide a theoretical perspective that helps explain our empirical observations, rather than deriving the tightest possible upper bound on contextual mutual information.

While your suggested formulations, $||H^{\text{out}}||_1 - \mathbb{E}[||H^{\text{out}}||_1]$ and $H(q)$, may indeed yield a tighter upper bound, they are significantly more difficult to estimate and optimize, particularly during dynamic fine-tuning. In contrast, our use of $||H^{\text{out}}||_1 - ||H^{\text{in}}||_1$ provides a more tractable and empirically estimable surrogate that still captures the core trend central to our analysis.

Q5: More rigorous statement in Theorem 3’s condition and Corollary 5’s proof. & Suggestion for using ”activation decay” & Definition of $\sigma$.

RQ5: Thank you for your valuable suggestion. We will revise the corresponding statements and definitions for greater clarity in the revised version.

Q6: Question about Figure 7.

RQ6: We would like to clarify that Figure 7 is not intended to validate the theoretical analysis. Rather, it is inspired by the theoretical insights and aims to empirically investigate further, from the perspective of contextual relevance, why excessive increases in energy loss are accompanied by reward hacking.

Q7: Comparison with directly penalizing the L1 norm of the output.

RQ7: Thank you for your comments. Following your suggestion, we conducted additional experiments on Llama3-8b to compare our approach with directly penalizing the output L1 norm. The results, shown in Figure R11, demonstrate the advantage of our approach.

We hypothesize that by penalizing the L1 norm difference between the input and output of the final layer, our approach constrains only the final layer, thereby preserving optimization flexibility. In contrast, directly penalizing the output norm inevitably constrains the preceding layers, limiting the model’s optimization capacity.

We appreciate your positive feedback on the clarity, novelty, theoretical insights, and compelling experimental results of our paper. We will address each of your comments and concerns below and also in our revised manuscript.

---

Q1: Why does the comparison method PPO with length penalty still suffer from reward hacking？

RQ1: Thank you for your valuable feedback. As discussed in Lines 367-369 on Page 7 of the manuscript, PPO with length penalty can only capture hacking patterns characterized by longer response lengths, such as excessive redundancy. However, it is not effective in addressing hacking patterns that are unrelated to response length, such as excessive caution. Therefore, PPO with length penalty still suffers from reward hacking to some extent. The related case study can be found in Appendix L of the manuscript.

Q2: What specific advantages does the proposed EPPO algorithm have over the ensemble-based methods?

RQ2: Thank you for your insightful question. As discussed in Lines 292-296 on Page 6 of the manuscript, although the ensemble-based RM method can significantly enhance the robustness of reward modeling, it may still be susceptible to spurious features in reward modeling and distribution shifts during the RL stage, potentially leading to reward hacking. In contrast, our proposed EPPO algorithm directly focuses on the neuron behavior within the LLM that is related to reward hacking, enabling more effective mitigation. Furthermore, our EPPO is more efficient compared to ensemble-based RM methods, as the latter often requires loading multiple reward models during the RL process, which significantly increases resource demands, especially for large-scale LLMs.

Q3: Computational overhead analysis.

RQ3: Thank you for your valuable suggestion. Following your advice, we report the training times for both PPO and EPPO on two tasks, using a single node equipped with 8 A100-SXM80GB GPUs in the table below. As shown, while our EPPO does incur a slight increase in training time, it significantly improves training performance, as demonstrated in our paper.

Q4: Intuitive reason why the L1 norm is used in this work instead of the L2 norm.

RQ4: Thank you for your insightful question. The primary consideration for using the L1 norm instead of the L2 norm is that the L1 norm is more robust to outliers. This robustness to outliers allows the L1 norm to more reliably identify relevant features while minimizing the impact of noise, making it a more appropriate choice for analyzing the hidden state representations in LLMs.

Thank you for recognizing our research perspective. We will address your comments below.

---

Q1: Does the energy loss phenomenon occur in other layers?

RQ1: Thanks for your thoughtful comment. We agree that, in theory, excessive energy loss at any layer could reduce contextual relevance and contribute to reward hacking. However, extensive experiments show that only the final layer consistently exhibits a strong trend across different LLMs and datasets. Evidence is provided in Figures R1, R2, and R3.

We speculate that the final layer, being closest to the output, encapsulates richer semantic information, making it a reliable indicator of overfitting or reward hacking. While earlier layers may show similar trends, their behaviors are influenced by model architecture, training strategies, and data, making patterns less stable and harder to generalize.

Q2: How to reconcile the theoretical framework with the experimental findings?

RQ2: Great question! Our theoretical explanation holds when energy loss stays below the threshold, where increased loss compresses contextual mutual information, providing a theoretical rationale for the energy loss phenomenon. Conversely, while increased energy loss exceeding the threshold may raise the upper bound, it does not necessarily correlate with enhanced contextual relevance. Why increased energy loss is linked to reward hacking in such scenarios remains an open question.

We would like to clarify that our main contribution is the empirical findings of the energy loss phenomenon and the corresponding RL regularization design, while the theoretical analysis is an exploratory attempt to explain the empirical findings. In the revised version, we will tone down the theoretical component and highlight the empirical contributions to better reflect the core value of this work.

Q3: The root cause of reward hacking and the role of EPPO in addressing it.

RQ3: We agree that reward hacking stems from flaws in reward design. However, in practice—especially in RLHF—it is challenging to obtain a perfect reward model due to overfitting, misspecification, and misgeneralization. In such cases, RL regularization techniques are necessary to guide the model's optimization toward human-desired behavior, even with imperfect reward models. EPPO is designed to address the limitations of imperfect reward models by incorporating RL regularization.

Q4: What would happen to the energy loss using a well-designed reward function?

RQ4: As discussed in RQ3, obtaining a well-designed (perfect) reward model in RLHF is challenging. To simulate this, we use a stronger reward model based on LLaMA3-70B as the well-designed function. The corresponding energy loss and hacking sample distributions after RLHF are shown in Figure R4. We observe that with a well-designed reward model, final-layer energy loss increases only moderately—unlike the sharp rise seen with an ill-structured reward model.

Q5: The fundamental reason behind energy loss growth in PPO training when an ill-structured reward function is used.

RQ5: Thanks for your comments. The growth in energy loss is primarily an empirical observation, consistently found across various LLMs, datasets, and tasks. To explore its causes, we first provide a theoretical proof that, in certain scenarios, increased energy loss tends to suppress contextual relevance (i.e., Theorem 3)—a key aspect of reward hacking. We then empirically demonstrate that this effect is widely observed across all scenarios (i.e., Figure 7 in the manuscripts).

We suspect that the fundamental reason behind energy loss growth lies in overfitting to reward model–favored patterns—a phenomenon we associate with reward hacking. This overfitting prioritizes alignment with the reward model’s biases over accurately capturing user intent, leading to deterioration in contextual relevance and a significant reduction in neural activation, manifested as excessive energy loss.

Q6: Comparison with early-stopped PPO.

RQ6: Following your suggestion, we compared EPPO with early-stopped PPO based on GPT-4 evaluations. As shown in Figure R5, EPPO consistently outperforms early-stopped PPO, demonstrating its stronger resistance to reward hacking.

Q7: Discussion about reward shaping approach in RL research.

RQ7: Thanks for your comment. As far as we know, reward shaping in RL is typically used to address the sparse-reward challenge by providing more informative reward signals, which differs from our focus on mitigating reward hacking in RLHF scenarios.

Notably, EPPO can be seen as a form of reward shaping, as it uses the internal neural behaviors of the LLM as auxiliary signals to adjust the reward during RL.