When Maximum Entropy Misleads Policy Optimization

Thank you for your positive feedback and questions. Indeed, we focused on understanding the principles behind potential limitations of MaxEnt (SAC being its best performing and most widely-used version), complementing most existing results on the benefits of entropy regularization. Although we phrase the effect as "misleading", we showed that it can be either beneficial or harmful in different environments (Section 6.2 discusses its benefits). Consequently, the formulation of the algorithm SAC-AdaEnt is intended more as an ablation study mechanism to empirically test the validity of our theoretical analysis in practice, rather than as a new candidate of MaxEnt algorithms. At the same time, we definitely agree with the reviewer that evaluating the algorithm on more baselines and hyperparameter changes can strengthen the empirical understanding and contribution of the paper. Here we respond to the questions directly, and will extend our paper along these lines accordingly.

Q: (More MaxEnt baselines)

How does the failure mode extend to other entropy-based RL methods? A more detailed comparison between SAC and other entropy-based RL methods (e.g., MPO, Soft Q-Learning) would provide a broader picture.

A:

We will follow your suggestion and expand the experiment section with more baseline MaxEnt algorithms. Here we provide results from additional experiments comparing with Soft Q-learning (SQL) and with adaptive entropy (SQL-AdaEnt). We observe that the overall performance of SQLs generally underperforms compared to SAC, and SQL is affected by the misleading effect in the Vehicle and Quadrotor environments as well, in which case SQL-AdaEnt also improves the performance.

In the table below, we summarize the mean and variance of the policy return in the environments of Vehicle, Quadrotor (as examples of negative misleading effect) and Hopper (positive misleading effect). We see that SAC generally performs better than SQL across all environments, and the effect of AdaEnt tends to be similar on SQL and SAC. That is, in Vehicle and Quadrotor, where SQL and SAC struggled, the AdaEnt component led to better performance. The improved performance of SQL-AdaEnt on Hopper may be due to SQL being a relatively weak baseline.

The results align with the predictions of our theoretical analysis, which showed that the misleading effect arises from the maximum entropy objective itself. As long as an algorithm aims to optimize policies to reduce divergence from the Boltzmann policy which is optimal under the MaxEnt objective, its performance can be negatively affected in situations that we analyzed in the paper.

Q: (More comparisons and ablation study)

Clarifying how adaptive entropy tuning compares to alternative exploration techniques (e.g., intrinsic motivation, Bayesian exploration) would add value. How sensitive is the failure mode to the choice of reward scaling and environment structure? Could benefit from conducting ablation studies on entropy-related hyperparameters, such as temperature scaling, to analyze their role in failure cases.

A:

These are very important suggestions that we will follow to strengthen the empirical analysis in the paper. In general, our experience is that policy optimization methods that encourage more exploration generally underperform in control environments where low-entropy policies are the key to success, compared to conservative and more greedy approaches such as PPO. On the other hand, similar to our discussion in Section 6.2, in many environments the "misleading" effect of MaxEnt and enhanced exploration strategies can be crucial to performance. Based on the theoretical analysis framework that we proposed in this paper, we believe it is promising to develop fine-grained analysis methods that take into account of both the specific structure of an environment to inform the hyperparameter choices and exploration strategies.

Q: (Related Work)

The paper does not cite Ahmed et al., "Understanding the Impact of Entropy on Policy Optimization" (ICML 2019)

A:

The paper was cited in the Related Work section on Line 90 as (Ahmed et al. 2019). We will add more discussion in the paper for this and similar analysis on the benefits of entropy regularization.

Thank you for your positive feedback and questions. We agree with the reviewer that comparisons with wider range of baselines will enhance the empirical analysis, and we will expand the paper accordingly. Our current formulation of SAC-AdaEnt is mainly intended for empirically testing the theoretical analysis, to show how the misleading effect practically affects policy optimization (in both negative and positive ways). As our theoretical analysis shows, the effect is rooted in the use of the MaxEnt objective, and thus we believe the principles are applicable to MaxEnt algorithms in general. We discuss additional experiments with Soft Q-Learning below as an example that reflects this.

Q: (PPO with entropy)

(Entropy in PPO) is treated as a secondary regularizer rather than a core objective. This distinction highlights why PPO may avoid some of the pitfalls of overemphasizing entropy.

A:

Thank you for pointing this out: it exactly aligns with our understanding. Our core construction of entropic bifurcation relies on the fact that the objective of policy optimization in MaxEnt algorithms is to match the Boltzmann distribution, which fundamentally changes the goal of the optimal policy and enables the misleading effect of entropy. In policy gradient methods using entropy as exploration strategies, the objective for policy optimization is unaffected, and thus entropy regularization does not lead to misleading effects as how we formulated in the paper.

Q: (SAC-auto-alpha v.s. SAC-AdaEnt)

In Fig.11, the SAC-auto-alpha refers to automated adjustment? Could you compare the SAC-AdaEnt to the SAC-automated-alpha-tuning?

A:

Yes, in Fig.11, SAC-auto-alpha refers to automated entropy adjustment. We show more results in the next question, and will update the plots in the paper along with AdaEnt applied to other MaxEnt baselines, as you suggested. Note that SAC-auto-alpha applies a uniform entropy adjustment across all states while SAC-AdaEnt adapts the entropy individually per state.

Q: (Baselines)

The choice of baselines is narrow; It would enhance the paper if extending the adaptive mechanism to other MaxEnt algorithms."

A:

We will follow the reviewer's suggestion and expand the experiment section with more baseline MaxEnt algorithms. Here we provide results from additional experiments comparing with Soft Q-learning (SQL) and with adaptive entropy (SQL-AdaEnt). We observe that the overall performance of SQLs generally underperforms compared to SAC, and SQL is affected by the misleading effect in the Vehicle and Quadrotor environments as well, in which case SQL-AdaEnt also improves the performance.

In the table below, we summarize the mean and variance of the policy return across Vehicle, Quadrotor (as examples of negative misleading effect) and Hopper (positive misleading effect). We see that SAC generally performs better than SQL across all environments, and the effect of AdaEnt is similar on SQL and SAC. That is, in Vehicle and Quadrotor, where SAC struggled, the AdaEnt component led to better performance. The improved performance of SQL-AdaEnt on Hopper may be due to SQL being a relatively weak baseline.

The results align with the predictions of our theoretical analysis, which showed that the misleading effect arises from the MaxEnt objective itself. As long as an algorithm aims to reduce divergence from the Boltzmann policy, the performance can be negatively affected as analyzed in the paper.

Q: (Scalability)

While the adaptive mechanism is promising, measuring discrepancies requires a “global” understanding at each state. In high-dimensional environments, ... computationally expensive.

A:

Yes, we formulated SAC-AdaEnt to test the theory in practice, acknowledging that it is not intended as a scalable replacement for SAC. As we shown in Section 6.2, the misleading effects of entropy may explain some of the key benefits of SAC, and thus we do not always need to eliminate the effect. We believe a promising direction in challenging environments is to use baseline SAC and PPO algorithms to understand the gap between Q-plain and Q-soft landscapes first, and then modulate the use of entropy on specific regions in the state space to minimize the need for global probing. Consequently, we focused mainly on analyzing the principles of how entropy affects policy optimization, in the hope that the understanding leads to more nuanced policy optimization strategies in challenging environments.

Thank you for your feedback and questions. We will reorganize the writing to make the core claims clear from the beginning of the paper. The toy example seems to have obscured the core results, which we further explain here.

Q: Large $\alpha$ value?

• In fact, the entire argument of the paper relies on a large enough value of $\alpha$. If $\alpha$ was not large, then there would not be any misleading effect at all.

• The analysis in Appendix A is meaningless if the authors assume because they assume a fixed and large value of alpha.

• SAC already solves by having a tunable entropy coefficient. The analysis with a fixed high value does not offer any useful insights.

A:

We do not assume large fixed $\alpha$ values in the core results.

We agree with the reviewer that under a large $\alpha$ value it seems obvious that the learning problem is changed. This is why in Section 3 we derive the general result: the misleading effect can be shown with arbitrarily small positive $\alpha$. All theorems in Section 5 treat $\alpha$ as a variable in the definitions and proofs to construct the bifurcation extension for misleading effects.

To see how: for an arbitrary $\alpha$, the optimal MaxEnt policy follows $\pi^*(a|s)=\exp(\alpha^{-1}Q(s,a))/Z(s))$ with normalizing factor $Z(s)=\int \exp(\alpha^{-1}Q(s,a))da$ (see [Haarnoja, 2018b] Eq (4)). The Q terms are thus canceled out in the state values:

$

V(s)&=\mathbb{E}_{a\sim\pi^}[Q(s,a)-\alpha \log(\pi^(\cdot|s))]\\ &=\mathbb{E}_a[\cancel{Q(s,a)}-\alpha\cdot\bigg(\cancel{\log(\exp}(\alpha^{-1}\cancel{Q(s,a)}))-\log Z\bigg)] \\ &=\mathbb{E}_a[\alpha \log Z]\\ &=\alpha \log(Z)

$

Consequently, the misleading effect preserves as long as the value ordering between good and bad states incorporates the scaling effect of $\alpha$. Moreover, with our definition of the bifurcation extension, if MaxEnt is misled for any small $\alpha$, then with any $\alpha'\geq \alpha$ the MaxEnt is misled as well. Thus the results apply to auto-tuning by choosing $\alpha$ to be smaller than the final weight after the decay schedule in SAC with autotuning.

Specifically, the toy example is designed to allow the use of any positive $\alpha$. For instance, say $\alpha=0.00001$, then we would multiply with $\alpha^{-1}$ in the exponents on Line 638 and 642 in Appendix A.1.1. Thus, to maintain the misleading effect we can scale all rewards by $\alpha$ to cancel out $\alpha^{-1}$, which preserves the same integrals and the ordering of $\log Z(s_g)<\log Z(s_b)$. Thus, in the example $Q(s_0,a\in A_1)=\gamma V(s_g)=\gamma\alpha \log(Z(s_g))<\gamma\alpha \log(Z(s_b))=Q(s_0,a\in A_2)$. Namely, the MaxEnt-optimal policy still prefers the wrong action range $a\in A_2$. Note that this scaling is why in theoretical analysis of MaxEnt the temperature $\alpha$ is typically omitted (original SAC paper [Haarnoja, 2018a], remark after Eq(1)).

All experiments in Section 6 are run with SAC with small $\alpha$ and autotuning (Line 923 Table 2).

We will update the paper to emphasize these points. Thank you for pointing out the concern.

Q: Complex Dynamics?

• When a more realistic dynamics model for the quadrotor is used... — the term more realistic is vague and hand-wavy.
• The selected benchmarks seem to be arbitrarily selected and complex dynamics is used without any justification of what is complex.

A:

The difference in the dynamics of quadcopters between easy and complex is described in Appendix C.2 and we will further add the full equations. We agree with the reviewer that it is hard to quantify complex dynamics, so we should refer to them as environments that require low-entropy control policies to succeed, which can be observed more directly by plotting the advantage landscapes as in Section 6.1.

The benchmarks are selected for visualizing concretely when entropy affects policy updates, as shown in Section 6. We also showed that the misleading effect can also benefit learning (Section 6.2). Our goal is to understand when MaxEnt helps and when it hinders policy optimization.

Q: No performance issues in SAC?

• The works cited in this paper do not give conclusive evidence of MaxEnt algorithms always struggling in important control problems.
• Tan & Karakose (2023) are cited to claim that SAC delivers suboptimal solutions in comparison to PPO in complex control problems, which is not shown in those papers.

A:

We do not aim to show that MaxEnt algorithm always struggles, but to understand when the entropy terms may hinder learning.

We definitely agree that SAC is widely observed to outperform policy gradient, which makes it important to understand in what cases they may underperform. The cited practical robotics papers are examples of such cases. In particular, Tan & Karakose (2023) shows in Figure 6 that SAC is outperformed by other methods. We can remove the reference if it is not a convincing example.

Thank you for your positive feedback and questions. We agree that more discussion and analysis can be done for SAC-AdaEnt, and we provide some more details below. Indeed, we formulated SAC-MaxEnt less as an algorithm that is intended to replace SAC, but more as an ablation mechanism that tests the validity of the theoretical analysis in practical learning environments. We believe the principled understanding of the misleading effect of MaxEnt algorithms (and how it can both positively and negatively affect policy learning) opens up the possibility of developing more customized policy optimization procedures for challenging environments that exploit their specific structures and value landscapes.

Q: (SAC-AdaEnt)

Will SAC-AdaEnt decrease performance in tasks where SAC generally succeeds?

A:

Following the question, we performed additional experiments on the environments in the paper where SAC generally succeeds: Hopper, Obstacle2D, Acrobot (also attached results for Vehicle and Quadrotor, in which SAC fails). The summary of performance comparison is shown in the table below, and we will add the plots and more discussion in the paper.

We observe that SAC-AdaEnt led to less mean return compared SAC as well as higher variance. Note that our design of SAC-AdaEnt maintains the entropy-based exploration component in SAC (only changing the policy optimization objective). Thus the overall performance still benefits from exploration, and the higher variance is the result of the advantage landscape being less dominated by entropy. The obstacle2D and acrobot environments are lower-dimensional, and thus exploration itself is likely sufficient for discovering successful trajectories, and thus the performance has not degraded in SAC-AdaEnt.

Overall, we believe the experiments show that in environments where SAC performs well, SAC-AdaEnt can still rely on entropy to support exploration, while the performance may benefit less from the tendency of the standard SAC in escaping action regions with raw high Q-values.

Q: (High-dimensional RL)

Were entropy-traps also expected or observed in high-dimensional image-based RL tasks?

A:

Entropy-traps are fundamentally dependent on the reward structure and the environment dynamics in the MDP, in terms of whether the success of learning relies on low-entropy policies. Thus the misleading effects can definitely be observed in high-dimensional RL problems. We believe the key to observing entropy traps may not always be about the dimensionality of the state space, but more crucially determined by the dimensionality of the action space -- when the action space is high-dimensional, it is more likely that the feasible policies form a low-entropy distribution over the space, thus making it easier to observe the misleading effect of MaxEnt. We believe such problems can arise frequently in high-dimensional RL problems in general, such as fine-tuning of language models, foundation Vision-Language-Action models, as well as diffusion policies.