State Entropy Regularization for Robust Reinforcement Learning

We thank the reviewer for the thoughtful and encouraging review. We appreciate the time and care dedicated to evaluating our work, and will make sure to incorporate all suggestions and clarifications into the last version.

Weaknesses

1. Quality: Clarifying the Scope of Theorem 4.1

Thank you for raising this point, we understand how the scope of Theorem 4.1 may have been unclear. To clarify, we say that a kernel-uncertainty problem defined by $\tilde{\mathcal{P}}(\mathrm{MDP}, \pi)$ is equivalent to a regularized objective defined by $\Omega(\mathrm{MDP}, \pi)$ if for every MDP and policy $\pi$ it holds that $\min\_{\tilde{P} \in \tilde{\mathcal{P}}}\mathbb{E}\_{\rho\_\tilde{P}^\pi}[r] = \mathbb{E}\_{\rho\_P^\pi}[r] + \Omega$. This notion of equivalence is the same as the one used earlier in the paper when showing the relationship between reward robustness and entropy regularization, and aligns with definitions in prior robustness literature (e.g., [1]). Here, we focus on problems where both the uncertainty set $\tilde{\mathcal{P}}$ and the regularization $\Omega$ depend only on the nominal kernel $P$ and the policy $\pi$. In this setting, exhibiting a single MDP where the equality fails for all $\tilde{\mathcal{P}}$ and $\Omega$ is sufficient to disprove universal equivalence.

The intended takeaway of Theorem 4.1 is not that equivalence fails for all MDPs, but rather that for every uncertainty which is a function only of $P$ and $\pi$, there does not exist a regularizer—also depending only on $P$ and $\pi$—that can represent the robust return for all MDPs. We will rephrase the theorem in the revision to better reflect this intended level of generality.

2. Quality: Minor Comments

Thank you for pointing this out—we will correct the spelling to L’Hôpital’s rule.

Regarding the CVaR result, we included it as a theoretical complement to highlight an important limitation of state entropy in risk-averse settings. We agree that it may feel out of the main paper's thread and will reduce its emphasis in the revision to keep the narrative more focused (e.g., remove the preliminary mention in the main text).

3. Significance: Theoretical Contributions

While we do provide novel individual results, we agree that a primary contribution lies in the collective insight of our analysis.

For robust RL, we show state entropy regularization induces robustness to spatially correlated perturbations—a class of perturbations that is highly relevant in settings like transfer learning, yet remains sparsely addressed in the robust RL literature.

Additionally, maximum state entropy is well-studied for exploration and coverage, but never explicitly intended for robust RL. Here we highlight the robustness benefits and practical considerations associated with this regularizer, which we believe sheds new light on this important line of work. Our analysis clarifies the type of robustness it implicitly provides and emphasizes key practical aspects—namely, its sensitivity to rollout budgets for reliably realizing these robustness benefits.

References:

[1] Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022.

I thank the authors for their time and effort in providing a rebuttal response to my review.

I appreciate their clarification regarding Theorem 4.1 and now understand the nature of their claim. I would concur that some amount of rewording in the claim and proof would go along way to clarifying the result for other readers.

I also thank the authors for their candid framing around the significance of their contributions, and was glad to hear that we are in alignment regarding the holistic nature of the contributions.

It seems there is one other reviewer who scored the paper negatively. I will inspect their review and your rebuttal response to them independently. For now, my concerns have been addressed and I see no reason to adjust my score.

Thank you very much for the careful and constructive review. We will make sure to reflect the reviewer's suggestions and clarifications in the revision.

Questions

1. Guidelines for Choosing Between PER, SER, and SEAR

We thank the reviewer for this interesting and practical question.

These regularizers have additional benefits beyond robustness (e.g., stability and exploration), but these are outside the scope of this paper. From a robustness perspective, the choice of regularization should mainly depend on prior belief about the deployment environment: policy entropy (PER) suits better to small, uniform perturbation, as it spreads stochasticity along a dominant trajectory, whereas for larger, spatially correlated disruptions, state entropy (SER) promotes broader coverage across multiple well-performing trajectories.

In the absence of clear prior knowledge, our results suggest that, when properly tuned, SER does not harm nominal performance and does not diminish the robustness benefits of PER, so it can be incorporated.

2. Scalability of SER in High-Dimensional State-Space

Thank you for raising this important remark. As discussed in the paper e.g., when using the k-NN estimate, state entropy estimation indeed becomes more statistically demanding as the dimensionality of the state space increases. In practice, this can be mitigated by computing entropy over a subspace of relevant state features selected based on prior knowledge of the test environment, or by estimating entropy over latent state representations, as demonstrated in prior work [1,2,3,4,5]. These approaches help scaling SER to high-dimensional problems.

Weaknesses

1. Clarifying the Notion of Local vs. Global Perturbations

We appreciate the reviewer’s comment and recognize that the distinction between “local” and “global” could benefit from more consistent wording, particularly when using these terms to describe types of perturbations and behaviors. We will keep this in mind and use clearer, more precise language in the revision.

A more accurate way to phrase our results is that SER improves robustness to spatially correlated perturbations—or equivalently, against adversaries that exploit global properties of the agent’s behavior across the state space. In contrast, PER primarily improves robustness against perturbations that are not strongly state-specific or spatially structured.

For example, in MiniGrid, the wall-type perturbation places obstacles in highly correlated locations, creating a major local blockage. In contrast, in the spread-out perturbation, each cell is obstructed independently with a small uniform probability, leading to more scattered, uncorrelated disruptions.

Each regularizer induces a behavior that aligns with the type of robustness it provides: to handle spatially correlated perturbations, an SER-agent learns to take more diverse trajectories, exhibiting more “global” behavior, as expected when maximizing state entropy. Conversely, a PER agent spreads randomness only along a single dominant trajectory, exhibiting more “local” behavior that yields robustness primarily against smaller, more uniform perturbations.

2. Improving Clarity and Detail in Appendix Derivations

We thank the reviewer for the helpful suggestion. We will expand the appendix derivations by adding the missing intermediate steps.

Specifically, for the proof in Appendix A.1 - equation 13 implies that $\min_{\tilde r \in \tilde R}\mathbb{E}\_{\rho^\pi}[\tilde r] = \mathbb{E}\_{\rho^\pi}[r]-\epsilon + \min_{\Delta r} (-\langle d^\pi,\Delta r^\pi \rangle +\text{LSE}_{\mathcal{S}}(\Delta r^\pi))$. The fact that negative entropy is the convex conjugate of LSE implies that $\min\_{\Delta r} (-\langle d^\pi,\Delta r^\pi \rangle +\text{LSE}\_{\mathcal{S}} (\Delta r^\pi)) = \mathcal{H}\_{\mathcal{S}} (d^\pi)$. By substituting this in we get our desired result $\min\_{\tilde r \in \tilde R} \mathbb{E}\_{\rho^\pi}[\tilde r] = \mathbb{E}\_{\rho^\pi}[r]-\epsilon + \mathcal{H}\_{\mathcal{S}} (d^\pi)$.

References:

[1] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, 2021.

[2] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, 2021.

[3] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with value-conditional state entropy exploration. In Advances in Neural Information Processing Systems, 2023.

[4] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, Younggyo Seo. Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration. In Advances in Neural Information Processing Systems (NeurIPS) 2023

[5] Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto. Reinforcement Learning with Prototypical Representations. In International Conference on Machine Learning, 2021.

Thank you for the reply, I have no further questions, and would keep my score as it is.

We thank the reviewer for the thoughtful and constructive feedback. Below, we respond to the main points raised below, and will incorporate relevant clarifications and improvements into the revised version.

Questions

1. Clarifying the Positioning and Significance of our Analysis

We thank the reviewer for the question and appreciate the opportunity to clarify.

While previous studies have focused on the connection between robustness and policy entropy, our work is the first to thoroughly characterize the robustness properties of state entropy regularization—including its benefits, limitations, and interaction with policy entropy—both theoretically and empirically. In particular, we show that state entropy naturally induces robustness to spatially correlated perturbations, a class of real-world deviations (e.g., in transfer learning) that remains underexplored in the robust RL literature—not only in the context of entropy regularization.

Moreover, maximum state entropy is commonly used for practical purposes such as exploration and coverage [1,2,3,4,5], not explicitly for robustness. By analyzing the type of robustness it provides and its practical challenges—especially its reliance on rollout budgets—we aim to inform both the theoretical understanding and empirical use of this regularizer.

2. Additional Experiments

We indeed had the opportunity to experiment with additional domains beyond those presented in the main paper and plan to incorporate these results in the revision.

• We first ran an additional experiment on Ant in MuJoCo, where the goal is to progress as far as possible along the $y$-axis within an episode. At test time, we introduce a hurdle perpendicular to this axis. To continue progressing, the agent must jump over the obstacle. As shown in the table below, the agent trained with state entropy regularization significantly outperforms the unregularized agent and the one regularized with policy entropy.

• We also ran an experiment illustrating reward robustness in continuous control. In this experiment conducted in Pusher, we randomized the puck's target location at test time by uniformly sampling it within a circle around the nominal goal. As shown in the table below, the agent trained with state entropy regularization achieves better performance under this variation compared to other baselines.

• We finally extended the original Pusher experiment by adding action-robust baselines, which seem to be standard in robust RL for continuous control [6]. As shown in the table below, state entropy regularization significantly outperforms these baselines. Nonetheless, we must emphasize that we do not expect state entropy to consistently outperform robust RL methods across all types of perturbation. However, this result—along with our accompanying analysis—suggests that it can be particularly effective in scenarios involving spatially correlated perturbations, something that standard methods are not designed to address.

3. On the Practicality of Kernel-Robust RL

Thank you for raising this point. Generalizing RL agents to environments that differ from training is a fundamental challenge with many important applications (e.g., sim-to-real transfer). While multiple formulations and approaches address this challenge (e.g., [9]), one prominent and well-studied framework is robust RL, which explicitly models uncertainty and optimizes for the worst-case performance (e.g., [10, 6, 11]). Within this framework, kernel robustness remains a widely used and practical formulation—particularly in settings where test-time shifts are hard to anticipate and performance must remain reliable under a range of plausible deviations.

We also clarify that our negative result regarding kernel-robust MDPs does not imply that this setting is intractable in practice. Rather, it shows that no regularizer depending only on the policy and transition kernel can capture the robust return across all MDPs. In practice, effective solutions exist for practically useful uncertainty sets (e.g., [6] presents an effective solution for a rectangular uncertainty set).

Weaknesses

1. On the Novelty of the Results

We acknowledge that our analysis builds on similar tools used in prior work on policy entropy and robust RL—a connection the reviewer rightly notes is explicitly acknowledged in the main paper. That said, while we adopt similar techniques, the results themselves are novel. Furthermore, we believe the insights and perspectives developed throughout the paper—summarized concisely in our response to Question 1—are themselves original and offer meaningful value to the community.

2. On Tabular/Discrete Assumptions

We thank the reviewer for the thoughtful comment. Analyzing the tabular case is a standard approach in the literature, as it allows us to characterize robustness in a tractable setting while still offering insights applicable to more complex domains. We also note that most of the results can be extended to the continuous case with relatively minor adjustments, and we will add a note on this in the revision.

3. Minor Comments

We thank the reviewer for the suggestion and will revise the text to clarify key terms while maintaining readability.

4. State Entropy for Robustness Under Noisy Estimation

The reviewer is making a great point that the estimation error of the entropy may degrade the robustness guarantees in practice. We can quantify the degradation by specializing the results of Th. 3.1 (reward robustness) and Th. 3.3 (kernel uncertainity) for the estimated entropy regime. Let $\hat d$ be the empirical estimate of $d^\pi_P$ obtained with $n$ sampled trajectories. We can prove that

holds with probability $1 - \delta$ for $\delta \in (0, 1)$. Then, we can plug the right-hand side in place of $\mathcal{H} (d^\pi_P)$ into Eq. 6 and 7 to obtain sample-based high porbability lower bounds on the robustness. The sample-based bounds approximate the exact entropy results at a $O(n^{-1/2})$. We will add these results to the paper together with their proof, which is based on centration of the empirical disitribution [7] and continuity bounds of the entropy [8].

5. Regarding Experimental Domains

We thank the reviewer for the comment. Please see our response to Question 2, where we present additional experiments we plan to include in the revised version.

References:

[1] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, 2021.

[2] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, 2021.

[3] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, and Younggyo Seo. Accelerating reinforcement learning with value-conditional state entropy exploration. In Advances in Neural Information Processing Systems, 2023.

[4] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, Younggyo Seo. Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration. In Advances in Neural Information Processing Systems (NeurIPS) 2023

[5] Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto. Reinforcement Learning with Prototypical Representations. In International Conference on Machine Learning, 2021.

[6] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pp. 6215–6224. PMLR, 2019.

[7] Tsachy Weissman, Erik Ordentlich, Gadiel Seroussi, Sergio Verdu, Marcelo J. Weinberger. Inequalities for the L1 Deviation of the Empirical Distribution. 2003. [8] Andreas Winter. Tight uniform continuity bounds for quantum entropies. 2015.

[9] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. 2023. A Survey of Zero-shot Generalisation in Deep Reinforcement Learning. J. Artif. Int. Res. 76 (May 2023).

[10] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pages 2817–2826. PMLR, 2017.

[11] Uri Gadot, Kaixin Wang, Navdeep Kumar, Kfir Y. Levy, and Shie Mannor. 2024. Bring your own (non-robust) algorithm to solve robust MDPs by estimating the worst kernel. In Proceedings of the 41st International Conference on Machine Learning (ICML'24), Vol. 235. JMLR.org, Article 576, 14408–14432.

Thank you for taking the time to carefully review our paper and provide constructive feedback. We appreciate the reviewer's comments and will incorporate them into the revised version.

Questions

Before answering the reviewer's questions below, we emphasize that these aspects have been thoroughly studied in prior works on state entropy maximization [e.g., 1, 2, 3, 4, 5, 6, 7] and fall outside the scope of our work.

1. Is there a better way to estimate the visitation probabilities/entropy?

Possibly, but this is not the focus of our work. In the RL setting, k-NN entropy estimation [1, 2, 3, 4, 5] outperforms alternative methods such as computing the entropy from KDE [6] or learned density models such as VAEs (e.g., [7] mentioned by the reviewer). It has become a standard approach for state entropy maximization, thus justifying our choice.

2. How would you actually maximize reward + visitation entropy? As this reward now is non-markovian (policy dependent)?

To maximize the reward + visitation entropy, we follow state-of-the-art approaches from [1, 3, 4, 5] by combining the external reward with an "entropy"-reward derived from the k-NN estimator (implementation details are reported in Appx. B.1). We understand the reviewer's concern: the entropy-reward is technically non-Markovian. However, [1, 3, 4, 5] have shown that it can practically be optimized using standard RL techniques.

Limitations

1. Applicability as a major limitation

The method used for optimizing the state entropy regularization as analyzed in this paper has been successfully applied in challenging tasks, including high-dimensional continuous control [2] and visual domains [1, 3, 5]. Therefore, we do not think applicability is a concern and we focus our study on other aspects (robustification induced by the regularization). In principle, our findings should apply to any domain in which state entropy maximization has been successfully deployed.

Weaknesses

1. Physical examples of the reward uncertainty set

Thank you for raising this point. Since the reviewer's comment may reflect two distinct concerns, we address both interpretations below—one regarding the interpretation of the uncertainty set, the other regarding reward uncertainty experiments on a physical domain.

1.1. Bridging Theory and Intuition for the Reward Uncertainty Set

We acknowledge that the uncertainty set may initially seem abstract and we will clarify its intuition in the revised version. Specifically, as suggested, we will add a worked example on a simple two-state, two-action MDP illustrating that the method is robust against an adversary with a soft exponential budget (the paper already includes a basic illustrative example in Figure 1). In addition, we believe that highlighting how the uncertainty set interpolates between more familiar $\ell_\infty$-like (localized, uniformly bounded) and $\ell_1$-like (global, budgeted) perturbations can provide further interpretability. Finally, while the set itself couples rewards through a log-sum-exp form, we would like to emphasize that the worst-case perturbations remain intuitive: policy entropy allows only local statewise attacks, whereas state entropy enables global attacks concentrated on frequently visited states.

1.2. Empirical Illustration of Reward Uncertainty in Pusher

If the reviewer’s comment refers to the absence of physical examples involving reward perturbations, we have prepared an additional experiment in the Pusher domain that we will include in the revised version. In this experiment, we simulate reward uncertainty by randomizing the puck’s goal location at test time, uniformly sampling from a circle around the nominal target. This setup captures a natural class of perturbations arising from slight spatial shifts in task objectives. As shown in the table below, the agent trained with state entropy regularization achieves better performance under this variation compared to other baselines.

2. Regarding the Use of k-NN State-Occupancy Entropy Estimation

Please see our answer to Question 1.

3. Additional Robust Baselines and Domains

We thank the reviewer for raising this important point.

First, we extended the original Pusher experiment by adding a standard robust RL baseline [8]. As shown in the table below, the agent trained with state entropy regularization significantly outperforms this baseline. While we do not expect state entropy to consistently outperform robust RL methods across all types of perturbations, this result—along with our analysis in the main paper—suggests that it is particularly effective against spatially correlated perturbations, which are not the primary focus of most standard approaches.

Second, we added an additional experiment on the standard Ant benchmark in MuJoCo. In this setting, the agent is trained to progress along the $y$-axis. At test time, we introduce a hurdle perpendicular to this axis that the agent must jump over. As shown in the table below, the agent trained with state entropy regularization significantly outperforms both the unregularized agent and the one regularized with only policy entropy.

4. Missing References

We thank the reviewer for pointing out these relevant works. We will include both in the revised related work section.

References:

[1] Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems, 2021.

[2] Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate. In AAAI Conference on Artificial Intelligence, 2021.

[3] Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, 2021.

[4] Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, Younggyo Seo. Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration. In Advances in Neural Information Processing Systems (NeurIPS) 2023

[5] Denis Yarats, Rob Fergus, Alessandro Lazaric, Lerrel Pinto. Reinforcement Learning with Prototypical Representations. In International Conference on Machine Learning, 2021.

[6] Elad Hazan, Sham Kakade, Karan Singh, Abby van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning 2019.

[7] Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., & Salakhutdinov, R. (2019). Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274.

[8] Chen Tessler, Yonathan Efroni, and Shie Mannor. Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pp. 6215–6224. PMLR, 2019.