Maximum Next-State Entropy for Efficient Reinforcement Learning

Dear Reviewer,

Thanks for your valuable comments. We hope the following statement can address your concern.

W1: Further clarification of example on aging equipment in introduction.

A for W1: Yes, we have revised this section in the introduction to provide further clarification. In our setup, before learning the policy, the equipment already has saturation or deadzone effects (caused by aging equipment or design redundancy). During policy learning and deployment, the action space remains unchanged.

Our work focuses on policy learning when there is redundancy in the action space, rather than on adapting the policy when the action space itself changes.

W2: Abuse of notation for reward function.

A for W2: Yes, we have modified the notation to ensure consistency throughout the paper.

Q1: There are some differences between MNSE and SAC when effective action proportion is 100%

A for Q1: Yes, there are differences in performance between MNSE and SAC when the effective action proportion (EAP) is 100%, and they are not exactly equal. There are two reasons for this:

• Even when the actuator's torque and the input commands are perfectly linear, the change in the state is not necessarily linear. MNSE is designed to eliminate such redundancy, leading to better performance in tasks like HalfCheetah.
• At EAP=100%, due to model learning and gradient training, the action mapping layer learned by MNSE may have a slight discrepancy from the true identity function (where actions = inner actions), which results in small performance differences, as observed in tasks like Ant.

Q2: How can we guarantee the mu variable is positive related with the entropy?

A for Q2: Providing a rigorous mathematical analysis is challenging. However, we have included numerical results for different sigma values, as shown in Figure. 6 in the Appendix. A. The results indicate that next-state entropy increases with higher mu values, demonstrating a positive correlation between mu and entropy.

Q3: Will maximum next-state entropy benefit the policy learning in discrete action space environments?

A for Q3: As discussed in Section 4, in discrete action spaces, the fewer redundant actions there are, the smaller the gap between next-state entropy and policy entropy. This insight suggests that evaluating actions and removing those with identical effects within the discrete action set can help improve exploration efficiency.

Q4: Can you explain in more details on how to derive the content of equation (8)

A for Q4: The content of equation (8) is derived using the Change of Variable Theorem.

The Change of Variable Theorem states that if you have a random variable $X$ with a probability density function $p_X(x)$, and you apply a transformation $y = g(x)$, then the probability density of the new variable $Y$ can be obtained by adjusting the original density using the Jacobian of the transformation:

Q5: Why the parameters of the inverse dynamic of inner policy are optimized first, instead of the ones of the mapping layer?

A for Q5: This is a technical decision related to the initialization process. At the beginning, the inverse dynamics network is randomly initialized, while the mapping layer is initialized as an identity function (i.e., actions = inner actions). Optimizing the inverse dynamics network first helps avoid the negative impact of the random initialization of the inverse dynamics network on the updates to the mapping layer. By optimizing the inverse dynamics network first, we ensure that the mapping layer can later benefit from a more stable inverse dynamics model, leading to more effective and efficient learning.

Dear Reviewer,

Thank you for the reviewer’s valuable suggestion! We have added the assumption $\sigma>1$ in the toy example (Section 4.1, Line 184). Based on this assumption, we have provided theoretical analysis and numerical results in Appendix A (Line 734) to demonstrate that the $\mu$ variable is positively related to the entropy.

We sincerely appreciate your feedback, which has helped make our paper more rigorous.

Best,

The Authors

Thank you for your efforts! I’ve raised my score accordingly.

Q1 & Q2: Why choose SAC as backbone framework? Why DSPG, CBSQL, and max-min entropy framework were not included as baseline in the experiment section?

A for Q1 & Q2: SAC is a successor to Soft Q-Learning (SQL) and integrates techniques such as double Q-learning and temperature adjusting (based on stable-baselines3). It is one of the most widely used maxEnt RL algorithms in the community. We believe that making improvements and developments based on the SAC algorithm is highly significant for the community.

The DSPG method (2019) introduced pioneering techniques like DoubleSampling into soft Q-learning. However, the SAC framework (based on stable-baselines3, 2021) has already integrated and refined techniques like double Q-learning.

The CBSQL method (2021) introduces pseudo-count density estimation to Soft Q-Learning, enabling more precise temperature adjustment. However, this method was only evaluated in discrete action spaces (Atari), and it does not show significant advantages in complex tasks in continuous action spaces.

The max-min entropy framework (2021) differs from traditional MaxEnt methods by constructing an exploration policy and separating exploration and exploitation. This added complexity makes it difficult to fairly compare this method with ours.

Q3: The correlation between next-state entropy and the total entropy of visited states.

A for Q3: Here are the key differences between our method (maximizing next-state entropy) and the state entropy maximizing methods mentioned:

The works mentioned in [2, 3, 4, 5, 6] directly construct policies (which can even be deterministic) to achieve better state coverage. However, they do not consider the entropy of policies.

In contrast, our approach builds on the entropy of stochastic policies, which not only accelerates learning and prevents premature convergence to suboptimal solutions but also induces a smoother objective that connects solutions and enable the use of larger learning rates [7].

Next-state entropy is an extension of policy entropy in environments with redundant action spaces. Policy entropy encourages action diversity while next-state entropy extends this idea to measure the diversity of effects caused by actions. Our method inherits the benefits of policy entropy while overcoming inefficiencies caused by action redundancy.

[1] Henaff M, Raileanu R, Jiang M, et al. Exploration via elliptical episodic bonuses[J]. Advances in Neural Information Processing Systems, 2022, 35: 37631-37646.

[2] Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., & Salakhutdinov, R. (2019). Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274.

[3] Guo, Z. D., Azar, M. G., Saade, A., Thakoor, S., Piot, B., Pires, B. A., ... & Munos, R. (2021). Geometric entropic exploration. arXiv preprint arXiv:2101.02055.

[4] Islam, R., Ahmed, Z., & Precup, D. (2019). Marginalized state distribution entropy regularization in policy optimization. arXiv preprint arXiv:1912.05128.

[5] Hazan, E., Kakade, S., Singh, K., & Van Soest, A. (2019, May). Provably efficient maximum entropy exploration. In International Conference on Machine Learning (pp. 2681-2691). PMLR.

[6] Liu, H., & Abbeel, P. (2021). Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34, 18459-18473.

[7] Ahmed Z, Le Roux N, Norouzi M, et al. Understanding the impact of entropy on policy optimization[C]//International conference on machine learning. PMLR, 2019: 151-160.

Dear Reviewer,

Thanks for your valuable comments. We hope the following statement can address your concern. We add supplementary experimental results as part of our response.

W1: Analyzing performance across a broader range of EAP settings.

A for W1: We conducted additional experiments to analyze the performance at EAP values of 20%, 40%, 60%, 80%, and 100%. This analysis provides a more comprehensive view of the algorithm's robustness and adaptability to different entropy thresholds. The results is shown as follows:

Table 1. Performance across a broader range of EAP settings.

W2: Further experiments focused on pure exploration tasks—such as in maze environments would be valuable.

A for W2: We conducted additional experiments with EAP=20% on pure exploration tasks in maze environments, including maze2d-umaze-v0, maze2d-medium-v0, and maze2d-large-v0. The results demonstrate that our method outperforms the baseline across these environments.

Table 2. Additional experimental results in maze environments.

W3: Comparisons with more recent approaches (from 2022 or 2023) could further strengthen the relevance and appeal of this work.

A for W3: Yes, we have added E3B (2022) [1] as a comparison method in our experiments. The results of these comparisons are provided below.

Table 3. Comparisons with E3B(2022) in MuJoCo.

Dear Reviewer,

We have added additional explanations and experiments for our methods. We are wondering if our response and revision have cleared your concerns. We would appreciate it if you could kindly let us know whether you have any other questions. We are looking forward to comments that can further improve our current manuscript. Thanks!

Best regards,

The Authors

Dear Reviewer,

Thank you for your thoughtful feedback on our paper. With only two days remaining in the discussion period, we kindly ask that you review our responses to ensure we have fully addressed your concerns. If you find our responses satisfactory, we would greatly appreciate it if you could reconsider your rating/scoring.

Your engagement and constructive input have been invaluable, and we truly appreciate your time and effort in supporting this process.

Best regards,

Authors

Dear Reviewer,

Thanks for finding our work is well written and significant for advancing research. The points you raised are explained in the following.

W1: How do you ensure the accuracy of the dynamic model estimations in MNSE and avoid overfitting?

A for W1:

We ensure the accuracy of the dynamic model estimations in the following ways:

• Choice of distribution:
  We adopt discrete multinomial distributions rather than Gaussian distributions for the inverse dynamic model. This choice better captures the dynamic characteristics of real-world physical systems.

• Iterative updates:
  The inverse dynamic model is continuously updated throughout the whole training process. This iterative refinement ensures that the model adapts and maintains accuracy as training progresses.

To avoid overfitting, we employ the following strategies:

• Continuously collected data:
  During the iterative training of the dynamic model, the replay buffer $D$ continuously incorporates new samples collected by the policy. This ongoing data augmentation enhances diversity and improves the model's generalization ability, effectively preventing overfitting.

• Regularization:
  We include a weight decay term in the Adam optimizer to constrain the L2-norm of the model parameters, further mitigating overfitting risks.

W2: Could you provide guidance on how to reasonably select the hyperparameters to optimize the algorithm's performance?

A for W2:

• Selection of hyperparameters in SAC backbone: Our algorithm, MNSE, is developed based on the SAC algorithm from the RL Baselines3 Zoo. All hyperparameters (e.g., learning rate, buffer size) are consistent with the SAC defaults in RL Baselines3 Zoo, as these have already been optimized. This ensures a fair comparison with baseline methods.

• Selection of unique parameters in MNSE: For our algorithm, a key hyperparameter is $N$, the number of parameters in the piecewise linear function. As illustrated in Figure 5, the algorithm's performance improves as $N$ increases. However, beyond $N \geq 20$, the performance stabilizes. For control tasks similar to Mujoco or Metaworld, we recommend setting $N=20$.

Dear Reviewer,

We have added additional explanations for our methods. We are wondering if our response and revision have cleared your concerns. We would appreciate it if you could kindly let us know whether you have any other questions. We are looking forward to comments that can further improve our current manuscript. Thanks!

Best regards,

The Authors

Dear Reviewer,

Thank you for your thoughtful feedback on our paper. With only two days remaining in the discussion period, we kindly ask that you review our responses to ensure we have fully addressed your concerns. If you find our responses satisfactory, we would greatly appreciate it if you could reconsider your rating/scoring.

Your engagement and constructive input have been invaluable, and we truly appreciate your time and effort in supporting this process.

Best regards,

Authors

W5: The entropy of the next state is never shown.

A for W5: We use a toy example similar to Section 4 where we calculate the next-state entropy for both MNSE and SAC methods. The table below shows the next-state entropy across different steps (from 10k to 150k).

From the table, it is clear that both MNSE and SAC show an initial rise in next-state entropy (NSE), which promotes exploration. After the model converges to an optimal solution, the NSE decreases, reinforcing exploitation. Our method (MNSE) quickly reaches a high next-state entropy, demonstrating stronger exploration capabilities than SAC, and is able to converge more quickly to the optimal solution.

Table 1. Next-state entropy of MNSE and SAC.

Q1.1: Why not simply use the log likelihood of the inverse dynamics model as intrinsic reward?

A for Q1.1: MinRed method [7] uses the log likelihood of the inverse dynamics model as intrinsic reward and we use it as a baseline in our paper. However, we demonstrate that our approach outperforms MinRed as Fig. 3 shown. The key advantage of our method lies in introducing an action mapping layer after the inner action, which directly shapes the action space. In contrast to MinRed, which uses additional intrinsic reward to train the agent, our forward approach is more direct and effective.

Q1.2: Why not simply learning a forward model of the MDP, and using the log likelihood of that model as intrinsic reward?

A for Q1.2: In high-dimensional state spaces, directly learning a forward model of the MDP is extremely challenging. In comparison to the state space, the action space is relatively smaller, making the training and application of inverse dynamics models more common and feasible, as seen in prior works [8, 9].

Q2 & Q3: What is the advantage of using the piecewise linear function in our method rather than a normalizing flow as action transformation and What is the advantage for using the discretized multinomial distribution rather than a normalizing flow (or auto-encoder + ELBO for learning) as the inverse dynamics model?

A for Q2 & Q3: In our theoretical derivation, any invertible mapping can serve as the action mapping function. On the other hand, the inverse dynamics model can be estimated using various approaches. In our experiments, the piecewise linear function and the discretized multinomial distribution have demonstrated performance surpassing the baseline. We appreciate the reviewer’s suggestion and plan to explore the integration of normalizing flows and auto-encoder + ELBO in future work to tackle more complex problems.

[1] Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., & Salakhutdinov, R. (2019). Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274.

[2] Guo, Z. D., Azar, M. G., Saade, A., Thakoor, S., Piot, B., Pires, B. A., ... & Munos, R. (2021). Geometric entropic exploration. arXiv preprint arXiv:2101.02055.

[3] Islam, R., Ahmed, Z., & Precup, D. (2019). Marginalized state distribution entropy regularization in policy optimization. arXiv preprint arXiv:1912.05128.

[4] Hazan, E., Kakade, S., Singh, K., & Van Soest, A. (2019, May). Provably efficient maximum entropy exploration. In International Conference on Machine Learning (pp. 2681-2691). PMLR.

[5] Liu, H., & Abbeel, P. (2021). Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34, 18459-18473.

[6] Understanding the impact of entropy on policy optimization[C]//International conference on machine learning. PMLR, 2019: 151-160.

[7] Action redundancy in reinforcement learning[C]//Uncertainty in Artificial Intelligence. PMLR, 2021: 376-385.

[8] Estimating q (s, s’) with deep deterministic dynamics gradients[C]//International Conference on Machine Learning. PMLR, 2020: 2825-2835.

[9] Learning action representations for reinforcement learning[C]//International conference on machine learning. PMLR, 2019: 941-950.

Dear Reviewer,

Thanks for your valuable comments. We hope the following statement can address your concern.

W2: Discussion with prior works focused on maximizing the entropy of all visited states

A for W2: We will add discussions of these works in the related works section and explicitly clarify the differences in objectives and methodologies between these studies and our approach.

Here are the key differences between our method (maximizing next-state entropy) and the state entropy maximizing methods mentioned:

The works mentioned in [1, 2, 3, 4, 5] directly construct policies (which can even be deterministic) to achieve better state coverage. However, they do not consider the entropy of policies.

In contrast, our approach builds on the entropy of stochastic policies, which not only accelerates learning and prevents premature convergence to suboptimal solutions but also induces a smoother objective that connects solutions and enable the use of larger learning rates[6].

Next-state entropy is an extension of policy entropy in environments with redundant action spaces. Policy entropy encourages action diversity while next-state entropy extends this idea to measure the diversity of effects caused by actions. Our method inherits the benefits of policy entropy while overcoming inefficiencies caused by action redundancy.

W3: Errors in some equations and notations:

A for W3:

We have corrected the errors in the equations and notations and provided clarifications to address any areas of potential confusion:

• a: Thanks for pointing out this, we have modified the notation to ensure consistency throughout the paper.

• b: $P^{\pi}(s' \mid s)$ represents the probability of transitioning to the next state $s'$ given the current state $s$ under the policy $\pi$. The relationship between these two quantities can be expressed through the Bellman equation. Given a policy $\pi$, the relationship between the state transition probability $P^{\pi}(s' \mid s)$ and the action-selection probability $\pi(a \mid s)$ is as follows:

• c: To avoid confusion with the constant $\pi$, we use $\pi\_{policy}$ to represent the policy. The policy follows a normal distribution, and we modify the expression as follows:

• d: We have added a definition of inverse dynamics of inner policy in the paper. The inverse dynamics model $p_{\text{inv}}^{\phi}(e\mid s, s')$is a function that predicts the inner action $e$ required to transition from a current state $s$ to a next state $s^{\prime}$.

• e,f,g: We acknowledge the missing expectation notation over the state $s$ in equations and have revised the paper to include it.
  All instances have been uniformly corrected in the revised version to address these issues.

• h: Yes, the $x$-variable in the equation corresponds to the inner action $e$ . We map the inner action $e$ to the action $a$ using the piecewise linear function $f$.

• i,j: We have standardized the notation throughout the paper to avoid any confusion.

W4: Discussion on the gap between action entropy and the next-state entropy is unused afterwards.

A for W4: As you pointed out, our discussion on the gap between action entropy and next-state entropy theoretically derives the condition under which non-redundant policies (and action spaces) would make them strictly equivalent. However, this condition is very strict and difficult to satisfy in cases of stochastic transitions.

This discussion motivates us to propose our approach in Section 5. Instead of pursuing strict equivalence, we quantify the gap between action entropy and next-state entropy using a parameterized action mapping layer. We then use optimization to gradually minimize this gap, providing a more practical solution.

Dear Reviewer,

We have added additional explanations and experiments for our methods. We are wondering if our response and revision have cleared your concerns. We would appreciate it if you could kindly let us know whether you have any other questions. We are looking forward to comments that can further improve our current manuscript. Thanks!

Best regards,

The Authors

Dear Reviewer,

Thank you for your thoughtful feedback on our paper. With only two days remaining in the discussion period, we kindly ask that you review our responses to ensure we have fully addressed your concerns. If you find our responses satisfactory, we would greatly appreciate it if you could reconsider your rating/scoring.

Your engagement and constructive input have been invaluable, and we truly appreciate your time and effort in supporting this process.

Best regards,

Authors