RETHINK MAXIMUM STATE ENTROPY

Other Comments

We humbly disagree with the way in which you presented the literature, as it does not consider any detailed algorithmic steps or theoretical relationships. In our framework, Hazan et al. (2019) provide the foundational algorithmic and theoretical structure, with the redundant $\eta$. This structure is not limited to entropy but applies to all smooth and bounded reward functions. We show that MaxEnt-LA can be seen as a special case of MaxEnt-H, achieved by using the kNN density estimator, setting $\eta \rightarrow 0$, and updating memory at each step. With the bridge between MaxEnt-H and MaxEnt-LA, other works you mentioned can be easily connected. For example, the difference between Lee et al. (2019) and MaxEnt-LA lies solely in the intrinsic reward function, i.e., KL divergence to a uniform distribution versus kNN entropy. RE3 simply replaces the pre-trained encoder with a random one compared to MaxEnt-LA. Yarats et al. (2021) further improves the encoder for visual representation by iteratively training the encoder and exploring with MaxEnt-LA. Each of these methods can be seen as a modification of the others, all within the framework of Algorithm 1 and Theorem 1. We show that MaxEnt-H and our MaxEnt-V theoretically support MaxEnt-LA and other related works.

In contrast, MEPOL and Guo et al (2021) reduce the problem by imposing a finite-horizon constraint. Mutti and Restelli (2020) constrain the number of steps, and Mutti et al. (The Importance...) assume a unique optimal action within a finite-horizon MDP. Each of these methods shifts the general MaxEnt problem into a different one.

In summary, the reviewer demonstrates a general understanding of MaxEnt methods but lacks a deep understanding of the details, except a few related works by Mutti et al. This has led to several misunderstandings regarding the literature. This highlights the importance of our paper: such misunderstandings would not arise if the reviewer had started from our paper to learn about maximum state entropy, rather than from MaxEnt-H, MaxEnt-LA, or MEPOL, etc.

We appreciate your response but encourage you to carefully review the main theorem in MaxEnt-H and the primary reward function in MaxEnt-LA (you may find Seo et al., 2021 useful for its simplicity). Most of the questions raised concern algorithmic details related to these elements. It is important to note that, given the kNN density estimation, MaxEnt-V is algorithmically identical to MaxEnt-H with $\eta \rightarrow 0$ and a hot-starting initialization. Similarly, MaxEnt-V is algorithmically equivalent to MaxEnt-LA when using a random encoder (Seo et al., 2021) and without the replay buffer freezing trick. These three methods can be transitioned into one another with at most five lines of code.

MEPOL

The statement "the slight difference does not change the nature of the problem" is neither formal nor correct. We can interpret $H\left(\frac{1}{T}\sum_{t=0}^{T}d_t^{\pi}\right)$ as a finite-horizon, discount-free ($\gamma \rightarrow 1$) version of MaxEnts’ objective, $H\left(\sum_{t=0}^{\infty}d_t^{\pi}\right)$, only if all induced trajectories have the same fixed length $T$. However, this assumption does not hold in environments such as Mujoco or in real-world scenarios, where stopping conditions (e.g., death, crash) are present. Specifically, if a trajectory $\tau$ terminates early at step $t = T(\tau) < T$, then $H\left(\frac{1}{T}\sum_{t=0}^{T}d_t^{\pi}\right)$ sums the probabilities over only the $T(\tau)$ steps in the trajectory, but still divides by the fixed $T$. This would incorrectly encourage the generation of longer trajectories. In fact, the finite-horizon version of MaxEnt should be expressed as $H\left(\sum_{t=0}^{T(\tau)}d_t^{\pi}\right)$, where $T(\tau) < T$.

More importantly, the MaxEnt methods discussed in our paper do not share key algorithmic techniques with MEPOL. The fundamental components of MaxEnt-H, MaxEnt-LA, and MaxEnt-V are dense reward functions, $r(s)$, and our paper demonstrates that these reward functions are identical when kNN is used ($r(s) = \log(\|s - s^{k\text{NN}}\|_2^p)$) given by vanilla kNN estimator. Each state receives a reward value compared to all historical states $D = [s_i]$. This simple reward function can be used in any RL algorithms. In contrast, MEPOL employs a trajectory-wise data buffer $D = [\tau^t_i, s_i]$, where $\tau^t_i$ the previous states within the same episode, and then a weight is computed to implicitly implement importance-weighted (IW) kNN estimator. As a result, MEPOL cannot be directly implemented in other off-policy RL algorithms, such as SAC, without further modification.

Therefore, we cite it in the related works section of our paper, rather than discussing it in detail in our paper.

APT uniform mixture

As we addressed in the previous comment, MaxEnt-LA trains a SAC with the non-stationary reward function for 1,000,000 steps. Although MaxEnt-LA reports only the final policy, i.e., $\pi_{1,000,000}$, we can still obtain the mixture of 1,000,000 policies from the training history by saving the SAC parameters. Recall the definition of a mixed policy in MaxEnt-H: "The policy $\pi_{mix}$ is one where, at the first timestep $t = 0$, we sample policy $\pi_i$ with probability $\alpha_i$, and then use this policy for all subsequent timesteps." The induced states of the mixed policy with uniform sampling can be obtained by sampling multiple policies from the 1,000,000 policies and executing them.

In fact, this is unnecessary. The induced states of the mixture are theoretically equivalent to all the states visited during the training process.

Tuning $\eta$ and Theorem 1

Given kNN density estimation, MaxEnt-V is algorithmically identical to MaxEnt-H with $\eta \rightarrow 0$ and a hot-starting initialization. Therefore, the convergence guarantees for a uniform mixture are already provided in Theorem 1 of our paper. The distinction is because that the main Theorem 4.1 of MaxEnt-H is based on the following settings: $\varepsilon_1 = 0.1\varepsilon$, $\varepsilon_0 = 0.1\beta^{-1}\varepsilon$, $\eta = 0.1\beta^{-1}\varepsilon$, and $T = \eta^{-1} \log \left(10B\varepsilon^{-1}\right)$. This setting is quite unrealistic, as we cannot directly control $\varepsilon_0$ or $\varepsilon_1$. If we eliminate these setting, Theorem 4.1 of MaxEnt-H would be in the similar expression Theorem 1 in our paper, with $\varepsilon = B e^{-T\eta} + 2\beta\varepsilon_0 + \varepsilon_1 + \eta\beta$ when $\eta \rightarrow 0$ (see lines 314 to 317).

If we adopt a similar setting to "cheat," we can simply set $\varepsilon_1 = 0.1\varepsilon$, $\varepsilon_0 = 0.1 \beta^{-1}\varepsilon$, and $T = 1.43(B + \beta[\rho + \ln(T)]) \varepsilon^{-1}$. The corresponding guarantee, expressed in a similar form to Theorem 4.1, becomes $T > 1.43(B + \beta[\rho + \ln(T)]) \varepsilon^{-1}$, or equivalently $T/(\beta \ln T + \text{Constants}) > 1.43 \varepsilon^{-1}$.

It is important to note that the goal of Theorem 1 is to compare regret and sample complexity. It represents a correction to Theorem 4.1 itself, incorporating a uniform mixture. Theorem 4.1 is inadequate for describing a uniform mixture, as it multiplies by $\frac{1}{\eta}$ during the derivation of $\varepsilon$.

Boundedness

Recall the definition of $B$-boundedness: $\|\nabla R(X)\|_\infty \leq B$, where $R$ represents the particle (kNN) entropy and $X$ lies within the state space. In this case, $B$ definitely exists, and we can compute it using brute force based on all historical state distributions. However, given that we operate in a large continuous space with millions of state points, obtaining a large value of $B$ is particularly challenging.

Discounted setting

In the experiments, the episodes are generated from a discounted MDP, consistent with the approach used in MaxEnt-H.

Related works

We agree with the suggestion to include additional related works. We have already cited Seo et al. (2021), and we will also cite Yarats et al. (2021) as a similar method to Seo et al. (2021), with differences in the encoder architecture. Additionally, we will include Mutti et al. (2022), which discusses scenarios involving unique optimal actions.

Some claims look subjective and lack strong support

1. This is presented in Theorem 1 of our paper. MaxEnt-V can be thought of as MaxEnt-H with uniform sampling. The only differences lie in the definition of the reward function and the method of initializing $\pi$ in each iteration. Besides, another major problem of $\eta$ is that the sampling probability $\alpha_t = \eta(1-\eta)^{t-1}$, with a fixed $\eta$, can never satisfy the condition $\sum_{t=0}^T \alpha_t = 1$. More details and corresponding experiments can be found in line 231 to 239 and line 352 to 377.
2. You have completely misunderstood Theorem 1 (as well as the main theorem of MaxEnt-H), which provides a theoretically grounded gap $\varepsilon$ between the current policy and the global theoretical optimal policy, similar to MaxEnt-H, rather than an "upper bound on the sub-optimality." "Freezing the reward" ensures that SACs are trained for an adequate number of steps to optimize subproblems within the Frank-Wolfe-like algorithm, rather than pursuing a smaller upper bound. We recommend a thorough review of Lemma 1 and Theorem 1 for clarification.
3. We do not assume strict smoothness and boundedness, but rather constrained with $B$ and $\beta$. $\beta$-smoothness and $B$-boundedness are not solely determined by Eq. (6) without knowing the distribution of kNN distances. Given the data points, we can always identify the maximum gradient difference $B$ and the maximum smoothness factor $\beta$. The key issue is whether $B$ and $\beta$ are appropriate for the specific environment. In Mickey-Mouse settings, such as in small DAG, data points are sparse, and there are no constraints on distribution shifts between adjacent nodes, leading to ridiculously large of $B$ and $\beta$. In contrast, in robotic environments, actions and states are continuous, with millions of dense data points, and the underlying $B$ and $\beta$ cannot exceed physical constraints such as velocity or acceleration. Thus, it is quite reasonable to assume this in a complex continuous environment.

For $\varepsilon_0$, the kNN entropy estimator consists of the kNN density estimator (see Eq. (9)) and a Monte Carlo estimate of the Shannon entropy ($-1/N \sum \log d$ and $-1 \sum d \log d$). $\varepsilon_0$ represents the error of the density estimates. The expression $H_{kNN} = -1/N \sum \log p$ is not merely an estimate of Shannon entropy, but also represents the particle entropy itself (refer to \cite{ref} for a detailed explanation of what constitutes an 'entropy'). Therefore, Theorem 1 provides tolerance for this entropy without considering the Monte Carlo estimate. We will clarify this by adding a hat to Eq. (6) and the corresponding explanation in the text.

The empirical analysis looks very far from the standards of the community

It is not true

1. The curves we provide represent the average scores from 4 runs. In the revised PDF, we have updated the curves along with the corresponding variances.

2. We have implemented the official versions of MaxEnt-H and MaxEnt-LA, using the same evaluation metric and Mujoco setup as in MaxEnt-H. The only difference is that we consistently use the default settings of Mujoco. In contrast to MaxEnt-H's experiments, which train with safety constraints but inducing states without them—an apparent inconsistency—our approach ensures a fair comparison by applying the same conditions throughout. In fact, we are the first to evaluate these two most important MaxEnt methods within the same empirical setting in a consistent manner, eliminating any auxiliary modules such as pre-trained encoders.

We appreciate your detailed response; however, we respectfully disagree that the paper is limited. We believe there has been a misinterpretation of our work, as well as of several influential contributions in the MaxEnt literature, due to a focus only on Mutti's work in a non-standard manner, leading to conclusions based on factual errors. We elaborate on these points in detail as follows:

Misunderstandings of the literature

1. The description "MaxEnt-H is mostly a theoretical tool, while MaxEnt-LA falls into a stream of practical methods" is neither formal nor accurate. One of the main contributions of our paper is to prevent such misunderstandings. We establish a theoretical and algorithmic relationship between these two methods, rather than presenting an abstract distinction, providing a step-by-step explanation. Both methods can transition into one another by adjusting hyperparameters (such as $\eta$, the frequency of updating the replay buffer, and the initialization of $\pi$ in each epoch).

Crucially, MEPOL (Mutti et al., 2021) is NOT premilitary to MaxEnt-LA. MaxEnt-H is NOT premilitary to MEPOL either. MEPOL dose NOT address the same problem.

From a motivational perspective, MEPOL seeks to maximize the entropy of the average state distribution $H\left(\frac{1}{T}\sum_{t=1}^{T}d_t^{\pi}\right)$, defined over episodes (note the index $t$ within an episode), while MaxEnt aims to directly maximize the state distribution $H(d^{\pi})$. Theoretically, MEPOL reduces the general MaxEnt problem to a finite-horizon setting, which requires a fixed hyperparameter (episode length) $T$ for each training process. In experiments, MEPOL has to remove the default termination rules for safe learning in Mujoco to maintain a fixed horizon length, which leads to illegal poses. In contrast, unconstrained MaxEnt methods allow for varying episode lengths in each run. Intuitively, MEPOL partly encourages exploration of novel episodes or trajectories, while MaxEnt focuses on exploring novel states.

From an information-theoretic perspective, MEPOL establishes a projection from the state space to a space defined by the average probabilities within trajectories. Such projections noramlly results in fundamentally different information-theoretic metrics and properties. Further details on these information-theoretic definitions may be found in the follows.

Alfréd Rényi. On measures of entropy and information. 1961.

Principe, Jose C. Information theoretic learning: Renyi's entropy and kernel perspectives. Springer Science & Business Media, 2010.

From an implementation perspective, MEPOL fundamentally does not present the algorithmic ideas used in Liu et al. or even MaxEnt-H. The key component of MaxEnt-LA is simply a non-stationary intrinsic reward function, which can be integrated into any off-policy RL algorithm (such as Q-learning, DQN, SAC, etc.) by substituting or augmenting the extrinsic rewards, much like the Approximation Oracles in MaxEnt-H. In contrast, the core of MEPOL is an implicit expression of the gradient of the trajectory-wise state entropy defined by itself, constrained by a KL divergence, within the framework of TRPO.

Therefore, it is not correct to position MEPOL between MaxEnt-LA and MaxEnt-H. It is sufficient to mention MEPOL briefly, along with its subsequent works, in the related works section.

2. The key component of MaxEnt-LA is a non-stationary intrinsic reward function, which can be implemented into any off-policy RL algorithm. During the training process, it iteratively appends a new sample $(s, s', a, r, d)$ to the replay buffer and updates SAC for one step. Consequently, the mixture of policies corresponds to the SAC models at all checkpoints. The induced states are simply the states visited during the training process. We will also clarify this tip in our paper to prevent confusion among readers who may not be familiar with DRL methods and experimental procedures.

3. This has been addressed in the first point. The finite horizon setting fundamentally alters the problem, particularly in relatively realistic environments such as Mujoco.

[1] Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate.

[2] Alfréd Rényi. On measures of entropy and information. 1961.

[3] Principe, Jose C. Information theoretic learning: Renyi's entropy and kernel perspectives. Springer Science & Business Media, 2010.

We appreciate your detailed response, but we humbly disagree that the contribution of this paper is limited. This paper is the first to bridge two of the most important tracks on maximum state entropy (MaxEnt), helping to clarify the disorganized literature within this community. This work help prevent further misinterpretations, such as those made by the reviewer, in fundamentally misreading previous works including MaxEnt-H, MaxEnt-LA, and MEPOL [1], which have led to conclusions based on factual errors. We elaborate on these points in detail as follows.

Weaknesses:

We believe you misinterpret the relationship between MEPOL and MaxEnts discussed in this paper. MEPOL is a particularly misleading work for readers seeking to understand MaxEnt, as it claims to improve MaxEnt-H by introducing kNN. Works like MaxEnt-LA cite it, but in fact, these works only reference MEPOL in passing, acknowledging its early connection to kNN, without ever directly comparing it or discussing it in detail. This has led some to assume a strong connection, which is, in fact, incorrect. MEPOL is NOT premilitary to MaxEnt-LA.

From a motivational perspective, MEPOL seeks to maximize the entropy of the average state distribution $H\left(\frac{1}{T}\sum_{t=1}^{T}d_t^{\pi}\right)$, defined over episodes (note the index $t$ within an episode), while MaxEnt aims to directly maximize the state distribution $H(d^{\pi})$. Theoretically, MEPOL reduces the general MaxEnt problem to a finite-horizon setting, which requires a fixed hyperparameter (episode length) $T$ for each experiment. Consequently, MEPOL has to remove the default termination rules for safe learning in Mujoco to maintain a fixed horizon length, which leads to illegal poses. In contrast, MaxEnt methods allow for varying episode lengths in each run.

We can interpret $H\left(\frac{1}{T}\sum_{t=0}^{T}d_t^{\pi}\right)$ as a finite-horizon, discount-free ($\gamma \rightarrow 1$) version of MaxEnts’ objective, $H\left(\sum_{t=0}^{\infty}d_t^{\pi}\right)$, only if all induced trajectories have the same fixed length $T$. However, this assumption does not hold in environments such as Mujoco or in real-world scenarios, where stopping conditions (e.g., death, crash) are present. Specifically, if a trajectory $\tau$ terminates early at step $t = T(\tau) < T$, then $H\left(\frac{1}{T}\sum_{t=0}^{T}d_t^{\pi}\right)$ sums the probabilities over only the $T(\tau)$ steps in the trajectory, but still divides by the fixed $T$. This would incorrectly encourage the generation of longer trajectories. In real worlds, the finite-horizon version of MaxEnt should be expressed as $H\left(\sum_{t=0}^{T(\tau)}d_t^{\pi}\right)$, where $T(\tau) < T$.

From an information-theoretic perspective, MEPOL establishes a projection from the state space to a space defined by the average probabilities within trajectories. This differs significantly from state entropy $H(d(s))$. The design of such projections generally results in fundamentally different information-theoretic metrics and properties. Further details on these information-theoretic definitions can be found in the [2, 3].

From an implementation perspective, MEPOL fundamentally does not present the algorithmic ideas used in Liu et al. or even MaxEnt-H. The fundamental components of MaxEnt-H, MaxEnt-LA, and MaxEnt-V are dense reward functions, $r(s)$, and our paper demonstrates that these reward functions are identical when vanilla kNN density estimation is used ($r(s) = \log(\|s - s^{k\text{NN}}\|_2^p)$). Each state receives a reward value based on all historical states $D =[s_i]$. This simple reward function can be used in any RL algorithms. In contrast, MEPOL employs a trajectory-wise data buffer $D = [\tau^t_i, s_i]$, where $\tau^t_i$ denotes the previous states within the same episode, and then a weight is computed to implicitly implement importance-weighted (IW) kNN estimator, i.e., $H(\bar{d}^{\pi}_T|\bar{d}^{\pi’}_T)$, where $\bar{d}_T = \frac{1}{T}\sum_0^Td_t^{\pi}$. As a result, MEPOL cannot be directly implemented in other off-policy RL algorithms, such as SAC, without further modification.

Therefore, we menioned it in the related work section without further discussion.

CLARITY and QUALITY:

1. We recommend that the reviewer set aside any prior misconceptions about MEPOL and read MaxEnt-H, MaxEnt-LA, and our paper in sequence. We believe some misunderstandings arise from the confusing notions in previous works. For instance, in the symbols $i$, $t$ and $T$ We have reorganized these elements for clarity.
2. We will present them in a concise manner.

Crucially, we do not believe it is appropriate to reject a paper solely because its claims and methods are perceived as too simple or lacking complex modifications, regardless of its correctness. The Heliocentrism, for instance, should not be dismissed simply because it is conceptually simpler or only slightly modifies the center while removing epicycles from the Geocentrism.

We appreciate your valuable response. We address your concerns as follow.

Weakness

1. Compared to MaxEnt-H in non-tabular experiments, our method should consistently perform better. We have shown that $\eta$ should always be very small, and more critically, the sampling probability $\alpha_t = \eta(1-\eta)^{t-1}$, with a fixed $\eta$, can never satisfy the condition $\sum_{t=0}^T \alpha_t = 1$.
   Compared to MaxEnt-LA, the performance of MaxEnt-V will be outperformed if the reward function $r_t$ changes slowly as $t$ increases. In this case, MaxEnt-LA can be seen as maximizing an approximately stationary reward function over recent steps. Consequently, there is no need to freeze the replay buffer as in MaxEnt-V, since doing so reduces sample efficiency. This situation typically arises in environments with small state spaces, such as small mazes or simple graphs. Recall that we adopt kNN (particle) entropy. Since the nearest neighbors of visited states do not change significantly, $r_t$ will become an approximately stationary reward function soon.

2. Apologies for the confusion. The value of $\varepsilon$ as $\eta \rightarrow 0$ is exactly given by Eq. (12) in Theorem 1 of our paper. The only distinction is that we specifically aim to maximize kNN (particle) entropy, eliminating the need to discuss $B$ and $\beta$. MaxEnt-H fails to give Eq. (12) due to the multiplication by $\frac{1}{\eta}$ in the derivation of $\varepsilon$. Consequently, the $\varepsilon$ obtained by MaxEnt-H, i.e., $\varepsilon = B e^{-T\eta} + 2\beta\varepsilon_0 + \varepsilon_1 + \eta\beta$, cannot be used to describe the scenario as $\eta \rightarrow 0$.

3. Apologies for the confusion. In each epoch (indexed by $t$ in the MaxEnt-V and MaxEnt-H algorithms), we train MaxEnt-H and MaxEnt-V for 500,000 steps. For MaxEnt-LA, the total number of training steps is set to $10,000,000$, ensuring that the SACs are updated an equal number of steps. Each epoch indicates $500,000$ steps. The states in Figs. 4 are obtained as follows: For MaxEnt-V and MaxEnt-H, we sample $\pi$ from the set of mixed policies $\pi_1, \pi_2, \dots, \pi_t$ to induce 100,000 states. An epoch corresponds to the update of the reward function $r_t$ for MaxEnt-H and MaxEnt-V, along with the corresponding training steps for SAC in MaxEnt-LA. For MaxEnt-LA, there would be millions of steps and corresponding policies in the set, as $r_t$ is updated in each step. Therefore, we can simply sample 100,000 states from all historical states to obtain the induced states across all historical policies. This approach is theoretically equivalent to recording all policies and executing them uniformly.

4. We will improve this expression, as you are correct. SACs can also explore by setting the rewards to be consistently zero. SACs aim to maximize $H(p(a|s))$, while MaxEnt aims to maximize $H(p(s))$. The key difference between the two approaches lies in the spaces they seek to explore: the state space for MaxEnt and the action space for SAC.

5. Thank you for your recommendations regarding the related works; we will certainly improve this section. More specifically, [1] is highly related to MaxEnt-LA. The key difference is that [1] minimizes the KL divergence to a prior uniform distribution $\rho$, which requires knowledge of the state space. In contrast, [3] provides an implicit variational approximation of the state distribution, whereas MaxEnt-LA uses a non-parametric kNN approach. [2] differs in that it aims to maximize the entropy of the average state distribution, $H\left(\frac{1}{T}\sum_{t=1}^{T}d_t^{\pi}\right)$, rather than $H(d(s))$. This formulation reduces the general MaxEnt problem to a finite-horizon setting, which necessitates a fixed hyperparameter (episode length) $T$, similar to MEPOL [4]. Therefore, it should be considered as a trajectory-wise entropy maximization.

We appreciate your valuable response. It is our oversight that we did not pay sufficient attention to the detailed description in experiments. We have added more details to address your concerns, as outlined below.

Experimental details and seeds

The curves we provide represent the average scores from 4 runs. In the revised PDF, we have updated the curves along with the corresponding variances. In each epoch (indexed by $t$ in the MaxEnt-V and MaxEnt-H algorithms), we train MaxEnt-H and MaxEnt-V for 500,000 steps. These numbers were selected because they are sufficiently large to achieve a small $\varepsilon_1$, rather than manually setting an $\varepsilon_1$-accurate stopping criterion. For MaxEnt-LA, the total number of training steps is set to $10,000,000$, ensuring that the SACs are updated an equal number of times.

The states in Figs. 4 and 5 are obtained as follows: For MaxEnt-V and MaxEnt-H, we sample $\pi$ from the set of mixed policies $\pi_1, \pi_2, \dots, \pi_t$ to induce 100,000 states. An epoch corresponds to the update of the reward function $r_t$ for MaxEnt-H and MaxEnt-V, along with the corresponding training steps for SAC in MaxEnt-LA.

For MaxEnt-LA, there would be millions of steps and corresponding policies in the set, as $r_t$ is updated in each step. Therefore, we can simply sample 100,000 states from all historical states to obtain the induced states across all historical policies. This approach is theoretically equivalent to recording all policies and executing them uniformly. The projection functions used for entropy calculation are consistent across methods and remain the same for all methods within each environment. (If these functions were different, the entropy values would vary significantly even for the same states.) The projection is used solely for entropy calculation purposes.

About NGU

The NGU can be considered an intrinsic reward function, $r_{ngu}$, designed to help avoid local optima in task-driven rewards, $r_{task}$. It is an auxiliary neural network irrelevant to the SAC and outputs a single reward value based on historical episodes. NGU requires no recurrence, as its episodic memory is not autoregressive.

To implement NGU, users can simply add the outputs of the NGU to the regular task-driven rewards without modifying the SAC. The combined reward is given by $r_{task} + \alpha r_{ngu}$, where $\alpha$ is a hyperparameter. SAC is then trained using these combined rewards, which results in larger $r_{task}$ values. In our experiments, we treat $r^{LA}$ and $r^{H}$ as $r_{task}$ and add $r_{ngu}$ in Step 3 of the algorithms. The objective of NGU is to increase the values of $r_{LA}$ and $r_{H}$ after convergence, without considering the value of $r_{ngu}$.

In our experiments, we found that this trick leads to larger $r_{LA}$ and $r_{H}$ values for all three methods, including MaxEnt-LA, Therefore, we implement this with $\alpha = 0.5$.

Downwards Tick

As discussed in the first point, we do not record all historical states in the entire training process for MaxEnt-V. What we did is to save policies (SACs) themselves using checkpoints, and then execute these policies to induce states to obtain figures and curves. Therefore, the unique state visited is not accumulated since we resample induced states after each epoch.

Something More

We greatly appreciate that you have read our paper in detail. We may adopt the name "Unified MaxEnt" for the framework.

Historically, informal descriptions and misinterpretations, such as "MaxEnt-H is just a theoretical tool, while MaxEnt-LA belongs to a stream of practical methods," have been prevalent. We are the first to provide a theoretical and algorithmic blueprint that connects most of the influential works in maximum state entropy.

This paper is not aimed at introducing tricks to improve performance by 4 or 5 % on a specific dataset. The experimental results are both predictable and explainable, as outlined in Theorem 1.

We appreciate your valuable response. We have added more details to address your concerns, as outlined below.

NGU

If we understand correctly, you may suggest that the episode-based NGU reward should only be effective when using RL algorithms with recurrent value functions, such as Recurrent Replay Distributed DQN, to maximize it. However, this is not strictly necessary, as a widely used technique involves optimizing memory-based non-stationary rewards directly with normal off-policy RL. MaxEnt-LA itself serves as an example: Seo et al. (2021) maximize the non-stationary $r^{LA}$ directly using non-recurrent A2C and RAD.

The reason why it works in episodic settings can be found in [1,2]. In brief, the reward is automatically transferred to a dense non-recurrent one via an implicit Monte-Carlo estimation when we sample a batch from the replay buffer to update agent. The non-recurrent agent would simply be encouraged to discover states within episodes that contain diverse states.

Therefore, what we did is straightforward: we compute $r_{ngu}$ using Eq. (2) from the NGU paper in state space, setting $\alpha_t$ in Eq. (1) of the NGU paper to a constant value of 1, since $r^{LA}$ already serves as the long-term novelty detection bonus. Then, $r_{ngu}$ is added directly to $r^{LA}$, allowing us to obtain a larger $r^{LA}$ in each iteration. The primary objective remains to maximize $r^{LA}$. We set $k$ to 3 for both $k$NN and use the same formulation as Eq. (6) from Seo et al. (2021). The $||\dots||_2$ denotes the norm for all $k$NN distances of s.

Apologies for the confusion. It is more accurate to say that NGU serves as the intrinsic reward for another intrinsic reward, $r^{LA}$ (our goal). It was our oversight that we did not give sufficient attention to this, as we considered step 4 as "(approximately) computing the optimal $\pi$ to maximize $r^{LA}_t$", similar to the approach in MaxEnt-H.

[1] Y. Efroni, N. Merlis, and S. Mannor, “Reinforcement learning with trajectory feedback,” in Proceedings of the AAAI conference on artificial intelligence

[2] Z. Ren, R. Guo, Y. Zhou, and J. Peng, “Learning long-term reward redistribution via randomized return decomposition,