Adaptive HL-Gaussian: A Value Function Learning Method with Dynamic Support Adjustment

Thank you very much for your questions and suggestions. We will respond to each of them as follows:

1、About the significance and importance of the topic

First, although AHL-Gaussian is an incremental improvement over HL-Gaussian, HL-Gaussian has been shown to exhibit scaling laws, making it a highly promising foundational algorithm. Secondly, as the reviewer correctly pointed out, AHL-Gaussian has the potential to continue improving with more training steps. For example, in the Atari tasks, AHL-Gaussian still shows an upward trend at 5M steps. In Gym Mujoco, AHL-Gaussian also maintains an upward trend in tasks like Humanoid and HumanoidStandup.

Moreover, we believe that this approach is not only applicable to the specific method of HL-Gaussian but also holds potential for extension to other Distributional RL methods that require a predefined support interval, such as C51 or ORDQN.

2、About the dynamic process of the interval bound $\xi$ and the projection error

We would like to thank the reviewer for your suggestions. In the revision, we have added Section E to present the curves of $\xi$ and the projection error.

It can be observed that, in almost all tasks, the projection error exhibits a steep increase during the early stages. This is primarily due to the small initial interval range and the mismatch with the rapid growth of the value function in the early stages. As training progresses, the projection error gradually decreases to a sufficiently small value, indicating that the support interval $\xi$ effectively encompasses the value function. This phenomenon aligns with our theoretical predictions.

Regarding the support interval bound $\xi$, it converges to a fixed value in most Atari tasks. However, in some Mujoco tasks, $\xi$ continues to show an upward trend even at 5M steps, particularly in HumanoidStandup and Humanoid. Interestingly, the scores for these two tasks also exhibit an upward trend, suggesting that as training progresses, the overall performance is likely to continue improving.

3、normalized the reward and use $[0,1/1-\gamma]$

We have added this non-learning baselines for further comparison in Section F. Figure 19 shows that this method performs significantly worse than AHL-Gaussian across multiple tasks. This is likely because directly setting the support interval to $1/(1-\gamma)$ causes unnecessarily large projection errors during the early stages of training, when the received reward values are relatively small compared to the interval range. This negatively affects training performance. Moreover, obtaining a reliable prior estimate of the maximum reward is challenging in practice.

About 9:

We will provide a detailed comparison of the Q-function learning process under the traditional framework and the distributional RL framework to align our understanding:

1. In the traditional Q-learning framework, $Q_{\text{trad}}(s, a)$itself is the variable to be learned and iterated. In the distributional RL framework discussed in this paper, $Q_{\text{distri}}(s, a) = \sum_{i=1}^{m} \hat p_i(s, a) z_i$, where $\hat p_i(s, a)$ is the variable to be learned and iterated. Although the forms of the Q-functions differ slightly, $Q_{\text{distri}}$ can be understood as a more complex representation of the Q-function, without any fundamental difference in meaning.

2. In the traditional Q-learning framework, $Q_{\text{trad}}(s, a)$ is learned by fitting the Bellman equation through optimizing the MSE loss (or Bellman residual):

$$\min_{Q_{\text{trad}}} \left[ Q_{\text{trad}}(s, a) - \hat{\mathcal T} Q_{\text{trad}}(s, a) \right]^2.$$

In contrast, in the framework discussed in this paper,$Q_{\text{distri}}$ is learned by optimizing the KL-divergence between [\hat p_i(s, a)] \) and [p_i(s, a)] $. Here,

$$\hat{\mathcal T} Q_{\text{distri}}(s, a) = r(s, a) + \gamma Q_{\text{distri}}(s', a'),$$

where $a' \sim \pi(s)$ or ( a' = \max_a Q_{\text{distri}}(s', a) $, and$ [p_i(s, a)] $is obtained by projecting$\hat{\mathcal T} Q_{\text{distri}}(s, a) $ on the support interval. From this, it can be seen that although the two frameworks differ in the form of the Q-function, the ultimate goal remains the same: to ensure the Q-function satisfies the Bellman equation. Therefore, determining the performance of the policy evaluation algorithms under the two frameworks only requires evaluating whether the Bellman equation is satisfied, i.e.,whether the Bellman residual is small.

3. However, during the projection of $\hat{\mathcal T} Q_{\text{distri}}(s, a)$ into $[p_i(s, a)]$, an inappropriate choice of the interval may introduce extra projection errors, thereby increasing the Bellman residual and affect negatively the policy evaluation process. The impact of the projection error has been elaborated in detail in Proposition 3.1. This work thus aims to minimize the projection error by adjusting the interval.

4. To further demonstrate the influence of projection error in each policy evaluation step on the final learned Q-value, we have added Section B.3 in the appendix. These results aim to show the cumulative effects of projection error at each policy evaluation step and highlight the importance of optimizing the projection error during training.

About the relationship between solution and theory

Our theory conveys the following key points:

First, Proposition 3.1 demonstrates that HL-Gaussian introduces projection error, which increases the Bellman residual and thus negatively affects policy evaluation, leading to the conclusion that reducing projection error is necessary.

Additionally, we have added Section B.3 in the appendix to discuss the cumulative effect of the projection error in one-step policy evaluation on the algorithm's final performance, highlighting the importance of minimizing projection error during each policy evaluation step.

Further, Theorem 3.2 estimates the numerical range of the projection error and shows that the magnitude of this error depends on the positional relationship between the Bellman target $\mu$ and the interval $[v_{\text{min}}, v_{\text{max}}]$, summarized in the key finding highlighted in the gray box. This finds reveal that the projection error increases markedly if $[v_{\text{min}}, v_{\text{max}}]$ does not match the current target Q values.

Finally, it is precisely because of our theoretical findings that we are inspired to treat $[v_{\text{min}}, v_{\text{max}}]$ as learnable variables and adaptively adjust them by optimizing projection error, forming the basis of our algorithm design.

We believe that our theoretical insights and algorithm design are logically consistent and clearly structured.

The reason we do not optimize the number of bins $m$is that this would lead to a time-varying network architecture, which poses implementation challenges. Our ablation study also shows that the algorithm's performance is robust when $m$ is within a certain range.

Besides, we have also added Section E in the appendix to visualize the training curves of projection error and the learned interval and their trends also align with our theoretical predictions.

Thank you very much for your questions and suggestions. We will respond to each of them as follows:

About the unclear notations

We apologize for any confusion the reviewer may have had regarding some of the notations in the paper. We will clarify each of them below.

About 1: $\mathcal{T}^*$ and $\mathcal T^\pi$ are defined here as Bellman operators. . $\mathcal T^{*} Q(s_t,a_t)$ and $\mathcal T^\pi Q(s_t,a_t)$ represent the Bellman backups of $Q(s_t,a_t)$, and equations (1) and (2) represent the Bellman equations. This is the standard definition in RL and is widely used in RL literature.

About 2 and 3: In this paper, $Q(s, a)$ is represented as the expected value of a random variable $Z(s,a)$, where $Z(s,a)$ follows a categorical distribution . The discrete values of this categorical distribution consist of $m$ locations $[z_1, \cdots, z_m]$, each lying within the interval $[v_{\text{min}}, v_{\text{max}}]$. We have emphasized in the main text that $Z$ is a random variable, not a distribution, and provided the definition of $\delta(\cdot)$, highlighted in blue. We believe the revised explanation is more rigorous and easier to understand.

About 4 and 5: We first introduce the basic assumption of Distributional RL methods: the random variable $Z(s,a)$ corresponding to $Q(s,a)$ lies within a bounded region $[v_{\text{min}}, v_{\text{max}}]$ and follows a categorical distribution $[\hat p_1(s,a), \cdots, \hat p_m(s,a)]$. At the same time, the Bellman target $\mathcal{T}Q(s,a)$ is assumed to be the expectation of some random variable $Z'(s,a)$, where $Z'(s,a)$ follows a categorical distribution $[p_1(s,a), \cdots, p_m(s,a)]$, which also explains why the condition $\sum_{i=1}^{m} p_i(s,a) z_i = \mathcal{T}Q(s,a)$ needs to hold in equation (139).

However, if the pre-specified $[v_{\text{min}}, v_{\text{max}}]$ is unreasonable, such that the true $Q$ values may fall outside this range, particularly when the Bellman target is more likely to fall outside $[v_{\text{min}}, v_{\text{max}}]$, the assumptions of the existing framework become invalid. In this case, the equation $\sum_{i=1}^{m} p_i(s,a) z_i = \mathcal{T}Q(s,a)$ may no longer hold.

Therefore, we propose an adaptive adjustment of $[v_{\text{min}}, v_{\text{max}}]$ to ensure that the equation $\sum_{i=1}^{m} p_i(s,a) z_i = \mathcal{T}Q(s,a)$ holds with only minimal error, thereby better enabling the application of Distributional RL methods for learning.

About 6: We sincerely apologize for the typo in the proof of Proposition 3.1 and for the lack of further explanation regarding some of the symbols, which led to confusion for the reviewers. We have corrected the expression in this part of the proof in the revision and highlighted it in blue. In fact, we introduced the notation $[p_1(s,a), \cdots, p_m(s,a)]$ as $p_{\mu}$ and $[\hat p_1(s,a), \cdots, \hat p_m(s,a)]$ as $h_x$ simply to apply Lemma B.1. Additionally, the assumption that $h_x$ has supports in $[v_{\text{min}}, v_{\text{max}}]$ is a fundamental assumption for these methods, as described in the previous response.

About 7: Thank you very much for pointing out the issue with the imprecise definition of $\delta$. In fact, even when $\mu$ does not belong to $[v_{\text{min}}, v_{\text{max}}]$, we can still assume the existence of bins, allowing us to define $m_0$ and $\delta$. This does not affect the subsequent proof. We have added a footnote in the main text to clarify this point.

About 8: Since $\delta$ represents the distance between $\mu$ and the center of a bin, $m_0$, it can be either positive or negative. Therefore, $\mathcal{E}_{discretization}$ can also be both positive and negative, which means it can be less than 0.

About 10: We have removed the inappropriate expression in Line 777. o(1) here represents a constant much smaller than 1, and we have provided its definition in Line 215.

Dear Reviewer uVAy:

We sincerely thank you for your efforts and valuable feedback on reviewing our paper. We noticed that you have not yet participated in the discussion, and we would like to ask if you have any additional comments or questions that we can address collaboratively. We have revised the manuscript based on your insightful feedback and are eager to address any remaining concerns.

Please let us know if you have any further thoughts or questions. We look forward to continuing our discussion.

Best regards,

Authors

Thank you very much for your response. We are pleased to know that our revised manuscript and replies have addressed some of your concerns. Below are our responses to the remaining concerns:

Question 1: $Z(s_t,a_t)$ is a random variable that follows a categorical distribution, where the probability of taking the value $z_i$ is $\hat p_i(s_t, a_t)$. We followed the definition in paper [1] to write $Z(s_t,a_t)$ as $Z(s_t, a_t) = \sum_{i=1}^m \hat p_i(s_t, a_t) \cdot \delta_{z_i}$. However, we agree that this expression may cause misunderstanding, so we have removed it.

Question 2: We suspect there might be a difference in how we define traditional TD error. We infer that, in your context, traditional TD error strictly requires the Q-value to be a scalar rather than the expectation of a random variable Z. However, in our context, the Q-value can be the expectation of a random variable, as long as the error takes the form $[Q(s,a)-\mathcal{T}Q(s,a)]^2$, rather than the cross-entropy induced by $p$ and $\hat p$. Thus, our Proposition 3.1 provides a comparison between these two error forms.

From a different perspective, within HL-Gaussian, the Q-value can still be interpreted as a scalar rather than the expectation of a random variable. The distinction lies in how this scalar is computed: previously, Q value was directly the output of a neural network (NN), whereas now it is obtained by applying a softmax function to the output of the NN's last layer, followed by a dot product with $\{z_1, \cdots, z_m\}$. This modification introduces a more sophisticated NN structure. Consequently, we believe it is reasonable to express $[Q(s,a)-\mathcal T Q(s,a)]^2$ as traditional TD error, as it continues to represent the loss function that traditional RL methods aim to optimize.

Question 3: We believe your understanding that "the theory primarily highlights that a fixed interval introduces projection error, while the algorithm addresses this by performing gradient descent to optimize the interval and minimize the error" is correct. However, we would like to further highlight the intrinsic connection between the theory and the algorithm:

1. Our theory identifies the source of projection error and explains how it influences the projection error. Based on this insight, we choose to optimize the interval range, which is the root cause, instead of selecting some other unrelated variables.

2. Furthermore, our theory predicts that a reasonable interval range should precisely cover the current Q-values without being too large to cause unnecessary errors. This also explains the necessity of dynamically adjusting the interval range as the Q-values evolve.

Thank you again for your feedback and suggestions. We sincerely hope this response addresses all your concerns.

[1] Stop regressing: Training value functions via classification for scalable deep rl.

Thank you very much for your quick response. We greatly appreciate the discussion process as it will certainly help make our work better understood and explained.

1. First, we apologize for the wording "new questions" in our previous reply. What we intended to express was the questions you raised in the new reply, not additional questions. We have already revised the wording in our previous reply.

2. The expression $\sum_{i=1}^m \hat p_i(s_t, a_t) \cdot \delta_{z_i}$ is meant to represent a categorical distribution, not an expectation. Therefore, a more accurate expression would be $Z(s_t, a_t) \sim \sum_{i=1}^m \hat p_i(s_t, a_t) \cdot \delta_{z_i}$, where $\sim$ replaces the equality sign $=$. We also agree that the previous equality was not accurate, and as such, we have removed it in the revision and replaced it with a description to reduce the potential for misunderstanding.

3. Your understanding of "the projection error discussed in the paper stems solely from the difference between MSE and cross-entropy loss, as both in Proposition 3.1 assume values are expectations of categorical random variables" is correct. However, we would like to explain more about TD-error and our insights.

• Firstly, we understand that the TD error should be induced by the algorithm and the underlying function space. When the algorithm restricts the function space used to express $Q$, the TD error should be the MSE loss between $Q$ and $\mathcal T Q$, where $Q$ is in the chosen function space. The error between the Q-value generated by the algorithm and the true $Q^{\pi}$ should be defined as the "approximation error." This is, of course, important but does not belong to the category of TD error.

• Secondly, in the widely used neural networks (NNs), additional constraints such as ReLU activation functions and layer normalization are commonplace. These constraints undoubtedly influence the output range of the neural network. Therefore, although HL-Gaussian restricts the Q-value range to $[v_{\min}, v_{\max}]$, it is essentially no different from the above-mentioned treatments.

• Finally, as you mentioned, constraining the Q-values to $[v_{\min}, v_{\max}]$ does indeed introduce error, i.e., "approximation error." Therefore, our approach adjusts $[v_{\min}, v_{\max}]$ to "reduce the error introduced by target projection," ensuring that the Q-value always stays within the range of $[v_{\min}, v_{\max}]$. This approach helps to mitigate the "approximation error." Although this point has not yet been explicitly stated in the theoretical expression, it is indeed a key insight in our theoretical work.

We hope our reply helps clarify your concerns, and please do not hesitate to let us know if you have any new comments.

Thank you very much for your questions and suggestions. We will respond one by one as follows:

1、 About the interval choice for baselines

First, we would like to clarify a misunderstanding by the reviewer regarding the interval selection for the baseline algorithms. In fact, we adopted different interval-setting strategies for Atari and Gym Mujoco environments:

For the Atari environment, since the rewards are normalized to $[-1,1]$ and [1] recommends a fixed interval of $[-10,10]$ as the optimal hyperparameter, we chose $[-10,10]$ as the hyperparameter for the baseline.

For the Gym Mujoco environment, we fine-tuned an optimal interval for each task. The fine-tuning process is described in detail in the last paragraph of Section C in the appendix of the paper. Of course, limited hyperparameter search cannot guarantee the discovery of the true optimal hyperparameter. This thus highlights one of the advantages of our method: it eliminates the need for hyperparameter tuning and is robust to hyperparameter settings.

[1] Stop regressing: Training value functions via classification for scalable deep rl.

2、 About other non-learning baselines

We would like to thank the reviewer for the suggestions. In response, we have added several non-learning baselines for further comparison in Section F.

Method 1: The interval bound $\xi$ is set as the maximum value of all current Bellman targets, multiplied by a larger coefficient $\eta = 2$.

From Figure 17, it is evident that this approach performs comparably to AHL-Gaussian on certain tasks. However, it exhibits significant shortcomings in tasks such as Swimmer and HumanoidStandup. Specifically, the target values in the Swimmer task fluctuate drastically, making this "overly sensitive" adjustment method unable to converge to a reasonable interval, which severely impacts training performance.

Method 2: The support interval is set to $[\frac{r_{\min}}{1-\gamma}, \frac{r_{\max}}{1-\gamma}]$, where $r_{\min}$ and $r_{\max}$ are observed during training.

As shown in Figure 18, this method also demonstrates disadvantages in tasks such as Swimmer and HumanoidStandup. This is likely because it relies on discovering effective rewards during training, rendering it unsuitable for tasks with highly fluctuating reward signals or those that are challenging to explore.

Method 3: This method normarlizes the reward and sets the support interval at $[-\frac{1}{1-\gamma}, \frac{1}{1-\gamma}]$.

Figure 19 shows that this method performs significantly worse than AHL-Gaussian across multiple tasks. This is likely because directly setting the support interval to $1/(1-\gamma)$ causes unnecessarily large projection errors during the early stages of training, when the received reward values are relatively small compared to the interval range. This negatively affects training performance. Moreover, obtaining a reliable prior estimate of the maximum reward is challenging in practice.

In summary, while these non-learning-based approaches achieve good performance on certain tasks, they have inherent limitations that make them unsuitable for a wide range of tasks. In contrast, our approach offers a more general and formal framework.

3、What types of problems is AHL-Gaussian more suitable for?

As outlined above, AHL-Gaussian may surpass parameter fine-tuning methods in scenarios characterized by highly fluctuating reward signals, challenging exploration, and limited prior knowledge.

Dear Reviewer KKWb:

We sincerely thank you for your efforts and valuable feedback on reviewing our paper. We noticed that you have not yet participated in the discussion, and we would like to ask if you have any additional comments or questions that we can address collaboratively. We have revised the manuscript based on your insightful feedback and are eager to address any remaining concerns.

Please let us know if you have any further thoughts or questions. We look forward to continuing our discussion.

Best regards,

Authors

Thank you very much for taking the time and effort to read our response and provide new comments. We appreciate your recognition of our theoretical contributions, and we would like to offer further clarification regarding our experimental contributions.

• First, your acknowledgment that "AHL-Gaussian, on average, performs slightly better than non-learning/non-adaptive baselines" indicates that our method has good generalizability and consistently outperforms many alternatives.

• Additionally, you expressed uncertainty about whether "it can serve as a substitute for problem-specific fine-tuning of the support interval." In fact, the experimental results on Gym Mujoco tasks have clearly shown that our approach is superior to baselines with fine-tuned support intervals.

• Furthermore, we believe that if an algorithm requires task-specific fine-tuning of hyperparameters, its scalability and generalization ability in practical applications are limited. In contrast, our approach is robust to such task-dependent hyperparameter sensitivity, allowing it to achieve strong performance with any initial support interval, regardless of the task. This practical advantage is also one of the key experimental contributions of our method.

We hope our reply addresses your concerns and please let us know if you have any further thoughts or questions.

Thank you very much for your questions and suggestions. We will respond to each of them as follows:

1、About the theoretical insights on why quantization should help in training

First, as the primary handling technique in distributional RL methods, quantization has been extensively discussed in C51 [1] for its benefits during training. Specifically, by learning the distribution of Q values instead of merely predicting the scalar Q values, it can better handle approximation errors, reduce chattering caused by policy updates, and mitigate state aliasing, thus improving training stability. Additionally, the distribution itself provides a rich set of predictions, allowing the agent to learn from multiple predictions rather than solely focusing on an expected value. Moreover, the distributional perspective introduces a more natural inductive bias framework for reinforcement learning, enabling the imposition of assumptions on the domain or the learning problem itself. This discussion has been included in the appendix of the revision.

Second, quantization changes the optimization objective from an MSE loss to a cross-entropy loss (CE loss). The advantages of the HL-Gaussian-based CE loss used in this paper have been explained in Lines 157-161 and Lines 176-183, namely: CE loss facilitates a more efficient path to the optimal solution with a reduced number of gradient steps. The related theoretical analysis can be found in [2].

In summary, quantization contributes to training through two dimensions: robust representation and efficient optimization.

However, the aforementioned advantages do not account for the projection error introduced by mapping the Bellman target onto the categorical distribution due to an inappropriate interval range. In fact, as demonstrated in Proposition 3.1 of the main text, projection error affects policy evaluation. Furthermore, we have added a section in the appendix discussing how projection error impacts the algorithm's final performance. These findings suggest that while quantization offers stable representations and efficient optimization, the projection error in its application remains overlooked, leaving room for further improvement.

To address this, this paper aims to reduce projection error by dynamically adjusting the interval range, enabling quantization to achieve better practical outcomes.

[1] distributional Perspective on Reinforcement Learning.

[2] Investigating the Histogram Loss in Regression.

2、About the theoretical guarantee on performance of the algorithm

We have added Section B.3 to discuss the impact of projection errors in HL-Gaussian on the final performance of the algorithm. In particular, Theorem B.1 reveals a linear relationship between the projection error during each policy evaluation step and the final suboptimality gap, highlighting the importance of reducing this error.

3、About more complex tasks

We have added experiments on two complex and sophisticated control tasks DMControl Figner-Spin and Fish-Swim to showcase the advantages of AHL-Gaussian, please refer to Section G in the appendix for more details.

4、How does the choice of bin quantization(number and varying size) affect the performance of the algorithm? Why not adaptively optimize them as well?

• In term of the number of bins $m$：

First, we have already demonstrated the impact of $m$ on AHL-Gaussian in Figure 8. Figure 8 shows that AHL-Gaussian generally performs robustly regardless of $m$, although there is a slight performance drop on a few tasks when $m$ is set too low.

Furthermore, adaptively optimizing $m$ during training is not practical, as the output of the Q-network is $m$-dimensional. If $m$ changes dynamically, the structure of the neural network would also need to change dynamically, which would undoubtedly pose significant challenges for both training and inference.

• In term of varying size：

We are not entirely sure what the reviewer means by "varying size". We speculate that you might be referring to the bin width or the support interval range.

In fact, when the number of bins $m$ is given, the bin width and the interval range are equivalent. We have already verified in Figure 1 that the performance of HL-Gaussian varies significantly under different interval ranges. Based on this observation, in this paper we proposed the dynamic optimization of the interval range (i.e., bin width) to address the differing requirements of interval ranges across environments and the dynamic evolving process of the value function.

Dear Reviewer egeR:

We sincerely thank you for your efforts and valuable feedback on reviewing our paper. We noticed that you have not yet participated in the discussion, and we would like to ask if you have any additional comments or questions that we can address collaboratively. We have revised the manuscript based on your insightful feedback and are eager to address any remaining concerns.

Please let us know if you have any further thoughts or questions. We look forward to continuing our discussion.

Best regards,

Authors