Generalized Gaussian Temporal Difference Error for Uncertainty-aware Reinforcement Learning

On Risk-averse Weighting

Related to Weakness 3 and Question 2

Theorem 2 establishes second-order stochastic dominance (SSD) among GGD random variables, where distributions with larger $\beta$ values are preferable under a risk-averse framework due to their more less spread-out and predictable nature. The connection to Equation (4), $\omega^\text{RA}_t=Q^\beta_t$, is as follows:

1. Theorem 2 states that for two GGD variables $X_1\sim\text{GGD}(0,\alpha,\beta_1)$ and $X_2\sim\text{GGD}(0,\alpha,\beta_2)$ with $\beta_1 \leq \beta_2$, $X_2$ exhibits SSD over $X_1$. Intuitively, this means that larger $\beta$ values lead to tighter, less dispersed distributions.
2. Considering the objective in risk-averse weighting, we prioritize more predictable samples (less dispersed, smaller aleatoric uncertainty) by assigning weights proportional to $\beta$. A direct application of SSD implies that higher $\beta$ values should correspond to higher weights, encouraging the model to focus on less noisy samples.
3. Now formulating the weighting according to above, to balance these considerations in Equation (4), we define $\omega^\text{RA}_t=Q^\beta_t$, where $Q^\beta_t$ represents the local $\beta$ value for the TD error at step t. This weighting ensures that samples with higher $\beta$ (less spread-out distributions) are weighted more heavily, aligning with the risk-averse preference established by Theorem 2. In addition, by normalizing the weight of each sample in a batch relatively, weighting ensures that the model appropriately accounts for the reliability of the estimate from each data point, resulting in robustness against inaccuracies in $Q^\beta_t$ estimate.

This weighting scheme aligns with SSD principles, leveraging GGD properties to construct a risk-averse weighting strategy that prioritizes less noisy samples.

On Temporal Difference Error Distributions

Related to Weakness 1 and Question 1

The mismatch of the Gaussian fitted PDF and the similar fitted variance in Hopper-v4 are due to the Gaussian distribution's inability to capture the higher-order moments of the empirical histograms. These observations underscore one of the paper's key motivations, the inadequacy of assuming normally distributed TD errors, and necessitate a more flexible modeling framework, such as GGD-based error modeling. The GGD explicitly incorporates a shape parameter $\beta$ that captures kurtosis and tail behavior, enabling it to model distributions like those in Hopper-v4, where higher-order moments play a critical role in distinguishing between training phases.

We agree that the NLL objective optimizes the distribution of TD errors for individual state-action pairs, not the global distribution. However, Figure 2 is intended to provide approximates of the overall trend, showing that even when aggregated, the TD errors exhibit significant deviation from normality. This aggregated view is a useful diagnostic for highlighting general trends, such as non-Gaussian characteristics and changes in tailedness over time, which motivate the need for a more expressive distributional model at the local level. Although such a plot cannot fully reflect the variability of individual distributions at each time step $t$, it effectively showcases the widespread non-normality of TD errors. Please note that we leveraged the property that the sum of GGD samples also follow GGD, despite the fact that aggregated generalized Gaussian distributed does not necessarily imply the generalized Gaussian property of each sample, to further supporting the aggregation of TD errors in Figure 2 to highlight the relevance of GGD modeling.

On Uncertainty Dynamics

Related to Weaknesses 2 and 3

Thank you for the opportunity to clarify.

We define "unexpected states and rewards" as regions encountered during exploration that deviate from states and rewards sampled by the agent's policy during convergence. Empirical support for this hypothesis is presented in Figure 2, which illustrates the evolution of TD error distributions during training. Specifically, in earier phases, broader and heavier-tailed distributions reflect the agent's exploration of less familiar states and rewards. Figures 6 and 7 also support these trends across different environments.

We agree that the explanation in Line 278 could benefit from additional evidence. The claim regarding aleatoric and epistemic uncertainty dynamics is supported by:

1. Theoretical Support: As par Kendall & Gal, 2017, epistemic uncertainty typically diminishes as the agent collects more data, narrowing the parameter space for value function approximation. On the other hand, aleatoric uncertainty, arising from environmental stochasticity, remains irreducible and becomes more prominent as training progresses.
2. Empirical Evidence: Figures 4 and 10 provide evidence for this interplay. The convergence of $\beta$ estimates toward lower and leptokurtic values (Figure 10) and the reduced coefficient of variation for $\beta$ (Figure 4) indicate stabilized estimates of tail behavior as epistemic uncertainty decreases during training. Specifically, Figure 4 illustrates the coefficients of variation (CV) for parameter estimates of $\beta$ during training. CV is defined as the ratio of the standard deviation to the mean of parameter estimates, offering a scale-invariant measure of stability. The CV values for $\beta$ consistently decrease as training progresses across all environments, reflecting greater stability in the estimation of the shape parameter. Epistemic uncertainty, which arises from a lack of sufficient data or exploration, manifests as high variance in parameter estimates during early training phases. As training progresses and the agent collects more data, this uncertainty diminishes, leading to more stable and consistent estimates of $\beta$, as shown by the reduced CV. On the other hand, Figure 6 and 7 tracks the evolution of TDE over training epochs for SAC and PPO variants. Across all environments, the experiments demonstrate that the TD error distributions become more leptokurtic (heavy-tailed) as training progresses, with decreasing $\beta$ value. Additionally, Figure 10 and our additional experiments suggests that in most cases, $\beta$ estimates decrease and converge within the leptokurtic bound. This trend indicates that the TD error distributions become more leptokurtic (heavy-tailed) as training progresses. The closed form of aleatoric uncertainty suggest that aleatoric uncertainty exhibits a negative proportionality to the shape parameter $\beta$ on an exponential scale, thus aligning with the hypothesis that aleatoric uncertainty increasingly dominates the error structure.

We revised the relevant paragraphs accordingly.

Regarding Equation (8), our BIEV mechanism addresses bias by assuming constant approximation bias in TD errors, as suggested in Flennerhag et al., 2020. This assumption allows us to focus on variance estimation to minimize the impact of bias on the overall uncertainty-aware objective.

On the Theoretical Contributions

Related to Weakness 4 and Question 3

We appreciate the reviewer's insights and acknowledge the need to delineate our theoretical contributions more clearly. While our work builds upon existing mathematical tools like the generalized Gaussian distribution (GGD) and uncertainty modeling in temporal difference (TD) learning, it introduces novel contributions specific to reinforcement learning (RL):

• Empirical Demonstration of Limitation in Prior Models: Section 3.1.1 empirically shows significant deviations from Gaussian assumptions of TD error distributions across various RL tasks, highlighting the necessity for more flexible modeling. This foundational observation was not established or systematically analyzed in prior works.
• Novel Application of GGD for Uncertainty Mitigation: We extend the GGD to aleatoric uncertainty modeling, providing a closed-form relationship between the shape parameter $\beta$ and uncertainty (Section 3.1). This integration, tailored for heteroscedastic TD errors, advances beyond traditional variance-based approaches.
• Theoretically Grounded Weighting Scheme: As detailed in Section 3.2, we propose a new batch inverse error variance (BIEV) weighting mechanism that incorporates kurtosis considerations for epistemic uncertainty. This theoretical enhancements refines existing weighting strategies by explicitly accounting for tail behavior.
• Risk-Averse Weighting via Stochastic Dominance: Leveraging second-order stochastic dominance properties of GGD (Theorem 2), we develop a risk-averse weighting strategy. This is a novel integration of SSD principles into RL for robust performance under heteroscedastic conditions.
• Broader Insights for Robust and Risk-Sensitive RL: Our insights provide theoretical and practical tools for designing robust RL algorithms that effectively handle tailed error distributions and improve epistemic uncertainty estimation (Section 5).

In summary, while some theoretical tools stems from prior work, their specific integration into TD learning and reinforcement learning tasks, along with the proposed theoretical and practical innovations, constitute the primary contributions of this work. These developments improve performance significantly and address critical limitations in existing uncertainty-aware RL methods, as evidenced by our experimental results in Section 4.

On Higher-Order Moments and Task Characteristics

Related to Question 7

Thank you for this thoughtful question. Below, we address how the method's performance relates to the characteristics of the TD-error distribution, including the role of higher-order moments, and clarify whether the method demonstrates greater improvements in environments with more pronounced or subdued higher-order moments.

The proposed method leverages the shape parameter $\beta$ in the GGD to explicitly model higher-order moments, such as kurtosis. This capability enables the method to effectively handle TD-error distributions with heavy tails (high kurtosis), outperforming Gaussian-based approaches that fail to account for extreme errors adequately. In environments with larger tails, i.e., leptokurtic, the method demonstrates significant performance improvements by explicitly addressing tail behavior. Unlike baseline Gaussian models, which often underestimate or ignore the impact of extreme errors, the proposed approach focuses on less spread-out samples, enhancing robustness to noisy or outlier TD errors. This is empirically evident in Figure 2, where environments such as Hopper-v4 and Ant-v4 exhibit heavy-tailed TD-error distributions and benefit significantly from the proposed method.

In contrast, for smaller tails (platykurtic distributions), the advantages of modeling kurtosis are less pronounced. However, the method remains effective due to its dynamic weighting scheme, which adjusts to the lower kurtosis by de-emphasizing higher-order moment terms when they are less relevant. This adaptability prevents overfitting to low-noise regions, a common limitation of variance-based baselines. As a result, the proposed method maintains robustness by continuously adapting to the evolving characteristics of the TD-error distribution.

Detailed Explanation of Equation 3

Relelated to Question 1

We appreciate the reviewer highlighting the need for a more self-contained explanation of Equation (3). Among other parts, we acknowledge that Equation (3) may lack sufficient context, and we make the explanation more self-contained.

On Epistemic Uncertainty Mitigation

Related to Questions 2 and 3

We appreciate the opportunity to clarify the role of ensembles in our method and their connection to epistemic uncertainty estimation.

Ensembles are employed in our framework to estimate epistemic uncertainty, which arises from limited exploration, i.e., insufficient data in the state-action space. Specifically, we use an ensemble of $k$ critics, each independently trained using the same TD error data but initialized with different random seeds. The variance among these critics' predictions $\mathbb{V}[\delta_t]$ serves as an empirical measure of epistemic uncertainty, capturing the disagreement across the ensemble. This ensemble-based variance is then used in the BIEV weighting in Equation (8). By penalizing high-variance predictions, the method prioritizes state-action pairs with more reliable, i.e., low-variance, estimates, improving robustness.

On Risk-averse Weighting

Related to Weakness 3 and Questions 2 and 4

Theorem 2 establishes second-order stochastic dominance (SSD) among GGD random variables, where distributions with larger $\beta$ values are preferable under a risk-averse framework due to their more less spread-out and predictable nature. The connection to Equation (4), $\omega^\text{RA}_t=Q^\beta_t$, is as follows:

1. Theorem 2 states that for two GGD variables $X_1\sim\text{GGD}(0,\alpha,\beta_1)$ and $X_2\sim\text{GGD}(0,\alpha,\beta_2)$ with $\beta_1 \leq \beta_2$, $X_2$ exhibit SSD over $X_1$. Intuitively, this means that larger $\beta$ values lead to tighter, less dispersed distributions.
2. Considering the objective in risk-averse weighting, we prioritize more predictable samples (less dispersed, smaller aleatoric uncertainty) by assigning weights proportional to $\beta$. A direct application of SSD implies that higher $\beta$ values should correspond to higher weights, encouraging the model to focus on less noisy samples.
3. Now formulating the weighting according to above, to balance these considerations in Equation (4), we define $\omega^\text{RA}_t=Q^\beta_t$, where $Q^\beta_t$ represents the local $\beta$ value for the TD error at step t. This weighting ensures that samples with higher $\beta$ (less spread-out distributions) are weighted more heavily, aligning with the risk-averse preference established by Theorem 2. In addition, by normalizing the weight of each sample in a batch relatively, weighting ensures that the model appropriately accounts for the reliability of the estimate from each data point, resulting in robustness against inaccuracies in $Q^\beta_t$ estimate.

This weighting scheme aligns with SSD principles, leveraging GGD properties to construct a risk-averse weighting strategy that prioritizes less noisy samples.

On Beta Estimation

Related to Question 5 and 6

Thank you for this insightful question regarding the use of $\beta$ as the sole estimated parameter in our framework and trade-offs involved.

While the GGD has three parameters, our method simplifies the problem by fixing $\alpha = 1$, resulting in computational efficiency and training stability. RL tasks are resource-intensive, and simultaneous optimization of $\alpha$ and $\beta$ introduces additional complexity and potential instability.

While fixing $\alpha$ reduces flexibility, $\beta$ still captures kurtosis and indirectly controls variance. This trade-off is reasonable in practice, as $\beta$ alone provides sufficient expressiveness for most RL scenarios, particularly those with heavy-tailed distributions. In addition, since $\alpha$ and $\beta$ are interdependent, e.g., variance $\sigma^2$ depends on both $\alpha$ and $\beta$, joint optimization may result in parameter updates that are inconsistent with each other, leading to poor uncertainty modeling. Our experiments in Figures 15 empirically support the choice of fixing $\alpha$. It is worth noting that the practical impact from fixing $\alpha$ is minimal, as $\beta$ still adapts dynamically to changes in TD-error distributions.

On the implementation

Related to Weakness 1

We appreciate the reviewer’s concern regarding the complexity of incorporating higher-order moments. To mitigate this challenge, our method is designed with simplicity and practicality in mind.

As mentioned in Remark 2, we only employ $\beta$ estimation for GGD error modeling, setting $\alpha=0$, to minimize computational overhead while preserving the expressiveness needed to capture kurtosis. Such simplification ensures the implementation aligns closely with existing variance-based methods, requiring minimal modifications to architecture or training pipelines. Additionally, empirical results, as shown in Figures 4, 8, and 10, demonstrate both coefficients of variation (CV) and estimated $\beta$ values converge stably, indicating the stable training process of our method.

On Task Applicability

Related to Question

Thank you for raising this important question regarding real-world applications. The proposed approach is particularly well-suited for tasks and environments requiring robust decision-making under uncertainty and non-Gaussian error distributions. Below, we provide examples of such scenarios from two key application areas:

1. Robotics: Tasks such as robot control, navigation, and manipulation in dynamic or unstructured environments often encounter:

   • Aleatoric Uncertainty: Stochastic dynamics, sensor noise, and imperfect actuators generate heavy-tailed error distributions.
   • Epistemic Uncertainty: Unexplored regions of the state space contribute to uncertainty during training.

   By dynamically modeling kurtosis, our method improves policy robustness, enabling robots to handle stochastic and uncertain scenarios more effectively.

2. Finance: Reinforcement learning applications in portfolio optimization and trading strategies involves:

   • Heavy-tailed Distributions: Financial returns and risk metrics often exhibit non-normal characteristics due to rare but impactful market events.

   • Uncertainty: Limited historical data and scarce market information introduce both aleatoric and epistemic uncertainties.

   By modeling the evolving tailedness of TD errors, our framework enhances risk-sensitive strategies, enabling better adaptation to volatile market conditions while managing risk effectively.

On Sensitivity to Reward Scales and Bounds

Related to Weakness 4 and Question 2

Our method focuses on the distribution of TD errors, which are not directly related to the reward scale. This is because estimation of TD errors is less biased than the reward (Flennerhag et al., 2020). Therefore, the sensitivity of the parameter estimation to reward scales is less of a concern in our method.

On the Contribution of the Paper

Related to Weaknesses 2 and 3

We acknowledge the reviewer's concern about the novelty of our work and the effectiveness of the proposed method.

Our work is motivated by the limitations of conventional temporal difference (TD) learning methods that assume a zero-mean Gaussian distribution for TD errors. Specifically focusing on the shape of the TD error distribution, we propose a novel uncertainty-aware objective function that minimizes the negative log-likelihood of the generalized Gaussian distribution (GGD) of the TD errors. From the existing literature on uncertainty-aware reinforcement learning (RL), which utilizes variance head for uncertainty estimation (Kendall & Gal, 2017, Mai et al., 2022), we extend the method to include higher-order moments, specifically kurtosis, to enhance the description of the TD error distribution.

We investigate this extension both empirically and theoretically, providing insights into the effects of higher-order moments on the TD error distribution. Based on those results, implications of our work to mitigate both aleatoric and epistemic uncertainties in TD learning lead to improved performance across various settings.

Contrary to concerns about increased complexity, our method requires only a single additional network head to estimate $\beta$, as conventional variance estimation requires. This makes the proposed method straightforward to integrate into existing architectures. We believe this simplicity is a strength of our method, as it allows practitioners to easily extend their uncertainty-aware TD learning methods to include higher-order moments, without significant changes to the existing framework.

On the Choice of the Generalized Gaussian Distribution

Related to Weakness 1 and Question 1

We appreciate the reviewer's comment on the choice of the GGD and the potential for exploring other distributions.

The GGD was selected due to its flexibility in capturing varying tail behaviors through the shape parameter $\beta$. This aligns well with the observed heavy-tailed distributions in TD errors, as shown in Figures 2, 6, and 7. While alternatives such as $q$-Gaussian distributions could also be considered, they introduce similar parameter estimation challenges without offering significant advantages over the GGD.

Methods like particle-based distributional RL (Bellemare et al., 2017, Nguyen et al., 2020) model the entire return distribution, often focusing on its variance. Our work differs in emphasizing the shape of the TD error distribution, particularly its tailedness, which is crucial for understanding the uncertainty in TD learning. Furthermore, parametric approaches are computationally more efficient, making them more suitable for online learning in RL.

We agree that incorporating skewness or handling multimodal distributions are also interesting extensions. However, our experimental results indicate that TD error distributions are unimodal in our setups, making GGD a practical and effective choice for this work. Additionally, as we hypothesized in Lines 273-276, exploration in RL can lead to heavy-tailed distributions, making the tailedness of the distribution more critical for understanding the uncertainty in TD learning than skewness.

On Training Stability

Related to Question 3

We try to show the stability of the training process by providing coefficients of variantion (CV) of the $\beta$ estimates over time in Figures 4 and 8. The CV of $\beta$ estimation are lower than variance estimation, which indicates greater stability in estimating the shape parameter of the GGD. Please note that the convergence of $\beta$ estimation, given in Figure 10, is also more stable than variance estimation.

To further ensure stability, we employed the following techniques (Note the Implementation section in Appendix C.):

1. Normalized Parameter Range: We applied the softplus function, a smooth approximation to the ReLU function (Dugas et al., 2000), to the outputs, to constrain $\beta$ is constrained within a range to avoid extreme values that could destabilize training
2. Simplified Loss Computation: We modify the NLL loss for the GGD by employing $\mathbb{Q_\beta}$ as a multiplier rather than as an exponent, to reduce computational overhead and mitigate numerical instability.

These measures, combined with empirical evidence provided in the paper, support the robustness and stability of the optimization process.