Latent Trajectory: A New Framework for Actor-Critic Reinforcement Learning with Uncertainty Quantification

[W1]

• Thank you for pointing out the unclear term "unknown". We will change the term "unknown actor network" to "non-fixed actor network", where the existing uncertainty quantification algorithms can only deal with one neural network. When it comes to actor-critic algorithms, the dependency between two network parameter update will introduce additional complexity in uncertainty quantification.
  We are aware of the types of uncertainty: aleatory uncertainty and epistemic uncertainty. Since we are working on the uncertainty of the V-function, which is an expectation of the cumulated rewards, the uncertainty we considered is epistemic uncertainty.

[W2]

• For a given actor network parameter $\theta$, the policy $\pi_\theta$ is fixed. The stationary distribution of transition tuples is uniquely determined by the environment and the policy, hence, we assume that we can drawn transition tuples independently from the stationary distribution $\pi(x|\theta)$.

• Given a policy $\pi_\theta$, the value function $V_\psi$ is learned to approximate its true value $V^{\pi_\theta}$, therefore, the network weights $\psi$ can be viewed as a random variable conditioned on $\theta$. In our paper, $\pi(\psi|\theta)$ represents ``the conditional distribution of $\psi$ given $\theta$''.

• In the setting of online RL, we have access to infinite data from the interaction between the agent and the environment. Like traditional Bayesian inference, if the sample size is infinity, the posterior distribution will degenerate to a point. Hence, the role of the "pseudo-population size" is to prevent the degeneration of the conditional distribution $\pi(\psi|\theta)$, which is denoted by $\pi_\mathcal{N}(\psi|\theta)$.

• Eq(3) integrates out the critic network parameter $\psi$ to mimic the policy gradient objective, while taking account of the uncertainty of $\psi$ conditioned on $\theta$. The uncertainty within $\pi_\mathcal{N}(\psi|\theta)$ comes from the transition generating process and $\psi$ updating process.

[W3]

• Shih & Liang (2024) focuses on DQN type methods, which only involves one deep neural network. We extend the SGMCMC framework to actor-critic methods, which has to consider the convergence of both neural networks. Since two network parameters affect each other during the training, this extension is highly non-trivial.

[W4]

• The coverage rate is generally regarded as the most rigorous metric for uncertainty quantification because it directly evaluates the alignment between predicted uncertainty and actual outcomes. It rigorously tests calibration, ensures reliability in probabilistic predictions, and detects overconfidence or underconfidence in models.

• The PyBullet environment also features a continuous action space and serves as a more implementation-friendly alternative to MuJoCo. The latent trajectory (LT) framework can be seamlessly integrated into existing algorithms, such as PPO, by simply replacing the critic optimization step with our SGMCMC sampling step.
  Moreover, the LT framework can perform uncertainty quantification even after an existing algorithm has converged (post-training inference). This is possible because the LT framework does not require any changes to the network architecture or the introduction of additional parameters, ensuring compatibility and ease of adoption.

General reply
Thank you for providing extensive references, which we have thoroughly reviewed. From our understanding, the uncertainty of value functions must be rigorously quantified in a distributional sense. However, as highlighted in our meta-reply, the uncertainty quantification methods in these papers lack rigorous validation. Specifically, none of the referenced works have verified that the “uncertainty” they claim to measure is accurate. For instance, no evidence is provided to confirm that the confidence intervals they produce achieve the desired coverage rates.

Additionally, these Bayesian methods fail to fully account for the dynamics of the RL system. For example, in Algorithm 1 of the Bayesian Bellman Operator (BBO) paper, samples are continuously generated in a dynamic system but are incorrectly treated as identically distributed. Bayesian methods aim to introduce uncertainty into the MAP estimates by using random priors, which can lead to randomized estimates. However, there are no theoretical guarantees for posterior consistency or for the reliability of downstream inference based on these estimates. The BBO paper adopts a similar strategy, further highlighting this limitation.

[Q1] How does the proposed method fit into the general model-free Bayesian RL framework that precisely characterises uncertainty in value functions? How does relate to the posterior over under these frameworks? If it differs from existing characterizations, ie [1]-[4], what theoretical, algorithmic or empirical advantages does it offer over a full Bayesian approach for characterizing uncertainty in value functions?

[A1] In Algorithm 1 (LT-A2C), as shown in Eq. (14), the actor network update remains identical to the A2C algorithm. The only modification lies in the critic network sampling step. Notably, the LT framework does not require any changes to the network architecture to achieve uncertainty quantification. Similarly, by replacing the critic network update step in PPO, we can extend the framework to obtain LT-PPO.

The advantages of our approach and the limitations of existing Bayesian methods are thoroughly discussed in the meta reply. Additionally, Bayesian methods often face significant challenges when extending their theoretical results to deep neural network settings. These methods typically rely on linear approximation assumptions, which restrict their scalability and practical applicability.

In more complex settings, Bayesian methods frequently resort to using approximated MAP estimates instead of sampling algorithms. This compromise sacrifices their ability to perform uncertainty quantification, undermining a key advantage of Bayesian approaches.

[Q2] Can the authors approach be used to derived Bayes-optimal policies? If not, why does their uncertainty quantification prevent this?

[A2] We can derive Bayesian version of our algorithm by adding prior information on the actor network parameter, and this will not affect the performance of the proposed algorithm in uncertainty quantification. Also, we would like to point out that we proved the convergence (in probability) for the policy network and the weak convergence of the actor network. Adding a prior on the actor network will not change this theoretical network.

[Q3] Can the authors extend their empirical evaluation to include other methods that quantify uncertainty in their value functions?

[A3] We will include comparisons with other actor-critic methods in the uncertainty quantification (UQ) experiments. However, it is important to note that these existing algorithms are typically based on Q-learning methods combined with an actor network, and they do not treat the dynamic system as an integrated whole. In (Shih & Liang, 2024), it was demonstrated that existing Bayesian Q-learning methods struggle to accurately estimate Q-functions and construct reliable confidence intervals. We will make sure to address this limitation in the related works section of our revised manuscript.

[W1] The proposed LTF only compares with the vanilla A2C method, while there have been many AC-based methods proposed, such as [W1,W2]. I recognize that the proposed LTF introduces a new perspective by treating the value parameters as a latent variable, while the experiments are lacking. In particular, the experiments in Section 4.1 are conducted on rather new environments (by Liang et al.). The lack of comparison with other AC methods (e.g., in experiments and/or related works section) makes the results and performance gain less convincing. If the comparison is not possible or not necessary, please clarify the reasons.

response: In [W1], the algorithm is built on linear assumptions, while our algorithm leverages a deep critic network, enabling it to model nonlinear dynamics and adapt to more sophisticated tasks effectively. In [W2], the algorithm is specifically designed for offline RL, which operates under the assumption of a fixed dataset. This is fundamentally different from our online RL setting, where data is dynamically generated during training. These differing assumptions make direct comparisons using the same baseline challenging and less meaningful.

[W2] The proposed LTF is largely based on SGMCMC algorithms that are proposed in Liang et al., 2022a; Deng et al.,2019, while the major contribution of the LTF is less clear. From my understanding, the main contribution is on the A2C settings, which pose unique challenges for uncertainty quantification. It will be very beneficial for the authors to explicitly state the key challenges of applying SGMCMC to A2C settings and how their approach addresses these challenges.

response: First of all, formulating RL as an adaptive SGMCMC problem is highly non-trivial, which needs to conceptualize the transition tuples as latent variables. Consequently, this leads to an ideal transition-trajectory independent algorithm for RL. We united the 2-step optimization problem (policy control and policy evaluation) to solving a single integral equation (eq.(3)).

Secondly, applying the adaptive SGMCMC algorithm to the A2C setting is highly challenging. Specifically, the gradient evaluation for $\nabla_{\psi_k} \log \pi_{\mathcal{N}}(\psi_k|\theta_{k-1})$ is highly non-trivial, which involves an importance weighting step and simulation of auxiliary samples.

[Q1] What is the parameter in Eqn. 10?
[A1] Equation (10) is used to draw auxiliary samples of $\tilde{\psi}$, and $\tilde{\psi}$ can be understood as its parameter.

[Q2] What are the relation between $\epsilon_{k,l}$ and $\epsilon_k$ in Algorithm 1?
[A2] Sorry, it is typo: $\epsilon_{k,l}$ needs to be replaced with $\epsilon_k$.

[Q3] How to select $\mathcal{N}$ in practice?
[A3] This is not a critical issue for the proposed algorithm. It can be chosen as a reasonably large number, ensuring the resulting confidence interval to be reasonably narrow as the simulation goes on.

[Q4] Why the variance in Figure 7 for LT-A2C is not significantly reduced if the uncertainty is effectively evaluated during update (e.g., HopperBullet, Evaluation reward)?
[A4] The variance of the reward can be influenced by various factors, including update frequency, mini-batch size, and environment complexity, among others. The primary focus of this experiment is to demonstrate that incorporating the Latent Trajectory (LT) framework does not negatively impact the practical performance of the actor-critic algorithm. Importantly, the LT framework achieves accurate uncertainty quantification without compromising the algorithm’s effectiveness or efficiency.

[W4] Connecting uncertainty quantification to performance. While the escape environment discusses the relationship of how LTF leads to better MSE of value functions, how this translates to better performance in the escape environment is not clear. Conversely, on other environments, the paper does not provide a clear analysis of how the uncertainty quantification of Q-values leads to better performance.

response: Uncertainty quantification (UQ) is crucial for understanding the entire RL system. In the indoor escape environment, we observe that an accurate value function approximation, indicated by a low mean squared error (MSE), facilitates more diverse action exploration across optimal policies (see Figure 3). In contrast, bias in value function estimation negatively impacts the policy control step, causing the policy network to converge prematurely to a local minimum without adequately exploring other optimal policies, as illustrated in Figure 4. In the PyBullet experiments, the only distinction between A2C and LT-A2C lies in the policy evaluation step. The performance improvements achieved by LT-A2C highlight the advantage of accurate policy evaluation, enabled by the Latent Trajectory framework.

[W5] It is also not so clearly apparent that LT-A2C has smaller seed variability than A2C, as the paper claims. The performance difference is largely imperceptible in the plots, considering the confidence intervals.

response: Our two experiments focus on different aspects of the proposed Latent Trajectory (LT) framework:

• Theoretical Validation in the Indoor Escape Environment:
  In this experiment, we demonstrate the theoretical rigor of the LT framework. Specifically, we show that LT-A2C achieves the expected coverage rate of 95% for the confidence intervals of the value function $V^*(s)$. This validates that our framework provides reliable uncertainty quantification, a critical requirement for theoretical soundness.

• Performance Improvement in Pybullet:
  The second experiment focuses on demonstrating the practical performance of the Latent Trajectory (LT) framework. By integrating the LT framework into an actor-critic algorithm and evaluating it on a suite of Pybullet tasks, we show that the framework maintains or slightly improves policy performance compared to standard actor-critic methods.

The key takeaway is that the LT framework achieves correct uncertainty quantification without compromising performance. While performance improvements are not the primary focus, our results underscore that accurate uncertainty estimation can be achieved without trade-offs in algorithm efficiency or effectiveness.

[Q1] Can you build a connection to prior works in uncertainty quantification?
response: The connections with Bayesian RL and distributional RL have been established in our replies to your comment W1 and the meta reply. We will add the related work to our revision shortly.

[Q2] How would you make this paper more approachable to a wider audience? Currently, it requires knowing a lot of prior work to make sense of the motivation, algorithm, and convergence proofs. The paper should be largely self-sufficient when reading.
response: Due to the page limit, we will incorporate your comments into the revision by adding extra materials into the Appendix.

[W1] Lack of related works and comparison. It is hard to position this paper in the context of prior work as it lacks a discussion of how this work relates to existing RL algorithms that model the uncertainty of Q-values. The paper should discuss how this work is different from distributional RL methods, which also model the uncertainty of Q-values. The paper should also discuss why SGMCMC is the suitable method for the problem at hand, in contrast to prior work. This made the paper particularly hard to understand as a reader.
response: Refer to our common reply regarding Bayesian RL. Bayesian RL and distributional RL (e.g., QR-DQN) address different aspects of uncertainty: Bayesian RL quantifies the uncertainty in Q-networks, whereas QR-DQN focuses on quantifying the uncertainty of rewards. In this paper, we focus exclusively on quantifying value function uncertainties; therefore, distributional RL falls outside the scope of this work. Furthermore, as demonstrated in Shih & Liang (2024), QR-DQN fails to construct accurate confidence intervals for true Q-values. Conversely, Bayesian methods heavily rely on techniques such as bootstrapping or Gaussian processes (suitable primarily for linear representations), which are challenging to extend effectively to deep neural networks both theoretically and numerically.

The proposed adaptive SGMCMC algorithm is particularly suitable for RL because of the following facts:

• Comprehensive Handling of RL Dynamics:
  The proposed algorithm fully accounts for the dynamic nature of reinforcement learning. At each time step, new transition tuples are generated based on the updated policy, leading to a continuously evolving distribution of transition tuples. Consequently, the target distribution for the critics in the SGMCMC method also changes at every time step due to updates in the actor network. We have proven the convergence of the actor network, which in turn ensures the convergence of the critic network’s distribution.

• Independence from Transition Trajectories:
  Our algorithm treats transition tuples as latent (or auxiliary) variables, integrating them out when updating the actor network. This approach guarantees that the convergence of the actor network is independent of the transition trajectory. In contrast, many existing reinforcement learning methods, such as Bayesian RL, rely heavily on trajectory-dependent computations.

• No Additional Complexity or Structural Changes:
  The proposed algorithm does not require modifications to the network architecture and maintains the same computational complexity as existing methods. This simplicity makes it easily applicable to large-scale problems and real-world applications.

[W2] The problem of uncertainty quantification is not well-motivated. The paper does not provide a clear motivation for why it is important to model the uncertainty of Q-values. It mentions "accurately quantifying the uncertainty of the value function has been a critical concern for ensuring reliable and robust RL applications", but does not provide any concrete examples of why this is the case or precisely in what scenarios this is important. Ideally, the paper should analyze the limitations of existing RL algorithms that do not model the uncertainty of Q-values and provide examples of scenarios where this leads to suboptimal performance, and how the proposed method addresses these limitations in experiments.

response: Thank you for your suggestion. The importance of uncertainty quantification has been well addressed in existing works like distributional RL and Bayesian RL. However, the "uncertainty" is not rigorously defined and verified. In this paper, we provide a much more effective method than the existing ones in uncertainty quantification; specifically, our method results in highly accurate coverage rates for value function.

[W3] No results on PPO The paper mentions that PPO suffers from severe miscalibration issues in both the actor and critic, but does not provide any results on PPO to demonstrate this.
response: Sorry for the confusion, we intend to state that the PPO leads to less accurate coverage rates and higher prediction error for value function. This has been demonstrated in Table 1 and Figure 6.

3. Rigorously define and verify the uncertainty of critic network estimation

In this paper, we rigorously define the uncertainty in the critic network’s estimation in a distributional sense. Specifically, we argue that the value function estimates produced by the critic network should form a non-degenerate distribution centered around the true value function of the optimal policy. Latent Trajectory framework theoretically guarantees the critic network to converge weakly to a stationary distribution $\pi(\psi|\theta^*)$ corresponding to the optimal policy $\pi_{\theta*}$.

Moreover, our experiments demonstrate that our algorithm consistently achieves the nominal 95% coverage rate for the 95% confidence intervals of the value function. This empirical result ensuring the reliability of the uncertainty estimates produced by our algorithm.

Appendix

As shown in Table 1, our approach achieves accurate coverage rates for the value function, demonstrating both its correctness and effectiveness; in contrast, none of the existing methods yield accurate coverage rates.

Table 1

Reference

[1] Shih, F., & Liang, F. (2024). Fast Value Tracking for Deep Reinforcement Learning. In The Twelfth International Conference on Learning Representations.

[2] Osband, I., Aslanides, J., & Cassirer, A. (2018). Randomized prior functions for deep reinforcement learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (pp. 8626–8638). Curran Associates Inc., Red Hook, NY, USA.