Dual Critic Reinforcement Learning under Partial Observability

Thank you for your valuable feedback. We provide the following clarifications in response to your comments.

Weakness 1: About the insight.

... the proposed method is somewhat derivative of two other well-known methods. ... there isn't enough insight ...

In this study, we address the issue of high variance stemming from an over-reliance on state information. We demonstrate that state information is not always advantageous for training purposes. In many environments, relying solely on partial observations can yield superior results compared to using state information.

Incorporating state information can offer significant advantages for training, as it often encapsulates high-level features that improve the learning of $V(h,s)$ compared to $V(h)$. For example, in an extreme scenario where the state information $s$ represents the true value of $V(h)$, the agent can effortlessly infer this true value through $V(h,s)$. Specifically, in the Empty Navigation task, when state information indicates the agent's relative distance to the goal, the value function learning process is significantly streamlined. In addition, the reward function is defined based on the state rather than the observation. Consequently, learning for the standard critic necessitates implicit consideration of the distribution $b(s|h)$. In contrast, the oracle critic bypasses this requirement by directly learning the reward, as the distribution is explicitly provided through the sampling operation during the policy gradient calculation.

However, state information is not always advantageous for training. When the state information contains significant noise, adding excessive noisy features to the input can hinder learning the value function. In unfamiliar environments, whether state information genuinely supports the learning process remains to be determined.

To address this issue, we introduce a DCRL framework that leverages the strengths of two critics. We provide theoretical evidence indicating that DCRL mitigates the learning variance while maintaining unbiasedness. Moreover, our approach incorporates a dynamic weighting mechanism, which can be interpreted as a form of lower-bound-soft-Q-learning, distinguishing it from conventional linear weighting methods. Compared to Recurrent AC and UAAC methods, our proposed DCRL demonstrates substantial improvements.

Weakness 2 and Question 1: About the variance reduction in policy gradient.

... the variance of the policy gradient Monte-Carlo estimator ... how the dual critic architecture affects the variance of the policy gradient estimator?

We fully understand your concerns regarding the variance of the policy gradient Monte Carlo estimator. Traditional Actor-Critic methods employ the reward-to-go term to compute the policy gradient. Nonetheless, the variance of the reward-to-go tends to be high, which has led researchers to substitute this term with the Q-value to mitigate variance in the standard policy gradient method, i.e., $Q(h_t, a_t) \nabla_{\theta}\log\pi_{\theta}(a_{t}|h_{t})$. Similarly, in our study, we focus on the policy $\pi(a|h)$, which is linked to a unique $Q(h, a)$ but can also correspond to multiple $Q(h, s, a)$ values due to the non-one-to-one relationship between state and history. Consequently, the variance of the policy gradient derived from $Q(h, a)$ is lower than that obtained from $Q(h, s, a)$. As stated in Theorem 1 of our paper, $Q_{\mathrm{dual}}(h, s, a)$ in DCRL achieves a reduction in this variance compared to $Q(h, s, a)$ by integrating the standard critic, thereby decreasing the variance of the policy gradient.

Weakness 3 and Question 2: About the $\beta$.

... reduce variance all the way to zero by setting $\beta=1$ ... when to clip $\beta=0$ ... the motivation behind the clipping of $\beta$ ... How was the clipping method for $\beta$ chosen?

In response to weakness 1, we elaborated on the advantages of employing the oracle critic $V(h, s)$, which incorporates state information for policy training.

The motivation behind the clipping mechanism is to leverage the standard critic when opportunities for policy enhancement remain following updates from the oracle critic. This approach can be interpreted as a form of lower-bound soft Q-learning, wherein only samples exhibiting a positive advantage contribute meaningful insights into the optimal soft Q-value. To validate this concept, we implemented a simplified weighting method that employs fixed ratios to weight the two critics (referred to as DCRL No Clip Version). As illustrated in Figure 1 of the rebuttal PDF, this method shows performance improvements in only certain environments, confirming that the dynamic weighting mechanism of DCRL is crucial.

To evaluate the impact of the parameter $\beta$, we conducted ablation experiments testing various values of $\beta \in \\{1/5, 1/3, 1/2, 2/3, 4/5\\}$. The choice of $\beta$ demonstrates notable stability. This stability is primarily attributable to the characteristics inherent in lower-bound soft Q-learning.

Weakness 4:

... the experiments would also benefit from stronger baselines...

Thank you for this valuable suggestion. We have followed your advice and considered the commonly used POMDP baseline DreamerV3.  We compared the performance of DCRL, DreamerV3, and additional baselines across eight tasks in the MiniGrid environment. Figure 3 of the rebuttal PDF illustrates the average scores from five training seeds over $1e5$ steps for each method. The results indicate that DCRL consistently outperforms DreamerV3 across all eight tasks. Importantly, DCRL requires fewer computational resources and less training time than DreamerV3.

Thank you for your insightful and thoughtful reviews, which have significantly improved the quality of our paper. If you find that these concerns have been resolved, we would appreciate it if you would reconsider your ratings of our paper.

Thanks again for your time and effort in reviewing our paper! As the discussion period is coming to a close, we would like to know if we have resolved your concerns expressed in the original reviews. We remain open to further feedback and are committed to implementing additional improvements if necessary. If you find that these concerns have been resolved, we would appreciate it if you would consider reflecting this in your rating of our paper!

Thank you for your constructive comments and suggestions. Below, we provide detailed responses to each point.

Weakness 1:

... harsh jump-discontinuity ... numerical instabilities.

We fully understand your concerns. The numerical instability arises primarily from the variance of $V(h, s)$ due to the expectation of $V(h, s)$ is equal to $V(h)$. Figure 8 in the appendix illustrates that our DCRL demonstrates a more stable value distribution and experiences fewer numerical instabilities than UAAC, attributable to its reduced variance.

Weakness 2:

... degenerate case of ct-de with N=1.

We appreciate the valuable insights you have provided. The degenerate case of CTDE when $N=1$ corresponds to a standard POMDP. In this context, the well-known multi-agent actor-critic algorithm COMA, utilizes both state information ($s$) and history ($h$) as inputs for the critic during centralized training. Conversely, during execution, the actor relies solely on history as its input. This situation can be interpreted as a specific instance of the UAAC framework. We have incorporated relevant discussions on this topic in the revised manuscript.

Weakness 3.1 and Question 3:

... when one could expect this method to be most useful.

What environments should we expect the method to do well in (and not so well)?

Incorporating state information can offer significant advantages for training, as it often encapsulates high-level features that improve the learning of $V(h,s)$ compared to $V(h)$. For example, in an extreme scenario where the state information $s$ represents the true value of $V(h)$, the agent can effortlessly infer this true value through $V(h,s)$. Specifically, in the context of the Empty Navigation task, when the state information indicates the agent's relative distance to the goal, the value function learning process is significantly streamlined. Furthermore, the reward function is defined based on the state rather than the observation. Consequently, the standard critic's learning necessitates implicit consideration of the distribution $b(s|h)$. In contrast, the oracle critic bypasses this requirement by directly learning the reward, as the distribution is explicitly provided through the sampling operation during the policy gradient calculation.

However, state information does not always enhance training. When the state information contains significant noise, introducing excessive noisy features to the input can hinder learning the value function. In unfamiliar environments, whether state information genuinely supports the learning process remains to be determined.

The motivation behind this work is that state information may be accessible during training in many scenarios. However, introducing a state could potentially lead to bias or variance. In most cases, state information offers higher-level features that assist agents in decision-making. In these situations, DCRL enhances the process by further reducing variance. Conversely, when state information and partial observations correspond one-to-one, the oracle critic demonstrates no variance; DCRL may become ineffective under these circumstances.

Weakness 3.2

... more information may slow training ... do not have access to the full global state ...

The scope of our study encompasses situations where state information is accessible during training. This assumption is prevalent in previous works and reflects real-life common scenarios.

We provide the runtime in seconds for each method below regarding the potential issue of increased training duration due to additional information. DCRL requires more time to execute compared to other baselines, primarily because it trains an additional critic network. Nevertheless, DCRL demonstrates significantly higher sample efficiency and performs better with fewer environment steps, thus compensating for the slower wall time.

Table 1: Comparison of DCRL and other baselines in MiniGrid. The first number represents the mean ± std of performance, while the second number indicates the runtime in seconds.

Question 1:

... $V^{\pi}(h)$ has nonzero variance.

We apologize for the misunderstanding caused by a typo in our paper. In line 155, we clarify that $V(h,s)$ possesses nonzero variance, while $V(h)$ has zero variance.

Question 2:

How does the method perform over a range of values of $\beta$?

To address your concerns, we conducted ablation experiments on the values of the $\beta$ parameter ($\beta \in \\{1/5, 1/3, 1/2, 2/3, 4/5\\}$). Figure 1 of the rebuttal PDF illustrates that the choice of $\beta$ exhibits notable stability. This stability primarily arises from the characteristics of lower-bound-soft-Q-learning. Consequently, we selected $\beta = 1/2$ as the parameter for reporting our results in the paper, as it consistently exhibits superior performance.

Thank you for your thoughtful review and insights into our work, which have significantly improved the quality of our paper. If you find that these concerns have been resolved, we would appreciate it if you would reconsider your ratings of our paper.

I would like to thank the authors for their detailed responses to my comments. I would also like to congratulate the authors for having produced useful additional results, particularly the ablation on $\beta$ in the short space of time.

My opinion is that overall the paper has merit and the additional analyses have allayed some of my concerns. In light of the fact that parts of the main insights exist albeit within the multi-agent reinforcement literature and that the corresponding methodology may be applied in the degenerate, single-agent case means that the contribution is somewhat constrained. In MARL, there are also techniques to learn how best to interpolate or switch between using a value function with global state inputs and that with local inputs which are worth mentioning e.g. [1]. This is an important consideration as it is likely not a priori known which environments would benefit from global information during training should this information be available. For this reason, I will keep my score as is.

Other points

• I recommend the authors use a different notation for the value functions $V^\pi(h,s)$ and $V^\pi(h)$ and similarly for the action-value functions.

• I think my earlier point about jump-discontinuities can be addressed by showing that $V^\pi_{\rm dual}$ is a smooth function w.r.t. the update parameters of $V^\pi(h,s)$ and $V^\pi(h)$ whenever the latter two functions are differentiable and, showing the corresponding statement for $Q^\pi_{\rm dual}$ .

[1] Mguni, David Henry, et al. "Mansa: Learning fast and slow in multi-agent systems." International Conference on Machine Learning. PMLR, 2023.

Thank you for your prompt response. We also appreciate your recognition of our supplementary experiment and are glad to have addressed some of your concerns. We will incorporate your suggestions regarding notation and jump discontinuities in our revision of the manuscript. For your remaining questions, we provide point-to-point responses:

Q1: Regarding the relevance to MARL problems.

We apologize for the misunderstandings that have arisen. The setup presented in our paper differs from the MARL problem formulation.

In MARL, the optimization objective is to maximize the reward under the global states, optimizing each agent accordingly. These methods mostly presuppose that the combination of optimized actions under partial observations leads to optimality under global states. However, this assumption does not always hold, even when $N=1$. The joint-action learner can not be decomposed into independent actors. The policy that maximizes rewards under partial observations does not necessarily align with the policy that maximizes rewards under global states, i.e., $Q(h, a) \neq Q(s, a)$ [DRQN2015Arxiv]. In contrast to MARL, the optimization objective in the POMDP problem discussed in this paper does not rely on this assumption, focusing instead on maximizing rewards under partial observations. Therefore, compared to our DCRL, the applicability of MARL methods in POMDP contexts is limited.

Q2: Regarding the effectiveness of state information.

We fully understand your concerns. In POMDPs, rewards are directly defined based on state information rather than on observations. Consequently, learning the value function $V(h,s)$ with the reward as the target is typically simpler than learning $V(h)$. This characteristic makes state information effective in most practical applications, a conclusion that numerous previous studies have empirically validated [UAAC2022AAMAS, Suphx2020Arxiv, PerfectDou2022NIPS, Honor-of-Kings2020NIPS]. Additionally, our DCRL leverages the advantages of state information while reducing variance, significantly enhancing training efficiency. Extensive experiments demonstrate that our DCRL consistently outperforms methods that do not incorporate state information across all tested tasks.

Thank you very much for your contribution in clarifying our method. Please feel free to let us know if you have any further concerns.

Reference

• [DRQN2015Arxiv] Hausknecht Matthew, et al. Deep recurrent q-learning for partially observable MDPs. Arxiv, 2015.
• [UAAC2022AAMAS] Andrea Baisero, et al. Unbiased asymmetric reinforcement learning under partial observability. In AAMAS, pages 44–52, 2022.
• [Suphx2020Arxiv] Junjie Li, et al. Suphx: Mastering mahjong with deep reinforcement learning. Arxiv, 2020.
• [PerfectDou2022NIPS] Guan Yang, et al. PerfectDou: Dominating Doudizhu with perfect information distillation. In NIPS, pages 34954–34965, 2022.
• [Honor-of-Kings2020NIPS] Deheng Ye, et al. Towards playing full MOBA games with deep reinforcement learning. In NIPS, pages 621-632, 2020.

Thanks again to the authors for their detailed response.

My comment about the relationship with MARL is that the centralised training with decentralised execution (CT-DE) paradigm closely resembles the central idea being presented, albeit for a distributed setting with a different need to aggregate local information. In particular, the methods that stem from CT-DE degenerate into something similar when considering the case of $N=1$. While it is true that in MARL, one has to consider how to best promote the so-called individual-global-max condition, in the degenerate $N=1$ case this may not be a concern, I believe as the local and joint policy coincide (even if the critic is centalised). Note also that the solution to a POMDP is in general, different to the solution when the agent is endowed with the missing data (which recovers the MDP setup) - this is not specific to the multi-agent setting.

I would like to thank the authors for their response to point 2 - this is a helpful clarification.

Thank you very much for your feedback! We are glad to have addressed some of your concerns.

Indeed, there are situations where the local and joint policy coincide, this holds ture when the state encompasses history (rather than just the observation). The centralized critic is equivalent to using $V(h, s)$, which theoretically ensures that the policy is unbiased and converges to the optimal solution. Since MARL primarily deals with cooperation and competition among multiple agents, partial observability often stems from other agents. When the problem is reduced to $N=1$, most tasks fall into this situation.

In POMDPs, it is more common for the state not to fully include history, which is equivalent to using $V(s)$. The state alone does not typically reveal whether the agent has previously collected the necessary information and thus cannot adequately indicate if the current state is favorable or adverse. The policy is biased and may not converge to the optimal solution. The UAAC paper has proved it in Theorem 4.2 and includes a toy example to illustrate the concept in Appendix B.2.

We appreciate your constructive feedback and apologize for the lack of clarity in our description. We will provide a more detailed explanation in the revised version of our manuscript.

Thank you for your encouraging words and constructive feedback. We appreciate your time reviewing our paper and provide point-by-point responses to your comments below.

Question 1:

There is a typo in the definition of $R(h,a)$ in line 122.

We greatly appreciate your pointing out the issues. We have corrected the typos in the revised version.

Question 2:

Can you explain in more detail how the weighting mechanism between the oracle critic and the standard critic is designed to reduce variance? Are there specific conditions or thresholds for adjusting the weights?

To address the high variance issue arising from an over-reliance on state information, we propose a weighting mechanism that leverages the benefits of two critics. As demonstrated in Theorem 2, the no-variance standard critic can effectively mitigate the variance introduced by the oracle critic, thereby reducing overall variance.

The weight adjustment is governed by the advantage function of the standard critic. When the advantage function yields a non-positive value, $\beta$ is clipped to $0$. Conversely, when the advantage is positive, $\beta$ activates the standard critic to accelerate training and reduce variance. Theorem 5 establishes that this weighting mechanism is a form of lower-bound-soft-Q-learning. This approach significantly enhances the performance of DCRL while maintaining high stability concerning hyperparameters.

Question 3:

Is it necessary to ensure the alignment of $V(s)$ and $V(h,s)$ in DCRL? if yes, how to ensure it?

Thank you for this question. The answer is no. It is unnecessary to ensure the alignment of $V(s)$ and $V(h,s)$. In the POMDP setting, the policy operates based on partial observations, resulting in a standard policy gradient derived from $V(h)$. Maintaining the policy gradient's unbiasedness is crucial when incorporating state information to enhance training. This stipulates that the value function must be unbiased concerning $V(h)$ without considering its relationship with $V(s)$. Our DCRL framework emphasizes two critics: $V(h)$ and $V(h,s)$. It has been established that $V(h,s)$ is unbiased about $V(h)$. Furthermore, Theorem 1 demonstrates that $V_{dual}(h,s)$ in the DCRL framework is also unbiased.

Thank you very much for recognizing our work! Your time and effort are greatly appreciated!

Thank you very much for your constructive comments and suggestions. We have revised our paper accordingly. Below, we present detailed responses to each point.

Weakness 1 and Question 1:

Even assuming that you have the state information during training is a strong assumption. You might not always have that. A lot of works do not make that assumption and perform pretty well.

Do you think in most real world settings you described in the introduction, you will have access to the state information for training the oracle critic?

We fully understand your concerns. To address your concerns, we included a more detailed explanation of the assumptions regarding state information accessibility and provided a representative real-world scenario (autonomous driving) of the POMDP problem.

Incorporating state information during the training process is a common assumption adopted by many previous works. In POMDP problems such as MiniGrid and MiniWorld, researchers often use state information to assist agents during training [UAAC2022AAMAS, Believer2023ICML]. In various partially observable games, including card games (e.g., Mahjong, DouDiZhu) and MOBA games (e.g., Honor of Kings), we can readily obtain the state information (opponents' hands or attributes) during training. Consequently, researchers often leverage this information to enhance agent training, resulting in significant performance improvements [Suphx2020Arxiv, PerfectDou2022NIPS, Honor-of-Kings2020NIPS].

In autonomous driving applications, due to the high costs of specific sensors and the challenges of acquiring high-definition maps in some areas, existing solutions mainly rely on end-to-end visual models [Tesla, PhiGent]. During the training phase, autonomous driving companies typically equip a small fleet of training vehicles with expensive sensors and use the collected high-definition maps and other information as state information to facilitate the training of these end-to-end visual models. This approach aids in constructing 3D environments, thereby enhancing the performance of autonomous driving systems [BEV2023TPAMI].

Weakness 2:

There should have been comparisons to commonly used POMDP works like Dreamer [] and others. But the comparisons were made only with (in some words) different variants of the algorithm.

Thank you for this valuable suggestion. We have followed your advice and considered the commonly used POMDP baseline DreamerV3 [DreamerV32023Arxiv].  We compared the performance of DCRL, DreamerV3, and additional baselines across eight tasks in the MiniGrid environment. Figure 3 of the rebuttal PDF illustrates the average scores from five training seeds over $1e5$ steps for each method. The results indicate that DCRL consistently outperforms DreamerV3 across all eight tasks. Importantly, DCRL requires fewer computational resources and less training time than DreamerV3.

Thank you for your thought-provoking and discussion-worthy suggestions, which have significantly improved the quality of our paper. If you find that these concerns have been resolved, we would appreciate it if you would consider reflecting this in your rating of our paper.

References

• [UAAC2022AAMAS] Andrea Baisero, et al. Unbiased asymmetric reinforcement learning under partial observability. In AAMAS, pages 44–52, 2022.
• [Believer2023ICML] Andrew Wang, et al. Learning belief representations for partially observable deep RL. In ICML, pages 35970–35988, 2023.
• [Suphx2020Arxiv] Junjie Li, et al. Suphx: Mastering mahjong with deep reinforcement learning. Arxiv, 2020.
• [PerfectDou2022NIPS] Guan Yang, et al. PerfectDou: Dominating Doudizhu with perfect information distillation. In NIPS, pages 34954–34965, 2022.
• [Honor-of-Kings2020NIPS] Deheng Ye, et al. Towards playing full MOBA games with deep reinforcement learning. In NIPS, pages 621-632, 2020.
• [Tesla] Tesla AI Day. [Online]. Available: https://www.youtube.com/watch?v=j0z4FweCy4M
• [PhiGent] PhiGent: Technical Roadmap. [Online]. Available: https://43.132.128.84/coreTechnology
• [BEV2023TPAMI] Hongyang Li, et al. Delving into the devils of bird's-eye-view perception: A review, evaluation and recipe. IEEE TPAMI, 2023.
• [DreamerV32023Arxiv] Hafner Danijar, et al. Mastering diverse domains through world models. Arxiv, 2023.

We appreciate your constructive comments, which have significantly enhanced the quality of our manuscript. Below, we provide a point-by-point response to your feedback.

Weakness 1:

The primary weakness of this paper is novelty. While I can appreciate that the dual critic architecture reduces variance, the proofs seems fairly obvious. In my opinion, the main novel contribution of this paper then relies on the weighting mechanism and the empirical results showing that DCRL can beat AC, UAAC, and also recurrent A2C. The novelty therefore seems marginal.

Thank you for your valuable feedback. In this study, we address the critical issue of high variance resulting from an over-reliance on state information, a topic that has been largely overlooked but is highly relevant to the POMDP community. To mitigate this issue, we propose a DCRL framework that harnesses the strengths of two critics. We provide theoretical evidence demonstrating that DCRL mitigates the learning variance while maintaining unbiasedness. Moreover, our DCRL incorporates a dynamic weighting mechanism, which can be interpreted as a form of lower-bound-soft-Q-learning, distinguishing it from conventional linear weighting methods. This mechanism substantially improves DCRL performance while maintaining high stability concerning hyperparameters.

We conducted additional experiments to validate our claims: 1) We implemented a simplified weighting method that uses fixed ratios to weight the two critics (referred to as the DCRL No Clip Version). 2) We performed ablation experiments on the values of the $\beta$ parameter ($\beta \in \\{1/5, 1/3, 1/2, 2/3, 4/5\\}$). Figure 1 of the rebuttal PDF illustrates that the DCRL No Clip version shows performance improvements in only some environments compared to UAAC. In contrast, DCRL consistently enhances performance across all tested environments, demonstrating that the dynamic weighting mechanism of DCRL is crucial. Additionally, the choice of $\beta$ exhibits notable stability. We selected $\beta = 1/2$ as the parameter for reporting our results in the paper, as it consistently exhibits superior performance.

Weakness 2:

Additionally, asymmetric actor critic (AC) was proposed to utilize additional state information to learn a policy more efficiently over the standard recurrent A2C where the critic only uses history information. UAAC then corrects the bias in AC by incorporating both the state and the history. However this paper proposes going back to the standard recurrent A2C method for one of the critics to achieve unbiasedness. It would have been interesting to compare two dual critic architectures: 1) V(h) and V(h,s) and 2) V(s) and V(h,s). Although the second approach would be biased, I would assume that using V(s) would lead to more efficiency gains. This seems to be an obvious comparison that was not considered.

We greatly appreciate your suggestion. Following your suggestion, we conducted ablation experiments comparing two different dual architectures: 1) $V(h)$ and $V(h,s)$ (our DCRL), and 2) $V(s)$ and $V(h,s)$ (referred to as the DCRL State Version). Figure 2 of the rebuttal PDF illustrates that the DCRL State Version demonstrates commendable performance among the various DCRL variants. However, the biased characteristics of the DCRL State Version contribute to instability in its effectiveness. Notably, our DCRL achieves superior training performance in three out of four environments. These findings further highlight the significance of our DCRL framework.

We appreciate your constructive feedback. These questions are very inspiring and discussion-worthy. The quality of the manuscript has been greatly enhanced by incorporating these and other reviewers' valuable suggestions. If you find that these concerns have been resolved, we would appreciate it if you would consider reflecting this in your rating of our paper.