Informed POMDP: Leveraging Additional Information in Model-Based RL

Theoretical justification for the improved convergence speed

We agree that asymmetric learning lacks theoretical motivations for exploiting the additional information, even in model-free RL. This is discussed extensively in Section 5.1 by Baisero & Amato (2022) and more recently in Section III.C of Sinha & Mahajan (2023). Despite our method being rooted in the theory of sufficient statistic (or information state) proposed by Subramanian et al. (2022), proving a faster convergence will be challenging. It would probably involve the notion of approximate sufficient statistic, known as approximate information state in Subramanian et al. (2022), to bound the performance of the dynamic program given the error on the learned distributions. This would require to express the error on the observation distribution that is implicitly encoded in the information distribution, through the exploitation of the motivating inequality $I(s', i' | h, a) \geq I(s', o' | h, a)$. It is thus quite an ambitious program that we are considering as a future work.

We also completely agree that the information could hurt learning, notably by containing irrelevant variables or exogenous variables. Moreover, the conditional information distribution may be more complex to approximate in practice than the observation distribution, while not being that useful to the control task. These reasons are now discussed in details in Section 4. We also want to point out that the presence of irrelevant or exogenous variables is a problem that may also concern the state / observation in symmetric RL, and that is well studied in the exogenous RL literature. While this body of work asks the question of what is necessary in the state, the challenge in recurrent RL is still about finding a sufficient representation of the history. Reconciling the apparent tension between these considerations is obviously a fantastic avenue for future work, but is out of the context of this article in our opinion.

We also want to remind that we have shown for a simple environment in which partial observability is quite challenging (need of inferring its orientation from noisy position observations) that the speed of learning was clearly proportional to the quality of the additional information. This is, in our honest opinion, already a convincing empirical demonstration of our hypothesis. We have extended this study, as explained in the following.

Empirical study for justifying the improved convergence speed

We agree that we should have added more experiment in order to further demonstrate empirically our hypothesis. We thus have added some results in that direction in Appendix E. First, in Appendix E.2., we show for a pathological example that the Informed Dreamer can learn near-optimal in environments in which the Uninformed Dreamer is completely unable to learn, at request of Reviewer SwZJ. Concerning the proposition of generalizing the study about the quality of the information to other domains, we consider it to be non feasible. Indeed, in the other domains, we only get access to the Markovian state, and it would be difficult to artificially construct an information $i$ that provides partial information about the state but that still makes the observation conditionally independent of the state given it. However, we propose to generalise this to the other Mountain Hike environments, which offer very similar conclusions, see Appendix E.1. This constitutes in our opinion a rather convincing empirical evidence in favour our our hypothesis.

Questions

No confidence interval in Figure 4(b). Does this mean that the standard error is 0?

No it doesn't, we decided not to display them for the sake of readability. We now stated it clearly in our article.

Does the proposed method introduce new hyperparameters?

No, we do not introduce any new hyperparameter (the log likelihood of the observation is just replaced by the log likelihood of the information in the losses), and we kept exactly the hyperparameters as in the original Dreamer release.

Minor

I would suggest to add the uninformed baseline in Figure 4(b) as well for comparison.

Excellent idea, it is done.

Typo: Section 3.1 the discount factor \gamma \in [0,1[

Thank you, it is fixed.

Double citation in the second paragraph of introduction: Gregor et al. 2019.

Thank you, it is fixed.

Figure 4 can be made larger.

It has now become difficult because of the additional discussions that we added to the article. But since we have considered additional environments at your request for the analyses of the quality of information, you can now consult these larger figures in Appendix E.

References

• Avalos, R., Delgrange, F., Nowé, A., Pérez, G. A., & Roijers, D. M. (2023). The Wasserstein Believer: Learning Belief Updates for Partially Observable Environments through Reliable Latent Space Models. In European Workshop on Reinforcement Learning.
• Baisero, A., & Amato, C. (2022, January). Unbiased Asymmetric Reinforcement Learning under Partial Observability. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems.
• Sinha, A., & Mahajan, A. Asymmetric Actor-Critic with Approximate Information State.
• Subramanian, J., Sinha, A., Seraj, R., & Mahajan, A. (2022). Approximate information state for approximate planning and reinforcement learning in partially observed systems. The Journal of Machine Learning Research, 23(1), 483-565.

Dear Reviewer,

Thank for your valuable feedback. We are glad to read that you found the paper promising and appreciated the novel learning paradigm. Below, we responds to your concerns with some clarifications, explanations or discussions that we added to the paper. Notably, we clarified the novelty of the paper, proposed to better discuss the hypothesis of the information improving convergence, and added additional analyses to better support this hypothesis, as requested. Moreover, we identified and fixed an ambiguous presentation of the result in the tables (conflicting with the figures) that may have lead you to wrong conclusions about the standard error of the mean.

If these clarifications happens to address some of your concerns, or that you agree that these additional results and discussions strengthen the paper, would you be willing to raise your rating of this first step in asymmetric model-based RL?

Thank you for your time and consideration.

Best regards,
The authors.

Novelty

While we agree that asymmetric learning is not novel, it was completely left unexplored in the model-based setting. To the best of our knowledge, this work is indeed the first to exploit the additional information for learning a model of the environment, together with a concurrent work (Avalos et al., 2023). In addition, this work is the first to propose a method within this more reasonable setting of non-Markovian additional information. Moreover, while the theory of sufficient statistic recently developed by Subramanian et al. (2022) found a lot of interest in the partially observable RL literature, this work is the first step in extending this theory to asymmetric learning, together with a concurrent work (Sinha & Mahajan, 2023). Finally, this work is the first to link (symmetric) recurrent world models with the theory developed by Subramanian et al. (2022), and generalizing this to approximate sufficient statistic, or approximate information state in Subramanian et al. (2022), could pave the way for deriving convergence guarantees for model-based RL in POMDP.

We would also like to emphasise that the apparent simplicity of the theorem and algorithm generalizations are a direct consequence of the key conditional independence requirement, that can always be met in practice. Indeed, even when the eventual additional information $o^+$ does not make the observation $o$ and the state $s$ conditionally independent, it is possible to design such an information $i = (o, o^+)$ that would satisfy this independence. We would also like to clarify the fact that the information is not necessarily a deterministic function of the state.

Limited performance gain

We agree that the performance gain of the informed Dreamer is not as important as we might have hoped. However, we disagree that the result are not conclusive for several reasons. First, in several environments, it can be seen that the Informed Dreamer reaches the final performance of the Uninformed Dreamer in about half the number of environment steps. We also have conducted a new experiment, at the request of Reviewer SwZJ, in which we see that there exist environments for which the Informed Dreamer converges to a near-optimal policy while the Uninformed Dreamer completely fails to learn anything, see Appendix E.2. Moreover, in environments for which the Informed Dreamer converges at a higher return, the standard errors are generally lower than the order of magnitude of the difference in performance. Our ambiguous presentation of the tables might have mislead you in interpreting the standard deviation as the standard error of the mean in tables. This was a mistake not to indicate clearly what these confidence intervals represent, and we fixed it now. If you think that it is preferable to report the standard error of the mean in the tables, we obviously agree to modify this. The smaller scale of the standard error of the mean can be visualised in Figures 4 to 6, in which the confidence interval is the standard error. We apologise for this confusion.

That being said, motivated by your remark, we also propose to better discuss the current limitations of the proposed adaptation in Section 5, and to better explain the avenues for future works in the conclusion. Notably, we highlight the limitations of our ELBO learning objective, in which the variational encoder is conditioned on the observation only, limiting the expressiveness of the reconstructed distribution. Future work may improve on this aspect by having a second encoder that is not used in the recurrence and thus not required at execution time. While this would come at the cost of reconstructing the observation distribution, it may also be possible to implement the latter without training the observation encoder to reconstruct the observation, but by KL-regularising its distribution to the information encoder distribution.

Dear Reviewer,

Thank your for your valuable feedback. We are pleased to see that you appreciated our new framework and this first step in asymmetric model-based RL. Even if you found the algorithmic contribution minor and the generalised theorem quite trivial, it is nice to read that your appreciated that the method was supported by the theory. Below, we give an explanation that may address your first concern about applicability, and we provide a discussion for explaining both theoretically and practically the limited performance gain that we also propose to add to the article, along with some propositions of future works for improving this in the conclusion.

If you agree that our answers and added discussions addressed some of your concerns, would you be willing to improve your rating for this work, which is both a first step in the theory of sufficient statistic for asymmetric learning and in model-based asymmetric learning?

Thank you for your valuable feedback and for your consideration.

Best regards,
The authors.

Assumption of conditional independence and applicability

While we agree that assuming state observability at training time is strong in some environments, we would like to point out that we are able to deal with any kind of additional information, which is a much more reasonable assumption, and a novelty. In other words, the problem that we propose to tackle is a strict generalization of the standard learning paradigm for POMDP. Even when the eventual additional information $o^+$ does not make the observation $o$ and the state $s$ conditionally independent, it is possible to design such an information $i = (o, o^+)$ that would satisfy this independence. We would also like to emphasise that the apparent simplicity of the theorem and algorithm generalizations are a direct consequence of this key conditional independence requirement, that can always be met in practice. Finally, to the best of our knowledge, this work is the first to propose a method within this more reasonable setting of non-Markovian additional information.

Minor change with respect to Dreamer and limited performance gain

While we agree that, when introducing this conditional independence requirement for the information, the theory motivates a very slight modification of existing algorithm, we really see this work as a first step towards developing asymmetric model-based RL methods. In the light of your remark, we propose to better discuss the current limitations of the proposed adaptation in Section 5, and to better explain the avenues for future works in the conclusion. Notably, we highlight the limitations of our ELBO learning objective, in which the variational encoder is conditioned on the observation only, limiting the expressiveness of the reconstructed distribution. Future work may improve on this aspect by having a second encoder that is not used in the recurrence and thus not required at execution time. While this would come at the cost of reconstructing the observation distribution, it may also be possible to implement the latter without training the observation encoder to reconstruct the observation, but by KL-regularising its distribution to the information encoder distribution.

Clarifications on the difference with the original Dreamer

We apologise for the important change in notation with respect to the initial Dreamer algorithm. We tried to find a comprise between those used in the motivating theory of Subramanian et al. (2022) and those used in the practical algorithm of Hafner et al. (2023). While we know that reviewers are not required to read the appendices, we believe that Appendix C may help disambiguating notations. Indeed, in view of this change in notations, we have written a precise pseudocode for the Informed Dreamer in Appendix C, that specifically highlight differences with the original Dreamer algorithm.

References

• Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023). Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104.
• Subramanian, J., Sinha, A., Seraj, R., & Mahajan, A. (2022). Approximate information state for approximate planning and reinforcement learning in partially observed systems. The Journal of Machine Learning Research, 23(1), 483-565.

References

• Baisero, A., & Amato, C. (2022, January). Unbiased Asymmetric Reinforcement Learning under Partial Observability. In Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems.
• Sinha, A., & Mahajan, A. Asymmetric Actor-Critic with Approximate Information State.
• Subramanian, J., Sinha, A., Seraj, R., & Mahajan, A. (2022). Approximate information state for approximate planning and reinforcement learning in partially observed systems. The Journal of Machine Learning Research, 23(1), 483-565.

Dear Reviewer,

We are pleased to read that you found our generalised formalisation of asymmetric learning natural and well motivated. It is also nice to read that your are enthusiastic about the established connection between asymmetric predictive models and sufficient statistics, and that your liked the simplicity of our method.

We agree with your concerns on the justification for why the convergence speed would improve, and propose some explanations to address these below. While the theoretical motivation is indeed not satisfactory yet, we elaborate on the difficulty of establishing it but propose several avenues for future work in that direction. For now, we see this work as a first step in asymmetric model-based RL, as well as in the theory of sufficient statistic in asymmetric learning.

If our explanations and the discussions added to the manuscript happened to address some of your concerns, would you be willing to improve your rating for the article?

Thank you for your valuable review and for your consideration.

Best regards,
The authors.

Theoretical justification for the improved convergence speed

We agree that asymmetric learning lacks theoretical motivations for exploiting the additional information, even in model-free RL. This is discussed extensively in Section 5.1 by Baisero & Amato (2022) and more recently in Section III.C of Sinha & Mahajan (2023). Despite our method being rooted in the theory of sufficient statistic (or information state) proposed by Subramanian et al. (2022), proving a faster convergence will be challenging. It would probably involve the notion of approximate sufficient statistic, known as approximate information state in Subramanian et al. (2022), to bound the performance of the dynamic program given the error on the learned distributions. This would require to express the error on the observation distribution that is implicitly encoded in the information distribution, through the exploitation of the motivating inequality $I(s', i' | h, a) \geq I(s', o' | h, a)$. It is thus quite an ambitious program that we are considering as a future work. We now better discuss this compelling future work in our conclusion.

We also want to remind that we have shown for a simple environment in which partial observability is quite challenging (need of inferring its orientation from noisy position observations) that the speed of learning was clearly proportional to the quality of the additional information. This is, in our honest opinion, already a convincing empirical demonstration of our hypothesis. Moreover, we extended this study in Appendix E.1 for the other Mountain Hike environments and it provides similar conclusions.

Characterization of the usefulness of the additional information

You have raised the interesting point of characterizing the usefulness of the additional information. It is indeed easy to see that badly designed additional information could hurt training, because the latter may contain irrelevant or exogenous information. Moreover, the conditional information distribution may be more complex to approximate in practice than the observation distribution, while not being that useful to the control task. These reasons are now discussed in details in Section 4. We also want to point out that the presence of irrelevant or exogenous variables is a problem that also concerns the state / observation in symmetric RL, and that is well studied in the exogenous RL literature. While this body of work asks the question of what is necessary in the state, the challenge in recurrent RL is still about finding a sufficient representation of the history. Reconciling the apparent tension between these considerations is obviously a fantastic avenue for future work, that we mentioned in our conclusion.

Comparison to related works in asymmetric learning

We agree that it would have been richer to gather these results of additional asymmetric learning methods in the article. However, these model-free RL algorithms would not benefit from the sample-efficiency of model-based RL methods such as Dreamer, and a fair comparison would be made difficult. Moreover, we emphasise that none of these methods were designed to deal with non Markovian additional information. Finally, we will not be capable of providing a fair comparison of these methods before the end of the discussion period.

Comparison of the observation reconstructions

While we agree that our article may lack more qualitative visualization of the results, other than the learning curves, we are unfortunately unable to provide this comparison. Indeed, one of the advantage of our method is that it does not require to reconstruct the observation but only the (hopefully more compact) information. Consequently, we cannot compare the quality of the reconstructions, but we known that the distributions of the observation is encoded in the distribution of information.

Discussion of failures with respect to the theory and in practice

We addressed your last remark by better discussing the failures that arise, both from the theory perspective in Section 4 of the article and from the practical implementation perspective in Section 5.

First, we want to point out that proving a faster convergence when learning the state/information distribution instead of the observation distribution is challenging. Such considerations would probably involve the notion of approximate sufficient statistic, known as approximate information state in Subramanian et al. (2022), to bound the performance of the dynamic program given the error on the learned distributions. This would require to express the error on the observation distribution that is implicitly encoded in the information distribution, through the exploitation of the motivating inequality $I(s', i' | h, a) \geq I(s', o' | h, a)$. It is thus a very ambitious task that we are considering as a future work. We now better discuss this compelling future work in our conclusion.

It is more easy to explain why considering (badly designed) additional information could hurt training. Indeed, it is straightforward to see that the information can contain irrelevant or exogenous variables. Moreover, the conditional information distribution may be more complex to approximate in practice than the observation distribution, while not being that useful to the control task. These reasons are now discussed in details in Section 4. We also want to point out that the presence of irrelevant or exogenous variables is a problem that also concerns the state / observation in symmetric RL, and that is well studied in the exogenous RL literature. While this body of work asks the question of what is necessary in the state, the challenge in recurrent RL is still about finding a sufficient representation of the history. Reconciling the apparent tension between these considerations is obviously a fantastic avenue for future work, that we mentioned in our conclusion.

We also better discuss in Section 5 the limitations of our ELBO learning objective, in which the variational encoder is conditioned on the observation only, which is another plausible explanation for the limited performance gain. Further work may improve on this aspect by having a second encoder that is not used in the recurrence and thus not required at execution time. We have made that clear in our conclusion.

Questions

Why does the reward decrease over time?

There is not guarantee of monotonic improvement with such policy gradient algorithms, especially when the world model that provides the latent representations to the policy is trained jointly with the policy.

In (10), it may be helpful to say that $I$ is the mutual information.

Thank you, it is fixed.

In 3.1, there's a typo in the range of $gamma$.

Thank you, it is fixed.

Beyond settings where $i=s$, what are practically relevant scenarios where $s \rightarrow i \rightarrow o$.

As example for which $s \neq i$, we can consider a grasping robot for which the information are some scene variables that can be provided during training (e.g., objects types and positions) and allow to give the observation distribution but are not Markovian (e.g., objects velocities are not included).

Do you have examples where $s$ is not conditionally independent of $o$ given the information?

Yes, any situation in which a candidate information $\bar{i}$ (or $o^+$) is drawn independently of $o$ given $s$. For example when $\bar{i}$ is an additional realisation of the observation $o$, because you have the opportunity to query your sensor $O(o | s)$ several times at training time. In this case, the conditional independence between $o$ and $s$ given $o^+$ does not hold. However, as explained in Section 3 (with notation $o^+$), whatever the candidate information $\bar{i}$, the information $i = (\bar{i}, o)$ can be used and satisfies the conditional independence.

References

• Subramanian, J., Sinha, A., Seraj, R., & Mahajan, A. (2022). Approximate information state for approximate planning and reinforcement learning in partially observed systems. The Journal of Machine Learning Research, 23(1), 483-565.

Dear Reviewer,

We are pleased to read that you found the asymmetric learning setting well motivated, and that you appreciated the theoretical foundations as well as the flexibility of the proposed method. Below, we propose additional experiments and discussions to address your concerns and provide answers to your questions.

If you happened to find that our additional results and explanations address your concerns and questions, would you be willing to improve your rating for this work, that in our opinion is an interesting first step both in the theory of sufficient statistic for asymmetric learning and in model-based asymmetric learning?

Thank you for your valuable feedback and for your consideration.

Best regards,
The authors.

Comparison to related works in asymmetric learning

We agree that it would have been richer to gather these additional result in the article. However, these algorithms would not benefit from the sample-efficiency of model-based RL methods such as Dreamer, and a fair comparison would be made difficult. Furthermore, we will not be capable of providing a fair comparison of these methods before the end of the discussion period.

No clear overperformance after convergence in some environments

We first want to further emphasise the partial observability of the informed policy, which means that the optimal informed policy has the same performance as the optimal uninformed policy. This can be clarified in the paper if you think it is worth emphasizing the focus on improving the speed of convergence.

Nevertheless, we agree that beyond improving the speed of convergence, such informed objectives could permit learning in challenging environment where standard algorithms would not learn. First, we want to highlight that we already observed this, by obtaining a better performance after convergence in nearly all environments of the Velocity Control benchmark. In order to fully address your remark, we considered harder version of one of the most basic Pop Gym environments: Repeat First. In this benchmark, the agent is observing noise, and is rewarded for outputting the observation that it got $k$ time steps ago. While we had only considered the default Easy version ($k = 4$) of the environment so far, we now considered the Medium ($k = 32$) and Hard ($k = 64$) versions of this environment. As can be seen from the results in Appendix E.2, the Informed Dreamer clearly learns near-optimal policies while the uninformed Dreamer does not learn at all.