Posterior Sampling via Langevin Monte Carlo for Offline Reinforcement Learning

We thank the reviewer for acknowledging our contributions and for the insightful questions.

The presented approach captures pessimism through posterior sampling. While innovative, one could question whether this form of pessimism adequately represents the complex nature of uncertainties found in the offline dataset, particularly given the non-stationary distributions that can arise from varied data collection policies.

This is indeed a very insightful question. The short answer is we do not know yet for sure. We speculate that this form of pessimism in our algorithms is likely not tightly capturing the uncertainties in the offline dataset as there is still a gap of $\sqrt{d}$ of our bounds for linear MDPs in the worst-case scenarios (please see the "Interpolating Bounds" paragraph on page 7). Closing this gap of approximate posterior sampling (perhaps with better algorithms that can tightly capture sufficient pessimism for offline RL, per your suggestion) is left as a future direction.

We thank the reviewer for the positive reviews.

What is the difference between the convergence of Langevin Monte Carlo and the convergence of the RL posterior sampling under the offline setting? Will the former lead to the latter?

To clarify the context, we assume that you are questioning the relationship between the Langevin Monte Carlo (LMC-PPS) way in Algorithm 2 and the pessimistic posterior sampling way in Algorithm 1. Note standard posterior sampling in [US21] does not have any frequentist guarantee. Please let us know if you are implying other stuff. We are happy to elaborate on that more.

Responding to the relation between the two algorithms, we discussed this at length in the second paragraph of Section 6. For the reviewer’s convenience, we summarize it here. In our settings, LMC-PPS and exact posterior sampling (PPS with exact posterior samples, not with approximate samples via LMC) achieve the same guarantees. This shows a promising benefit of LMC-PPS, where we can employ first-order sampling methods (e.g., LMC) without compromising the statistical guarantees. This is immensely meaningful as, in many cases, exact posterior sampling is much more expensive to obtain (in our settings, as formally discussed in Section 6) or even intractable, while LMC is simply a noisy gradient-based method that can efficiently apply to any differentiable model.

Why is LMC, instead of SGLD, used in Algorithms 2 and 3? Can mini-batch data be used in simulations of the proposed algorithm?

We can replace LMC with SGLD, where the use of stochastic gradients introduces additional complexities that are typically managed by the standard tools in LMC [1]. In essence, we only need to account for the additional source of stochasticity stemming from the random batch selection in the analysis. However, it's important to note that this point is ancillary to our main argument. Consequently, we introduce LMC for the sake of clarity in exposition.

In our simulations, we did, in fact, utilize mini-batch data (SGLD). Moreover, we discuss in Section D.2 in our appendix for a thorough description of the experimental setup and training particulars. We will enhance the elucidation of the relationship between LMC and SGLD in the final version.

Reference:

[1] AS Dalalyan, A Karagulyan. User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient. Stochastic Processes and their Applications, 2019

Dear reviewer jpdm,

As the author-reviewer discussion period will end soon, we will appreciate it if you could check our response to your review comments. We are confident that we have addressed your concern in the rebuttal. We haven't hear from you, could you let us know if your concerns have been addressed? If our response resolves your concerns, we kindly ask you to consider raising the rating of our work. If not, we are very happy to provide further explanations. Thank you very much for your time and efforts!

We thank the reviewer for the positive reviews and the constructive feedback. We will revise our paper accordingly based on your suggestions.

I do not see particularly new ideas from the submission, either in the algorithm or in the analysis. Thus, the novelty of the paper is limited.

For algorithms, Lin-LMC-PPS and Neural-LMC-PPS in our paper are both novel for offline RL. Especially, the use of linear auxiliary models in our Neural-LMC-PPS is non-standard. For analysis, we did not claim to invent any new technical tools and we built upon some of the existing techniques used for linear MDP [JYW21] and Neural MDP [NTA23] to analyze our problem settings. However, the fact that our algorithms are completely different from their algorithms by nature and that our bounds improve upon the bounds of [JYW21] and [NTA23] in the respective MDP models shows that we cannot use their techniques ``as is'' and that we need a new analysis treatment. The key technical challenge that is absent from both [JYW21] and [NTA23] is to show whether we can obtain pessimism only from approximate posterior samples. To address this challenge (and obtain such improvement consequently), the key idea is to separate the distributional shift problem from the estimation problem (Lemma A.3 in our appendix) and leverage the (near-)linear structures of linear MDP and neural MDPs to control the noises propagated from approximate posterior samples.

The work in Uehara and Sun [US21] considers the setting where the representation of state-action is unknown, whereas the submission assumes the feature representation function is known. I think it is not fair to claim the improvement from [US21].

We apologize for the confusion. We compared only with Section 8 of [US21], which studies the model-based Posterior Sampling for offline RL where there is no representation learning (the rest of [US21] is not related to posterior sampling). We will revise our paper accordingly.

How is the regression objective related to posterior sampling? How is this a "Langevin" Monte Carlo method?

The regression objective is the negative log-likelihood function of the posterior (see footnote 7 in our paper). If one simply samples from $\theta\sim \exp(-L_h(\theta))$, then it corresponds to the standard posterior sampling (with uninformative prior). However, direct sampling from the general $\exp(-L_h(\theta))$ could be intractable, so the alternative method would be adding noise to the gradient decent step of the negative log-likelihood function, which is essentially Langevin Monte Carlo (also check eqn (4) of [WT11]).

I do not see any particular contribution from the NTK part to reinforcement learning theory. It is a good add-on result though.

We study the statistical benefits of Posterior Sampling via Langevin Monte Carlo for offline RL in neural MDPs using NTK. Linear MDPs and neural MDPs (with NTK) are two different models that cover two different aspects of scenarios we might encounter. The guarantees in linear MDPs do not imply any guarantees in neural MDPs (in fact, roughly speaking, the bound for neural MDPs is worse than that for linear MDP by a factor of $\sqrt{H}$); thus, considering both settings gives two orthogonal views of the landscape of Posterior Sampling via Langevin Monte Carlo for offline RL.

I wonder how this method performs on some offline deep-RL benchmarks.

While the current scope of our paper focuses only on the theoretical understanding of the offline LMC algorithm, the empirical performances of our algorithm in large-scale scenarios remain an interesting and non-trivial question that we leave as future work.