Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning

Thank you for your review and feedback.

Regarding the “incremental advancement” point, we strongly disagree with this point. We do share the motivation and use of the contraction mapping condition with Kolter (2011), but our approach differs greatly after that. Firstly, the approach in Kolter (2011) only works for very small MRPs. Our approach uses a completely new technique for satisfying the contraction mapping condition and includes a policy projection step that is crucial for policy optimization. We will make sure to emphasize this further.

Regarding the poor performance results, what is unique about our method is that, unlike the state-of-the-art offline RL techniques, our method does not regularize the policy towards the data policy. For that reason, we expect conservative methods and explicit policy regularization methods to out-perform our method on near-optimal datasets. In our (updated) small-scale results and the “random” D4RL results we show that as the dataset becomes less optimal, our method outperforms other methods. We will try to make this point clearer.

Your questions regarding additional datasets is a great point. We tried to stick with common benchmark datasets because we didn’t want the datasets to seem contrived. But adding additional datasets might further illustrate that point.

Regarding the ablation study question, this is a good suggestion. From our informal tests, without the policy projection, POP-QL diverges and when combining CQL with POP-QL we get some slight improvements, but we will conduct these experiments formally and add them to the results.

Finally, regarding the lack of more recent literature, we will make sure to include some of the most recent publications in the space.

Thank you for your review.

Regarding the Frozen Lake performance, hopefully our updated results illustrate the point better.

Regarding your question about value-approximation error, the challenge here is that except in small tabular cases, we cannot accurately estimate the true value function. Thus, any approximation of true “value-error” is only as good as our function approximator. Do you have any suggestions on how to illustrate a reduction in approximation error?

Since we state that $q$ is a distribution, we chose not to explicitly state that $q>=0$ and $\sum q = 1$ since we thought that was implied. But we can add that in if it makes it clearer.

Good catch on the proof of Lemma 1! This is meant to refer to Theorem 2 from Kolter (2011). In a previous draft of the paper, we restated this theorem, which is why there was a missing reference.

Regarding the derivation bug, we do in fact keep track of the normalization term, even though we don’t explicitly state it in the paper. In fact, ignoring the normalization term leads to a very different result. Below is our step-by-step derivation. We’ll try to make this clearer.

Your point about the baselines is a good point and something we struggled with. The reason is around the “Lagrange threshold” parameter discussed in the Appendix F of Kumar et al. (2020). The goal of this parameter is to automatically tune the conservatism parameter, \alpha, to the specific task (similar to the automatic entropy tuning of SAC). However, when using the library provided by the authors (https://github.com/young-geng/JaxCQL), using the automatic \alpha tuning significantly reduces performance for some of the tasks compared to a fixed \alpha. Reading through the paper you sent, it looks like Li et al. (2023) used a fixed \alpha parameter, which is different from the parameters reported in Kumar et al. (2020). For our next revision, we will re-run the CQL experiments using a fixed \alpha parameter and report both numbers.

Regarding your question on equation 24, this is the gradient with respect to the Q-network not the true Q-function. We tried to make the equation less cluttered by removing the parameter subscript, but that clearly made it more confusing. We will make that clearer.

Thanks for the note on using two different notations for KL. We fixed that in the newest version.

Our analytical solution (equation 9) can be written as:

$q^{\star}(s) = \frac{1}{C} \ \mu(s) \exp \left(\operatorname{tr} Z^T F(s)\right)$

With $C = \sum_s \mu(s) \exp \left(\operatorname{tr} Z^T F(s)\right)$. Plugging this into equation 8 yields.

$\max_{Z \succeq 0} \mathrm{KL}(q^* \| \mu)-\operatorname{tr} Z^T \mathbf{E}_{q^*}[F(s)]$

$\max_{Z \succeq 0} \ \mathbf{E}_{q^*} [ \log \left(\frac{1}{C} \frac{q^*(s)}{\mu(s)} \right) ] - \operatorname{tr} Z^T \mathbf{E}{q^*} \left[ F(s) \right]$

$\max_{Z \succeq 0} \ \mathbf{E}_{q^*} \left[- \log(C) + \log \left( \exp \left(\operatorname{tr} Z^T F(s)\right) \right) \right] -\operatorname{tr} Z^T \mathbf{E}{q^*}[F(s)]$

$\max_{Z \succeq 0} \ - \log(C)$

$\max_{Z \succeq 0} \ - \log\left(\mathbf{E}_{\mu} \exp \left(\operatorname{tr} Z^T F(s)\right)\right)$

Thank you for your review.

Thank you for pointing out the missing literature. We agree we should spend more time discussing other off-policy methods and will make sure to include those papers.

And thank you for pointing out mistakes regarding mixing up off-policy and offline RL. These are typos and we will fix them.

The reason we chose to evaluate the fully offline setting is because the challenges of distribution shift are more pronounced. Unlike our method, all the best performing offline RL methods have some regularization back to the data policy. For that reason, of course our method will not be able to compete with policy-regularization or conservative methods on near-optimal datasets. However, our (updated) small scale results and the “random” D4RL results illustrate that this method can outperform policy-regularization and conservative methods when the datasets are severely sub-optimal. We will try to further highlight this.

Regarding your point about IS methods, our understanding is most IS methods (like many other off-policy methods) assume the support of the data and candidate policies match, otherwise there would be division-by-zero issues. However, in the offline RL case, this assumption is often violated. Could you elaborate on what part you disagree with?