Dual RL: Unification and New Methods for Reinforcement and Imitation Learning

We thank the reviewer for the detailed and thoughtful review. We have uploaded a revision with all the suggested typos corrected. The changes are marked in red. Please find the clarification to the queries below:

1. There seems to be an implicit assumption that $d^O$ is non-zero everywhere

We had earlier discussed the assumption of sufficient coverage of $d^O$ over $\mathcal{S}\times \mathcal{A}$ above Equation 32, but make this more clear in the revised paper. The degeneracy without this assumption is expected as a result of using importance sampling (requires coverage on importance sampled distribution) as well as $f$-divergences which may be ill-defined without enough support. We discuss a potential solution for this in Theorem 2 (see Assumption 1 in theorem 2) where we constrain the search set of policies within the support of offline dataset (similar to the idea of pessimism). In this case, the best we can expect to do is return a policy with highest performance within the dataset.

2. It is unclear what the left-most figure in Figure 1(a) represents and how it is related to the squared distribution gap plot right next to it.

We have updated the caption in the revised paper as well as included a detailed explanation of the experiment in Appendix H.3. Please let us know if you need more clarification.

3. In Table 2, it is unclear how the authors determined to highlight some of the entries with blue bold texts as an indication of the most-performant policy.

We have updated the highlighting to be more accurate and have highlighted all offline imitation algorithms that lie within the standard deviation of the algorithm with the highest mean performance. The revised paper mentions this in the caption now.

4. Similarity in Table 3, it is unclear what rules were applied to highlight entries in blue.

We have updated the highlighting to mark all offline RL algorithms within one point performance of the top-performing algorithm. Note that in the offline RL experiments, the results extend the performance reported by prior works and the standard deviation is unavailable. Comparing using standard deviation would require rerunning all the baselines, which we believe will take more than the discussion time limit and compute capacity available to us at the moment. We have updated the revised paper to mention this in the caption now.

5. Moreover, I do not see a huge performance gap between f-DVL and prior methods, and it is hard to judge whether the proposed RL methods empirically outperform existing ones.

The locomotion benchmark in D4RL for offline RL is simpler with performance to certain extent saturated, compared to the Antmaze or Kitchen environments. A detailed discussion on the properties of these environment and datasets can be found in [2]. We notice that $f$-DVL outperforms in 6/9 tasks in antmaze and kitchen environments which pose a harder challenge of stitching suboptimal trajectories compared to recently proposed IQL[3], XQL[4] and DT[5]. We believe that these improvements over prior work are significant by the standards used in prior offline RL papers.

One of the contributions through $f$-DVL is also to provide another axis of improvement in a stable learning alternative to XQL. Indeed, using statistical metrics like IQM would be more suitable but unfortunately prior works in offline RL(XQL, IQL, etc) have used mean as an estimate, and computing IQL would require rerunning all the baselines which would take more time and more compute budget than we have available.

6. I do not see how one can directly derive equation (14) from equation (5), as a rewriting of dual-V with the temperature parameter.

Note that Equation 14 and Equation 5 are not equivalent - we consider a simpler objective for optimization with identical properties. Indeed the discount factor in $\hat{Q}$ (Eq. 14) remains unchanged and $\lambda$ is used as a substitute for the temperature parameter ($\alpha$) of the second term (multiplying by $\frac{1-\gamma}{1-\lambda}$, results in exactly the same first term as Equation 5 and the weight on second term $\frac{\lambda(1-\gamma)}{1-\lambda}$ acts as a tunable temperature parameter). A more detailed discussion about this conversion is in Section G.4 in the appendix.

7. In equations (32) to (34), can you elaborate on why the interchangeability principle holds in this specific case?

Consider the unconstrained optimization in Equation 33 w.r.t $d$. The inner optimization objective is an upper semicontinuous concave function. The idea is simply that for every $s,a$ in the expectation one can pointwise maximize $d$, and the maximizer is guaranteed to exist by assumption of concavity and upper semicontinuity. So taking the maximum inside or outside the expectation will lead to same maximizer per $s,a$. We refer the reviewer to Lemma 1 in [1] for a formal proof.

We thank the reviewer for taking the time to provide a thoughtful and thorough review of the paper. We uploaded the revised paper with the changes highlighted in red. Please find the clarification to the queries below:

1. Proposition 4: The authors write that CQL can be cast as a dual-Q problem with a right choice of f-divergence (Pearson). However, in the proof of Proposition 4 in Appendix D.1, Eq (70) actually corresponds to the objective of ATAC [Cheng et al. 2022], not CQL.

We appreciate reviewer's suggestion of expanding the family of algorithms unified by dual-RL by including ATAC[1]. We have updated Proposition 4 and it's proof to point out the connection to ATAC.

We believe Conservative Q-learning (CQL) and ATAC share some similarities in their objectives, which makes it hard to discard CQL from this family. The reviewer mentions that CQL does not formulate their objective as a min-max game. However, we believe this is not completely correct. Directing the reviewer's attention to Equation 3 in CQL, a min-max game is proposed between an action-proposal distribution $\mu$ and the Q-function. The authors point out that $\mu$ can be taken as the last policy iterate (also used in their implementation for high-dimensional action spaces) making it particularly close to the objective coming out of dual-RL. A straightforward difference between CQL and ATAC is the reliance on Gradient-Descent Ascent vs the Stackelberg game strategy of updating the two-player game. We have included both ATAC and CQL in Proposition 4 in the revised paper.

2. Eq (109) is not just a summary, and that "we summarize the result of the derivation so far" can be misleading. But this can be easily fixed though, by applying strong duality before moving forward.

Thanks for pointing this out. We have made the correction.

3. Validity Eq (10). In the proof, the author assume $d_0 = d_E$. I think that this is a fairly strong assumption, as it implies that the initial state at the test time would be drawn from the same distribution of the expert.

We think there might be a misunderstanding here. We do not assume $d_0=d^E$ but rather assume $d_0=d^S$. We clarify that Eq.10 is a simplified objective and the full derivation under $\chi^2$ can be found in Appendix E.3. In the full derivation we downweigh the $Q$-function on states obtained from mixture distribution of offline data ($1-\gamma$) and expert data ($\gamma$), and the actions obtained from current policy. In practice, the simplified objective contains the offline data term only as our offline data is a superset of expert data and $\gamma$ is close to 1.

The offline distribution $d^S$ is a wider distribution encompassing the starting state $d_0$ distribution. This strategy allows us to learn a near-optimal policy from a wider initial state distribution and should not be detrimental to function representation with sufficient capacity. A similar strategy is common in (eg. [2], off-policy RL [3],[4] ).

We have corrected Eq.10 with the missing constant of 1/4 and also updated the sentence before Eq.10 to say that we are presenting a simplified objective here.

4. Bug in Proof of Proposition 3.

We apologize for the phrasing of this section in the appendix leading to confusion. We have rewritten the paragraph to increase clarity. To summarize answers to your concerns briefly:

• We have corrected that $f'$, $(f')^{-1}$ are not strictly increasing (only non-decreasing), but note that we do not need them to be strictly increasing for the following proof.
• A common issue when dealing with $f$-divergences is the limited domain of $\mathbb{R}^+$ on which $f$ divergences are defined and subsequently limits the domain and range of $f'$ or ${f'}^{-1}$. For us to be able to create an optimization objective defined on $\mathbb{R}$, we utilize a surrogate extension of ${f'}^{-1}$ on $\mathbb{R}$. This was initially expanded upon in the practical algorithm section but we have now made this change in Section 6 as well.
• $(f')^{-1}$ for TV divergence (which the reviewer points out) is not well defined and $f^*$ for RKL and Sq. Hellinger divergences are discontinuous on $\mathbb{R}^+$. These divergences require special treatment and have expanded this discussion in the proof of Proposition 3. This treatment was earlier discussed in our proposed practical objective for $f$-DVL in Section 6.

I wanted to discuss the possibility of moving some of the interesting experiments from your appendix into the main text of your paper.

Please don't worry too much about this comment. I think your work is really good, and this is just a suggestion from a writing perspective. Feel free to disregard it if it doesn't suit your needs, and don't feel any pressure about it.

In terms of shortening some parts of the paper, I feel that the sections on basic MDP definitions, value functions, and Bellman operators are a bit lengthy. Perhaps you could consider condensing them slightly.

We thank the reviewer for taking the time to provide a thoughtful and thorough review of the paper.

We have made the changes for all the suggested typos in the revision. The changes are highlighted in red. We appreciate the reviewer's attention for pointing these out. We provide clarification to reviewers queries below:

1. Figure 1 (a) is quite hard to follow. Please refine the caption to explain what the floats and the colors mean.

We have elaborated on the experiment in Figure 1 (a) in the caption of the revised paper as well as added a detailed explanation of the experiment setup in Appendix H.3.

2. The first paragraph of Section 7.1 lacks supportive evidence for this claim. Incorporating illustrative figures and evaluations would substantiate this statement.

We want to request clarification on this question. It seems like Section 7.1 is the experimental setup section and perhaps the reviewer meant some other section.

3. Filling the replay buffer arbitrarily with expert data at the beginning is not fair for an ADP-style algorithm. Please rewrite this paragraph.

In this section of the appendix, our intention was to compare popular ADP-based off-policy methods vs dual-RL/DICE style off-policy methods. Any off-policy algorithm, by definition, should be able to make use of any kind of data obtained from the environment -- expert data is a special case. We have reworded this section to make our claims more accurate since as the reviewer points out considering one special case is not enough to say the the algorithm is brittle.

4. On page 40, the statement "f-DVL leads to noticeable improvements even in the online RL benchmarks" appears overstated. And please broaden the discussion of related works in the online RL setting beyond SAC.

We have corrected our claim to say that we are competitive with SAC and XQL in online RL experimeents. In this work, our focus was limited to offline setting and we leave a more in-depth analysis of online RL algorithms for future work. We have added the suggested recent online RL methods in the related work section.

5. Some crucial experimental outcomes, currently in Appendix H, should be integrated into the main paper for better accessibility and coherence, especially those results referenced and analyzed within the main paper.

We appreciate the reviewer's suggestion. We acknowledge that our appendix contains a number of interesting experiments. In this work, we were constrained by the page limit to strike a balance between facilitating understanding, demonstrating our contributions concisely, and providing empirical evidence to support the theoretical contributions. Currently, we take $\approx 0.6$ page for related works and $\approx 1.4$ page for preliminary. We believe introduction should not be considered a part of prior work and a similar format can be found in a number of influential ICLR works [1,2,3]. We would be happy to receive suggestions on what the reviewer thinks might be traded off in favor of more experiments.

6. Better not to claim "New" in the title directly. Just saying "A unified framework" is ok.

We clarify that the use of the word 'New' was to signify that we leveraged the existing framework of duality in RL to propose new and improved algorithms for offline RL and IL and not to signify the novelty of the duality framework.

We appreciate the reviewer's suggestions and are happy to incorporate other suggestions into our submission. Please let us know if there are other issues or concerns we can address. We would be happy to expand this discussion.

References:
[1]: Eyuboglu, Sabri, et al. "Domino: Discovering systematic errors with cross-modal embeddings." arXiv preprint arXiv:2203.14960 (2022).
[2]: Flennerhag, Sebastian, et al. "Bootstrapped meta-learning." arXiv preprint arXiv:2109.04504 (2021).
[3]: Ren, Tongzheng, et al. "Latent variable representation for reinforcement learning." arXiv preprint arXiv:2212.08765 (2022).

We thank the reviewer for the thoughtful review. The modifications are marked in red in the paper revision. We address your concerns and questions below.

1. While f-DVL outperforms XQL, its gains over other offline RL methods like IQL are marginal. The gains are not as significant as claimed over the full spectrum of baselines.

The locomotion benchmark in D4RL for offline RL is simpler with performance to a certain extent saturated, compared to the Antmaze or Kitchen environments. Our proposed approach is competitive in these domains. A detailed discussion of the properties of these environments and datasets can be found in [1]. We notice that $f$-DVL outperforms in 6/9 tasks in antmaze and kitchen environments which pose a harder challenge of stitching suboptimal trajectories compared to recently proposed IQL[2], XQL[3], and DT[4]. We believe that these improvements over prior work are meaningful by the standards used in prior offline RL papers. One of the contributions through $f$-DVL is also to provide another axis of improvement in a stable learning alternative to XQL. We would like to bring attention to the fact that XQL (ICLR 23) is a more recent method than IQL (ICLR 22) and our proposed method drives direct improvement over XQL.

2. The presentation of empirical results could be improved by using standardized metrics, showing confidence intervals, and increasing clarity around performance highlights.

We appreciate the reviewer's suggestions and note that the following changes have been made in our revised paper: a) In the results for imitation learning (ReCOIL) we highlight all methods that lie within the standard deviation of the best-performing method. b) In the offline RL (f-DVL) results we highlight methods that are within 1 performance point of the best-performing method. Additionally, we have also added standard deviations for our method in Table 10. Indeed, using statistical metrics would be more suitable but unfortunately prior works in offline RL like XQL have used mean as an estimate. We extend prior results in our work and computing metrics like Inter-Quartile mean (IQM) or std-dev would require rerunning all the baselines which would take more time and more compute budget than we have available.

Questions:

1. In Eq. 10, is the reward assumed to be zero in the Bellman consistency term?

To clarify for Eq. 10, we are in the setting of imitation learning where we are not provided with task rewards but rather only expert demonstrations. Eq. 10 is a direct consequence of Theorem 1 which provides a principled objective for imitation learning signifying that bellman backup with zero rewards in conjunction with a contrastive loss on Q is sufficient and pricipled for imitation learning.

2. Can you expand on how ReCOIL estimates the policy visitation and reward for the results in Sec 7.1?

We have added a detailed explanation for estimating policy visitation in Appendix H.3 in the revised paper and also edited the caption of Figure 1 to give more clarity. In brief, any dual-RL objective written in the $Q$-form allows us to extract density ratios by considering the inner optimization of Equation 9.

For reward estimation, the paragraph below Theorem 1 explains how rewards can be inferred from learned Q-function and policy. In brief, given $Q^*$ and $\pi^*$, the rewards can be extracted as $r(s,a)= Q^*(s,a)-\mathcal{T}^{\pi^*}_0(Q^*(s,a))$

3. Also clarify if Fig 1 and Fig 10 use OPOLO or SMODICE for comparison.

We appreciate the reviewer pointing out this typo, we correctly mentioned in the figures that we used SMODICE but incorrectly in the text that OPOLO was used for comparison. We have made this correction in the text. SMODICE can be understood as an offline counterpart to OPOLO and in our work we focus on the offline setting. A more detailed discussion of this experiment and baselines can be found in Appendix H.3.

4. What do the distributions D and $d^O$ refer to in Eq. 12 and 13? Are they both $d_{mix}^{E,S}$?

We appreciate the reviewer bringing attention to this typo. They are both indeed $d_{mix}^{E,S}$ and we have fixed this in the revised paper.

Please let us know if our clarifications resolve your questions and concerns. We would be happy to expand the discussion on any remaining specific concerns about design choices, and other technical details . Since the review is posted near the end of the discussion period we would greatly appreciate if we are made aware of any remaining concerns and issues soon.

References:
[1]: Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv preprint arXiv:2004.07219 (2020).
[2]: Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).
[3]: Garg, Divyansh, et al. "Extreme q-learning: Maxent RL without entropy." arXiv preprint arXiv:2301.02328 (2023).
[4]: Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." Advances in neural information processing systems 34 (2021): 15084-15097.