Enhancing Offline Reinforcement Learning with an Optimal Supported Dataset

We would like to express our heartfelt gratitude for the priceless feedback you have given us. We highly value your viewpoints and believe they will greatly improve our paper. We will carefully address all of your specific questions in our forthcoming response. To make it easier for you to identify the changes, we have highlighted them in blue in our revised version.

Q: About the symbols

A: We apologize for not providing you with a clearer understanding of these symbols. $V$ is the optimization space for the neural network $\nu$ that we are studying, which is a common assumption in this type of problem, as described in [1]. The dataset $D_m$ is also a common assumption, as referenced in [2]. $d^{*}_{\alpha}$ represents the optimal support of the data distribution, and we have highlighted its definition in the revised manuscript.

Q: About Assumption 5 and assumptions in Theorem 6

A: Assumption 5 is a technical assumption to ensure the continuity of $\hat L_\alpha (\nu)$ for all $\nu \in V$. It is a constraint on the optimization space $V$ for $\nu$. Remark 3 associated with Assumption 5 implies that some desired $V$ can be as follows: if the advantage $e_{\nu_\alpha^*}$ induced by the optimal distribution $d_\alpha^*$ is not equal to $\alpha f'(0)$ in the region not visited by $d_\alpha^*$, then a neighborhood of the optimal $\nu_\alpha^*$ can be chosen as the desired $V$. This assumption is not difficult to verify and does not impose strong assumption to the practical problems.

Theorem 6 does not introduce any new assumptions, but instead requires that the optimal distribution $d^*_0$ when $\alpha=0$ satisfies the common concentrability assumption， Assumptinon 4. Other specific choices regarding $\alpha$ and $f$ are both easily satisfied.

Q: About the sample complexity $O(\epsilon^{-4})$

A: We acknowledge that we have not proven the sample complexity to be optimal for OSD, as we aimed for practicality. For offline RL methods based on linear programming, although there are some works that can prove the sample complexity to be optimal, they either introduce multiple optimization objectives[2], leading to algorithm instability in practice, or impose constraints that are difficult to apply in real-world scenarios [3]. Therefore, the practicality of OSD is not possessed by these methods.

However, one of our future goals is to delve into practical algorithms that offer optimal sample complexity.

Q: About the novelty of our method

A: There does exist some work which utilize a reweighing strategy to enhance offline RL methods. The more common practice is to adopt exponentiated advantage or return estimates as importance weights, with different methods of advantage estimation [4-6]. While simple and intuitive, these methods do not have a theoretical guarantee of obtaining the near-optimal supported policy. On the contrary, we utilize the optimal density ratio as the importance weight, which is totally different and more theoretically justified. The most pertinent study to our research is [7] conducted during the same period. This study also suggests utilizing optimal density ratios as weights. However, it is worth mentioning that [7] primarily focuses on the scenario where $\gamma=1$. Consequently, they propose a different algorithm to estimate the optimal density ratio, but no evidence is provided to support its optimality. Our method targets the more common case of $\gamma<1$ and theoretically guarantees learning the near-optimal density ratio. Additionally, when combined with typical offline RL algorithms, it can achieve near-optimal supported policies, which is not possessed by the aforementioned works.

[1] Offline reinforcement learning with realizability and single-policy concentrability

[2] Optimal conservative offline rl with general function approximation via augmented lagrangian

[3] Revisiting the Linear-Programming Framework for Offline RL with General Function Approximation

[4] Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting

[5] Offline Prioritized Experience Replay

[6] Exponentially Weighted Imitation Learning for Batched Historical Data

[7] Beyond uniform sampling: Offline reinforcement learning with imbalanced dataset

We want to sincerely thank you for the invaluable feedback you have provided us. We greatly appreciate your perspectives and are confident that they will contribute to the enhancement of our paper. We will systematically address all of your specific inquiries in our upcoming response. To facilitate your review of the modifications, we have marked them in blue in our revised version.

Q: About clarity and presentation

A: We apologize for the logical leaps and typographical errors in some of our statements. As a result, we have made revisions to the methodology section. On one hand, we have provided a more detailed explanation of the derivation of equation (3) (now equation (5). On the other hand, we have made changes to the parts you mentioned, hoping to alleviate any confusion or doubts you may have.

Q: About the model learning in complex domains

A: For complex domains, we have chosen to use the single-transition estimator to approximate $e_{\nu}$. This choice is made because the MDPs involved in these tasks are approximately deterministic, which means that the bias issue is not a concern as discussed in Corollary 3 in [1]. Therefore, we can rely on the single-transition estimator to provide reliable results in this situation. We have previously clarified this in Appendix C and now emphasized it by providing an explanation in section 5.2.

[1] OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation

Q: About why the regularization term does not change the optimality and its functions

A: The reason why the regularization term does not change the optimality has been proven in Lemma 4 in Appendix A, please refer to it for more details.

The introduction of the regularized term in the optimization objective, $L_{\alpha}(\nu)$, is crucial as it brings about properties similar to strong convexity. This plays a significant role in ensuring that the solution obtained from optimizing $L_{\alpha}(\nu)$ is an approximate global optimum. On the other hand, if we were to remove the regularization term, the resulting optimization objective $L_{\alpha}(w^{*}_{\nu}, \nu)$ would lack strong convexity, the objective function near the optimal value may be too flat, making it difficult for us to bound the difference between the learned solution and the optimal solution.

We would like to express our heartfelt gratitude for offering us valuable feedback. Your thoughtful comments are highly valued as they will undoubtedly help improve our paper. In our upcoming response, we will carefully address each of your specific questions. To ensure clarity, we have indicated all the changes and additions made in our revised version by highlighting them in blue.

Q: About the comparison with OptiDICE

A: We sincerely apologize for the oversight of not including a direct comparison between OSD-BC and OptiDICE for complex tasks in the main text. Due to space limitations, we had to defer the comparison to Appendix D and E.

• In Appendix E Table 3, we have provided an explicit comparison between OSD-BC and OptiDICE. The results indicate that they perform comparably in the maze2d domain, but OSD-BC has a remarkable advantage over OptiDICE in the Locomotion domain.

• Furthermore, we conducted ablation experiments on the regularization coefficient $\beta$, as shown in Appendix D Figure 2(b). The variant with $\beta=0$ corresponds exactly to OptiDICE+CQL. It is worth noting that OSD-CQL consistently outperforms OptiDICE+CQL across all three tasks. This highlights the advantage of the OSD method over OptiDICE when incorporated with behavior regularization-based offline methods.

To provide a more comprehensive comparison between OSD and OptiDICE in complex tasks, we have added a paragraph in Section 5.2 of the main text to discuss the comparison between the two methods.

Q: OSD-DICE requires additional learning of a model compared to OptiDICE, and it is not certain whether the performance improvement over OptiDICE is worth learning a model in complex domains.

A: For complex domains, we have chosen to use the single-transition estimator to approximate $e_{\nu}$. This decision is based on the fact that the MDPs involved in these tasks are approximately deterministic, which means that the bias issue is not a major concern as discussed in Corollary 3 in [2]. Therefore, we can rely on the single-transition estimator to provide reliable results in this situation. We explained this point previously in the Appendix C, and now we have highlighted it by providing an explanation in section 5.2.

Q: For finite domains, while the squared regularization seems to improve a lot, but using a model for unbiased estimator does not seem to improve much. It is counterintuitive as model learning makes the estimator unbiased, while squared regularization does not change the optimal solution (in theory).

A： For a finite MDP, the single transition estimator $e_{\nu}$ is essentially replacing the true transition $P$ with the empirical estimator $\tilde{P}$ obtained from the dataset. While there is some bias in $\tilde{e}_{\nu}$ and the induced objective, the bias is generally small when the dataset size is sufficiently large. This observation helps explain why the performance of model learning is often not that bad compared to not performing model learning.

However, it is important to note that the inclusion of model learning is theoretically necessary. This is because model learning allows us to bound the bias, and the upper bound is solely dependent on the size of the candidate function class $|\mathcal{P}|$ and the size of the dataset $N$.

Q： About the reason to use additional regularization term

A: We propose the addition of a regularization term to enhance the convexity of the optimization objective (5), aiming to transform it into an approximately strongly convex function. This regularization term plays a crucial role in guaranteeing that the solution obtained from optimizing (5) is an approximate global optimum. Conversely, if we were to eliminate the regularization term, the resulting optimization objective would lack strong convexity, the behavior of the objective function near the optimal solution may be too flat, making it difficult for us to bound the difference between the learned solution and the optimal solution, potentially leading to an alternative optima. To clarify the intention behind this regularization term, we have included a description Section 4.2.

Q: Comparison with [1]

A： Thank you very much for pointing out this relevant work. We have compared this work in the main text and supplemented experiments comparing it. The experiments show that the OSD method still has advantages compared to it.

[1] Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets

[2] OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation

Q:Explanation of bias issue induced in single-transiton estimator of $e_{\nu}$.

We apologize for not being able to explain the bias issue in the main text more clearly. The background of this question is as follows.

The original optimization objective is $L_\alpha \left(w_\nu^*, \nu\right)$ which is defined in (4) in the revision. OptiDICE utilizes a single-transition estimation $\tilde e_\nu = r(s,a) + \gamma \nu(s') - \nu(s)$ to approximate $e_\nu$ in (4),
which thus solves an empirical objective $\hat L_\alpha(w_\nu^*,\nu)$. However, due to the non-linearity of $(f')^-1$ and the double-sample problem, it has been proven by Jenson Inequality in Corollary 3 of [1] that $\hat L_\alpha(w_\nu^*,\nu) \geq L_\alpha\left(w_\nu^*, \nu\right)$, where equality holds when the MDP is deterministic. Therefore, a bias arises for OptiDICE.

However, in complex domains, we actually still adopt the single-transition estimator to approximate $e_\nu$, as the MDPs are approximately deterministic on these tasks, and the single-transition estimator is simple and reliable in this situation. This has been explained in Section 5.2.

[1] OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation

We would like to express our heartfelt gratitude for the valuable feedback you have given us. Your insights are highly valued and will undoubtedly help us improve our paper. We will address each of your specific questions in a systematic manner in our next response. In order to make it easier for you to see the changes we have made, we have highlighted them in blue in our revised version.

Q： More comparison with SOTA methods

A: Thank you very much for your advice. We have added EDAC, and some recent reweighing methods as new baselines, as shown in Appendix F, Table 4. The results indicate that OSD method outperforms these baselines in most of the datasets.

Q： About the explanations and verifications of two key designs

A: We apologize for not being able to clearly demonstrate the roles and effects of the two core designs. Their respective functionalities are explained as follows：

• The objective of model learning is to mitigates the bias issue that arises in OptiDICE. This is achieved by ensuring that the empirical objective and the expected objective can be closely controlled, especially when the sample size $N$ is large. Additionally, the error associated with this treatment depends solely on the size of $|\mathcal{P}|$ and $N$.

• The introduction of the regularized term in the optimization objective, $L_{\alpha}(\nu)$, is crucial as it brings about properties similar to strong convexity. This plays a significant role in ensuring that the solution obtained from optimizing $L_{\alpha}(\nu)$ is an approximate global optimum. On the other hand, if we were to remove the regularization term, the resulting optimization objective $L_{\alpha}(w^{*}_{\nu}, \nu)$ would lack strong convexity, the behavior of the objective function near the optimal value may be too flat, making it difficult to bound the difference between the learned solution and the optimal solution, potentially leading to the presence of alternative optima.

In order to provide a clearer explanation of their functions, we have explained the motivation and effects of these two core designs in Section 4.2.

Regarding their practical effects in experimental settings, we have already shown their orthogonal functionality in tabular MDPs in Section 5.1. As for the experimental utility in complex tasks, the verification has been deferred in Appendix D and E due to space limitations. To be more highlighted, we have added a paragraph in Section 5.2 to discuss the contribution of the regularization term.

Q: About the novety of two key designs

• Firstly, both of these designs play a crucial role in theory. OptiDICE, by utilizing a single transition estimator, leads to a biased empirical objective of the original objective. To tackle this issue, we introduce an approximated transition estimator, which allows us to control the error between the empirical objective and the expected objective. The error associated with this approach depends solely on the size of $|\mathcal{P}|$ and $N$. However, even with this modification, solving the new empirical objective may not guarantee the attainment of the global optimal solution. To overcome this limitation, we incorporate a regularization term that does not impact the solution but enhances the convexity of the optimization objective. This regularization term ensures that the learned solution is indeed the optimal solution. Therefore, although the implementation of these two designs is not complex, they are necessary conditions to ensure the optimality of the algorithm.

• Secondly, the proof process is not straightforward. On one hand, we need to demonstrate that the introduction of the regularization term does not change the true optimal solution. On the other hand, due to the complexity introduced by equation (5), it does not strictly adhere to convexity principles, making it impossible to directly apply the properties of strong convex functions. Consequently, we must thoroughly analyze the characteristics of equation (5) in various scenarios, particularly in the vicinity of the optimal solution, to acquire the assurance of optimality.

• Finally, our framework goes beyond the mere solution of the optimization problem $\min_{\nu}\widehat{L}_{\alpha}(\alpha)$ by enabling the incorporation of the learned optimal weights with BC or behavior regularization-based algorithms. This integration enhances the versatility and effectiveness of our framework. Moreover, we offer a theoretical guarantee, as demonstrated by Theorem 10 and Remark 11, that the resulting policy derived from our framework will not be inferior to the optimal policy supported by the dataset. This novel contribution represents a significant advancement that has not been previously explored.

Thank you sincerely for providing us with constructive feedback. We greatly appreciate your insights as they will undoubtedly contribute to the enhancement of our paper. In response to your specific inquiries, we will systematically address each one in our subsequent response. To facilitate clarity, we have highlighted all modifications and additions in blue in our revision.

Q: About the lack of baselines and related works discussion

A: Thank you for your suggestion. We have discussed and compared the literature mentioned by you in Section 2 of the main text. Additionally, we have included the results of these baselines in Appendix F, Table 4-5. The results indicate that OSD-CQL outperforms these baselines in most of the datasets.

Q: About the implementation details of the baselines.

A: We have added implementation details about the baseline in the revised paper, please refer to Appendix G. It is worth mentioning that both the TD3BC-based baselines and the CQL-based baselines are indeed implemented based on [Sun], and the hyperparameters are consistent with osd-TD3BC and osd-CQL, which ensures a fair comparison. We have uploaded our source code package, which includes our algorithms, baseline code, and execution scripts.

Q: About the hyperparameter search results for baselines.

A: We have added the method and results for selecting hyperparameters of the baselines. Please refer to Appendix G.

Q: About the lack of standard deviation for baseline scores.

A: We apologize for not including the std in Table 1 in the main text due to space limitations. For the newly added experiments, we present the std, as shown in Table 4-5 in Appendix F.

Q: About the comparison with OptiDICE

A: We apologize for not presenting the comparison between OSD-BC and OptiDICE for complex tasks in the main text. Actually, the comparison has been deferred in Appendix D and E due to space limitations.

• Firstly, the explicit comparison between OSD-BC and OptiDICE is put in Appendix E Table 3. The results show that they perform comparable in maze2d domain, but OSD-BC has a significant advantage over OptiDICE in Locomotion domain.
• Additionally, we conducted ablation experiments on the regularization coefficient $\beta$, as shown in Appendix D Figure 2(b). The variant with $\beta=0$ corresponds exactly to OptiDICE+CQL. It can be observed that OSD-CQL consistently outperforms OptiDICE+CQL across all three tasks. which highlights the advantage of OSD method over OptiDICE when imcorperated with behavior regularization based offline methods.
• In order to provide a clearer and more explicit comparison between OSD and OptiDICE in complex tasks, we have added a paragraph in Section 5.2 of the main text to discuss the comparison between the two.

Q: About the siginificance of OSD-RL

• Firstly, as shown in Table 1 in the main text, OSD consistently brings significant improvements to the baselines without any reweighing treatment, take CQL based experiments as an example, the total performance gain is about 300+ points on maze2d domain, 100+ points on locomotion domain and 100+ points on Antmaze domain.
• Secondly, from Table 1 again, OSD has remarkable advantages over other mainstream reweighing methods, including top10%, AW and RW, espeically incorperated with behavior regularization based methods. This further validates the effectiveness and universality of OSD.
• At last, we also compare OSD-CQL with additional baselines as the reviewer suggested. The results show that OSD-CQL still has remarkable advantages compared to these newly added baselines.

Q: About the difference between OptiDiCE + SquareReg and OSD-DiCE

A： We apologize for not providing a more detailed explanation about the differences between OSD-DICE and OptiDiCE + SquareReg in the main text. Both OSD-DICE and OptiDiCE + SquareReg use squared regularization in the optimization objective. The difference lies in the fact that OSD-DICE utilizes an approximator $\widehat{P}$ in a realizable function class $\mathcal{P}$ to approximate the advantage function $e_{\nu}$ and OptiDiCE + SquareReg uses the empirical $\tilde{P}$ to approximate $e_{\nu}$. Detailed construction method for $\tilde{P}$ is put in Appendix B. The comparison between the two is to verify the effect of approximate advantage in the tabular case.

Q: About exmperiments for more tasks

A: Thank you for your suggestion. We have added experimental results on some mixed datasets. See Table 5 in Appendix F for more details.

Thank you very much for your recognition of our work and your constructive suggestions. Our response is as follows.

Q: About explanation of Equation (3)

A: We are sorry that we could not make the source of equation (3) clearer. We have provided the background and context of equation (3) (now equation (5)) in Section 4.1 of the main text, highlighted in blue. We hope this helps you better understand the rationale and consequences of this optimization objective.