Learning from Sparse Offline Datasets via Conservative Density Estimation

We thank the reviewer for careful review and valuable suggestions.

• W1. Why does the Slater's condition of optimization problem in eq.(2)~(4) hold.

  Thanks for pointing out this question. Here the Slater's condition is that there exists a strictly feasible $d(s,a)$ s.t. the equality constraint in eq.(3) holds while constraint in eq.(4) is satisfied with strict inequality. One strictly feasible solution is empirical dataset distribution $d^D(s,a)$: (1) Bellman flow constraint $\sum_a d^D(s,a)=(1-\gamma)\rho_0(s) + \mathcal{T}_* d^D(s)$ holds because the dataset is sampled from real environment; (2) $d^D(s,a)=0<\epsilon \mu(s,a),\forall s,a\in\text{supp}(\mu)$ by definition of $\mu$. Therefore, the Slater's condition holds.

  We have added it in appendix A.1 in revision to make our statement more clear.

• W2. RHS of eq.(7) may not exist when $\tilde{A}(s,a)<0$

  Actually, it still holds when $\tilde{A}(s,a)<0$ because we assume $(f')^{-1}(x)>0, \forall x\in \mathbb{R}$ in Assumption 1. In practice, we adopt soft-chi function as $f$ as mentioned in training details in page 7, where

  $$f_{\text{soft}-\chi^2}(x) = \begin{cases} x\log x -x +1, 0<x<1\\\\ (x-1)^2/2,x\geq 1 \end{cases}$$

  Therefore,

  $$(f')^{-1}(x) = \begin{cases} e^{x},x<0\\\\ x+1,x\geq 0 \end{cases}$$

  This satisfies the above assumption and thus we still can compute RHS of eq.(7) when $\tilde{A}(s,a)<0$.

• W3. Set $d^{*}(s)$ as a uniform distribution on the state in the successful trajectories. Why is it close to $\tilde{w}(s,a)\hat{d}^D(s,a)$.

  $d^*(s)$ denotes the state distribution of optimal policy $\pi^*$, and $\pi^*$ should always succeed in any trajectory. So we believe it is reasonable to approximate $d^*(s)$ as the empirical distribution of the given successful trajectories.

  The empirical distribution can be close to the state margin of $d^*(s,a)$ (i.e., $\sum_a \tilde{w}(s,a)\hat{d}^D(s,a)$): the state-action pair $(s,a)$ in failed trajectories should have a smaller importance ratio $\tilde{w}(s,a)$ than $(s,a)$ in successful trajectories; therefore, the density of $s$ from successful trajectories should also be much larger than $s$ from failure ones. Meanwhile, since it is infeasible to do summation of $\sum_a \tilde{w}(s,a)\hat{d}^D(s,a)$ over $a$ in continuous space to directly compute the state margin, we use the approximation for easy computation.

• Q1. How are the OOD actions sampled? Do you just sample a random action and reject it if it is in distribution?

  Yes, we uniformly sample an action and reject it if it is in distribution.

• Q2. Does Slater's condition also hold for optimization problems dealing with infinite-dimensional vectors (i.e., functions)?

  Yes. In fact, the stationary distribution $d:\mathcal{S}\times\mathcal{A}\rightarrow [0,1]$ is indeed a function if we consider continuous state-action space, which is the case of our experiment. Meanwhile, we can observe that the empirical dataset distribution $d^D(\cdot, \cdot)$ is still strictly feasible for optimization in eq.(2)~(4). Therefore, the Slater's condition still holds.

• Minor issues.

  Thanks for your careful check, we have corrected them in revision.

Thanks again for your feedback. We hope our response addresses your main concerns.

• Q1. The performances on sparse-mujoco with different cut-off percentiles.

  Since CDE exceeds the baselines by a large margin on "-medium-expert" tasks, we test the performances with 90 percentile and 50 percentile as cut-off thresholds on "-medium" level tasks. The results are as follows:

  The success rate of CDE and baselines with cut-off percentile=90

  	BCQ	CQL	IQL	TD3+BC	CDE (Ours)
  halfcheetah-medium	25.8±11.7	88.0±1.8	56.0±3.6	48.5±21.2	81.2±2.9
  walker2d-medium	34.8±10.6	1.5±0.7	3.3±2.1	2.5±3.5	53.5±14.8
  hopper-medium	1.2±0.8	65.3±6.3	0.0±0.0	0.0±0.0	74.2±5.7

  The success rate of CDE and baselines with cut-off percentile=50

  	BCQ	CQL	IQL	TD3+BC	CDE (Ours)
  halfcheetah-medium	73.7±2.5	99.3±0.9	93.5±1.1	37.0±24.1	92.5±2.8
  walker2d-medium	38.0±1.2	47.6±17.1	18.7±2.5	15.7±10.8	67.0±5.0
  hopper-medium	9.3±1.9	90.0±0.8	6.0±7.8	0.0±0.0	88.0±5.7

  When the percentile decreases, the reward becomes denser, and the threshold of success is lower. Therefore, the most baselines have higher success rate. We notice that CDE keeps relative high performance with 90 cutoff percentile. Meanwhile, CDE still obtains comparable performances with 50 percentile (CQL performs well in original 75 percentile setting).

• Q2. The performance on dense reward settings.

  We directly adopt the reported scores of baselines from original paper or D4RL benchmark and compare CDE with them.

  	BCQ	CQL	IQL	TD3+BC	CDE
  halfcheetah-medium	40.7	49.1	47.4	48.3	43.3±2.9
  walker2d-medium	54.5	82.9	78.3	83.7	73.8±4.4
  hopper-medium	53.1	64.6	66.3	59.3	51.2±3.7
  halfcheetah-medium-expert	64.7	85.8	86.7	90.7	75.6±7.2
  walker2d-medium-expert	110.9	109.5	109.6	110.1	107.7±10.4
  hopper-medium-expert	57.5	102.0	91.5	98.0	108.6±4.8

  While CDE may not outperform the baselines in dense reward settings, it achieves a significantly higher success rate when reward is sparse, which indicates that the baseline performances are predominantly reliant on dense rewards, thereby highlighting the effectiveness of CDE for sparse-reward offline data.

References:

[1] Provably good batch off-policy reinforcement learning without great exploration

[2] Bellman-consistent pessimism for offline reinforcement learning

[3] Conservative q-learning for offline reinforcement learning

[4] Mildly conservative Q-learning for offline reinforcement learning

We thank the reviewer for valuable feedback.

• W1. Some clarifications are required in the introduction section.

  • Why is over-pessimism a bigger problem for high-dimensional state-action spaces?

    Our statement is based on (1) the accurate data distribution estimation is more difficult when space is high-dimensional, and (2) it is relatively harder to overcome over-pessimism issue with large estimation error on data distribution [1][2]. But we also concede that the over-pessimism is not necessarily more severe when space is high-dimensional. We have modified our statements accordingly and thanks for pointing it out.

  • Why do regularization methods struggle with the tuning of the regularization coefficient?

    We do agree many offline RL methods can achieve good performances on different tasks with similar hyper-parameters such as IQL or TD3+BC. However, there are also some methods [3][4] requiring finetuning the regularization coefficient.

    Although the cited paper is DICE-based, it adds a f-divergence regularization in the objective. This paper also finetunes the coefficient in different tasks.

  Thanks for your careful check. We update statements in revision and make them more rigorous now.

• W2. The considered tasks are not thorough.

  Thanks for your questions. We add more experiment results as follows.

  • Mujoco medium-replay tasks.

    	BCQ	CQL	IQL	TD3+BC	CDE (Ours)
    halfcheetah-m-r	91.0±2.2	48.0±14.9	70.7±3.7	61.0±16.3	95.2±3.5
    walker2d-m-r	85.7±8.4	78.3±11.2	4.0±4.9	84.4±7.3	90.4±5.0
    hopper-m-r	91.4±5.2	87.0±5.1	0.0±0.0	89.0±4.1	93.3±4.6

    CDE exceeds the compared baselines on medium-replay tasks.

  • Adroit cloned tasks.

    	BCQ	CQL	IQL	CDE (Ours)
    pen-cloned	44.0	39.2	37.3	56.9±13.2
    hammer-cloned	0.4	2.1	2.1	3.3±1.7
    door-cloned	0.0	0.4	1.6	0.3±0.1
    relocate-cloned	-0.3	-0.1	-0.2	0.4±0.3

    We did not include them in our original manuscript because both the offline data (collected by a policy which is trained by behavioral cloning on human "tasks") and final performances are similar to "human" tasks. Our method still obtains the best or comparable performances on most tasks. We have added the new experiments in appendix B.3.2 in revision.

  • Antmaze and kitchen tasks.

    In antmaze and kitchen tasks, the initial distributions are significantly different between training and test settings, which violates the basic assumption of CDE. For the antmaze tasks, the starting point can be anywhere in the offline dataset but is always at the same corner when tested. The mismatch also happens on kitchen tasks. On the contrary, the optimization objective of CDE (eq.(6)) is given by initial distribution $\rho_0$ and $d^D$, requiring the consistency of initial distribution. As we mentioned in "Conclusion" section, one direction to overcome this inconsistency is to incorporate goal-conditioned setting, which we leave as a future work.

• W3. Some baselines are missing, e.g., DT and Diffuser.

  Thanks for your suggestion on baselines. We add comparison as follows (we report normalized score for maze2d and success rate for sparse mujoco).

  	Decision Transformer	Diffuser	CDE (Ours)
  maze2d-umaze	21.1±25.4	113.9±3.1	134.1±10.4
  maze2d-medium	29.7±13.3	121.5±2.7	146.1±13.1
  maze2d-large	41.7±14.1	123.0±6.4	210.0±13.5
  halfcheetah-medium	39.7±1.9	66.7±10.2	82.0±8.6
  walker2d-medium	10.3±2.1	46.3±16.7	53.0±11.7
  hopper-medium	63.0±19.9	94.0±2.8	85.5±5.7
  halfcheetah-medium-expert	0.0±0.0	0.0±0.0	95.2±2.9
  walker2d-medium-expert	0.0±0.0	74.0±11.0	97.0±2.8
  hopper-medium-expert	0.0±0.0	92.7±5.0	97.0±1.4

  Our method exceeds DT and diffuser on most tasks. Despite of its notable performances in dense reward setting, the performances of DT drop remarkably in sparse reward setting, indicating DT heavily relies on dense reward to train the transformer and supervise policy learning.

We thank the reviewer for valuable review and address the concerns as follows.

• W1. The effectiveness of "warm-up" stage for value function training.

  We use "warm-up stage" to refer to "policy update happens when the value function converges". We adopt warmup because (1) the value function is not well learned in the early stage and thus it cannot well guide the policy learning at that time, and (2) warm-up also reduces the computation overhead because policy update is stopped.

  Meanwhile, the warm-up stage does not influence the final performances significantly. We give an ablation study on maze2d tasks (w. warm-up means stopping policy update at the beginning; w.o. warm-up means training value and policy together from start) and present results as follows:

  Task	w. warmup	w.o. warmup
  maze2d-umaze	134.1±10.4	136.2±13.2
  maze2d-medium	146.1±13.1	141.7±17.3
  maze2d-large	210.0±13.5	205.4±15.4

• W2. More parameter studies. The study of max OOD IS ratio does not provide guideline.

  We include more parameter studies in appendix B.4 in original manuscript. We can observe that the performance of CDE is not sensitive to the most hyperparameters except max OOD IS ratio and mixture coefficient. For mixture coefficient, we can simply choose $\zeta$ to be close to 1 (e.g., set $\zeta=0.9$) by analysis in fig.3.

  For the max OOD IS ratio, we agree that this study does not directly offer guidance on selecting $\tilde{\epsilon}$ when encountering new tasks. Nonetheless, by consistently employing the same $\tilde{\epsilon}$ across all experiments, we demonstrate its broad applicability to a variety of tasks. Meanwhile, our parameter study in appendix B.4.1 also shows that CDE is relatively resilient to the change of $\tilde{\epsilon}$. But we also admit that we may need to fine-tune it for specific new tasks.

• W3. Compare CDE and CQL in detail, both theoretically and empirically.

  Thanks for your suggestions.

  As the reviewer mentioned, CDE and CQL apply pessimism on state-action occupancy distribution and Q-value function respectively. But in theory, we can also link them by the eq.(7), which shows that the optimal IS ratio is related to the value function. The main difference is that CQL adds an expectation of Q value on OOD region as a soft regularization with coefficient $\alpha$ while CDE enforces a direct constraint on IS ratio (can be also viewed as a constraint on advantage value by eq.(7)), which allows for more precise control on conservativeness for CDE.

  We also provides some empirical results on maze2d-large task*. We test the performances of CQL with different regularization coefficient $\alpha$.

  $\alpha$	0.1	0.5	1.0	5.0	10.0
  maze2d-large	-0.1±1.1	98.2±24.8	57.8±8.1	4.5±3.7	0.4±3.0

  We can observe that CQL does achieve high score with $\alpha=0.5$. However, compared to CDE (see table 10 in appendix for parameter study of CDE), CQL's performance exhibits extreme sensitivity to the choice of $\alpha$. This sensitivity creates a challenging balance in fine-tuning the hyperparameter, oscillating between over-pessimism and substantial extrapolation error in OOD regions.

  (*) We directly adopt results of CQL ($\alpha=5.0$) from D4RL benchmark in our original manuscript, which is a little different from the re-run results with $\alpha=5.0$.

Thanks again and we hope our response addresses your main concerns.

We thank the reviewer for valuable review and address the main concerns as follows.

• Precise differences with DICE could have been made more clear; more discussion of the effects of the additional constraint.

  We summarize the key differences of CDE from previous DICE method:

  1. In theory, CDE has an upper bound for concentrability ratio on OOD regions (Proposition 3). We also derive the upper bound in context of function approximation (Theorem 1). Therefore, the additional constraint avoids the arbitrarily large IS ratio and can mitigate issues caused by extrapolation error on OOD regions.
  2. We further derive the performance gap bound in Theorem 2, which is absent in previous DICE paper.
  3. Empirically, we verify the effectiveness of additional constraint on various tasks. The OOD constraint can reduce potential extrapolation error so we study the situations where offline data are scarce (i.e., the occupied state-action pairs are sparse in the whole space) and thus the OOD issue is more severe. From fig.1, we can find that CDE significantly outperforms optidice when offline data are limited.

• The theorem part can be better organized, e.g., to make the statement of theorem, assumption, and proof as a whole.

  Thanks for your suggestion. As you mentioned, we may not be able to insert proof into main text. So we repeat the statement of theorems in appendix to make them more readable and we also keep them in main test. The modification can be found in appendix part in revision.

Thanks again for your feedback and we will emphasize them in the paper. We are more than glad to clarify it if you have further questions.

References:

[1] Nachum, Ofir, and Bo Dai. "Reinforcement learning via fenchel-rockafellar duality." arXiv preprint arXiv:2001.01866 (2020).

[2] Lee, Jongmin, et al. "Optidice: Offline policy optimization via stationary distribution correction estimation." International Conference on Machine Learning. PMLR, 2021.