Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset

Dear Reviewer,

Thanks for finding our paper interesting and novel. We hope the following statement clear your concern.

W1 and Q1: Where does the performance gain come from? and Does simply filtering the dataset by high returns recover the same performance?

A for W1 and Q1: The reason for using Equation 5 instead of Equation 3 is that in the offline RL setting, we cannot directly solve the problem defined by Equation 3. For this reason, we attempt to adopt Equation 5 as an approximate alternative optimization problem.

The performance gain comes from two aspects: (1) Reduced dataset can eliminate a large amount of redundant data, making the learning process of the algorithm more efficient. (2) we balance data quantity with performance by focusing on data points that are aligned with the learned policy.

As suggested, we conduct ablation study in the D4RL (Hard) tasks by simply filtering the dataset by high returns. The experimental results in Table 1 show that simply filtering data with high returns is not sufficient to achieve good performance. This is because the task cannot be well-defined if only given the expert demos. Diverse trajectories can help specify the boundary of the task.

Finally, we have added the missing references as suggested in the revised version.

Table 1. Comparison with the simply filtering baseline in the D4RL (Hard) tasks.

Q2: What is the performance of ReDOR on the original version of D4RL?

A for Q2: As suggested, we conduct additional experiments on the D4RL (Original) tasks. The experimental results in Table 2 show that the reduced dataset on the original dataset can still bring performance gains to the algorithm.

Table 2. Performance on the D4RL (Original) tasks.

Q3: How is the hard dataset generated? How many datapoints are added to the dataset?

A for Q3: We added noise data generated by various behavioral policies to the original dataset to simulate the noise in data collection in the real world. In each dataset, the size of the added noise data is 20% of the original dataset.

Q4: How many datapoints are removed by ReDOR? What is the size of the reduced datasets?

A for Q4: In our experiments, each dataset reduces by approximately 70% to 90%. Specifically, for Walker2d and Hopper, there will be a reduction of 70%, while for Halfcheetah, there will be a reduction of 90%.

Q5: Is there a way to tune the resulting dataset size?

A for Q5: Yes, we can tune the size of the data subset by changing the approximation error $\epsilon$. For example, as $\epsilon$ increases, the precision of gradient matching decreases, resulting in a smaller data subset. Conversely, as $\epsilon$ decreases, the data subset becomes larger.

Q6: Is Fig 3, episode return = 99.5 for behaviors [2-7] correct or a bug?

A for Q6: Thanks for your suggestions. It is a bug and we have corrected it in the revised version.

Thanks again for the valuable comments. We hope our response has cleared your concern. We are looking forward to more discussions

Dear Reviewer,

Thanks for finding our paper novel, interesting, strong theoretical study, and empirical analysis. We hope the following statement clear your concern.

W1: Short discussion on model-based methods like MOPO and MoERL.

A for W1: Thanks for your suggestions, we have corrected it in the revised version.

W2: How is $F_{\lambda}(s)=L_{\rm max}- min_wErr_{\lambda}(w, S, L, \theta)$ used in the OMP.

A for W2: Since $L_{\rm max}$ is a constant, maximizing $F_{\lambda}(s)$ is equivalent to minimizing $Err_{\lambda}(w, S, L, \theta)$. Therefore, we adopt OMP to directly minimize $Err_{\lambda}(w, S, L, \theta)$.

W3 and Q2: If the method leads to the claimed reduction in complexity and Are the networks $Q_{\theta}$, $\pi_{\phi}$ networks first trained on the full dataset before starting with the subset selection?

A for W3 and Q2: Yes, the selected data subset reduces the computational complexity. Specifically, we first train $Q_{\theta}$ and $\pi_{\phi}$ on the full dataset before starting with the subset selection. Then, we load the pre-trained parameter $\theta, \phi$ to select the data subset and re-train offline RL methods on the reduced dataset from scratch. The experimental results on the paper show that reduced dataset speeds up the training process.

Minor:

• The table references do not match the table numbers: Thanks for your suggestions, we have corrected it in the revised version.

• Notations table in Appendix: Thanks for your suggestions, we have added the notation table in the revised version.

Q1: How is the weight $w_i$ or $\lambda$ decided during training and the parameters $L_{\rm max}, m, \epsilon$ chosen in practice?

A for Q1: In practice, $w_i$ and the indices $i$ are generated simultaneously by OMP. As for other parameters, we use uniform values across different tasks, $\lambda=0.1, m=50, \epsilon=0.01$. Since we use the OMP method to directly minimize $Err_{\lambda}(w, S, L, \theta)$, we do not need to set $L_{\rm max}$.

Q3: What is the empirical reduction percentage achieved in each dataset?

A for Q3: In our experiments, each dataset reduces by approximately 70% to 90%. Specifically, for Walker2d, there will be a reduction of 70%, while for Halfcheetah, there will be a reduction of 90%.

Q4: Why ReDOR performs better than a dataset containing only expert trajectories in Figure 1.

A for Q4: In the tasks depicted in Figure 1, we evaluate baselines on D4RL (Hard), which includes some suboptimal noisy data. We find that suboptimal data may lead to significant performance degradation due to distribution shifts. For this reason, ReDOR performs better than other baselines in the walker2d-expert-v0 task.

Q5: What do samples with highest TD Loss mean?

A for Q5: The $`Prioritize`$ baseline is designed based on the Prioritized Experience Replay for online RL [1]. Specifically, when the value function is updated with gradients, prioritizing data with larger loss values provides more information for policy learning. Inspired by this, in each round of data selection, we choose data with highest TD Loss as the data subset.

Q6: How does ReDOR perform on random datasets such as halfcheetah-random-v2?

A for Q6: As suggested, we conduct experiments on the random dataset, as shown in the Table 1. The experimental results show that the performance improvement on the random dataset is not significant, which is because there is a relatively small amount of high-quality data in the random dataset.

Table 1. Experimental results on the random dataset.

Q7: Why are the reduced dataset points more for category 6 when it is a subset of complete dataset?

A for Q7: In Figure 3, each component reflects a distinct skill of the agent. The category 6 represents the stepping, which is relatively important compared with other skills, hence more points are allocated.

We sincerely thank the reviewer again for the timely and valuable comments. We hope that our response and additional experimental results have cleared most of your concerns.

[1] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015.

Q9: Why use -v0 version?

A for Q9: TD3+BC algorithm in the original paper was evaluated on the -v0 version, so we also conduct experiments on the -v0 version, for no other reason.

Q10: The standard deviations of IQL.

A for Q10: We run the official code released by the IQL authors, supplementing the missing standard deviation in the paper.

Q11: Recent baselines in dataset selection of offline RL.

A for Q11: The baselines considered in this paper are common approach to data selection in supervised learning. This is also why we chose these methods as our baselines. As suggested, we evaluate the recent baselines (ADS [1]) in the D4RL (Hard) tasks. The experimental results in Table 4 show that ReDOR achieves better performance than ADS in most tasks. We thank the suggestions of the reviewer, and we will place the complete experimental results and discussion in the revised version later.

Table 4. Comparison with ADS in the D4RL (Hard) tasks.

Q12: More details on what is meant by the "Complete Dataset" baseline.

A for Q12: In the original D4RL tasks, the complete dataset baseline denote the original dataset. As for the D4RL (Hard) tasks, the complete dataset denote the original mixture of the desired dataset and the suboptimal dataset.

Thanks again for the valuable comments. We hope our additional experimental results and explanation have cleared the concern. We sincerely hope that the reviewer can re-evaluate our paper after seeing the our response. More comments on further improving the presentation are also very much welcomed.

[1] Wang, Da, et al. "Improving Generalization in Offline Reinforcement Learning via Adversarial Data Splitting." Forty-first International Conference on Machine Learning.

Dear Reviewer,

Thanks for your valuable comments. We hope the following statement can address your concern.

W1 and Q1: How to generate hard datasets.

A for W1 and Q1: We added noise data generated by various behavioral policies to the original dataset to simulate the noise in data collection in the real world. In each dataset, the size of the added noise data is 20% of the original dataset.

Q2: How was $Q_{\theta}$ formulated?

A for Q2: We first train $Q_{\theta}$ on the full dataset before starting with the subset selection. Then, we load the pre-trained parameter $\theta$ to select the data subset and re-train offline RL methods on the reduced dataset from scratch.

Q3: The results of simply selecting trajectories with top returns for comparison.

A for Q3: As suggested, we conduct ablation study in the D4RL (Hard) tasks by simply filtering the dataset by high returns ($m=50$). The experimental results in Table 1 show that simply filtering data with high returns is not sufficient to achieve good performance. This is because the task cannot be well-defined if only given the expert demos. Diverse trajectories can help specify the boundary of the task.

Table 1. Comparison with the simply filtering baseline in the D4RL (Hard) tasks.

Q4: What is the exact search space of the selected trajectories?

A for Q4: To balance the data quantity with performance, we aim to focus on data points that are aligned with the learned policy, avoiding performance degradation caused by suboptimal trajectories. As suggested, we conduct additional experiments by adding the same filter module in the baselines. The experimental results in Table 2 show that ReDOR still performs better than baselines equipped with the filter module. This is because the reduced dataset can eliminate a large amount of redundant data, making the learning process of the algorithm more efficient.

Table 2. Comparison with baselines in the D4RL (Hard) tasks.

Q5 and W1: The selection of $m$.

A for Q5 and W1: We thank the reviewer for raising the point. Due to poor communication among collaborators, the initial codebase was inadvertently uploaded.

Q6 and W2: The inner product operation in Algorithm 2.

A for Q6 and W2: $r$ is not scalar. In practice, we use the last layer of the gradients for neural networks and $r$ denotes the residual error, which is a vector and the shape is ($\cdot$,257).

Q7: The inequality in Algorithm 2 appears to be reversed.

A for Q7: Thanks for your suggestions, we have corrected it in the revised version.

Q8: Is there a reason why TD3+BC was chosen as the backbone offline RL algorithm for the MuJoCo tasks? and Would using IQL, as in the Antmaze tasks, provide a more consistent comparison?

A for Q8: TD3+BC algorithm is one of the well-known algorithms in the offline RL community, and we have chosen TD3+BC as the backbone. As suggested, we conduct additional experiments in MuJoCo tasks based on IQL. The experimental results in Table 3 show that IQL can still be adapted to Mujoco tasks with ReDOR.

Table 3. Experimental results of ReDOR (IQL) in the D4RL (Hard) tasks.

Dear Reviewer,

We have conducted additional experiments on ablations and baselines. We are wondering if our response and revision have cleared your concerns. We would appreciate it if you could kindly let us know whether you have any other questions. We are looking forward to comments that can further improve our current manuscript. Thanks!

Best regards,

The Authors

Dear Reviewer,

Thanks for your valuable and detailed comments. We hope the following statement clear your concern.

W1: The assumptions in Theorem 4.1 are rather stronger than scenarios in actual implementations.

A for W1: We thank the reviewer for raising the point. If the gradients of the algorithm diverge in practice, it would contradict our assumptions, and the selected data subset would no longer be valuable. However, there are currently various empirical techniques [1,2,3] that can overcome this issue, ensuring that the algorithm's gradients remain stable.

W2: If such approach would lead to a guaranteed good estimation of values/Q-functions.

A for W2: In practice, we found that if we use the standard Bellman backup errors, the gradients used for data selection can be unstable. On the other hand, if we use the relatively fixed target, it cannot lead to the good estimation of Q-functions. To address this issue, we first train $Q_{\theta}$ and $\pi_{\phi}$ on the full dataset based on the standard Bellman backup errors before starting with the subset selection. Then, we load the pre-trained parameter $\theta, \phi$ to select the data subset based on Equation 13, which can ensure relatively accurate and stable gradients.

W3: The theorems only consider the proposed method defined with classic TD loss.

A for W3: The theoretical analysis in Section 5 provides conclusions when the gradient approximation error is $\epsilon$. On the other hand, the techniques in Sections 4.2-4.3 can guarantee the gradient approximation error is lower than $\epsilon$. Therefore, the theoretical conclusion in Section 5 can be adapted into our algorithm.

W4: Why $S^k$ can always start from the cluster center of gradients.

A for W4: This is the assumption in our theoretical analysis. However, this assumption is not difficult to achieve in practice. We can first cluster the dataset and using the cluster center as the initial point before selecting the data subset within each cluster.

C1: Some index $t$ of should be included in the notation of $Q$.

A for C1: Thanks for your suggestions, we have corrected it in the revised version.

C2: Horizon $H$ is not explicitly defined.

A for C2: Thanks for your suggestions, we have corrected it in the revised version.

C3: There is not enough information for $L_{\rm max}$.

A for C3: Since $L_{\rm max}$ is a constant, maximizing $F_{\lambda}(s)$ is equivalent to minimizing $Err_{\lambda}(w, S, L, \theta)$. Therefore, we adopt OMP to directly minimize $Err_{\lambda}(w, S, L, \theta)$ and do not need to set $L_{\rm max}$.

C4: There lacks for an introduction to how KRLS, Log-Det and BlockGreedy are implemented in such offline RL settings.

A for C4: We implement KRLS, Log-Det and BlockGreedy based on the standard Bellman backup errors, and the specific data selection process remains consistent with the original paper.

We sincerely thank the reviewer again for the timely and valuable comments. We hope that our response have cleared most of your concerns.

[1] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020.

[2] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145, 2021.

[3] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021.

Dear Reviewer,

We have added additional explanations for our methods. We are wondering if our response and revision have cleared your concerns. We would appreciate it if you could kindly let us know whether you have any other questions. We are looking forward to comments that can further improve our current manuscript. Thanks!

Best regards,

The Authors

Thanks for the responses! Please see my follow-ups:

If the gradients of the algorithm diverge in practice, it would contradict our assumptions, and the selected data subset would no longer be valuable. However, there are currently various empirical techniques [1,2,3] that can overcome this issue, ensuring that the algorithm's gradients remain stable.

• First of all, in the current proposed method it has not been clearly introduced how such boundedness is guaranteed. In addition, either empirically or theoretically, it would be better if one can verify those boundedness assumptions are satisfied. Otherwise such strong assumptions may restrict the capabilities of the method.

A for W2

• So it means the target is actually biased? (not following the Bellman operator)?

• If some pre-training on whole dataset is needed, then the advantage of acceleration could not hold?

the techniques in Sections 4.2-4.3 can guarantee the gradient approximation error is lower than $\\epsilon$

• I see the techniques, but not the justification. For example, both theorems explicitly use the term 'TD loss', then why is it the case in Equation (14), that is not the TD loss?

Dear Reviewer,

Thanks for your reply! We hope the following statement can clear your remaining concerns.

Boundedness issue of gradients.

A: Thanks again for your suggestions. We also appreciate the valuable comments, which helped us significantly improve the paper's strengths.

Using TD loss or not.

A: Yes. In the TD loss version, all pre-training, similar to the primary procedure, has also been used besides using TD loss.

Training cost.

A: As suggested, we conduct additional experiments for the computational cost between different approaches (with or without pre-training). Specifically, we train TD3+BC and IQL on the reduced dataset generated by ReDOR (named TD3+BC (Reduce) and IQL (Reduce)). On the other hand, we train TD3+BC and IQL on the standard datasets (named TD3+BC (original) and IQL (original)). We record the time the algorithms reach the same performance. The experimental results in Table 1 show that the computational cost spent training on the reduced dataset is significantly lower than on the original dataset.

Note that the results for IQL are generally faster than those for TD3+BC because we are using the official code for IQL, implemented with JAX, which is faster than PyTorch. All experiments are conducted on the GeForce RTX 3090 GPU device. In the revised version later, we will conduct an empirical comparison between more approaches.

Table 1. Computational cost between different approaches (with or without pre-training). m denotes the minute.

Best,

The Authors

Thanks for the details.

Boundedness issue of gradients

• I appreciate that the authors explicitly clarify such issue as a limitation in the corresponding part of the paper.

Using TD loss or not

• The result attached looks interesting, roughly showing that consistency in such setting may not always be ideal. Just want to confirm, when adopting the TD loss version, i.e. the first column in the table, some pre-training similar as the primary procedure has also been used?

Training cost

• It's ok not to have such results currently, but could help for some later revision if an empirical comparison between different approaches (with or without pre-training) is provided.