Query-Efficient Offline Preference-Based Reinforcement Learning via In-Dataset Exploration

Dear Reviewer,

We thank the Reviewer for the constructive comments and clarify your concerns as follows. We would appreciate it if you have any further feedback.

W1: Does this paper violate the paper length limit?

A for W1: This paper does not violate the paper length limit. According to the ICLR guidelines, the reproducibility statement does not count toward the page limit.

Please refer to the author guideline (https://iclr.cc/Conferences/2024/AuthorGuide) for more details.

W2: How is a small suboptimality related to "query efficiency"?

A for W2: A smaller suboptimality bound indicates that we need fewer queries to achieve the same performance bound and, hence, better statistical efficiency. We use "query efficiency" to refer to the fact that our method aims to reduce the number of required preference queries. At each iteration, our method uses an online exploration objective to select a query to ask for preference labels, which can be seen as a special form of active learning.

W3.1: Gap between Algorithm 1 and Algorithm 2.

A for W3.1: We respectfully disagree with this assessment. Algorithm 1 is an extension of Algorithm 2 in the deep learning setting without significant gaps. Specifically,

1. Query Oracle. Both Algorithms 1 and 2 require querying the preference between two given trajectories, as standard in preference-based RL. There is no difference in the requirement for query oracles between Algorithms 1 and 2.

2. Optimistic Bonus. The optimistic bonus is used for selecting policy candidates that are possibly optimal, which aligns with the theoretical formulation for the near-optimal policy set as in Equation (8). In fact, Equation (8) can be rewritten as the policy set where every policy is an optimal policy under some optimistic Q-functions. Then, Algorithm 2 can be rewritten using an optimistic bonus, as in some prior works [2]. The variance-based bonus in Equation (12) reflects the uncertainty over the reward functions, which aligns well with the confidence set formulation as in Equation (7).

3. Expectile Loss. The expectile loss in Equations (14) and (15) is used as a conservative estimation for the value function, as proposed in IQL [1]. This aligns well with the use of PEVI in Algorithm 2, which uses a pessimistic value estimation to exploit the offline dataset.

W3.2 The intuitive explanation of adding clipped Gaussian noise.

A for W3.2: The motivation to add weights is to learn better credit assignments with trajectory-wise information. Since we are only allowed to query the preference between trajectories, it is possible that we can not learn proper temporal credit assignment with only fixed data.

For example, suppose we have two trajectories $\tau\_1$ and $\tau\_2$ that finish a task with two steps, knowing the preference between $\tau\_1$ and $\tau\_2$ give us no knowledge about how well they do at each step. Since we are not allowed to generate new trajectories, a natural idea is to give weights to different parts of the trajectory and ask for the preference again.

Specifically, we can give high weights for step 1 and low weights for step 2 to distinguish which trajectory performs better at step 1. Therefore, augmenting trajectories can help for better temporal credit assignment and efficient preference learning.

We have added experiments on Meta-world tasks to further demonstrate the advantage of using weighted trajectories for queries, as shown in Appendix G. We have made this point more clear in the revised version.

As to the choice of using clipped Gaussian noise, we clip the range of the weight to avoid large weights that can lead to high variance in the learning process, as commonly used in practice, like in TD3. We also demonstrate the importance of weight clipping in the chain MDP experiment shown in Figure 10 (a) in Appendix G. We have made this motivation more clear in the revised version.

Q1: What these numbers in the table 1 and 2 are?

A for Q1: The numbers in Tables 1 and 2 represent the (normalized) performance of the offline RL algorithm using the reward labels learned via the given preference-based reward learning algorithm. A higher performance indicates a better learned reward function.

Q2: Why 10 queries work well?

A for Q2: Theoretically, preference queries contain rich information such that learning from preferences is not harder than standard RL [3]. Our theoretical result shows that offline PbRL can be made more efficient in the number of queries by leveraging the knowledge from the offline dataset. A similar phenomenon is also observed in prior and concurrent works [4,5].

Another possible reason is that offline algorithms are known to be robust to the reward functions [6,7,8]. Therefore, we can have a good performance as long as the reward function learned from preference queries does not deviate too much from the true reward functions.

Thanks again for the valuable comments. We hope our additional experimental results and explanation have cleared your concerns, and we sincerely hope the reviewer can re-evaluate our paper based on our response. More comments on further improving the presentation are also very much welcomed.

[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

[2] Jin, Chi, et al. "Provably efficient reinforcement learning with linear function approximation." Conference on Learning Theory. PMLR, 2020.

[3] Wang, Yuanhao, Qinghua Liu, and Chi Jin. "Is RLHF More Difficult than Standard RL? A Theoretical Perspective." Thirty-seventh Conference on Neural Information Processing Systems. 2023.

[4] Shin, Daniel, Anca D. Dragan, and Daniel S. Brown. "Benchmarks and algorithms for offline preference-based reward learning." arXiv preprint arXiv:2301.01392 (2023).

[5] Anonymous. "Flow to Better: Offline Preference-based Reinforcement Learning via Preferred Trajectory Generation." https://openreview.net/forum?id=EG68RSznLT

[6] Shin, Daniel, Anca D. Dragan, and Daniel S. Brown. "Benchmarks and algorithms for offline preference-based reward learning." arXiv preprint arXiv:2301.01392 (2023).

[7] Li, Anqi, et al. "Survival Instinct in Offline Reinforcement Learning." arXiv preprint arXiv:2306.03286 (2023).

[8] Hu, Hao et al. “Unsupervised Behavior Extraction via Random Intent Priors.” arXiv preprint arXiv:2310.18687 (2023).

Q5 & Q6: Is an ensemble of reward functions going to give reasonable uncertainty estimates of the preference model? What if preferences have high aleatoric uncertainty?

A for Q5 & Q6: It is a common practice to use an ensemble of reward functions to estimate the uncertainty of the preference model, and they are known to lead to good empirical performance [4,5]. Under the Bradley-Terry preference model, the aleatoric uncertainty is accounted for as a soft probability rather than a one-hot probability. In general cases, advanced methods like bootstrapping have the potential to improve uncertainty estimation further and can separate aleatoric and epistemic uncertainties [6], which is an interesting direction for future work.

Q7: How is the optimization problem in (13) solved?

A for Q7: Equation (13) takes the pair of trajectories that maximize the value difference. Note that there is no optimization procedure (for continuous variables) needed, and we only need to search for the best candidate, which requires $O(M^2N^2)$ time, where $M$ is the number of sampled trajectories, and $N$ is the number of Q-values.

Q9: How does this policy error bound compare to, e.g., a G-optimal design and then PEVI?

A for Q9: Without weighted queries, pure offline methods are known to suffer from an exponential policy error bound [7]. When using weighted queries, a G-optimal design and then PEVI would yield a similar performance bound, but the second term consisting of the number of queries $K$ in Equation (10) would depend on an additional coverage coefficient [8]. This can lead to suboptimal performance when the dataset has poor coverage, which motivates us to introduce exploration into offline PbRL.

Q10: How does this policy error bound compare to the Pacchiano et al result?

A for Q10: Pacchiano et al. focus on the pure online setting, considering two cases: a known model and an unknown one to be learned online. Our result considers the offline case, where the model is estimated via an offline dataset, which requires a careful treatment balancing offline exploitation (for dynamics estimation) and online exploration (for preference learning).

Q11: What are the steps in the $x$ axis of Figure 3?

A for Q11: Figure 3 depicts the performance of the offline RL algorithm as a variable of the offline training procedure. The $x$-axis represents the training steps of the offline RL algorithm. We have made this clear in the revised version.

Thanks again for the valuable comments. We sincerely hope our additional explanations have cleared the concern. More comments on further improving the presentation are also very much welcomed.

[1] Faury, Louis, et al. "Improved optimistic algorithms for logistic bandits." International Conference on Machine Learning. PMLR, 2020.

[2] Filippi, Sarah, et al. "Parametric bandits: The generalized linear case." Advances in neural information processing systems 23 (2010).

[3] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. PMLR, 2018.

[4] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in neural information processing systems 30 (2017).

[5] Shin, Daniel, Anca Dragan, and Daniel S. Brown. "Benchmarks and Algorithms for Offline Preference-Based Reward Learning." Transactions on Machine Learning Research (2022).

[6] Chua, Kurtland, et al. "Deep reinforcement learning in a handful of trials using probabilistic dynamics models." Advances in neural information processing systems 31 (2018).

[7] Zhan, Wenhao, et al. "Provable Offline Reinforcement Learning with Human Feedback." arXiv preprint arXiv:2305.14816 (2023).

[8] Agarwal, Alekh, et al. "Reinforcement learning: Theory and algorithms." CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep 32 (2019).

Dear Reviewer,

Thanks for your valuable comments. We have provided additional experimental results and explanations to address your concerns, and we hope the following clarifications shed light on the points you raised.

W1: The motivation for the augmentation.

A for W1 The motivation to add weights is to learn a better credit assignment with trajectory-wise information. Since we are only allowed to query the preference between trajectories, it is possible that we can not learn proper temporal credit assignment with only fixed data.

For example, suppose we have two trajectories $\tau\_1$ and $\tau\_2$ that finish a task with two steps, knowing the preference between $\tau\_1$ and $\tau\_2$ give us no knowledge about how well they do at each step. Since we cannot generate new trajectories, a natural idea is to give weights to different parts of the trajectory and ask for the preference again.

Specifically, we can give high weights for step 1 and low weights for step 2 to distinguish which trajectory performs better at step 1. Therefore, augmenting trajectories can help for better temporal credit assignment and efficient preference learning. We have added experiments on Meta-world tasks to demonstrate further the advantage of using weighted trajectories for queries, as shown in Appendix G. We have made this point more clear in the revised version.

W2: Details of the chain MDP experiment.

A for W2: We thank the reviewer for pointing this out, and we gave a more detailed description of the chain MDP experiment in Appendix E in the revised version.

W3: Justification for the use of projected MLE and the definition of the projected MLE.

A for W3: By assumption, the parameter $\theta$ for the reward function is bounded, but an MLE estimator can generally be unbounded. Therefore, we project the MLE estimator to the space with bounded norms, ensuring the estimation is in the confidence set [1,2]. We have made this motivation more clear in the revised version.

W4: Typo errors.

A for W4: We thank the reviewer for pointing it out, and we have substantially revised the manuscript.

W5 & Q8: What the counterfactual trajectories are? What do counterfactual trajectories have to do with weighting?

A for W5 Q8: The counterfactual trajectories in Figure 2(c) refer to the two colored trajectories that are out of the dataset. We can not infer the preference between these two out-of-dataset trajectories if weighted queries are not allowed.

As explained in W1, by adding weights to trajectories, we can learn better credit assignments. We can use the former 2-stage task in W1 as an example. Without weighted queries, we can not infer the preference between the counterfactual trajectories $\tau'\_1$ and $\tau'\_2$, where $\tau'\_1$ acts as $\tau\_1$ at step 1 and acts $\tau\_2$ at step 2, and $\tau'\_2$ acts as $\tau\_2$ at step 1 and acts $\tau\_1$ at step 2. With weighted queries, we can identify the preference at each step. Therefore, weighted queries help reason about preferences between the counterfactual trajectories $\tau'\_1$ and $\tau'\_2$ without generating new trajectories.

Q1: What is $T$ in “querying with an offline dataset can be much more sample efficient when N >> T” at the end of Section 3.1?

A for Q1: Thanks for pointing it out. This is a typo, and it should be $K$, the number of comparisons. Intuitively, online preference-based RL requires online samples to learn both dynamics and the preference function. In contrast, in offline settings, the dynamics information is already contained in the dataset, and we only need to infer the preference function, which requires fewer samples than the online one.

Q2: What is ``True Reward'' in the table 1/2?

A for Q2: In Table 1/2, ``True Reward'' denotes the performance of offline RL algorithms under the original reward function of the dataset, which can be seen as the oracle performance for preference-based reward learning methods. We have added a detailed explanation in the revised version to enhance clarity.

Q3 & Q4: Can you explain why you augment the trajectories with a truncated Gaussian? Why a truncated Gaussian rather than some other random variable?

A for Q3 & Q4: The motivation for using weighted queries is explained in W1.

We clip the range of the weight to avoid large weights that can lead to high variance in the learning process, as commonly used in practice [3]. We conduct additional ablation studies to demonstrate the importance of weight clipping in the chain MDP experiment shown in Figure 10 (a) in Appendix G.

Using other random variables is also possible, but we found that Gaussian weights perform better in practice. Please refer to Appendix G's Figure 10 (b) for an ablation study.

Dear Reviewer,

Thanks for your positive and insightful comments. We provide clarification to your concerns below. We would appreciate it if you have any further feedback.

W1: More clear presentation.

A for W1: We thank the Reviewer for pointing this out. As suggested, we highlighted key takeaways in each section, added a summary of Pessimistic Value Iteration to the main paper, and provided more examples in the revised version.

W2: Whether the same gains observed in MetaWorld and Antmaze apply to other environments such as mujoco-gym and perhaps from human preferences too.

A for W2:

To investigate whether our method is generally applicable to other environments, we conduct experiments on three mujoco tasks, including hopper, walker2d, and halfcheetah. As shown in Table 1, our method outperforms the baseline method PT, especially on the hopper tasks.

Table 1. Additional experimental results on Mujoco tasks.

Q1: In definition 2, does the instantiation of linear MDP with the Bradley-Terry preference model depend on another $\phi$? Is the definition truly recursive?

A for Q1: A linear MDP with the Bradley-Terry preference model does not depend on another $\phi$. Definition 2 is only used to show that a linear MDP with the Bradley-Terry preference model naturally satisfies the definition of a generalized linear preference model. In linear MDP, the reward function is linear with respect to the feature map $\phi(s,a)$, i.e., $r(s,a)=\theta^T \phi(s,a)$, and the preference over trajectories $\tau\_1$ and $\tau\_2$ can be written as

$P(\tau\_1<\tau\_2) = \sigma ( \sum\_{s,a \in \tau\_1} \theta^T\phi(s,a) - \sum\_{s,a \in \tau\_2} \theta^T\phi(s,a))$ $= \sigma ( \theta^T(\sum\_{s,a \in \tau\_1} \phi(s,a) - \sum\_{s,a \in \tau\_2} \phi(s,a)))$

Therefore, by overloading the notation $\phi$ and let $\phi(\tau\_1,\tau\_2) = \sum\_{s,a \in \tau\_1} \phi(s,a) - \sum\_{s,a \in \tau\_2} \phi(s,a)$, a linear MDP with the Bradley-Terry preference model naturally satisfy the definition of generalized linear preference model.

Q2: What is $T$ on the theoretical guarantees section? and Does $K$ mean the number of queries?

A for Q2: Thanks for pointing this out. This is a typo, and it should be $K$ rather than $T$ in the theoretical guarantee section. Yes, $K$ means the number of queries. We have corrected this in the revised version.

Q3: Why do both baselines beat OPRIDE at $K>15$ for push-v2 in Figure 4? And is there a downside to OPRIDE as the number of queries goes up?

A for Q3: This is because push-v2 is a relatively simple task, and the performance saturates with a few queries. To verify this, We evaluate the performance of the algorithms with 30 queries on push-v2, and the result is shown in Figure 4. We can see that OPRIDE performs well compared with other methods when the number of queries goes up.

S: Nitpicks and suggestions.

A for S: We appreciate the valuable suggestions of the reviewer, and we have corrected them in the revised version.

Thanks again for the valuable comments. We sincerely hope our response has cleared your remaining concerns.

Dear Reviewer,

Thanks again for your valuable comments. We provide a response below for the points you raised.

W2 The improvement on some Mujoco tasks is relatively minor.

A for W2 This is because Mujoco locomotion tasks have a simple reward function such that a simple preference-based reward learning method can lead to a saturated performance. To verify this, we calculate the correlation coefficient between each dimension of the observation and the reward. As shown in Table 1, the observation can have a very high correlation (>0.9) with the reward in some dimension for every locomotion task, making the reward learning easy on these tasks. This phenomenon is also observed in concurrent works like [1]. Please refer to https://anonymous.4open.science/r/ICLR2024-7244-4A6F/README.md for a visualization of the correlation. Therefore, simple methods can also lead to reasonable performance on these tasks, and we believe that tasks like Meta-world, where learning the reward function is more challenging, are a better test bed for query efficiency.

Table 1. The correlation coefficient between each dimension of observation and the reward function.

Q1 Add the intermediate step to text

A for Q1 We thank the Reviewer for the suggestion, and we have added it to the manuscript.

We sincerely hope that our response can address your remaining concerns. More suggestions and discussions are welcomed.

[1] Anonymous. Flow to Better: Offline Preference-based Reinforcement Learning via Preferred Trajectory Generation. https://openreview.net/forum?id=EG68RSznLT