Fewer Questions, Better Answers: Efficient Offline Preference-based Reinforcement Learning via In-Dataset Exploration

Dear Reviewer:

Thanks for finding our method simple yet effective. The main issues regarding experiments and theory are addressed as follows.

W1: Could you demonstrate how to infer better query efficiency of OPRIDE than random query from Theorem 4?

A for W1: Using a proper query selection criteria is crucial in reducing the preference error in Theorem 4. Similar to exploration in bandit, random exploration does not lead to efficient inference of the optimal action and incurs large regret. Using random queries lead to a much larger reward function estimation error (second term in Equation (15)) and therefore less efficienct than OPRIDE.

W2: The motivation of the discount scheduling.

A for W2: Intuitively, a learned reward function is prone to overoptimization [1, 2], leading to overestimation of a policies' return and, therefore, a pessimistic estimation is required to avoid such exploitation of the imperfect reward function. For this reason, we propose to reduce the discount factor based on the variance of the value function estimates as a pessimistic mechanism, which is more effective than previous ensemble-based methods like REDQ. Without any pessimistic estimation, we will need much more queries so that the estimation reward function becomes good enough to resist such overoptimization.

We conduct additional ablation studies for the discount scheduling (IDE). The experimental results in Table 1 show that the discount scheduling is necessary for OPRIDE.

Table 1. Ablation study for OPRIDE.

W3: Hyper-parameter analysis on $m$.

A for W3: As suggested, we conduct additional ablation studies on the ensemble number $m$. The experimental results on Table 2 show that the performance of OPRIDE increases with the ensemble number $m$ since larger ensemble provides more diverse information. In addition, we find that OPRIDE performs sufficiently well with around $m=5$.

Table 2. Hyper-parameter analysis on $m$.

W4: Why this paper limits the query budget to 10 and Could you please provide some empirical results of more query budget like 200?

A for W4: This is because in our experiments, we find that 10 queries are sufficient to achieve good performance in the evaluated tasks. Further, we conduct experiments with more query budgets. The experiment results on Table 3 show that the performance of OPRIDE improves with the query budget increases, and the performance is sufficiently good with the 10 budget.

Table 3. Additional experiments with more query budgets.

W5: Does PT refers to random query and Could you please provide results of OPRIDE w/ random query?

A for W5: Yes, PT refers to random query. As suggested, we conduct experiments by replacing query selection module in OPRIDE with random query (named OPRIDE (Random)). The experimental results in Table 4 show that the in-dataset exploration query mechanism is necessary for OPRIDE.

Table 4. Ablation study for OPRIDE.

Thanks again for the valuable comments. We hope our response has cleared your concern. We are looking forward to more discussions.

[1] Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835–10866. PMLR, 2023

[2] Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024

Dear Reviewer,

Thanks for the quick reply. We will address your follow-up questions below.

Q1: Theoretical comparison between random query and the proposed query strategy.

A for Q1: Following the analysis of fitted Q-iteration [1], when using random queries drawn from a distribution $\mu$, and keeping another part of our algorithm the same, the suboptimality bound will have the following form:

$$\text{SubOpt}(\hat{\pi}) \leq \text{Offline Error} + \sqrt{\frac{H^2\kappa C(\mu) \log(K \Delta R )}{K}},$$

where $C(\mu) = \sup_{s} \sup_{\nu} \frac{\nu(s)}{\mu(s)}$. It is easy to show that $C(\mu)\geq |S| \geq d_{Elu}(\Delta R, \epsilon)$. So, using random queries leads to a worse bound by a factor of $\frac{C(\mu)}{d_{Elu}(\Delta R, 1/K)}$ on the reward error. Specifically, $C(\mu)$ becomes unbounded with an infinite state space, while $d_{Elu}(\Delta R, \epsilon)$ can remain bounded beyond linear function approximations. Intuitively, when we have a large hypothesis space for the value function, using random queries can be hopeless, while there may still be structures that allow strategic queries to be efficient.

Q2: The performance difference w.r.t random query strategy.

A for Q2: As suggested, we conduct additional Random query strategy experiments with various query budgets. The experimental results in Table 1 show that as the query budget increases, both Random and OPRIDE performance will gradually improve, and the gap between them will decrease progressively. This is because when the query budget is sufficiently large, OPRIDE's performance approaches the true reward. On the other hand, we find that the performance of OPRIDE with a small number of queries requires more queries from the Random strategy, which fully demonstrates the query efficiency of our method.

Table 1. Comparison with Random query strategy with various query budgets.

Best,

The Authors

References

[1] Munos, Rémi, and Csaba Szepesvári. "Finite-Time Bounds for Fitted Value Iteration." Journal of Machine Learning Research 9.5 (2008).

Dear Reviewer,

Thanks for your valuable comments. We hope the following statement can address your concern.

W1: Discussion of RLHF in the related work section.

A for W1: Thanks for your suggestion, we have corrected it in the related work section in the updated version.

W2, Q1 and Q2: Clarification of formalization.

• Why is it important to use the non-stationary definition?

We are not sure what exactly the reviewer means by "non-stationary" and assume that "non-stationary" means "heterogenous", that is, the transition function and the reward function can be different at each timestep. This is not crucial for the theoretical analysis but it makes the analysis more general. That is, homogenous MDPs are a special case of our analysis.

• Why do we need both Bellman operators? Bellman evaluation operator can be used to simplify our theoretical analysis, please refer to the proof of Lemma 5 for more details.

• Why are the returns in (6) not discounted? The return is not discounted because we consider the finite horizon case in the theoretical analysis.

• What's the impact of learning a return model vs a learning a reward model? A return model, or more accurately a value model, is crucial in selecting informative queries. Most prior works select informative queries by maximizing disagreement on the reward function, while we select queries by maximizing disagreement on the value function. Intuitively, disagreement over the reward function on the low-return regions does not affect the judgement of optimal policies, therefore it can be less efficient comparing to using disagreement over value functions.

• Do you learn a return function instead of a reward function in the experiments? We learn a value function over the reward function in the experiments and use the value function as query selection criteria.

• Variance-based Query selection is also scale-sensitive . We agree that variance-based query selection is also scale-sensitive. In the context, we are comparing to the disagreement-based method [1] which only consider the numbers of preference each candidate get, which is scale-insensitive.

W3 and Q3: Does the theoretical analysis really justifies their proposed criterion in Algorithm 1?

A for W3 and Q3: When the size of the dataset approaches to infinity, any online trajectory can be found in the offline dataset so that select queries from the dataset is equivalent to rollout online queries. Therefore, when $N>>K$, where $N$ is the size of the dataset and $K$ is the number of queries, our theoretical analysis is a proper approximation for the pure offline setting, and query selection principles still applies.

W4: Minor issues.

A for W4: Thanks for your detailed suggestion, we have corrected it in the updated version.

Thanks again for the valuable comments. We hope our response has cleared your concern. We are looking forward to more discussions.

[1] Shin, Daniel, Anca Dragan, and Daniel S. Brown. "Benchmarks and Algorithms for Offline Preference-Based Reward Learning." Transactions on Machine Learning Research.

Dear Reviewer,

We have added additional explanations for our methods. We are wondering if our response and revision have cleared your concerns. We would appreciate it if you could kindly let us know whether you have any other questions. We are looking forward to comments that can further improve our current manuscript. Thanks!

Best regards,

The Authors

Dear Reviewer,

Thanks for finding our paper novel, well-motivated, nice simplification, and thorough experimental results. We hope the following statement clear your concern.

W1: Key contributions are not clear enough.

A for W1: Thanks for your suggestion, we have corrected it in the introduction section in the updated version.

W2: Differences to prior work in the related work section.

A for W2: Thanks for your suggestion, we have explained more differences to prior work in the related work section in the updated version.

W3: Comparison to prior methods.

A for W3: While our method and the prior works (e.g., IDRL [1]) have similar connections, our approach introduces several novel contributions:

• Unlike IDRL, which relies on the Laplacian approximation and the Hessian matrix for posterior computation, our method leverages critic values for query selection, ensuring easier implementation and superior empirical performance, as demonstrated in our comparative experiments in the paper.

• Additionally, unlike IDRL, our method provides a concrete suboptimality bound and regret analysis, enhancing its theoretical robustness.

Thanks for your suggestion, we have added more discussion of the differences in the updated version.

W4 and Q2: Discussion of the theoretical results.

• Intuition for the proof technique or a proof sketch: We thank the reviewer for the suggestion. The proof consists of two main steps. The first step decomposes the suboptimality into two parts: the dynamical error and the reward function error. This is only possible when we use a pessimistic reward function (i.e., Equation (9) and the confidence set selection in Equation (12) ). The dynamic error part can be well-bounded when using a proper offline algorithm. We are interested in the reward function error due to limited preference data. The second step aims to bound such error with the explorative query selection criteria (i.e., Equation (8) and (14)). Without such a criteria, the reward function error part can be unbounded.

• Comparison of the theoretical results: Our result gives the first finite sample complexity bound for PbRL with a offline dataset and online queries with general function approximation. Compared to pure online setting [2,3], our result applies to offline dataset with online query setting; compared to linear setting [2], our result extend to general function approximation setting with finite Eluder dimensions. Compared to [1], our analysis give a rigorous complexity bound.

• Theoretical roots for variance scheduling: We thank the reviewer for raising the point. While the variance scheduling is an empirical approximation for the pessimism principle in Equation (12), there are theoretical evidence that they are equivalent in terms of regularization effect [4] and model-based pessimism principles [5].

W5: Evidence on the LLM settings:

A for W5: As suggested, we have conducted experiments on the LLM setting using $`opt-1.3b`$ and the $`rm-static`$ dataset. We use a total query budget of 1000 samples from the training dataset, which comprises 30000 samples in total. The following table shows the score for different methods from the reward model (trained with the full training dataset) on the evaluation dataset.

Table 1. Experimental results on the LLM setting.

Additionally, we assessed the performance of our method using $`gpt-4o`$ as the judge. The results clearly show that our method outperforms the baseline by a large margin.

Table 2. Experimental results on the LLM setting.

W6, Q5 and Q6: Choice of baselines and ablations:

• Results of IDRL on Antmaze, Figure 2 and Table 6 in Appendix: As suggested, we have added the experimental results of IDRL on Antmaze, Figure 2 and Table 6 in Appendix in the revised version. The experimental results show that our method achieves better performance than baselines.

• What about no variance scheduling? As suggested, we conduct ablation studies for OPRIDE. Specifically, we remove the variance scheduling (named OPRIDE w/o VDS) and the in-dataset exploration module (named OPRIDE w/o IDE) respectively. The experimental results in Table 3 show that the effectiveness of each part of our algorithm.

Table 3. Ablation study for OPRIDE.

W7: Discussion of practical downsides:

• Computational cost of training ensembles of reward models and policies: Thanks for your suggestion. We present the computational cost on the GeForce RTX 3090 GPU device, as shown in Table 4. The experimental results indicate that as the number of Ensembles $M$ increases, the computational cost does not increase significantly. This is because we have adopted a multi-head mechanism instead of separate training, thereby saving training time.

Table 4. Computational cost for OPRIDE with ensemble $M$, where h is the hour.

• Deal with less ideal data: In our experiments, we adopt the non-ideal data rather than ideal setting. Specifically, each dataset consists of 1000 trajectories. 50 trajectories of which are collected by the corresponding scripted policy added with a Gaussian noise $\mathcal{N}(0, 0.8)$ to increase diversity, and the rest 950 trajectories are collected with a policy that is a $\epsilon$-greedy variant to the former noisy policy and select random actions with probability $\epsilon=0.8$.

Q1 and Q7: How does the proposed method compare to typical approaches there and could dueling bandit algorithms be adapted to serve as baselines to compare your approach to?

A for Q1 and Q7: We use Q-values rather than returns in the selection phase. This is in stark contrast with bandit algorithms which does not consider credit assignments between different timesteps. Per the reviewer's suggestion, we added additional baselines that use dueling bandit algorithms as query selection criteria. As shown in Table 5, our method significantly outperforms the dueling bandit algorithm.

Table 5. Comparison with Dueling Bandit.

Q3: Will code for the experiments be released to enable others to reproduce the work?

A for Q3: Yes, the code is available in the supplementary material and our code will be released.

Q4: How many datapoints are the boxplots computed over in Figure 2?

A for Q4: We use five seeds for each datapoint in boxplots in Figure 2.

We sincerely thank the reviewer again for the timely and valuable comments. We hope that our response and additional experimental results have cleared most of your concerns.

[1] Lindner, David, et al. "Information directed reward learning for reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 3850-3862.

[2] Pacchiano, Aldo, Aadirupa Saha, and Jonathan Lee. "Dueling rl: reinforcement learning with trajectory preferences." arXiv preprint arXiv:2111.04850 (2021).

[3] Chen, Xiaoyu, et al. "Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation." International Conference on Machine Learning. PMLR, 2022.

[4] Jiang, Nan, et al. The dependence of effective planning horizon on model accuracy.

[5] Hu, Hao, et al. On the role of discount factor in offline reinforcement learning.

Dear Reviewer,

We have conducted additional experiments on LLM, computational costs and ablations. We are wondering if our response and revision have cleared your concerns. We would appreciate it if you could kindly let us know whether you have any other questions. We are looking forward to comments that can further improve our current manuscript. Thanks!

Best regards,

The Authors

Dear Reviewer,

Thank you for your thoughtful feedback and for noting that our previous response addressed most of your concerns. We deeply appreciate your engagement with our work.

Regarding the online query assumption in the theoretical analysis, we would like to clarify that it is only used to simplify the analysis and can indeed be relaxed. When using purely offline queries, we need to ensure that the queries are well-represented in the dataset. This introduces an additional error term on the order of $(\sqrt{1/N})$. This error is negligible when $N \to \infty$ (as it becomes zero, as in the online case) or when $N \gg K$ (as the error becomes much smaller than the reward error term). For a more rigorous description, please refer to This Link.

We sincerely hope this addresses your remaining concerns. We look forward to further discussion!

Best,

The Authors

Dear Reviewer,

Thanks for your valuable comments. We hope the following statement can address your concern.

W1 and W3: Proposed methods are loosely related and the theoretical results are poorly related with the practical performance.

A for W1: We respectfully disagree with this assessment. Our theoretical analysis suggests that there two critical requirements that leads to an efficient offline PbRL algorithm - a pessimistic reward function and an efficient explorative query selection criteria. We implement the former with the variance-based discount scheduling mechanism and the latter with the disagreement based criteria. These two modules are closely related to our theoretical analysis and both are shown crucial for efficient PbRL algorithms.

W2 and Q2: The novelty of this paper and comparison with previous works.

A for W2 and Q2: We respectfully disagree with this assessment.

• Our query selection is significantly different from PEBBLE, which focus on maximizing information with respect to the reward function, while we focus on the information with respect to the value function. Intuitively, disagreement over the reward function on the low-return regions does not affect the judgement of optimal policies, therefore it can be less efficient comparing to using disagreement over value functions. As shown in Table 1 and 2, our method significantly outperform reward-based query selection (OPRL).
• While there are many uncertainty-based method in previous literature like REDQ and MOPO, we are the first to apply it to the discount factor rather than the reward or value function. This is novel and comes with theoretical guarantees. As shown in our ablation experiment in Table 3, while being simple, such a change leads to significant performance improvement.

Table 1. Experiments of OPRIDE with uniformly selecting queries.

Q1: Why overoptimization of learned reward functions contributes to the low query efficiency of pbrl? How does the discount scheduling mechanism alleviate this?

A for Q1: Intuitively, a learned reward function is prone to overoptimization [1, 2], leading to overestimation of a policies' return and, therefore, a pessimistic estimation is required to avoid such exploitation of the imperfect reward function. For this reason, we propose to reduce the discount factor based on the variance of the value function estimates as a pessimistic mechanism, which is more effective than previous ensemble-based methods like REDQ. Without any pessimistic estimation, we will need much more queries so that the estimation reward function becomes good enough to resist such overoptimization.

Q3: More empirical elaboration on the assumption of finite Eluder dimension in the theory part and Why do we need this?

A for Q3: Intuitively, Eluder dimension is the "RL version" of VC dimension. Without a finite Eluder dimension, the reward function can be too large for us draw any meaningful conclusion by observing a small amount of samples. We would also to say that the space of finite Eluder dimension is in fact a general assumption that covers linear settings, etc and appears in many RL theory

Q4: It is found that querying with an offline dataset can be much more sample-efficient than pure online queries. How do such findings related to the algorithm pipeline, including in-dataset exploration and the discount scheduling?

A for Q4: This is a comment regarding the offline setting and the online setting rather than our specific algorithm. Our algorithm pipeline belongs to the former, which select queries with an offline dataset and therefore requires much less queries than pure online methods like [3].

Thanks again for the valuable comments. We hope our additional experimental results and explanation have cleared the concern. We sincerely hope that the reviewer can re-evaluate our paper after seeing the our response. More comments on further improving the presentation are also very much welcomed.

[1] Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835–10866. PMLR, 2023

[2] Zhu, B., Jordan, M. I., and Jiao, J. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. arXiv preprint arXiv:2401.16335, 2024

[3] Christiano, Paul F., et al. "Deep reinforcement learning from human preferences." Advances in neural information processing systems 30 (2017).

Dear Reviewer,

We have conducted additional experiments and explanations for our methods. We are wondering if our response and revision have cleared your concerns. We would appreciate it if you could kindly let us know whether you have any other questions. We are looking forward to comments that can further improve our current manuscript. Thanks!

Best regards,

The Authors