Preference Elicitation for Offline Reinforcement Learning

Dear Reviewer aVUF,

Thank you very much for taking the time to read our work and for your very positive feedback!

Dear Reviewer wreC,

Thank you very much for your detailed review and feedback. We address your comments and questions below.

Q1. Complex reward functions

Other than the relationship between the reward and preference function (Equation 1), no assumptions are made on the reward function in our theoretical analysis and method development. Our method is therefore agnostic to the form of the reward function, and we validate with our experiment on the HalfCheetah environment whose reward function is non-linear (Table 2, Figure 1).

As for sparsity (weakness #4), we simply propose a hypothesis that learning reward functions from preferences may be more challenging in general for this case, as all our baselines are inefficient in the Sepsis environment. Sim-OPRL remains much more efficient than other offline methods even in this case, which demonstrates its wider applicability.

Q2. Hyperparameter tuning.

Hyperparameters $\lambda_T , \lambda_R$ control the degree of pessimism in practice and could be considered equivalent to adjusting margin parameters $\beta_T , \beta_R$ in our conceptual algorithm proposed in Section 4. Since the exact values prescribed by our theoretical analysis cannot be estimated, the user must set these parameters themselves. Hyperparameter optimization in offline RL is a challenging problem in general, as the environment is not accessible to monitor policy performance, and as off-policy evaluation can be biased (Levine et al., 2020). As a result, we fixed $\lambda_T , \lambda_R$ for both our method and all baselines to avoid giving an unfair advantage to one or the other.

Our work focuses on sample efficiency, an important bottleneck in real-world applications where interaction with domain experts can be costly or time-consuming. Improving preference modeling and policy optimization falls outside the scope of our problem setting, presented in Section 3.

Q3. Additional baselines

We would like to clarify that our work is not concerned with preference modeling or policy optimization, but with efficient preference elicitation in the offline context. In Section 2 and Appendix B of our manuscript, we discuss related works contributing to the former research direction (see references below), but we note that their elicitation strategy is either based on a fixed preference dataset (equivalent to OPRL uniform sampling) or on uncertainty-sampling (equivalent to OPRL uncertainty sampling).

For instance, Brown et al., (2019, 2020); Park et al. (2022); Kim et al. (2023); Hejna and Sadigh, (2024) all assume access to a fixed preference dataset. PEBBLE (Lee et al., 2021) performs uncertainty sampling and also assumes access to the environment.

We benchmark our elicitation strategy with these two methods. To the best of our knowledge, however, there are no additional baselines for preference elicitation from offline data. Naturally, if you are aware of alternative offline elicitation methods, we would be happy to run these. We hope this answer also addresses your weakness #1.

Q4. Environment complexity and scalability

We are surprised by your comments and questions on our experimental setup, as we report empirical performance on a range of environments in Table 2 and Figure 1. For instance, the MuJoCo HalfCheetah and Minigrid environments are from the D4RL benchmark (Fu et al., 2020), which are identified as particularly challenging offline preference-based reinforcement learning tasks (Shin et al., 2022). As detailed in Appendix D, our environments consist of high-dimensional state spaces with continuous or discrete action spaces, follow complex transition dynamics, and have sparse rewards and termination conditions. This makes them representative of the challenge of learning a reward function and learning offline in a real-world, large-scale application. We hope this addresses your comments regarding weaknesses #2 and #3.

Dear Reviewer wreC,

Thank you again for your review. We were wondering if you had had the opportunity to read our rebuttal, as we look forward to discussing your follow-up thoughts before the end of the discussion period. We hope you will consider increasing your score if your concerns have been addressed.

Dear Reviewer BKXk,

Thank you very much for your positive feedback and for your review. We address your comments below.

Q1. Complexity of implementation.

While Sim-OPRL requires learning a transition model from the observational data for simulating rollouts, we note that this modeling step (and the ensemble needed for uncertainty estimation) is often also needed for model-based offline RL algorithms such as MOPO (Yu et al., 2020). To address your concern that our algorithm might not perform as well on complex environments, we find that Sim-OPRL also outperforms baselines on the complex D4RL datasets and in the Sepsis simulation (Table 2 and Figure 1).

We agree that modeling complexity increases as we go from uniform-sampling OPRL, to uncertainty-sampling OPRL (which now requires uncertainty estimation for the reward function), and finally to Sim-OPRL (which requires uncertainty estimation for both reward and transition functions). Considering our results in Table 2, we see that greater complexity leads to higher performance. For a method with “less computational overhead”, practitioners can therefore use one of our baselines, at the cost of performance.

Q2. Dependence on dataset optimality and feedback quality.

Dataset Optimality. We explore datasets with different levels of policy optimality, ranging from random policy (e.g., Halfcheetah-Random) to $\epsilon$-optimal (e.g., Sepsis). Further details on each dataset and environment are provided in Appendix D. We conclude that Sim-OPRL is more efficient than OPRL baselines in all cases.

Following Theorems 5.1 and 6.1, if the behavioral policy covers the optimal policy well, the concentrability terms will be smaller: fewer preferences are needed to achieve a target suboptimality. The reverse is true when the behavioral policy is far from optimal. We also discuss this in detail in our response to Reviewer tF7g. For a rigorous empirical analysis of performance as a function of dataset optimality, we propose an ablation in Figure 3b. We measure optimality through the density ratio coefficient which upper bounds $C_T$. Our empirical results are discussed in the paragraph starting line 512, and support the above theoretical analysis.

To complement this with a more complex environment, we also ran Sim-OPRL on Halfcheetah medium and medium-expert datasets. In the following table, we report the sample complexity achieved with these different datasets to reach a suboptimality gap of $\epsilon = 20$ over normalized returns. We reach the same conclusion as above: fewer preferences are needed as the dataset becomes more optimal. We will include this additional result in our revised manuscript.

Feedback Quality. Following prior empirical and theoretical work on preference elicitation for reinforcement learning (Chen et al., 2022, Zhan et al.,2023a and 2023b, Shin et al., 2022), our framework assumes the existence of a ground-truth reward function $R$, and that the feedback we receive follows the Bradley-Terry model determined by $R$. We formulate this problem in Section 3.

We therefore use each environment’s true reward function to generate preference feedback whenever our algorithms query it, again following prior work. Introducing noise in the feedback model violates this assumption, and therefore falls outside of the scope of our analysis.

Q3. Additional questions

Thank you for spotting these unclear elements and typos.

• The pessimistic transition and reward models are defined as follows: $\hat{R}\_{inf},\hat{T}\_{inf} = argmin_{\tilde{R} \in \mathcal{R}, \tilde{T} \in \mathcal{T}} \max_{\pi \in \Pi} V^{\pi}_{\tilde{T}, \tilde{R}}$. We will include this definition in our revised manuscript.
• The model refers to the transition model. We propose to rewrite the sentence for clarity: “While sampling from the offline buffer in OPRL is not sensitive to the quality of the transition model, good coverage of the optimal policy is needed from both transition and preference data to achieve low suboptimality.”
• Thank you very much for spotting this typo.

Thank you again for your positive feedback and interesting comments. We hope our response addresses your remaining concerns, and we would be very grateful if you would increase your score.

Dear Reviewer 8W5G,

Thank you very much for your positive feedback and for your review.

We are surprised by your comment on our experimental setup, as we report empirical performance on a range of environments in Table 2 and Figure 1. Among others, we explore environments from the D4RL benchmark (Fu et al., 2020) identified as particularly challenging offline preference-based reinforcement learning tasks (Shin et al., 2022), as well as a medical simulation designed to model the evolution of patients with sepsis (Oberst and Sontag, 2019). As detailed in Appendix D, these environments consist of high-dimensional state spaces with continuous or discrete action spaces, follow complex transition dynamics, and have sparse rewards and termination conditions. This makes them representative of the challenge of learning a reward function and learning offline in a real-world application.

Thank you again for your positive feedback. We look forward to hearing your thoughts in follow-up. We hope our response addresses your remaining concern and we would be very grateful if you would increase your score.

References

J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.

M. Oberst and D. Sontag. Counterfactual off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning, pages 4881–4890. PMLR, 2019.

D. Shin, A. Dragan, and D. S. Brown. Benchmarks and algorithms for offline preference-based reward learning. Transactions on Machine Learning Research, 2022.

Dear Reviewer tF7g,

Thank you very much for your positive feedback and for taking the time to review our paper.

We agree that the optimality of the behavior policy has an important effect on performance. We summarize our analysis and propose additional results in the following paragraphs.

Q1. Theoretical analysis

Your intuition about optimality affecting the quality of the transition model is right. For Sim-OPRL, as you noted, the transition model mostly needs to be accurate in the state-action space corresponding to the optimal policy, since that is where we generate rollouts and optimize the final policy.

Therefore, if the behavioral policy does not cover the optimal policy, the transition model will be less accurate in this region. Following our theoretical analysis, the concentrability term $C_T$ will be very large. This means the transition term in Theorems 6.1 will be large. More preferences $N_p$ will be needed to achieve the same suboptimality.

Conversely, if the behavioral policy covers the optimal policy well, the concentrability terms will be smaller. Fewer preferences would be needed to achieve a target suboptimality.

The same conclusions apply to SoTA offline preference elicitation methods (OPRL, Shin et al., 2022), as we find in Theorem 5.1 that their suboptimality depends on concentrability terms $C_T$ and $C_R$ for both transition and reward terms. Under a very suboptimal behavior policy, many preferences are needed to overcome the concentrability terms dominating the transition and reward terms respectively. As the behavior policy becomes closer to the optimal policy, we may recover good performance with fewer preferences.

Q2. Empirical analysis

Figure 3b in our manuscript proposes an ablation measuring sample efficiency as a function of the optimality of the behavior policy. We measure optimality through the density ratio coefficient which upper bounds $C_T$. Our empirical results are discussed in the paragraph starting line 512, and support the above theoretical analysis.

We also ran Sim-OPRL on Halfcheetah medium and medium-expert datasets. In the following table, we report the number of preferences needed to achieve a suboptimality gap of $\epsilon = 20$ over normalized returns. We reach the same conclusion on this environment: as the dataset becomes more optimal, fewer preferences are needed to achieve a given suboptimality.

We will include this additional result in our revised manuscript.

Thank you again for your positive feedback and insightful questions. We hope our rebuttal addresses your remaining concerns, and we would be very grateful if you would increase your score.

References

D. Shin, A. Dragan, and D. S. Brown. Benchmarks and algorithms for offline preference-based reward learning. Transactions on Machine Learning Research, 2022.