SEER: Towards Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation

We thank reviewer pCcb for the constructive comments. We value your insights and address each point you raised. Below are our point-wise responses:

Q1. The proposed algorithm seems to only modify the existing framework of maximum entropy RL a little bit and not very novel.

A1: Thanks for your question. The general framework of PbRL consists of the reward learning and policy learning. SEER employs SAC as its backbone algorithm in policy learning, similar to baselines like PEBBLE [1], SURF [2], and MRN [3]. Main Contirbutions lie in, SEER leverages the non-parametric model to sample informative trajectories and regularize neural Q function for feedback-efficient learning.

Q2. The paper only shows that empirical Q function is a lower bound of the ground-truth optimal Q function. I think the distribution-constrained loss is more reasonable if the authors can further show the empirical Q function is close to the ground-truth optimal Q function.

A2: Thank you for your question. In response to your concern about the empirical Q function's closeness to the ground-truth optimal Q function, we have conducted additional experiments. The results are as follows:

Table 1: MSE of $\hat{Q}$ and $Q_\theta$ across ten runs

This result represents the performance of the empirical Q and the neural Q across different tasks. As indicated by the result, the empirical Q closely approximates the ground-truth optimal Q value.

Q3. Why do the authors use the KL divergence between $\hat\pi(s)$ and $\pi_{soft(\theta)}(s)$ as the regularization term? A more intuitive choice would be the direct distance between $Q_{\theta}$ and $\hat{Q}$.

A3: Thanks for your thoughtful question. The decision to use KL divergence is rooted in our objective to align the learned policy as closely as possible with the optimal policy derived from the Q-function. This choice is inspired by the SAC, where the KL divergence is used to minimize the distance between the current policy and the exponentiated softmax of the Q-function, as detailed in [4] Equation 4.

Reference:
[1] PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. ICML, 2021.
[2] SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. ICLR, 2022.
[3] Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning. NeurIPS, 2022.
[4] Soft Actor-Critic Algorithms and Applications. ICML, 2018.

We thank reviewer pgTP for the constructive comments. We will give our point-wise responses below.

Q1. The main weakness with the paper is in the experimental evaluation. All of the other baselines test on the same broad suite of simulated robotics tasks. Since this paper follows from those papers and directly compares against them, those environments should definitely be included in the evaluation.

A1: Thank you for your valuable feedback. SEER primarily utilizes a discrete graph for conservative estimations, which inherently aligns more with discrete action domains and therefore is not very suitable for continuous action domains. However, we highlight that SEER demonstrates the benefit of regularization based on conservative estimation for feedback-efficient learning. This finding opens up a wide range of research possibilities for future studies. In terms of addressing continuous action scenarios, we are considering the adoption of alternative approaches, such as offline RL methods [1], to achieve conservative estimation without explicitly constructing a graph.

Q2. Another weakness is with the presentation. There are a lot of grammar mistakes throughout, for example the first sentence in abstract should be “One of the challenges… “ also in abstract “optimizing policies”, also citation (III and Sadigh) should be Hejna III and Sadigh. I think related works could be improved a lot. For example, the main baseline methods are not well-described in this section.

A2: Thank you for your valuable feedback regarding the presentation aspects of our paper. We will review the entire manuscript to correct these issues. Regarding the citations, we will ensure that all references, such as 'Hejna III and Sadigh', are correctly formatted and cited according to the required style guidelines.

Q4. How adaptable is SEER across different domains or problems? Can it be seamlessly integrated into other existing RL algorithms?

A4: Thanks for your question. SEER uses a discrete graph to obtain conservative estimations, which is not very suitable for continuous action domains. However, we have strategies to extend its application to continuous action scenarios. Firstly, one approach is to employ discretization techniques to transform continuous action spaces into discrete equivalents. This method is commonly used in contexts such as robotic arm manipulation, exemplified by RT-2 [2], and in complex games like DOTA [3] and StarCraft [4]. Secondly, SEER can be adapted using an action translator for generating continuous actions, such as the VMG model [5], involves using a graph-structured world model to convert actions via an action translator. The PbRL framework generally contains reward learning and policy learning, with the latter often directly employing standard RL algorithms. SEER is designed to be compatible with any value-based RL algorithm during the policy learning stage.

Q5. How does SEER handle potentially conflicting or inconsistent human preferences? Is there a mechanism to resolve such conflicts?

A5: Thanks for pointing our this. While directly resolving such conflicts was not the primary focus of this work, we recognize its importance in the robustness of SEER. To address this, we have conducted experiments to understand how SEER performs under conditions of noisy or conflicting data, which can be seen as a proxy for inconsistent human preferences. Specifically, we tested SEER in Push-5x5-1 and Strip-shaped with varying ratios of noisy data, including {0%, 10%, 15%, 20%, 25%, 30%}. This noisy data simulates scenarios where human preferences might be inconsistent or conflicting. The results, as shown in the following tables, indicate that SEER's performance degrades with increasing levels of noise. It fails with 30% noisy labels in Push-5x5-1 and with 25% noisy data in Strip-shaped.

Table 1: Performance of SEER with different noise levels

Table 2: Performance of PEBBLE with different noise levels

Reference
[1] Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv, 2020.
[2] Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv, 2023.
[3] Dota 2 with large scale deep reinforcement learning. arXiv, 2019.
[4] Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575.7782 (2019): 350-354.
[5] Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning. ICLR, 2023.

We are grateful to you for the feedback. We have endeavored to address the issues highlighted in your review. It would be helpful to know if the revisions we have made align with your expectations and satisfactorily address your concerns. If you find that they do, we would appreciate it if you could consider reflecting this in an updated evaluation of our work. Our aim is to ensure that the changes we have implemented effectively improve the paper's quality. We are fully open to further discussion and eagerly await any additional guidance or questions you may have.

Thank you, Reviewer AZLG, for your constructive comments. We appreciate the insights and suggestions provided. Below are our point-wise responses addressing each of your concerns:

Q1. I comprehend the author's assertion regarding the conservatism of the empirical Q-function. However, I am seeking clarification regarding the formal properties of the resulting policy, denoted as SAC $\pi_\phi$. Are we anticipating it to exhibit conservatism or optimism? Additionally, the author contends that it "aligns with human preference." Could this alignment be elucidated in a more rigorous manner?

A1: Thanks for your question. In our approach, the primary objective is to obtain an accurate Q function to enhance policy learning in PbRL. To this end, we employ an empirical Q-value, which embodies a conservative estimation approach, to constrain the neural Q function. This strategy is designed to accelerate and refine the policy learning process. Additionally, the term 'aligns with human preference' in this context implies that a more precise Q-value estimation correlates with greater alignment to human preferences within the PbRL framework.

Q2. The author state that "theoretically, we demonstrate that the empirical Q-function is a lower-bound of ...." in the Abstract. However, it is challenging to discern the precise details from Theorem 3.1. in a main text. Several elements remain undefined, such as the exact meanings of $Q_t$ and $\hat{Q}_t$ in the main text, as well as the underlying assumptions (e.g., do we need assumptions for rewards to say converges to?)

A2: Thank you for your valuable feedback. $Q_t$ represents the Q-values at time step t learned following the Bellman optimality equation, which is a standard approach in reinforcement learning for estimating the expected utility of actions in a given state. $\hat{Q}_t$ denotes the Q-values estimated at the same time step t but derived from our empirical approach as outlined in Eq. (3). The theorem and its proof are constructed under the general RL framework, without additional assumptions about their nature or convergence properties.

Q3. The proposed framework appears to be tailored for tabular settings, primarily due to its reliance on an empirical Q-function. How does the author intend to extend this approach to accommodate continuous settings, particularly in terms of both algorithmic and theoretical considerations?

A3: Thank you for your insightful question. To adapt our framework to continuous settings, there are several strategies. Firstly, it can consider the incorporation of discretization techniques to transform continuous action spaces into discrete equivalents. This method is common in similar contexts; for instance, RT-2 [1] successfully applies it for discretizing actions in robotic arm applications. Secondly, it can also introduce an action translator to generate continuous actions, akin to the VMG model [2], which uses a graph-structured world model and convert generated actions via an action translator.

Q4. In a related context, update (3) seems somewhat naive in addressing the data coverage concern. I agree it might be beneficial to differentiate between actions that have not been visited, actions that have been visited. But how can we distinguish actions that fall in between – perhaps those visited frequently versus those visited infrequently?

A4: Thank you for your valuable feedback regarding update rule (3) in our paper. In our method, we use $N(s,a,s')$ to represent the number of visits to different actions. The empirical dynamics $\widehat{p}(s'|s,a) = N(s, a, s')/\sum_{s'} N(s, a, s')$ already encapsulate a distinction in visit frequencies. For instance, actions that have been visited frequently will have a higher value of $N(s, a, s')$, reflecting in a higher empirical probability, whereas infrequently visited actions will have lower values. This differentiation is crucial for ensuring that our update mechanism does not overlook less explored actions.

Q5. Would you explain Line 8 in Algorithm 1 more? How did you sample a pair of trajectories? Does this correspond to the "Sampling informative trajectories part"?

A5: Thanks for your question. Yes, Line 8 indeed corresponds to the 'Sampling informative trajectories' part of our algorithm.

Reference
[1] Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv, 2023.
[2] Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning. ICLR, 2023.

We sincerely thank you for your constructive comments. And we would like to know whether we have addressed your concerns. If so, might you be able to update your rating to reveal this? Our goal is to ensure that our responses and modifications meet your expectations and enhance the quality of our work. We are eager to continue this dialogue and are available for any further discussion that may be helpful.