Meta-Reinforcement Learning with Adaptation from Human Feedback via Preference-Order-Preserving Task Embedding

We are grateful and indebted for the time and effort invested in evaluating our manuscript and for all the suggestions to make our manuscript better.

Weakness 1 and Q1

Answer: Thanks for pointing out the important observation. In both the existing context-based meta-RL methods, such as PEARL (Rakelly et al., 2019), and this paper, the ideal case of the meta-training can be using the optimal policy to reconstruct the decoder policy, and using the optimal policy to train the encoder. As you kindly mentioned, this can help avoid the error due to the current policy being not optimal. However, the optimal policies are not accessible, we have to replace the optimal policy reconstruction loss by the policy optimization loss.

In PEARL, the authors also use the policy optimization loss to train the output conditional policy (the decoder policy) and train the encoder simultaneously at the beginning of training. Although the policies during the training for the encoder are not optimal, the method is shown to be effective and efficient. One reason could be that although the policies are not optimal, they could provide an effective approximation to guild the optimization during the training to the correct direction. Therefore, we follow the same encoder-decoder training pattern in this paper, which keeps the algorithm statement concise.

Weakness 2 and Q2

Answer: Explanation of Equation (9), i.e., $(\hat{z}^{\prime}, \hat{z}^{\prime\prime})={\arg\min}_{z^{\prime},z^{\prime\prime} \in \mathcal{Z}_k}( \max \lbrace|\mathcal{Z}^{(1)}|,|\mathcal{Z}^{(2)}| \rbrace )$:

In Equation (9), $\mathcal{Z}^{(1)}=\lbrace z\in \mathcal{Z}\_k: S(z,z^{\prime} ) >\_\epsilon S(z,z^{\prime\prime} )\rbrace$ and $\mathcal{Z}^{(2)}=\lbrace z \in \mathcal{Z}\_k: S(z,z^{\prime\prime} ) >\_\epsilon S(z,z^{\prime})\rbrace$. When $z^{\prime}$ is preferred over $z^{\prime\prime}$, i.e., $S(z,z^{\prime} ) >\_\epsilon S(z,z^{\prime\prime} )$, any embedding in $\mathcal{Z}^{(1)}$ will satisfy the preference condition and remain to the valid embedding candidate. Similarly, when $z^{\prime\prime}$ is preferred over $z^{\prime}$, i.e., $S(z,z^{\prime\prime} ) >\_\epsilon S(z,z^{\prime} )$, any embedding in $\mathcal{Z}^{(2)}$ will remain as the valid embedding candidate. If the number of the remaining valid embedding candidates is smaller, the range of the valid embeddings will be quickly narrowed, and then the query efficiency will be higher. Therefore, the goal of Equation (9) is to make the number of the remaining valid candidates as small as possible. As we do not know what the queried preference is and which set of $\mathcal{Z}^{(1)}$ and $\mathcal{Z}^{(2)}$ will be remaining, we have to consider the worst case, i.e., the one between $\mathcal{Z}^{(1)}$ and $\mathcal{Z}^{(2)}$ with the larger size will be remaining. Therefore, Equation (9) first takes the set $\max \lbrace |\mathcal{Z}^{(1)}|,|\mathcal{Z}^{(2)}|\rbrace$ to pick the larger set in $\mathcal{Z}^{(1)}$ and $\mathcal{Z}^{(2)}$, and then minimizes its size.

Explanation of line 17 in Algorithm 2, i.e., $z^k= \mathop{\arg\max}\_{z \in \mathcal{Z}_k} {\sum}\_{(z^{\prime},z^{\prime\prime}) \in \mathcal{Q}\_k} \log \mathrm{Pr} [S(z,z^{\prime} )>S(z,z^{\prime\prime})]$:

In the $k$-th iteration of the human-in-the-loop adaptation, the candidate embedding set $\mathcal{Z}_k$ includes multiple valid embedding candidates. However, we need to pick one as the output for the $k$-th iteration. Therefore, line 17 is to use the maximum likelihood over all the valid embedding candidates in $\mathcal{Z}_k$ to determine the output task embedding $z^k$.

Q3

Answer: After query the preferences of $(z_1,z_2)$ and $(z_3,z_4)$, it will be beneficial from querying $(z_1,z_3)$. However, $(z_1,z_3)$ is not the most query-effective pair to be queried. As mentioned in the Answer to Q2, we use Equation (9) to determine which pair is the most query-effective.

Note that, in Algorithm 2, sampling two new embeddings other than $z_1, z_2, z_3, z_4$ in $Q$, incurs almost no cost, as it merely involves sampling from a normal distribution. However, the human preference is the most expensive step in the human-in-the-loop adaptation. Therefore, $(z_1,z_3)$ in $Q$ will not be used to query the human preference if it is not the most query-effective pair computed by Equation (9).

Weakness 3 and Q4

Answer: Our method cannot handle tasks in which the dynamics change over time within a single task. If the dynamics is changing over time within any single task. When the task is revealed to the agent, the agent needs to query a human for preference comparison gradually and optimize the policy based on all the historical preference queries. However, when the dynamics change over time, the historical preference queries from humans may become invalid, which leads to a wrong direction for the policy optimization. Solving tasks with dynamics changing over time is an interesting future work.

We are grateful and indebted for the time and effort invested in evaluating our manuscript. Thanks for the typo reminders and the suggestions to make our manuscript a better and stronger contribution.

Methods And Evaluation Criteria: One minor weakness in the evaluation is the experiments are limited to continuous control robotics tasks, and the task spaces are mostly target locations. In these setups the level of difficulty in generalizing to different goals may not be very high. How would POEM perform in more complex task spaces?

Answer: In the conducted experiment, MetaWorld-ML10, the task is defined by the type of manipulation rather than the target location. The task family includes multiple types of manipulation tasks. As shown in Figure 8 (page 17), the interactions between the robot and the environment vary across tasks, encompassing different types of manipulation such as ‘assembly,’ ‘basketball,’ and ‘door opening.’ The training task set in this benchmark is highly heterogeneous, making the task space complex. As shown in Figure 5, the proposed method achieves a 50% improvement over the baselines on MetaWorld-ML10.

Claims and Evidence: The approach is quite similar to the work proposed by ANOLE.

Answer: The partitioning of the embedding into reward embedding and policy embedding spaces is only an initial and minor design of the paper. The main contribution of the proposed method is that we train a preference-order-preserving task encoder, which establishes a connection between task embeddings and human preferences. This connection facilitates the efficient inference of task embeddings for new tasks during human-in-the-loop adaptation.

In terms of the algorithm design, as you pointed out, this paper first designs an encoder that partitions the embedding space into reward embedding space and policy embedding space. Second, based on the embedding space, we prove that an encoder exists under the embedding space partition, which holds the preference-order-preserving property (Property 1 in Section 4). Third, in Section 5, we train an encoder that enforces that Property 1 holds by using the preference loss term (equation (6) in line 241), which penalizes violations of Property 1. Fourth, in Section 6, the preference-order-preserving property of the encoder enables the task embedding inference from human preferences for the human-in-the-loop adaptation.

Note that all the above four steps of the algorithm design are different from ANOLE, which addresses the issue of ANOLE that the task encoder cannot capture preference-related features across tasks.

Theoretical Claims: The definition of "well-trained" is unclear. The policy reconstruction loss is not used. How to ensure the encoder-decoder is properly trained?

Answer: The definition of "well-trained" is given by the three assumptions of Theorem 1. Specifically, the assumption (i) (lines 375-377, Theorem 1) states that the posterior distribution is the normal distribution; the assumption (ii) (lines 377-379) requires that Property 1 holds, i.e., the preference-order-preserving property of the encoder holds; and the assumption (iii) (lines 379-381) requires that the optimal policy is accurately reconstructed.

To achieve the above three requirements, i.e., the encoder-decoder network is well-trained, we use the KL divergence loss term in line 273 to achieve assumption (i), use the preference loss in Equation (6) (line 240) to achieve assumption (ii), use the optimal reconstruction loss in Equation (5) (line 235) to achieve assumption (iii). Therefore, including the policy reconstruction loss, all the losses included in Section 4 are used to support Theorem 2.

Experimental Designs or Analyses a)

Answer: In the conducted experiment, MetaWorld-ML10, the tasks family has multiple types of manipulation tasks. As shown in Figure 8 (page 17), the interactions between the robot and the environment are different for different types of manipulation tasks, and therefore their state transition functions are different and change. For example, it is easy to see that the state transition in the "assembly" task and that in the "basketball" task are different. As shown in Figure 5, the proposed method achieves 50% improvement over baselines on MetaWorld-ML10.

Experimental Designs or Analyses b)

Answer: There are several works, such as [1,2], addressing the issue of imbalance in task distribution in meta-learning/meta-RL. Their approaches are agnostic to the meta-RL methods and can also be used to address the issue in this paper. However, this paper primarily focuses on a new problem of meat-RL with human-in-the-loop adaptation. Addressing the issue of imbalance in task distribution is an interesting future work.

[1] "Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks", 2020.

[2] "Improving Generalization of Meta Reinforcement Learning via Explanation", 2024.

Weakness 1

Answer: The numbers of steps between this paper and ANOLE are exactly the same for the ant task and the cheetah task. In both this paper and ANOLE (Ren et al., 2022), cheetah-vel uses 1M steps (check the first figure in Figure 5 in this paper and the second figure of Figure 1 in ANOLE paper), cheetah-fwd-back uses 2M steps (check the first figure in Figure 9 in this paper and the first figure of Figure 1 in ANOLE paper). In this paper, we also use 4M steps on the ant task (check the second and third figures in Figure 9 in this paper).

Weakness 3

Answer: Meta-RL with human-in-the-loop adaptation is to train a meta-model from the training tasks (T1, T2, T3), such that it can be adapted to new tasks with limited human preference queries. The learned meta-model is fixed during the meta-test phase. When a new task T4 arises, the meta-model learned from (T1, T2, T3) will be adapted to solve T4. Similarly, when a new task T5 arises, the meta-model will be adapted to solve T5. If any of the training tasks T1, T2, T3 arises in the meta-test, the meta-model is adapted to solve the task, which is easier than solving a new task T4.

We are grateful and indebted for the time and effort invested in evaluating our manuscript and for all the suggestions to make our manuscript a better and stronger contribution.

Methods And Evaluation Criteria 1: I could see issues with Property 1 being called into question in scenarios where similarity between task embeddings may not be sufficient to ensure policy transferability, such as long-horizon dependencies, however I believe the relaxation in equation (8), while explained to be used to account for noise in human preference selection, should also clarify this.

Answer: In this manuscript, we do not claim that, Property 1 always holds, and the similarity between task embeddings is sufficient to ensure the policy preference, for any encoder.

Instead, we claim and prove that in Section 3, under the MDP (including the case of long-horizon dependencies), there exists a task encoder such that Property 1 holds, i.e., there exists an encoder such that the similarity ordering of task embedding pairs is expected to align with human preference order. Next, in Section 4, we train an encoder that enforces that Property 1 holds under the encoder. To achieve this, we design a loss term for the encoder network, the preference loss term (equation (6) in line 241), which penalizes violations of Property 1. Therefore, although Property 1 may not be naturally satisfied without the training, once the encoder is trained under the supervision of the preference loss term, Property 1 holds approximately.

Furthermore, as you kindly pointed out, we also incorporate the relaxation in Equation (8) to account for noise in Property 1, enhancing the algorithm’s robustness.

Methods And Evaluation Criteria 2: I would appreciate more detail about how the human-in-the-loop was given the trajectories

Answer: During the meta-test (human-in-the-loop adaptation), a new task ${\mathcal{T}}_{new}$ is given. The agent explores the environment and provides two trajectories (the rewards along the trajectories are unknown) to the human. Then, the human will tell the agent which one is better, and the agent will adapt the policy according to this human feedback. One time of the human preference query is denoted as one iteration of the human-in-the-loop adaptation. The detail of the human feedback is introduced in Section 2, lines 95-109.

Experimental Designs or Analyses: The primary concern is not understanding how these environments were modified.

Question 2: Could the authors please explain the difference between the base Mujoco and their modified version?

Answer: Thanks for pointing out the confusion. In this paper, we do not modify the environment of Mujoco. Instead, we directly use the environment in Mujoco and design the reward functions for multiple tasks for the meta-RL setting. The details of the reward design are shown in Appendix C.2. To avoid the confusion, we will modify the "Modified Mujoco" to "Mujoco".

Suggestions: Despite the technical strength of this paper, I take a very strong issue with the related work being relegated to the appendix. Even if the authors only use half of a column to address it in the main paper, I believe it is critically important for previous and related work to be given its proper acknowledgment by the scientific community. This is the primary motivating factor behind my score.

Answer: Thanks for pointing it out. In the modified version, we will move the related work section in Appendix A (lines 608 to 662) to the main body of the paper.

In the related work section, we comprehensively review the works related to (i) meta-RL methods, (ii) RL from human feedback (RLHF), and (iii) methods for meta-RL with human-in-the-loop adaptation, and provide details of the comparisons between the problem settings, the problem formulations, and the algorithm designs in this paper and those in the existing works. Specifically, in (i), we discuss three categories of meta-RL methods and whether they can be applied to the problem of meta-RL with human-in-the-loop adaptation. In (ii), we discuss the existing methods for RLHF and the motivation of studying meta-RL with human-in-the-loop adaptation based on RLHF. In (iii), we discuss the existing methods for meta-RL with human-in-the-loop adaptation, including ANONLE and meta-reward, and discuss the differences between their algorithm designs and this paper.

Question 1: Could the authors explain how the policies were presented to the human-in-the-loop during evaluation? How many iterations of this were required per task?

Answer: Please refer to the answer to Methods And Evaluation Criteria 2 for the explanation of the human-in-the-loop adaptation.

As shown in Figures 5 and 10, we conduct at most 10 iterations for human-in-the-loop adaptation for each task, and it usually requires about 5 iterations to achieve the near-optimal performance per task.