VLP: Vision-Language Preference Learning for Embodied Manipulation

We thank reviewer R4Zt for the constructive comments. We will give our point-wise responses below.

W1: "A primary concern is the rationale behind the train-test task split in Meta-World ... as the rest of the experimental design is robust."

A: Thank you for your valuable feedback. The test tasks used in our paper are motivated by the test tasks in prior preference-based RL works [1,2,3,4,5]. However, we agree that using the standard ML45 benchmark can provide additional rigor and ensure broader reproducibility. To address this, we conduct experiments on the ML45 benchmark, training the vision-language preference model on its training tasks and evaluating on its test tasks. The results shown below demonstrate the strong generalization capability of our method on unseen tasks in ML45. This reinforces the robustness and adaptability of our framework regardless of task split. We have updated the manuscript and included these results and discussions in Appendix E.

Task	VLP Accuracy
Bin Picking	95.0
Box Close	90.0
Door Lock	100.0
Door Unlock	100.0
Hand Insert	100.0
Average	97.0

Q1: "The authors claim novelty in the architecture (line 071) ... clarify the specific architectural innovations that distinguish this approach?"

A: Thank you for highlighting this point. The architectural novelty lies in three key aspects that distinguish our approach from prior methods:

1. Vision-Language Preference Definition: We introduce language-conditioned preferences (intra-task, inter-language, and inter-video preferences) that generalizes the preference learning framework. Unlike prior work focused on state-based preferences [1,2,3,4,5], our method leverages video trajectories and language as flexible and universal conditioning inputs to define and learn preferences from multi-modal data.
2. The Preference Alignment Objective: The training objective explicitly optimizes the model to align video and language features according to our defined preference types. This alignment ensures that the learned preferences generalize effectively across tasks and instructions.
3. Cross-Modal Attention: The proposed vision-language preference model integrates video and language modalities using a cross-modal attention mechanism rather than naively combining video and language features. This mechanism not only extracts relevant features from both modalities but also aligns them to predict trajectory-wise preferences.

We have revised the manuscript to clearly highlight these points and their impact in Appendix E of the revised draft.

References

[1] PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training. ICML 2021.

[2] SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning. ICLR 2022.

[3] Reward Uncertainty for Exploration in Preference-based Reinforcement Learning. ICLR 2022.

[4] Few-Shot Preference Learning for Human-in-the-Loop RL. CoRL 2023.

[5] PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation. ICML 2024.

Q1: "How does the computational cost of training VLP compare to other approaches like R3M or VIP?"

A: Currently, we train VLP on Meta-World tasks using a single NVIDIA RTX 4090 GPU with 12 CPU cores for approximately 6 hours. For VLM methods like R3M and VIP, they usually requires pre-training on large-scale datasets like Ego4D, which can take several days with the same hardwares. However, since there lacks a large-scale preference dataset for embodied manipulation tasks, we cannot provide a direct comparison of the training cost between VLP and R3M or VIP.

Q2: "How sensitive is the model to the quality and diversity of language instructions? Is there a significant performance drop when using instructions generated by a less capable model than GPT-4V?"

A: Thanks for the question. We observe that generating diverse language instructions does not necessarily require strong VLMs like GPT-4V, even open-source Llama-3.1-8B-Instruct can accomplish this job since the language model is prompted with a diverse set of examples, following LAMP [1]. To evaluate this sensitivity, we conduct additional experiments using instructions from less capable model, such as GPT-3.5 and open-source Llama-3.1-8B-Instruct. The results in the following table show that the model's performance is relatively stable across different LLMs. We have included these results in the revised version in Appendix D to demonstrate the model's robustness to different types of models for generating language instructions.

Task	GPT-4V	GPT-3.5	Llama-3.1-8B-Instruct
Button Press	93.0	93.0	91.0
Door Close	100.0	100.0	98.0
Drawer Close	96.0	96.0	97.0
Faucet Close	100.0	100.0	100.0
Window Open	98.0	99.0	99.0
Average	97.4	97.6	97.0

References

[1] Language Reward Modulation for Pretraining Reinforcement Learning. ArXiv 2023.

We thank reviewer NSgJ for the positive support and constructive comments. We will give our point-wise responses below.

W1: "The evaluation is limited to relatively simple Meta-World tasks, without testing on more complex task domains (e.g., MANISKILL2 [1] and MyoSuite [2])."

A: Please refer to the subsequent glogal response.

W2: "The paper lacks comparison with human preference labels, which would validate the quality of the generated preferences against human intent."

A: We appreciate this suggestion. Although previous preference-based RL works mainly evaluate their methods using scripted preference labels, we agree that comparing our learned preferences with human labels would provide additional insights into the quality of the generated preferences. To address this, we collect human preference labels and conduct experiments with CPL, IPL and PIQL. The results in the following table show that our model's preferences align well with human intent, achieving high accuracy across different tasks. We have included these results in the Table 2 of the revised version to demonstrate the effectiveness of our method in learning preferences.

Task	P-IQL Human	P-IQL Scripted	P-IQL VLP	IPL Human	IPL Scripted	IPL VLP	CPL Human	CPL Scripted	CPL VLP	VLP Acc. Human	VLP Acc. Scripted
Button Press	93.1 ± 5.2	72.6 ± 7.1	90.1 ± 3.9	65.2 ± 7.2	50.6 ± 7.9	56.0 ± 1.4	85.0 ± 7.2	74.5 ± 8.2	83.9 ± 11.8	99.0	93.0
Door Close	79.2 ± 6.3	79.2 ± 6.3	79.2 ± 6.3	61.5 ± 9.4	61.5 ± 9.4	61.5 ± 9.4	98.5 ± 1.0	98.5 ± 1.0	98.5 ± 1.0	100.0	100.0
Drawer Close	63.7 ± 6.4	49.3 ± 4.2	64.9 ± 2.9	63.2 ± 4.6	64.3 ± 9.6	63.2 ± 4.7	54.1 ± 8.7	45.6 ± 3.5	57.5 ± 14.3	96.0	96.0
Faucet Close	51.1 ± 7.5	51.1 ± 7.5	51.1 ± 7.5	45.4 ± 8.6	45.4 ± 8.6	45.4 ± 8.6	80.0 ± 2.9	80.0 ± 2.9	80.0 ± 2.9	100.0	100.0
Window Open	69.7 ± 6.8	62.4 ± 6.4	69.7 ± 6.8	61.4 ± 8.6	54.1 ± 6.7	61.4 ± 8.6	99.1 ± 1.1	91.6 ± 1.7	99.1 ± 1.1	100.0	98.0
Average	71.4	62.9	71.0	59.3	55.2	57.5	83.3	78.0	83.8	99.0	97.4

W3: "The theoretical analysis assumes access to all possible segments, weakening its practical implications."

A: We acknowledge that the theoretical analysis requires the assumption of access to all possible trajectory segments, which introduces a gap between the theoretical guarantees and empirical applications. However, this assumption is essential for deriving the theoretical results. To address this, we presented the main theoretical analysis in the appendix, and we focus on the empirical results in the main paper to demonstrate the practical utility of the method. To solve your concern, we have clarified this point in the revised draft in the limitations in Section 6.

W4: "(minor) The paper does not report the performance of scripted policies, which would help establish an upper bound for task performance and validate the quality of collected expert demonstrations."

A: Thank you for highlighting this. We have conducted experiments using scripted policies on the same set of tasks used in the main experiments. The results in the following table show that the scripted policies achieve $100\%$ success rate on all tasks, providing an upper bound for task performance. Also, we have included these results in the revised version of the draft.

Task	VLP Accuracy
Button Press	100.0
Door Close	100.0
Drawer Close	100.0
Faucet Close	100.0
Window Open	100.0
Average	100.0

Dear Reviewer NSgJ,

As the deadline of the discussion period draws near, we would greatly appreciate your attention to our rebuttal. We would like to know whether we have adequately addressed your concerns. Your feedback is crucial to us, and we value the opportunity to address any concerns you have raised. The answer to W1 has been provided in the global response. If you have any further questions, we are more than happy to discuss.

Thank you for your time and consideration.

Best regards,

Authors

We thank reviewer LWSo for the constructive comments. We will give our point-wise responses below.

W1: "The theoretical claim seems to lack ... not mean that one approximates the other."

A: Thanks for the question. We give further justification for this point in the following.

In the theoretical analysis, we define $A^*_r(s_t^\sigma,a_t^\sigma)$ as the optimal advantage function of $\pi^*$ under the reward $r$. Then the regret preference model is defined as

$P\_{\rm regret}^*(\sigma^1 \succ \sigma^2 | r) = \frac{\exp \sum\_{\sigma^1} A^*\_r(s^{\sigma^1}\_t,a^{\sigma^1}\_t)}{\exp \sum\_{\sigma^1} A^*\_r(s^{\sigma^1}\_t,a^{\sigma^1}\_t) + \exp \sum\_{\sigma^2} A^*\_r(s^{\sigma^2}\_t,a^{\sigma^2}\_t)}.$

Then we denote the regret function as $-{\rm regret}(\sigma)=g\_\varphi(\sigma)=\sum\_\sigma A^*\_r(s^\sigma\_t,a^\sigma\_t)$, where the summation is calculated over the timeteps of segment $\sigma$.

In this case, we define the optimization objective as

$L\_1(g) = \mathbb{E}\_{\sigma\_1, \sigma\_2 \sim D\_r} {\rm CE} [ P\_{\rm regret}^{*}(\sigma^1 \succ \sigma^2 \mid r) ] = \mathbb{E}\_{\sigma\_1, \sigma\_2 \sim D\_r} {\rm CE} [\frac{\exp (g\_\varphi(\sigma^1))}{\exp (g\_\varphi(\sigma^1)) + \exp (g\_\varphi(\sigma^2))}],$

then we obtain that value of $g_\phi(\sigma)$ that estimates the negative regret function, i.e., we can obtain $g\_\varphi^*=\arg\min L\_1(g\_\varphi)$.

According to Eq. (11), we rewrite the actual optimization objective of our learning objective with samples from $D\_r$ as

$L\_2(f\_\psi) = \mathbb{E}\_{\sigma\_1,\sigma\_2 \sim D\_r} {\rm CE} [\frac{\exp (f\_\psi(\sigma^1))}{\exp (f\_\psi(\sigma^1)) + \exp (f\_\psi(\sigma^2))}].$

As a result, if $g_\varphi(\sigma)=f_\psi(\sigma)$, we have $L_1(g)=L_2(f_\psi)$. We also have $f^*_\psi=\arg\min L_2(f_\psi)=\arg\min L_1(g)=g_\varphi^*$ for the same preference dataset $D_r$. As a result, given the same preference dataset $D_r$, the proposed preference model $f_\psi$ can be considered as parameterized negative regret that approximates the true negative regret of the segment if both $L_1(g)$ and $L_2(f)$ are minimized.

However, in practice, $f_\psi(\sigma)$ and $g_\varphi(\sigma)$ are optimized under difference preference dataset. Specifically, $g_\varphi$ is defined under the preference dataset labeled by the ground truth reward function. In contrast, $f_\psi$ is optimized with preference data that has pseudo-labels $\tilde{y} \in \\{y^{\text{ITP}}, y^{\text{ILP}}, y^{\text{IVP}}\\}$ according to the optimality of segment and segment-language correspondence, which is an approximation of the ground truth preference labels.

W2: "I'm concerned that the simplicity ... weakening the paper's claims of generalizability."

A: We appreciate it for pointing out this concern. However, our framework explicitly focuses on learning relative preference relationships rather than absolute semantic similarities.

1. The preference model prioritizes alignment between videos and language instructions that are more relevant to each other. Even if a video and a language instruction from different tasks have some degree of similarity, our language-conditioned preferences do not treat such cases as positive examples compared with more similar pairs.

2. While ILP and IVP definitions rely on language conditioning, the cross-modal transformer actively aligns semantic relations between videos and language. This enables the model to capture and exploit similarities even when tasks differ, leveraging the flexibility of language-conditioned preferences.

3. Empirical results show that VLP effectively handles unseen tasks and languages instructions (as reported in Section 5). This demonstrates that our framework successfully learns generalized preferences rather than rigid task-specific labels.

Q1: "Comparing this work with RoboCLIP ... would strengthen the claims."

A: Thank you for the suggestion. We further conduct experiments using RoboCLIP on the five evaluation tasks and the results are shown in Table 4 of the updated draft. The comparison demonstrates that VLP outperforms RoboCLIP, highlighting the benefits of our vision-language preference learning. We have included a detailed comparison with RoboCLIP in the revised version to provide a more comprehensive analysis of the advantages of VLP over zero-shot video-input models.

Q2: "Regarding the second weakness ... share similar initial subtasks?"

A: Thank you for the questions.

(1) In the construction of video pairs for ITP, ILP, and IVP in the MTVLP dataset, we consider all combinations of video optimality levels: expert, medium, and random. Specifically:

• For ITP, we pair videos from the same task, where we assign preference based on the optimality of the trajectories (i.e., expert > medium > random).
• For ILP, videos from the same task are paired with language instructions from different tasks. These video pairs are treated as equally preferred under the given language instruction.
• For IVP, we pair videos from different tasks with a language instruction from either task, where the preference is assigned based on the alignment of video and task.

(2) Regarding medium-optimality, we define medium-optimality as a trajectory that completes part of the task (but not all). For example, in the task of Drawer Open, a medium level trajectory involves grasping the drawer handle but not fully pulling it out. In the task of Hammer, a medium level trajectory completes grasping the hammer but not hitting the nail. For initial subtasks, the task of Drawer Open and Door Open share similar initial subtasks, where the medium level trajectory for Drawer Open involves grasping the drawer handle but not fully pulling it out, while the medium level trajectory for Door Open involves grasping the door handle but not fully opening the door. However, although the initial subtasks may be similar, they have other distinct aspects that the target object is different during the initial subtask. We will provide more detailed examples of medium-optimality across the 50 tasks in the revised version to clarify this definition.

Q3: "Writing clarification suggestions."

A: Thanks for these suggestions. For (1), we have updated the manuscript to add the notations in Table 1, which is more clearer than describing with words. For (2), we have revised the confusious "language" term into "language instruction(s)" throughout the paper.

Thank you for the clarification and revisions. I was impressed with the comparisons with RoboCLIP. Also, regarding Q2, I see your point about the target object being different for each task. The Meta-World tasks appear to have different preconditions (e.g., a door is closed for the "Door Open" task) that can be easily verified using a video, making it difficult to find similar video segments from two different tasks. I agree that providing detailed examples of medium-optimality across the 50 tasks will greatly help clarify this issue.

Nevertheless, I would like to raise some additional questions regarding the theoretical analysis:

W1-continued:

I believe that the preference dataset used to optimize the preference model is crucial for determining the meaning of the learned preference model. Can we say that $f_{\psi}$ can be considered as $g_{\varphi}$, which approximates the true negative regret, even if it is optimized with a different preference dataset?

To illustrate my point, here is an example with a similar argument, using the same preference dataset for optimization. Let's think of an "immediate reward function", defined by $h(\sigma):= r(s_1^{\sigma}, a_1^{\sigma})$ for $\sigma = (s_1^{\sigma}, a_1^{\sigma}, \cdots, s_H^{\sigma}, a_H^{\sigma})$. Then we can derive a preference distribution using $h$.

$$P_{\text {imm.rew}} \left(\sigma^1 \succ \sigma^2 \mid r\right)= \frac{\exp \left(h \left(\sigma^1\right)\right)}{\exp \left(h \left(\sigma^1\right)\right)+\exp \left(h \left(\sigma^2\right)\right)}$$

Similarly to $L_1$ and $L_2$, we can define the optimization objective for $h_{\phi}$ as follows.

$$L_3(h_{\phi})=\mathbb{E}\_{\sigma_1, \sigma_2 \sim D_r} \mathrm{CE}\left[ P\_{\text {imm.rew}}\left(\sigma^1 \succ \sigma^2 \mid r\right)\right]=\mathbb{E}\_{\sigma_1, \sigma_2 \sim D_r} \mathrm{CE}\left[\frac{\exp \left(h_{\phi}\left(\sigma^1\right)\right)}{\exp \left(h_{\phi}\left(\sigma^1\right)\right)+\exp \left(h_{\phi}\left(\sigma^2\right)\right)} \right]$$

If we use the same preference dataset $D_r$ to optimize $h_\phi$, we will get $h_\phi^*=f_\psi^*=g_{\varphi}^*$. But this result, suggesting that the parameterized immediate reward is equivalent to the parameterized negative regret, sounds weird. I believe the problem arises from the assumption that the same preference dataset can be used for optimization. And if we use different preference datasets for optimization, we can no longer say that each parameterized preference model approximates the other.

Please kindly correct me if I have misunderstood or gotten something wrong.

Q4:
Additionally, it would be helpful to discuss the ground-truth reward function associated with the preference model $f_{\psi}$ to analyze the meaning of the learned preference distribution $P_{\psi}$.

In my understanding, $f_{\psi}$ is defined as a multi-task preference model that can compare two segments from either the same task or different tasks (e.g., $\sigma_1^1 \sim \mathcal{T}^1, \sigma_1^2 \sim \mathcal{T}^2$) based on any arbitrary language instruction (e.g., $l \sim \mathcal{T}$). Accordingly, the true reward function associated with $f_{\psi}$ should be defined as a single multi-task reward function, encompassing all $\mathcal{R}^\mathcal{T}$ for each task $\mathcal{T}$ and conditioned by the language $l$.

I believe the regret should be defined with this ground-truth, multi-task, and language-conditional reward function, but I'm afraid the definition for this reward function is not clear.

I would like to hear your thoughts on this point. Could you also share some intuition on how this ground-truth multi-task reward $\mathcal{R}\_{f_{\psi}}(\sigma \mid l)$ is structured in terms of each task-specific reward $\mathcal{R}^\mathcal{T}(\sigma)$?

Dear Reviewer LWSo,

As the deadline of the discussion period draws near, we would greatly appreciate your attention to our rebuttal. We would like to know whether we have adequately addressed your concerns. Your feedback is crucial to us, and we value the opportunity to address any concerns you have raised. If you have any further questions, we are more than happy to discuss.

Thank you for your time and consideration.

Best regards,

Authors

We thank Reviewer LWSo for the insightful feedback. We provide our point-wise responses below.

Q2-continued

Thanks for your feedback. We have updated the medium-optimality videos across the 50 Meta-World tasks on our website. Please refer to the website for details.

---

W1-continued: "Can we say that $f_\psi$ can be considered as ... different preference dataset?"

Thanks for the question. The answer is yes if the preference labels are the same for two preference datasets. Let us define two preference datasets in the same format as $\{(v_1, v_2, l, y)\}$, where video pair ($v_1$, $v_2$) and language description $l$ can be sampled randomly from their space, and $y$ denote whether $v_1$ is preferred than $v_2$ in such a language description. Then, (1) in the ground-truth preference dataset $D_r$, the preference label $y$ is given by comparing the ground-truth accumulated reward on videos, i.e., $R(v_1|l)$ and $R(v_2|l)$; while (2) in the preference dataset $D_{VLP}$ of VLP, the preference label $y$ is determined by the optimality (if $v_1$, $v_2$ are from the same task defined by $l$) or the video-language alignment (if $v_1$, $v_2$ are from different task and one video belongs to task defined by $l$, or both videos not belong to task $l$), which are defined by the pseudo-labels $\tilde{y}\in\\{y^{\text{ITP}}, y^{\text{ILP}}, y^{\text{IVP}}\\}$ in VLP. We remark that there are also other cases beyond these kinds of preferences defined in pseudo-labels of VLP, in which we use the generalization of learned preference model $f_\psi(v|l)$ to define its preference label.

In an ideal case, if the preference label of $D_r$ and $D_{VLP}$ are the same, then $f_\psi$ can be considered as the same as $g_\varphi$. In practice, $D_r$ and $D_{VLP}$ can be different in some cases beyond the definition of pseudo-labels $\tilde{y}$ depending on the generalization ability of the learned preference model.

W1-continued: "But this result, suggesting that the parameterized immediate reward ... sounds weird."

Thanks for the problem. We believe the definition of "immediate reward function" as $h(\sigma):= r(s_1^{\sigma}, a_1^{\sigma})$ is wrong since the immediate reward function can only be defined in a single time step. Specifically, the correct definition can be (1) $h(s_1, a_1):= r(s_1^{\sigma}, a_1^{\sigma})$ as the single-step reward function and $h(\sigma):=\sum_\sigma r(s_t^{\sigma}, a_t^{\sigma})$, which is the ordinary BT model. (2) $h(\sigma):=\sum_\sigma Q^*(s_t^{\sigma}, a_t^{\sigma})$ or $h(\sigma):=\sum_\sigma Q^*(s_t^{\sigma}, a_t^{\sigma})-V^*(s_t^{\sigma})$, where the two types of $h(\sigma)$ are actually the same in regret-based preference model since $V^*(s_t^{\sigma})$ is a constant. In the second case, $h(\sigma)$ approximates the negative regret, as we highlight in our theoretical analysis.

---

Q4: "Additionally, it would be helpful to ... learned preference distribution"

Thanks for the comment. Indeed, $f_\psi$ is defined as a multi-task preference model that can compare two segments from either the same task or different tasks (e.g., $\sigma^1 \sim \mathcal{T}^1, \sigma^2 \sim \mathcal{T}^2$) based on any arbitrary language instruction (e.g., $l \sim \mathcal{T}$). As a result, the reward function $r(s, a \mid l)$ is defined as a single-step reward that evaluates the immediate quality of a state-action pair $(s, a)$ based on the task's goal and the provided language instruction $l$, and $R(v|l)=\sum_v r(s,a|l)$ is defined as the undiscounted return. For a given task $\mathcal{T}$, the reward function depends on how well the state-action pair aligns with the task objective under the guidance of the language instruction. For example: If the instruction $l$ specifies "close the drawer," the reward for states where the drawer is closer to being closed will be higher. If the instruction $l$ specifies "open the window," the reward will prioritize states corresponding to window opening. The reward is inherently language-conditioned, which allows it to generalize across tasks. Formally, for a specific task $\mathcal{T}$, the single-step reward function $r(s, a \mid l)$ aligns with the task-specific reward $r^\mathcal{T}(s, a)$, but conditioned on the instruction $l$. For multi-task learning, the reward function is unified as: $r(s, a \mid l) = r^\mathcal{T}(s, a)$, where $\mathcal{T}$ is implicitly determined by the language instruction $l$.

We thank reviewer ARLJ for the positive support and insightful comments. We will give our point-wise responses below.

W1: "In the experiments, ... on several different environments."

A: Please refer to the subsequent glogal response.

W2: "Only five tasks in the Meta-work are evaluated among 50 tasks. This set of test tasks is not so challenging compared to 45 training tasks."

A: Thank you for your valuable feedback. The test tasks used in our paper are motivated by the test tasks in prior preference-based RL works [1,2,3,4,5]. To address this, we conduct experiments on the ML45 benchmark, training the vision-language preference model on its training tasks and evaluating on its test tasks. The results shown below demonstrate the strong generalization capability of our method on unseen tasks in ML45. This reinforces the robustness and adaptability of our framework regardless of task split. We have updated the manuscript and included these results and discussions in Appendix E.

Task	VLP Accuracy
Bin Picking	95.0
Box Close	90.0
Door Lock	100.0
Door Unlock	100.0
Hand Insert	100.0
Average	97.0

W3: "Why are RL-VLM-F and CriticGPT not compared?"

A: RL-VLM-F and CriticGPT are both online preference-based RL algorithms, while our method focues on offline preference-based RL setting.

W4: "The dataset construction itself may ... size is not big."

A: Thank you for raising this concern. While the trajectory sampling pipeline is straightforward, we would like to clarify that the contribution lies in addressing a critical gap in the field: the lack of high-quality, multi-modal preference datasets for vision-language preference learning. It introduces implicit preference labels conditioned on diverse language instructions, which are currently unavailable in existing benchmarks. By including trajectories of three distinct optimality levels (expert, medium, random) and pairing them with multiple language instructions, we ensure a varied and rich dataset. Our goal was to demonstrate the effectiveness of language-conditioned preference learning, not merely to build a large dataset. Future work could expand the dataset scale and diversity to facilitate broader research.

W5: "Figures can be improved."

A: Thank you for this feedback. Regarding Fig. 2, we understand it may appear standard. However, its primary purpose was to clearly present the architecture vision-language preference model. For Fig. 3, we have improved the clarity of the attention maps to illustrate the cross-modal attention mechanism. Please refer to the revised manuscript for these updates.

References

[1] PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training. ICML 2021.

[2] SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning. ICLR 2022.

[3] Reward Uncertainty for Exploration in Preference-based Reinforcement Learning. ICLR 2022.

[4] Few-Shot Preference Learning for Human-in-the-Loop RL. CoRL 2023.

[5] PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation. ICML 2024.

Dear Reviewer ARLJ,

As the deadline of the discussion period draws near, we would greatly appreciate your attention to our rebuttal. We would like to know whether we have adequately addressed your concerns. Your feedback is crucial to us, and we value the opportunity to address any concerns you have raised. The answer to W1 has been provided in the global response. If you have any further questions, we are more than happy to discuss.

Thank you for your time and consideration.

Best regards,

Authors