Thank you for your dedication to reading our work and providing thoughtful feedback. Below are our responses to the weaknesses and questions you raised.

Weaknesses

The main weakness is the novelty of this approach. The approach uses online-DPO style method to generate and leverage preference data to learn downstream policy. Direct preference optimization approaches have already been applied to many domains. Although the improvements in results in Minecraft domain are appreciable, the paper does not offer any specific analysis for the pertaining application, which makes this work incremental.

We believe that our approach represents a novel contribution as it is the first to propose using preference data for efficient post-training foundation policies. In fact, our approach is not limited to DPO; it can be applied to other preference learning methods. We add experiment of IPO[1] and SLIC. The results are presented in Appendix D. Additionally, our method requires a smaller amount of data and less training effort compared to traditional approaches, making it a resource-efficient solution. Furthermore, it offers broad applicability, such as serving as a continual learning method or integrating with higher-level planners. These capabilities underscore our contribution and demonstrate its potential impact across various open-world tasks and beyond.

[1] Azar et al. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics,pp. 4447–4455. PMLR, 2024.

Further, the method assumes trajectory generation for current policy in simulation, which might be cost-inefficient in real domains like robotics. This limits the applicability of the proposed approach in such scenarios.

We believe that scenarios with high rollout costs is actually a favorable situation for PGT application. Unlike online RL and other imitation learning algorithms, which require a large amount of interaction, PGT requires very little data, typically around tens to a hundred trajectories. Therefore, the cost of data collection is relatively low. At the same time, the smaller data volume makes the process of annotating preferences less costly.

The paper generates preference dataset using privileged information (preferences), however the comparison baseline Behavior cloning does not, which raises questions on the fairness of such comparison. It would have been interesting to see comparisons against other methods that leverage such preference data for policy learning like SLIC [1].

In fact, for the sake of fairness, when we used $N$ positive samples and $N$ negative samples to train PGT, we used $2N$ positive samples to train BC. Moreover, we have added experiments with SLIC as well. The results are included in Appendix D of the rebuttal version.

Thank you for your dedication to reading our work thoroughly and providing thoughtful feedback. Your insights have been instrumental in helping us refine and improve our research. Below are our responses to the weaknesses and questions you raised.

Weaknesses

Limited application: The paper conducted experiments solely on Minecraft Skill forage, leaving questions how it works in other goal-conditioned applications.

We believe that for many goal-conditioned policies, not limited to the open-ended world agent and robotics fields, PGT has potential applications. Due to constraints in time, computational resources, and model availability, we conducted experiments using two open-source foundation policies in the Minecraft domain. We consider other applications as future work.

This method can be hard to adopt in tasks where rolling out trajectories or obtaining preferences for them is costly

We understand your question. We believe that scenarios with high rollout costs are actually a favorable situation for PGT application. Unlike online RL and other imitation learning algorithms, which require a large amount of interaction, PGT requires very little data, typically around tens to a hundred trajectories. Therefore, the cost of data collection is relatively low. At the same time, the smaller data volume makes the process of annotating preferences less costly.

Utilizing negative trajectory samples seems critical in this method but it is already well known method. Unless there is a specific challenge to adopt such method in openworld task such as Minecraft, the contribution of this paper becomes marginal as it is just adopting good method in another applications.

We believe that our approach represents a novel contribution as it is the first to propose using preference data for efficient post-training foundation policies. This distinguishes our work from existing methods, even if they also leverage negative samples. Our method is applicable to nearly all goal-conditioned policies and is not limited to open-ended world environments or Minecraft. Additionally, our method requires a smaller amount of data and less training effort compared to traditional approaches, making it a resource-efficient solution. Furthermore, it offers broad applicability, such as serving as a continual learning method or integrating with higher-level planners. These capabilities underscore our contribution and demonstrate its potential impact across various open-world tasks and beyond.

Question

Due to the word limit, we have included the responses to the questions in the next block.

Questions

Can this method be easily adapted for other goal-conditioned tasks where the goal is not limited to a language prompt but could be, for example, an image—as in robotics tasks (as the authors mentioned), possibly Franka Kitchen [1,3] or UR3 BlockPush [2,3]? If not, an explanation of the limitations would help clarify the potential of the proposed method.

Thank you for your feedback. PGT can certainly be applied to other goal-conditioned tasks where the goal is not in natural language. In fact, Groot is a video-instruct foundation policy. We appreciate the two environments you provided, and we believe PGT is well-suited for these scenarios. We have added references to these papers.

We take [3] as a concrete example. [3] employs VQ-BeT to construct a goal-conditioned foundation policy where the goal is provided as a sequence of images. After reading the code from [3], we found that these images are mapped into a goal_seq via a _resnet_header. This is a latent representation of the goal. This is where PGT comes into play. The goal_seq serves the same function as the goal latent representation in our paper. We collect trajectories and label them as positive and negative samples, then train the goal_seq with PGT-loss. Suppose we obtain goal_seq' after training. goal_seq' can be used as a good latent to perform the task.

[3]:Lee, Seungjae, et al. "Behavior generation with latent actions." arXiv preprint arXiv:2403.03181 (2024).

It is unclear to me how PGT works. Could you elaborate more on PGT and how to generate g in the first place to compute LPGT(g,gref) ?

Take the GROOT policy and the "collect_wood" task as an example. GROOT follows an encoder-decoder architecture, taking video instructions as prompts. It encodes the video instructions with an encoder into a 512-dimensional latent vector, and the decoder uses this latent vector as a condition, acting as an auto-regressive policy network.

During the data collection phase, we first select a video about collect wood, which may be suboptimal for GROOT. We can encode the video using the encoder and obtain the latent goal, denoted as g_ref. We then collect trajectories of collecting wood generate by the decoder and g_ref. Next, we label the trajectories as either positive or negative.

In the training phase, we initialize g as g_ref and use it as the condition for the decoder, then use previously collected preference data with PGT loss to train only the latent goal g. During this training process, the parameters of the decoder remain fixed.

For each task, should we conduct iterative process which gathers trajectories and obtains preference them from human or reward models to get the optimal latent goal. If so, this method can be limited in generalization where the strength of the foundation model primarily lies.

Indeed, PGT trains a separate latent for each task, but PGT should be regarded as a plug-and-play module that generates a refined prompt, significantly enhancing the capabilities of goal-conditioned policies and even unlocking potential in the foundation model that ordinary prompting cannot. Since we only train the goal latent representation and do not alter the model parameters, PGT does not conflict with the generalization capabilities of the foundation model. When faced with unseen tasks, the foundation model can still function as a goal-conditioned policy seamlessly.

In Table 2, is there any specific or presumed reason for STE+ not working well compared to STE in hunt task?

1. To ensure consistency and convenience, we use the same hyperparameters for all tasks under the same loss and training parameter settings. We did not specifically tune hyperparameters for each task. Maybe the optimal hyperparameters differ from the default ones.
2. Actually STEVE-1 is not good at killing sheep. The quality of the collected synthetic data is not good enough.

How sensitive is PGT with respect to β in Eq. 2? which value is used for each task?

We conducted a hyperparameter search on the "collect wood" task and used the same set of hyperparameters for all the other tasks. We visualized the performance of the "collect wood" task under different values of $\beta$. The results showed similar performance when $\beta \geq 0.2$. We choose $\beta=0.6$ for the rest of tasks. The line graph of different $\beta$ and other hyperparameters used are listed in Appendix B.5.

Regarding weakness 3), could you explain how the proposed method differs from or improves upon existing applications of preference learning in open-world tasks?

Thank you for raising this important point. We addressed weakness 3 in our previous response. We hope our earlier explanation clarifies these differences.

We sincerely thank you for your valuable feedback and thoughtful comments! We truly appreciate the time and effort you have dedicated to reviewing our submission. Below, we provide detailed responses to your questions.

Weaknesses

Section A.2 about PGT loss mentions the assumption of a "Bradley-Terry-Model-liked oracle reward function" existing. I don't quite understand what it means to be a BT-model-liked reward function. We apologize for the unclearity. We will explain in detail.

Generally, we want to utilize human preference to finetune our policy. "Preference" is assumed to be a subjective choice from human but generated by an oracle reward function $r^*(\tau)$, which is inaccessible. $r^*(\tau)$ represents how well trajectory. $\tau$ performs given task. The better $\tau$ performs, the higher $r^*(\tau)$ is. "Bradley-Terry-Model-liked oracle reward function" in the article refers to this. It is worth mentioning that this oracle reward function is used only as a derivation target and is ultimately eliminated. Its role is to help us derive PGT loss.

We refine our writing and change some notations of Appendix A to make the derivation clearer. You can see the rebuttal version for more details.

Section A.2 what does it mean the task can be subjective or objective? This needs elaboration as I don't understand how it relates to the derivations.

Sorry for the misunderstanding here. This sentence is irrelevant to the derivation below. Our derivation is base on $r^*$. So it is fine to just ignore it.

"Task can be subjective or objective" means on data collection phase, the classification of trajectories can be based on human preference or reward from environment. Here reward from environment is the singal from environment, which aligns with RL literature. For example, in the task "collect wood", we can simply define the number of logs collected as reward. We use trajectories that collect more logs as positive trajectories, and trajectories that collect less as negative ones. We call this kind of task "objective".

For tasks like "explore climb", the agent needs to climb up a mountain, so no explicit reward can be obtained from the environment. Human labellers are required to annotate positive and negative trajectories. We call this kind of task "subjective".

This is perhaps is more of a opinion coming from a RL+embodied AI/robotics background but I find the term "Task Environment Generalization" to be a little misleading.

We appreciate your perspective, and we understand "Out Of Distribution(OOD)" might initially seem potentially misleading. Our generalization refers to performing the same task in different locations, which is a crucial ability for open-ended world agents. For example, agents are required to collect wood in different biomes, which means they need to find different kinds of trees in different locations. Although we stick to the original term, we still admire your perspective.

Table 2's choice of colors for improvement numbers vs worsened numbers is a bit strange to me (maybe a cultural thing). I would normally expect green values to be = good, red = bad, but it appears flipped here.

Thanks for this advice. We adopt it in the rebuttal version. It is truly a cultural thing :)

Questions

What were the initial prompts like used in Figure 4? I am not particularly surprised some finetuning on newly collected preference data for the wood collecting task would do well if the initial prompts were already fairly good.

We collect 5 different initial reference videos by humans. They are normally collected wood videos in Minecraft without cherry-picking or special tricks. The results of iteration 0 are the performances of these initial prompts. The horizontal dashed line represents the best-performing initial prompt, and in the first small figure, the starting point of the iterations curve coincides with the horizontal line, indicating that this represents the best-performing initial prompt. The starting points of the iteration curves for other initial prompts are all below the horizontal line, and we want to emphasize that after iterations, they all rise well above the horizontal line, meaning that they all outperform the best initial reference video recorded by humans.

Is it possible to map the finetuned prompt latent representation to human text? That would be interesting from an interpretability standpoint + may unveil qualitative/quantitative insights into which prompts work for a model and what tends to not work.

Although we have no way to convert the latent goal into text, that is a really interesting direction. In GROOT, the authors plotted the t-SNE of the latent representation. We print the t-SNE of our latent goal in different representations. Although we have not reached any conclusions, the latent goal space may have more properties worth exploring.

We sincerely appreciate the time and effort you have dedicated to carefully reading our paper, and we sincerely thank you for your valuable feedback. Below are our responses to the weaknesses and questions you raised.

Weaknesses

Minecraft environments is a good start, while more challenging tasks would be greatly appreciated.

We completely agree with your perspective. We believe PGT is well suitable for other goal-conditioned policies in domains like robotics.

For example, another reviewer mentioned a robotics environment UR3 BlockPush[1], for which we have provided implementation details.

[1] Lee, Seungjae, et al. "Behavior generation with latent actions." arXiv preprint arXiv:2403.03181 (2024).

The title of section 4 is not very informative. Please re-consider the title of section 4.

Thanks for your feedback. We revised the title of section 4 as Experiments and we hope this title will be more informative.

"Trajectory Preference" is serious. I briefly read the derivation of the PGT loss in section A.2. From my understanding, the π(τ;g) denotes the action given by the policy is conditioned on the goal g and the trajectory τ.

We are sorry for the misunderstanding. Assume $\tau = (s_t,a_t) (0 \leq t \leq N-1)$ is a $N$-step trajectory, $\pi(\tau;g)$ denotes the probability of generating $\tau$ conditioned on $g$ . We revise our notation as $\pi(\tau|g)$, and it can be expanded as:

Where $p(s_{i+1}|s_t,a_i)$ is transition dynamics, which are model-independent. Therefore, it does not exist in the final loss.

The revised version of derivation is in Appendix A. Despite some changes in notation, the final result remains the same.

Questions

If I understand it correctly that you do treat each τ as a single step to optimize under DPO, it is still required to model the whole task as a MDP?

We believe this is a common misunderstanding. MDP problems refer to settings where actions can be made based solely on the current state without requiring information from past states. When facing a non-MDP problem, the usual practice in RL/agent is to expand state variables to include more information, and regard it as an approximate MDP problem. In the last step of loss derivation, we decompose $\tau$ as $(s,a)$ pairs. This step leverages the MDP modeling, as it is this step that enables us to compute the loss.

Here we propose a further explanation: If there is a non-MDP environment where deciding on an action requires not only the current observation but also past observations, the state can be expanded to $[o_t, o_{t-1}, \dots, o_{t-h+1}]$ where $h$ is the context length. Under this modeling approach, PGT is effective and can be used in this non-MDP environment.

When annotating the trajectories, how to judge the quality of the trajectory if their goal g is different?

There is no need to compare trajectories with different goals. The PGT approach involves collecting several trajectories for a specific goal, then distinguishing these trajectories as positive and negative samples for training. When labelling, any two trajectories are relative to the same goal.

This is minor, why the counter-part of the Full Fine-Tuning in table 1 is the Soft Prompt? What does Soft indicate here?

Sorry for the unclearity here. Our method focuses on goal-conditioned policies. These policies encode goal to a latent representation. During training, we fixed the entire policy and trained the goal latent representation only, which is a 512 dimension vector. This is similar to soft-prompt in the field of Large Language Models, so we adopt the phrase here.

Table 1 tries to offer a comparison between BC-loss and PGT-loss, and we performed comparisons under both fine-tuning goal latent (which we called soft prompt) and full parameter fine-tuning scenarios. We found that, regardless of the setting or task, PGT-loss consistently outperformed BC-loss.

Moreover, for the ablation results regarding the use of soft-prompt fine-tuning versus full parameter fine-tuning, please refer to Figure 3.

It is possible to use foundation model itself (or any other foundation model) to judge the quality of the trajectory?

You are absolutely right. We propose a potential approach here: adding a judgment head within the policy to assess the quality of the current action or previous action. The cumulative score of every action in a trajectory would then be used as the overall score for that trajectory.

Using extra models to judge the quality of trajectories is another promising approach. For example, Video-to-text models, also known as video language models, can tackle various tasks like video question answering and video captioning. These models can be used to score trajectories.