Learning to Plan with Personalized Preferences

Dear Reviewer kiee:

Thank you for your thoughtful review and constructive feedback.

Some potential limitations include these results/claims are mainly validated via simulations, the real-world applicability of PbP is not quite clear.

Most of the claims in this paper are mainly validated via simulations, the real-world applicability of PbP is not quite clear. We acknowledge the weakness for the lack of real-world demonstrations. We would argue that our study has a different focus. The preference definitions defined, together with the simulation based on prior research in the community, can be viewed as simplified simulation of real-world human-AI interactions. We did not ground the task in more challenging settings involving real humans, for the reason that their preferences might change unpredictably and observations of their behaviors can be noisy and with errors. Instead, we focused on fundamental reasoning and planning tasks where a stable preference can guide the entire process. However, moving forward, it is indeed important to explore our current work in real-world scenes with diverse and unpredictable real-human behaviors. Adapting the pipeline to real-world scenarios is a valuable direction for future work.

One potential issue is that it is clear that there will be an error propagation in the two-stage learning, however, there is not a comprehensive or deep analysis from this perspective. However does the noisy preference prediction affect downstream planning?

We have shown the results in table 1, where the mid line results indicates using previously inferred noisy preference labels, and the bottom line results ground truth preference labels for downstream planning. There is a significant performance drop when using noisy preference labels.

How diverse are user preferences in the benchmark? Does the benchmark sufficiently cover cross-cultural differences?

Very important question. There are 290 pre-defined questions in PbP. There are certainly some corner cases or cross-cultural differences that not coverd in these scope. However, as defined, these preferences can be seen as high-level abstractions of action sequences. The generalization of our preference modeling doesn't necessarily depend on the scope of these pre-defined preference definitions. The model learns from action sequences and finally outputs actions as well. The pre-defined preferences mainly serve as a guide to help us sample demonstrations and help model planning. While they may not cover all corner cases, they provide a substantial enough range to serve as a benchmark to evaluate the baselines.
Besides, our environment naturally supports extensions like cross-culture differences or preference drifts. So long as the task can be formulated as action sequences and sub-structures exist in the action distributions, the methodology can be applied, no matter how domains vary, or humans have their own unique preference, or organize preferences according to cultural differences. Our proposed three-tiered hierarchical structure of preferences is designed from the perspective of how things happen in a household scenario.

There is often preference drift in practice, will the benchmark be able to handle evolving or conflicting user preferences over time?

Yes, as the learned preferences are mainly based on high-level abstractions of action sequences, as long as the user preference drift is demostrated and can be observed through his observations, his preference evolution could be learned and updated to the machine's policy.

In practice, the amount of data may be limited. In order to achieve reliable preference inference in real-world applications, is there any estimations on the number of demonstrations required?

Yes. This is exactly the figure 7 wants to show. We do ablation study on the number of demonstrations and show the results in various cases. Generally, more demostrations mean better performance. In our experimental settings, we found that approximately five demonstrations are typically enough. However, real-world applications may require further validation, as they involve more complex perception challenges and more nuanced human behaviors than our controlled experimental environment. We also agree that a more rigorous study shall be required in order to study accurate scaling laws.

We sincerely welcome your further feedback.

Dear Reviewer rU32:

Thank you for your thoughtful review and constructive feedback.

the motivations for formulating preference learning as a few-shot learning from demonstrations are technically naive. Relevant literature in embodied AI and preference learning needs to be reviewed.

Our formulation is based on two considerations:
i) Humans, even infants are found to have the ability to detect others' preference with only a few demonstrations. There do exist a series of literatures, especially in psychology, supporting this [1][2][3]. Recently in embodied AI, works like [7][8] demonstrate the importance of personalization in robot assistants and highlight the difficulty in preference inference and adaption. So to facilitate embodied AI, it is necessary to test such ability.
ii) Under our home assistant setting, it is nearly impossible to collect a larget amount of demonstrations for a specific person and task. And it is too tedious for users to choose or rank the preferred trajectories in performing everyday tasks. So in a realistic setting, learning from the observations of user behaviors is a more natural way rather than collecting preference data. There exists literature proposing similar opinions, like reducing query times as far as possible [5][6], or using few-shot learning from demonstrations for adapting [4]. We will add more literature discussion in the Intro part and Related work part.

[1] Choi et al. How do 3-month-old infants attribute preferences to a human agent?. Journal of experimental child psychology.
[2] Duh et al. Infants detect patterns of choices despite counter evidence, but timing of inconsistency matters. Journal of Cognition and Development.
[3] Baker et al. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour. [4] Verma et al. Adaptagent: Adapting multimodal web agents with few-shot learning from human demonstrations.
[5] Hejna et al. Few-shot preference learning for human-in-the-loop. PMLR.
[6] Yu et al. Few-shot in-context preference learning using large language models.
[7] Hwang et al. Promptable behaviors: Personalizing multi-objective rewards from human preferences. CVPR. [8] Hellou et al. Personalization and localization in human-robot interaction: A review of technical methods. Robotics.

It is unclear whether or how the M - the bird's eye view are utilized in the inputs. More details are needed explaining any explicit mappings other than A_i between the S_i and M values. The work also needs to specify whether this was a choice of design for such kinds of evaluations.

The scene maps are not used as the model input in our settings, as discussed in sec 4.2. It rather helps illustrate the overall process of robot behaviors. Its exclusion from model inputs is a deliberate design choice to focus on egocentric perception. There are no other explicit mappings other than A_i (can be human action or robot action). For evaluation, we kindly refer to examples and discussons to reviewer eCTV.

Generalization experiment

For generalization experiment, there could be randomly-sampled scenes and objects for the same preference. We will add side-by-side rendering in revision. Thank you for your suggestion.

It is essential to note that the VLMs have a frame of reference bias.

We will explicitly highlight this point together with the references in revision.

It would have been useful to know which levels of preference learning are more prone to failure or are difficult for what kind of models.

Table 1-3 in the paper reports model-level performance at different levels. Generally, all models struggle more when dealing with the sequence level than the option level, as in long-horizon preference reasoning dependencies between preference steps accumulate errors.

'gen' and 'orig' has been used for the same.

Sorry for the confusion. We will fix this confusion in revision.

What do the auxiliary preference tokens look like? How do they differ for symbolic models vs vision models?

The auxiliary preference tokens are preference labels we define in section 4.1. They are same for symbolic models and vision models for two-stage experiment setting.

Does option-level include action-level preferences as well?

Yes. Action-level preferences are related to single actions. We didn't explicitly include action-level preference in comparision for the reason that as commonly defined, action preferences are related to specific actions, such as drinking juice vs drinking coffee. A basic imitation policy can effectively learn these preferences where models only need to remember and repeat. We would like to focus on more complex preferences that cannot be easily addressed through simple copy/imitation, and to test models in scenarios where preferences involve more nuanced or context-dependent decisions.

We sincerely welcome your further feedback.

Dear Reviewer mzDn:

Thank you for your thoughtful review and constructive feedback.

No direct comparison with baseline retrieval-based or reinforcement learning based approaches.
Other Strengths And Weaknesses: a detailed comparison with RL based methods is missing.

Retrieval-based and RL-based approaches are intuitive candidates for this benchmark, but they face inherent limitations that make them unsuitable for the task. For Retrieval-based methods, they rely on matching input queries to similar patterns in a memory pool. Although they can retrieve relevant demonstrations, they fail to address the core challenge of the few-shot reasoning:

These scores are notably poor—only marginally better than DAG-based methods—because retrieval-based systems, while capable of accessing more demonstrations, cannot effectively generalize from historical data to new tasks. We will add this baseline to our paper.
RL-based approaches, especially meta-RL, on the other hand, often need millions of training steps and explicit reward functions for each task. The requirement contradicts with the few-shot nature of our task: we need an agent that can adapt to new tasks given only a few demonstrations in a complex environment, where current RL-based or meta-RL based methods have been not shown satisfactory results. Moreover, designing reward functions for each preference can be subtle and trivial, which is impractical as humans have very diverse and nuanced preferences.

no real-user evaluations validate whether these improvements translate to better human-AI interactions.
The benchmark relies on synthetic data, and no real-world demonstrations are included.
The paper also lacks Real-World Validation

We would argue that as a first of its kind in the study, our work focuses on developing an agent capable of the task in virtual environments before transitioning to real-world scenarios. The preference definitions defined, together with the simulation based on prior research in the community, can be viewed as simplified simulation of real-world human-AI interactions. While not grounded on the real world, our design still focuses on fundamental reasoning and planning tasks where a stable preference can guide the entire process. However, moving forward, it is indeed important to explore our current work in real-world scenes with diverse and unpredictable real-human behaviors. Adapting the pipeline to real-world scenarios is the future direction of our work.

need more experiments to see how the used metrics is inline with the literature. There is a lot of research on RL based preference learning (see literature section).

RL baselines: see claims above. Levenshtein distance: kindly refer to examples and discussons to reviewer eCTV.

Relation To Broader Scientific Literature. Essential References Not Discussed.

Thank you very much for your kind reminder.
Our work aligns with PbRL works most closely as we learn from demonstrated action sequences that implicitly encode user preferences. PbRL mainly utilizes human preferences as feedback from experts to replace numeric rewards to help models learn better [1]. Works like [2] explore goals defined in terms of human preferences between trajectory segments, [3] proposes modeling human preferences instead as informed by each segment's regret. They are definity works related to our research topic. We further extend beyond traditional preference-based RL settings in several ways. While most preference-based RL methods require extensive human feedback through pairwise comparisons or explicit reward signals, we focus on learning from minimal demonstrations that implicitly convey preferences. Second, rather than learning a single reward function or policy, our approach aims to identify and abstract generalizable preference patterns across diverse tasks and scenarios.
We will add such discussion and cite these related papers in our revision. And we will also include RObocasa in Sec 2.3 of our paper.
[1] Advances in Preference-based Reinforcement Learning: A Review.
[2] Deep reinforcement learning from human preferences.
[3] Models of human preference for learning reward functions.

The paper has limited Scope of Generalization

As far as we know, there is a lack of benchmarks for embodied tasks that include systematically-defined human preference or human behavior data. Thus, we also propose a benchmark to tackle this and see this as one of the contributions of our work. We will release the benchmark together with our baselines for the community to test and explore.

We sincerely welcome your further feedback.

Dear Reviewer eCTV:
Thank you for your thoughtful review and constructive feedback. We appreciate your recognition of the benchmark’s novelty and the clarity of our primary findings. We address your concerns point by point:

The proposed benchmark is novel within embodied AI. However, its significance is unclear, as simple symbol-based approaches already perform fairly well (Table 2), surpassing video-based counterparts.

Video-based parts are important in the benchmark. We mainly incorporate symbol-based parts to study how different modalities impact performance for more comprehensive evulation. However, the benchmark’s significance greatly lies in its scalability and realism. To ensure our research can address real-world challenges in perception and noise, we should simulate human behaviors and their preferences within complex, real-world conditions. Therefore, video-based parts are more critical for potential future extension, as real-world agents must process raw sensory inputs to infer preferences dynamically. Visual cues can offer nuanced insights into user preferences that are not explicitly stated in the form of text, making them invaluable for applications where understanding subtle, non-verbal user behaviors.
Indeed, as you pointed out, the effectiveness of symbol-based LLMs in preference learning tasks, as demonstrated in our experiments, underscores the relative ease with which these models can handle few-shot induction when provided with explicit action and preference labels. However, it still presents great challenge in real-world grounding. As information from raw sensory often includes a lot of noise, converting everything into well-organized texts is nearly impossible. The poor performance of video-based approaches also illustrates practical difficulties.

The primary finding from the benchmark—improved personalized planning through learned preferences—is clearly demonstrated, but not particularly novel on its own.

Thank you for acknowledging our findings. While the idea of improved personalized planning through learned preferences is intuitive, our main contribution lies in creating the first realistic benchmark that extends this concept to embodied AI, modeling human preferences through observable behaviors across hierarchical levels, and leveraging machine learning methods to solve preference-guided planning in a scalable and systematic way. We demonstrate comprehensive research across a broad range of settings, and spans a broad range of settings, revealing key insights—for instance, that symbol-based approaches show promise in scalability, though significant challenges persist in both preference learning and planning, which has not been previously identified in the literature. We hope our work can serve as a foundation for future research.

The proposed benchmark uses Levenshtein distance to measure discrepancies between generated and ground truth action sequences. However, the rationale for choosing Levenshtein distance is unclear and should be better justified.
Could you elaborate on the strengths of Levenshtein distance as an evaluation metric?

Levenshtein distance is chosen as it quantifies sequential alignment between generated and ground-truth actions, penalizing deviations that violate preferences (e.g., mismatch in task ordering, like AAABCBCA (gt) vs AABCBCAA (predicted). Note how similar they are. In this case, accuracy is 32.5%(very low) but Levenshtein distance is 1/8 (smaller is better, which indicates the result is almost perfect) ). Levenshtein distance is much more suitable in our settings. Unlike rigid exact-match metrics, it accommodates valid variations in execution while ensuring preference adherence. We will add more explanantion in sec 5.2. Actually, Levenshtein distance has been widely used in sequence comparison [1-3].

[1] Yujian, Li, and Liu Bo. "A normalized Levenshtein distance metric." PAMI.
[2] Gu, Jiatao, Changhan Wang, and Junbo Zhao. "Levenshtein transformer." NeurIPS.
[3] Fanello, Sean Ryan, et al. "Keep it simple and sparse: Real-time action recognition." JMLR.

There appears to be a mismatch between the paper's framing and its actual contribution. The paper is framed as a new method for preference-based planning, but its main contribution seems to be the empirical benchmark and evaluation.

We appreciate this observation. While we agree the benchmark is a key contribution, we would argue that our main contribution lies in a series of studies on machine learning of human preferences. The benchmark serves as a testbed to rigorously evaluate preference-based planning, enabling us to demonstrate novel and comprehensive research across a broad range of settings. Through extensive experiments, we provide new insights into preference-based planning.

We sincerely welcome your further feedback and suggestions to strengthen our work.