ADAPT: Actively Discovering and Adapting to Preferences for any Task

We thank the reviewer for their thoughtful and positive feedback, and the appreciation for the novelty of our approach, our evaluations and writing clarity.

We thank the reviewer for the positive feedback on our work. We are pleased that you found the ADAPT task design to be unique and well-evaluated. Below, we address specific questions and outline additional experiments conducted based on the suggestions provided.

Related work:

We have expanded the related work section to better contrast our approach with existing methods, as suggested by the reviewer. Thank you for highlighting these works.

Persona creation:

We have included all persona descriptions in Appendix B.3. The 16 personas were manually created and selected to represent a diverse range of user types, ensuring broad coverage of potential user preferences. While more personas can be added, we believe that the current set is diverse enough to enable a systematic study of adaptation to preferences.

Teacher model and user simulation:

Both the teacher model and the LLM-based user simulation are LLMs with access to ground truth user preferences, but they are prompted to perform different tasks. The teacher acts as an assistive agent, using preferences to plan tasks, while the user simulation responds to agent queries. The planner and persona prompts in Appendix C.1 and C.3 illustrate this distinction. We will further clarify this in the paper.

Reflection Hyperparameters:

The $\epsilon$-hyperparameters in our reflection mechanism balance the questions demonstrated by reflection and the physical actions demonstrated by the teacher model. We tuned the hyperparameters on the train data (avoiding data leakage) to keep informative questions while not overshadowing the actions. We empirically tune our parameters ($\epsilon_1$=0.2 and $\epsilon_2$=0.5), resulting in 37% of the dataset being questions and 48% being physical actions. This analysis was done entirely on the train dataset, and hence ensured that there was no data leakage. In the final paper, we will add a result doing a hyperparameter sweep and cross-validation for thoroughness.

Alternative baseline without any hyperparameters:

We appreciate the idea of exploring variations of Reflection-DPO that do not use hyperparameters. We tried two variants of our reflection mechanism which would not require hyperparameters, and would instead directly compare sampled probabilities. First, we created a ‘Question Advantage’ baseline that predicts the question if it has a positive utility ($P_{\pi_s} (a_t|q, x) - P_{\pi_s} (a_t|x) > 0$). Second, we created a ‘Mean Probability’ version that treats the predictions as multiple future hypotheses (asking the question then predicting teacher action and directly predicting teacher action), and compares the average probability of each branch, choosing the question if probability of that branch ($\frac{P_{\pi_s}(a_t|q, x) + P_{\pi_s}(q|x)}{2}$) is larger than the other ($P_{\pi_s}(a_t|x)$). We evaluated both variations and empirically found their performance to be lower than our reflection mechanism.

We find that the ‘Question Advantage’ baseline asks far too many questions. It is unable to distinguish between questions that gain information versus questions that simply increase the probability of teacher action (by mentioning the relevant object, for example). This leads to over-emphasis on questions, with the majority of the dataset consisting of questions, resulting in a model that asked questions and didn’t take enough actions. The ‘Mean Probability’ version requires the tokenwise probabilities of an action generation and a question generation to be mutually comparable. This fails in practice due to the sensitivity of LLM probabilities to generation lengths and other extraneous syntactic variables. We posit that new breakthroughs in uncertainty calibration [1] can enable such hyperparameter-free methods to succeed, but we leave it to future work to explore efficient calibration mechanisms for this domain.

We appreciate the reviewer's suggestion and will include these results in an appendix section on alternatives to our reflection mechanism.

[1] Liu, Xiaoou, et al. "Uncertainty quantification and confidence calibration in large language models: A survey." arXiv preprint arXiv:2503.15850 (2025).

We thank the reviewer for their detailed review and the appreciation of the value of the ADAPT benchmark creation and open-source sharing. We are grateful for your positive remarks regarding our benchmark's contribution and the novel methodology of Reflection-DPO. Below we address some of the concerns raised:

Increasing Complexity and Adding Tasks:

Our primary goal in designing ADAPT tasks was to ensure sufficient task complexity with many possible preferences, while keeping tasks long-horizon and open-ended. Expanding to different domains and adding new tasks currently requires resources, including manual effort for creating reward functions and computational resources for scaling experiments. Although modern LLMs could assist, manual verification and correction are often necessary to ensure correctness. We plan to address this expansion in future work and will note this limitation in Section 7.

Performance Results with SoTA LLMs:

Our evaluations focused on how different approaches using the same LLM perform on ADAPT, including prior research on active task-oriented questioning (STaR-GATE) and strong planning baselines (ReAct). We chose Llama 3.1 as the base LLM due to its satisfactory performance.

We agree that evaluating ADAPT across different LLM models would provide valuable insights. We are currently running experiments with the Qwen-2.5-72B model on ReAct, our strongest baseline, and will update this rebuttal with preliminary results. Final results will be included in the paper.

Measuring Satisfaction Rate per Question:

Measuring satisfaction rate per question is challenging due to the lack of a "ground truth" for whether a question directly relates to a preference or satisfies multiple preferences. For instance, if the agent asks, "Do you like low-fat or whole milk?" and the user responds, "I am trying to be healthy, so low fat," the LLM might use this to offer healthier options like whole-wheat cereal in subsequent interactions. In this case, one question and response are responsible for multiple preference satisfaction.

To approximate question efficiency, we consider the ratio of preferences satisfied to user input, i.e. the number of questions and initial goal specification $\frac{p_+}{n_q + 1}$ . However, this metric can favor models that satisfy preferences by chance without asking questions, or asking very few questions. With this caveat, here are the "efficiency of questions" per baseline:

As expected, baselines that ask few questions like ReAct are more efficient at this metric, while Reflection-DPO and Always Ask are less efficient. Note that we currently don’t explicitly penalize for asking extra questions, and merely aim to satisfy preferences (noted as a limitation in Section 7). In future work, we aim to increase the efficiency of our approach, aiming for models that can ask fewer questions while satisfying preferences. We will add this metric and discussion to Appendix.

Use of LLM Simulated User:

We use an LLM, prompted with a set of preferences, to simulate a user to enable closed-loop experimentation and reproducibility on the benchmark. The simulated user enables us to test different baselines with open-ended questions, and judge their performance in a systematic manner.

Our preliminary evaluations on using Qwen2.5-72B as a backbone for ReACT show similar results as Llama3.1, with marginally better performance by asking a slightly larger number of questions. The results are as follows:

We thank the reviewers and the area chair for their time and thoughtful feedback. We hope that our responses and additional results have addressed the concerns raised, and we respectfully invite the reviewers to reconsider their evaluations in light of these clarifications.