Introspective Planning: Aligning Robots' Uncertainty with Inherent Task Ambiguity

We thank the reviewer for the positive feedback, acknowledging that our study includes sufficient comparison and ablation experiments. We also thank you for the comment that the write-up is easy to follow. We are looking forward to receiving additional comments and feedback from you during the discussion phase.

We thank the reviewer for the thoughtful feedback!

The term "introspection" is not scientifically defined. Does introspective refer to a robotic agent, the LLM, or the proposed approach?

We use the term “introspection” as defined in [19], referring to the human ability to assess internal values and knowledge of abilities to guide domain-level reasoning. In our paper, we extend the usage by analogy to non-human agents—and in particular to language-enabled AI agents—and denote by “introspective planning” a decision-making procedure by which these agents “assess their own confidence regarding task compliance and safety for multiple candidate plans”.

The term “introspective” refers to the planning procedure followed by the agent. In our paper, this planning procedure relies on a large language model, but it is not identical to it. In fact, our planning procedure is introspective regardless of what LLM it uses. The LLM itself (e.g., GPT-4) is not intrinsically introspective but is queried in an introspective manner.

What is the novelty compared to [31]? [31] also reasons about uncertainty and ambiguity through MCQA. Is [31] introspective?

In [31], the LLM does not in fact reason about its own uncertainty, that is, it does not form any intelligible logical discourse about it. Instead, MCQA is merely reading out the final-layer activations of the language model, which the system designers (and not the LLM itself) then interpret as log-probability estimates (but these do not readily encode well-calibrated probabilities, hence the need for the conformal prediction calibration phase). In contrast with [31], our approach prompts the LLM to explicitly discuss, in natural language, its own uncertainty regarding the task compliance and safety of candidate options, and the generated discussion is then used to inform the MCQA readout. Our experiments show that this procedure drastically improves the planner's performance across multiple metrics.

We note that while our contributions are briefly alluded to at the start of Section 2 (line 63), they are actually outlined in more detail in the Introduction (lines 36–61). We thank the reviewer for their question, and we will emphasize the central importance of our method, which prompts the LLM using retrieval augmentation to generate explicit reasoning in natural language prior to the MCQA query to refine the uncertainty of the resulting judgment.

Line 48 indicates introspective planning provides a tighter statistical guarantee. However, no theoretical or mathematical proofs are provided.

We clarify that our claim regarding our method’s ability to provide a tighter statistical guarantee is empirical, not theoretical (the latter would require us to make theoretical proofs involving the outputs of LLMs, which would be beyond the scope of our paper). In particular, we provide a comprehensive benchmark comparison between our proposed introspective–conformal method and various baselines, including the previous state-of-the-art in conformal planning [31]. Our results show significant improvement in balancing task success rate and the robot's decision conservativeness.

Table 1 shows our method's performance with an 85% target success rate, achieving an over-asking rate of about 6%, compared to 51% for the baseline [31]. This means that the baseline [31] is inappropriately asking the user to provide clarification in 1 out of every 2 requests that are in fact unambiguous, while our method does so in just 1 out of every 16.

Similar improvements are evident in Fig. 4g. For any target success rate between 70% and 99%, our method achieves a significantly lower over-ask rate; and, equivalently, for any acceptable over-ask rate, our method can provide a higher statistical bound on success rate.

The latter result substantiates our empirical claim: if it is allowable for the robot to ask for unnecessary clarification in only 1 in every 2 requests, then the strongest statistical guarantee achievable through prior conformal planning [31] is 85% success rate, whereas with our method it is 99%. We will emphasize this more strongly in the final version of the paper to clearly justify our empirical claim about the tightness of statistical guarantees.

How does the proposed approach enable the new capability of modeling and addressing task safety?

Our approach introduces several key advancements for modeling and addressing task safety:

• We generate safety-relevant examples for the knowledge base (lines 197-203, 526-540).
• Our approach relies on human-provided valid options to guide the LLM in reasoning about compliance and safety, trusting that human label providers can account for safety when relevant. In Figure 1, we show how these human-provided labels incorporate safety considerations, allowing LLM to generate safety-aware reasoning (e.g., deducing that plastic is unsafe for direct heating on the cooktop).
• We explicitly prompt the LLM to consider safety when forming its introspective judgments at runtime. The prompt for generating the safety-aware knowledge base is provided in lines 653-654 and Table 9, and the knowledge retrieval prompt for inference is provided in Table 12.

We present qualitative results in Figures 3 and 10 to further illustrate the differences between [31] and our approach in addressing safety.

What robotics simulations or physical robots are used to create the safe mobile manipulator dataset?

Both our approach and KnowNo focus on high-level planning through human-labeled mobile manipulation scenarios. The creation of the original Mobile Manipulation, TableTop Rearrangement, and our safe mobile manipulation datasets does not rely on simulations or physical robots. However, these datasets can be utilized for physical experiments. Our approach can be readily combined with the framework in [1] that employs physical robots to execute LLM-chosen plans, although doing so is beyond the scope of our paper.

We thank you for the feedback, acknowledging that our proposed method shows significant improvement compared to prior works in terms of addressing uncertainty alignment. We also thank you for the question and for providing an opportunity to clarify our contribution and its significance.

What’s the difference between introspective planning and Chain of Thought? Why is the contribution significant?

Our approach does indeed utilize Chain of Thought, but it is significantly more advanced. We observed that simply applying Chain of Thought does not effectively address hallucinations and can often lead to overconfident responses. For example, in Table 1, the overstepping rate for Prompt Set + CoT is 30.8%, while our approach achieves a much lower rate of 3.8%. Introspective planning which utilizes retrieval augmentation effectively prompts language-enabled agents to proactively refine their own confidence regarding task compliance and safety for multiple candidate plans, with a guaranteed probability that the agent will either execute the actions desired by the user or ask an appropriate follow-up question to disambiguate the user’s intent (lines 52-55).

One of our core contributions is the construction of the knowledge base. We introduce a new, weakly supervised offline knowledge base construction method that guides the LLM to generate human-aligned introspective reasoning examples as post-hoc rationalizations of human-selected safe and compliant plans. This contribution is discussed in lines 39-42, 56-58 of the paper. As a result, our approach guides the LLM to generate more precise plans, as evidenced by state-of-the-art performance on three datasets across different domains with various metrics.

Additionally, we have created a new Safe Mobile Manipulation benchmark, which enhances previous mobile manipulation datasets by including safety-critical scenarios and introduces new metrics (ESR, NCR, UCR, OAR, OSR, UR) to evaluate a planner’s performance in terms of compliance, safety, and degree of conservatives. Our approach effectively reasons about both compliance and safety, achieving state-of-the-art performance.

Is the gain marginal when there is better underlying LLM?

We conducted experiments using both GPT-3.5 and GPT-4, and the results are presented in Tables 1-7 and Figures 4-7. Our observations indicate that introspective planning significantly improves performance regardless of whether we use a stronger model (GPT-4) or a weaker model (GPT-3.5). Introspective planning leverages the reasoning capabilities of the language model to achieve superior performance. Therefore, as the underlying LLM becomes more powerful, the potential for introspective planning to enhance performance increases.

We thank the reviewer for the positive and thoughtful feedback!

Why are there significant performance gaps between the direct and conformal predictions and what is the trade-off?

We thank the reviewer for pointing this out. We discussed the trade-off between direct and conformal prediction in lines 219-225. While direct prediction provides more accurate predictions, it cannot guarantee success. Conformal prediction can guarantee success but performs worse than direct prediction. We acknowledged this limitation in lines 318-319. We hypothesized that the performance gap between direct and conformal prediction could be due to the misalignment between the first-token probabilities (conformal prediction) and text answers (direct prediction) [2, 3], but this still requires extensive additional analysis. We agree that understanding and analyzing this performance gap more deeply is crucial and future work should aim to reduce it.

The paper focuses primarily on manipulation tasks that involve somewhat ambiguous items in a kitchen setting. It would be helpful to include other domains to better show the generalizability of the method.

In addition to mobile manipulation and safe mobile manipulation tasks, we also conducted experiments on the Tabletop Rearrangement domain. This task involves moving colored blocks and bowls on a table according to specific instructions. These instructions are designed to include ambiguities in attributes (such as alternative names for objects and colors), numbers (using vague terms for quantities), and spatial relationships (using general terms for directions and orientations). The experimental results for this dataset are provided in Appendix A, and a detailed description of this dataset is in Appendix B.

What should the human provide to generate our knowledge base?

To construct the knowledge base, we query the LLM to generate a set of candidate plans, conditioned on the task, the observation, and hand-crafted few-shot examples. The user then needs to provide all the valid options $\mathcal{G}_i$​ (alphabetic labels such as A, B, C, D). We prompt the LLM to produce rationales k_i based on these labels. Specifically, we use in-context learning with few-shot examples to guide the LLM in generating explanations of why certain options are valid according to the ground truth. We provided a detailed explanation of constructing the knowledge base in lines 74-85 and showed an example in Figure 1.

Should users be aware of the ambiguity and safety?

Yes, users should be aware of ambiguity and safety when providing the labels. This awareness ensures that our approach guides the LLM to generate uncertainty- and safety-aware reasoning, resulting in predictions that are both safe and compliant with the user’s request.

There may be bias in the user-generated knowledge base, especially for multiple users. It would be good to show how these affect the performance of the system. Perhaps evaluating a system with multiple users without sharing the intended goals would show how well the system aligns across multiple users and a single knowledge base.

We agree that reducing bias in the knowledge base is important. With less bias, we can use a smaller knowledge base to achieve high performance. When collecting data for the knowledge base, we ensured that users were aware of ambiguity and safety and had varied intended goals, thus maintaining high knowledge quality. Consequently, during inference, we evaluated the performance with users having different intended goals, and the results show that our system can effectively retrieve relevant and helpful knowledge to guide the LLM in reasoning about compliance and safety.

Does the knowledge base contain the explanation? Are these stored after construction?

Yes, the knowledge base contains the explanations $k_i$, which are stored after construction. These explanations serve as important introspective reasoning examples that guide the LLM to generate more human-aligned reasoning. This procedure is described in lines 76-85 and Algorithm 1. We will emphasize this point in the final version of the paper.

Is there a reason that the other metrics were not included in the evaluation of performance vs knowledge bas size? Why does the performance drop with the largest knowledge base and what was the variation in tasks contained in the knowledge base?

We thank the reviewer for acknowledging our studies in Appendix C. The goal of this section is to show how the size of the knowledge base influences performance. Among all the evaluation metrics, Exact Set Rate (ESR), which evaluates the model’s ability to generate precise responses, and Success Rate (SR) are the most representative and clearest for supporting our conclusions. We added a plot with additional metric results (OAR, OSR, NCR) on Mobile Manipulation in the pdf.

We evaluated performance on Mobile Manipulation and Safe Mobile Manipulation. With the maximum knowledge base size, there is a slight performance decrease in Mobile Manipulation, while performance remains almost the same for Safe Mobile Manipulation. We hypothesize that this slight decrease could be due to shortcomings in the retrieval process. Although our approach retrieves examples relevant to the target instruction, it does not always guarantee retrieving the most helpful knowledge. A larger knowledge base includes more diverse scenarios but we might also retrieve less relevant knowledge. However, we observed that these cases are rare and do not significantly impact overall performance.

What is the result when the user provides an ambiguous answer when the system seeks clarification?

We do consider scenarios where the user could have multiple intents. For example, if the user asks for "bring me a soda" when there are both Coke and Sprite, selecting either option is considered a success. This differs from "bring me that soda," where the user has a specific intent regarding which soda.

We thank the reviewer for the positive feedback, acknowledging that our method is sound, the writing is clear, and the proposed metrics comprehensively measure the planner's performance. We also thank you for recognizing the valuable experimental results we have presented.

The proposed method relies on a knowledge base with human supervised labels, which may require significant human efforts to construct. It might be useful to discuss the cost of such knowledge base.

In our approach, humans only need to provide labels (instead of explanation or reasoning), and we leverage the LLM to generate the reasoning based on these labels, which are then stored as knowledge. This significantly reduces the human effort required. Labeling the knowledge base data with $400$ examples takes approximately $3$ hours.

The usage of both introspective planning and conformal prediction might increase the computation load for inference. It might be useful to discuss the increased computation.

Using introspective planning does indeed increase the computation load for inference. We discussed the cost and efficiency in Appendix D. For our experiments on Mobile Manipulation with a calibration set size of 400, a test set size of 400, and a knowledge base size of 400, KnowNo requires 35k completion tokens and 376k prompt tokens in total. Our approach requires three times the completion tokens and twice the prompt tokens. Consequently, the total API cost for KnowNo is $4.8, while ours is approximately$9.8. Despite the increased cost, our method significantly improves the Exact Set Rate, with KnowNo achieving only 37% compared to our method's 94.5%.

In Eq.2, why choosing (N+1)(1-ε)/N?

To ensure the prediction set meets the desired coverage level with the specified confidence $1 - \epsilon$, we use the quantile threshold defined as follows:

$\hat{q} = \text{Quantile}(s_1, ..., s_N ; \frac{\lceil (N+1)(1-\epsilon) \rceil}{N})$

This means $\hat{q}$ is the $\lceil (N+1)(1-\epsilon) \rceil$-th smallest value among the nonconformity scores $s_1, \ldots, s_N$. When we compute $\hat{q}$, we are effectively selecting this specific quantile to ensure the desired coverage. For a new task $x_{\text{test}}$, we include all labels $y$ in the prediction set $\hat{\mathcal{G}}_{\text{test}}$ such that:

$\hat{f}(y \mid x_{\text{test}}, \mathcal{C}_{\text{test}}, k) \geq 1 - \hat{q}$

This criterion ensures that the prediction set $\hat{\mathcal{G}}_{\text{test}}$ includes all labels $y$ where the confidence level $\hat{f}$ meets or exceeds $1 - \hat{q}$. With this construction, we achieve the following coverage guarantee:

$\mathbb{P}(z_{\text{test}} \in \hat{\mathcal{G}}_{\text{test}}) \geq 1 - \epsilon$

This guarantees that the true intent $z_{\text{test}}$ is included in the prediction set $\hat{\mathcal{G}}_{\text{test}}$ with a probability of at least $1 - \epsilon$.

Line 135-141 highlights the difference of the proposed work vs [31]. Just to make sure I understand it correctly, the key difference is incorporating k in Eq.3?

Yes, the key difference compared to [31] is the incorporation of the introspective planning rationale $k_i$ in our framework. This rationale is generated through the process described in lines 74-85, and it significantly improves planner performance in both compliance and safety. It also refines LLM uncertainty and provides a tighter bound of guarantee.

Dear Reviewers,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. As the discussion period approaches its conclusion, we kindly remind you of the upcoming deadline. Please feel free to raise any points or questions that may require further clarification.

Your insights and feedback are invaluable to us. Thank you once again for your thoughtful review.