Enhancing LLM Planning for Robotics Manipulation through Hierarchical Procedural Knowledge Graphs

• W1: Focuses on table-top tasks; transfer to industrial or other domains untested.

It is possible that our pipeline could be adapted to other domains such as industrial assembly. Our core methodology may be applicable provided that suitable large-scale datasets are available for these domains. However, due to time constraints during the rebuttal period, we do not have sufficient time to construct new large-scale knowledge graphs, as this process requires substantial data collection and curation efforts. As mentioned in our limitations, the study is limited to household activities. Different domains would likely require domain-specific adaptations in entity types, relationship categories, etc.

• W2: No quantitative quality metrics (precision/consistency) for the final KG are reported.

We conduct experiments to verify the quality of our generated data. Specifically, we sample a subset(100task and 864 steps) of our HP-KG. We use three data quality metrics, including truly atomic actions precision (AAP), step ordering precision (SOP), and actions ordering precision (AOP).

To measure the atomic actions precision, we extracted all verbs from the HP-KG and manually created an atomic verb dictionary containing only atomic verbs (e.g., pick, turn on) while excluding non-atomic ones (e.g., arrange). We then calculated the percentage of atomic verbs in the generated actions based on this dictionary. For Steps Precision and Action Precision, we annotated the correct steps/actions ording and calculate the average percentage of correctly ordered steps and actions, respectively. Detailed results are presented in following Table. Experiments show that our LLM Agents can generate reasonable atomic actions and acceptable steps, helping to bridge high-level human instructions with atomic actions that robots can understand.

To better illustrate our HP-KG approach, we include a graph example in Figure 5 and detailed graph statistics in the Appendix of the Supplementary Material. We hope these supplementary materials may offer helpful context for understanding our HP-KG.

• W3: author mentioned "safety-critica" in Line 125, but no further information provided.

The term 'safety-critical' refers to the 'safety-critical guidelines' mentioned in Line 125. These are essential safety warnings and precautionary instructions that our procedure should incorporate to alert robot to potential accidents during procedure execution. Following this principle, we incorporated a Tips attribute when defining the properties of the Procedure. An example of a Tip is: "Handle the notebook carefully to avoid bending or tearing pages.

• W4: embedding + K-hop search latency on large graphs is not profiled.

We conduct experiments to evaluate the efficiency of our retrieval method. Specifically, we measure the average Instruction Encode Time (Encode Time),Retrieval Time, K-hop Time, Reranking Time, Graph Conversion Time and LLM Inference Time during the Top-3 and Top-5 search processes on the Actplan-1K[1].

The results are presented in the following table.

Results show that our retrieval process is highly efficient, with only minimal additional time overhead that is significantly less than MLLM inference time.

• Q1: Could you provide automatic or human-annotated precision/coverage statistics (e.g., % of actions truly atomic, % of step ordering correct) for HP-KG?.

Thank you for your valuable comment. We have conducted experiments to evaluate the precision and coverage statistics of HP-KG. Detailed results can be found in W2

• Q2: Do you foresee the same structure and pipeline scaling to industrial assembly or navigation tasks? Any preliminary results or challenges?

It is possible that our pipeline could be adapted to other domains such as industrial assembly or navigation tasks. The core methodology may be applicable provided that suitable large-scale datasets are available for these domains. However, due to time constraints during the rebuttal period, we do not have sufficient time to construct new large-scale knowledge graphs, as this process requires substantial data collection and curation efforts.

As mentioned in our limitations, the study is limited to household activities. Different domains would likely require domain-specific adaptations in entity types, relationship categories, etc.

• Q3: Also, I would encourage the trials on more challenging manipulation benchmarks like RoboCerebra.

Thank you for your insight. We further conducted experiments on a more complex benchmark. Since the RoboCerebra benchmark is not open source, we conducted our experiments on VLABench[2].

VLABench, which contains multiple complex tasks that involve common sense knowledge, physical rules, and reasoning capabilities. VLABench can evaluate not only VLAs but also MLLMs. In the MLLM evaluation setting, MLLMs are required to output the skills to be invoked and the corresponding parameters for each skill.

We select four task categories (CommonSense, Complex, M&T, PhysicsLaw), with each category containing 5-8 tasks. CommonSense, M&T, and PhysicsLaw categories evaluate the model's understanding of world procedural knowledge, while Complex categoriy, beyond world knowledge, demand long-horizon planning. We compare the accuracy of MLLMs enhanced with our HP-KG against those without such enhancement, with each task run for 50 trials. Results are presentsed on following Table.

Results demonstrate that our HP-KG can enhance MLLMs' planning capabilities by injecting procedural commonsense into the LLM context, thereby providing structured prior knowledge that facilitates robust planning in complex tasks.

• Q4: Please report average retrieval + sub-graph conversion time and memory footprint for top-3/top-5 on ActPlan-1K, and discuss impact on total inference.

Thank you for your valuable comment. We have conducted experiments to evaluate the time performance of the Top-k retrieval process. Detailed results can be found in W4. The results demonstrate that our retrieval process is highly efficient, incurring only minimal additional time overhead that is significantly less than MLLM inference time

• Q5: To mitigate potential bias from GPT-4o judging, could you include a small human study validating that HP-KG plans are indeed more feasible/executable?.

We conduct experiments to verify the feasibility of our plans of HP-KG. Specifically, we sample a subset (100 tasks) from our HP-KG and manually annotate whether each action is feasible or executable for the robot. We then calculate the average percentage of feasible actions, denoted as the feasible action rate (FAR).

Detailed results are presented in following Table. Experiments show that our LLM Agents can generate feasible atomic actions, helping to bridge high-level human instructions with atomic actions that robots can understand.

• Q6: The KG Retrieval reminds me the the in-context prompts used in CaP or Instruct2Act, can the authors provide more discussions

Instruct2Act presents an agent-based robotic manipulation framework that generates Python code based on task instructions, leverages this code to invoke vision tools (such as SAM) for environmental perception, and subsequently generates plausible robotic actions for low-level controllers to execute.

Our approach is orthogonal to these agent-based robotic manipulation frameworks and provides complementary capabilities. The constructed HP-KG can be seamlessly integrated into various planning generation systems, offering procedural commonsense knowledge to facilitate more rational action planning.

• Q7: Overall, due to the limited task complexity in the chosen RLBench, it did not fully demonstrate the effectiveness of the promised KG.

We conduct experiments on more complex tasks using VLABench. Detailed experimental results can be found in Q3.

[1] ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities. EMNLP 2024.

[2] VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks. ICCV 2025.

• W1: The anonymous link didn’t provide any code or data for us to review.

Thank you for your valuable feedback. Due to NeurIPS's policy this year, we cannot update our repository or submit any links during the rebuttal period. We commit to updating our repository and releasing the data immediately after the rebuttal period ends to further facilitate Procedural Knowledge development and research.

To better illustrate our HP-KG approach, we include a graph example in Figure 5 and detailed graph statistics in the Appendix of the Supplementary Material. We hope these supplementary materials may offer helpful context for understanding our HP-KG.

• W2: The lack of experiments or demos in real-world robots, which could decrease the contribution of the world.

Thank you for your sincere comment. Due to time and cost constraints, we have not yet deployed our approach to real-world robots. We will further extend our HP-KG to real-world applications in future work.

• W1: LLM agent-generated data lacks quality verification and examples.

Thank for your valuable comment. we conduct experiments to verify the quality of our generated data. Specifically, we sample a subset(100task and 864 steps) of our HP-KG. We use three data quality metrics, including truly atomic actions precision (AAP), step ordering precision (SOP), and actions ordering precision (AOP).

To measure the atomic actions precision, we extracted all verbs from the HP-KG and manually created an atomic verb dictionary containing only atomic verbs (e.g., pick, turn on) while excluding non-atomic ones (e.g., arrange). We then calculated the percentage of atomic verbs in the generated actions based on this dictionary. For Steps Precision and Action Precision, we annotated the correct steps/actions ording and calculate the average percentage of correctly ordered steps and actions, respectively. Detailed results are presented in following Table. Experiments show that our LLM Agents can generate reasonable atomic actions and acceptable steps, helping to bridge high-level human instructions with atomic actions that robots can understand.

To better illustrate our HP-KG approach, we include a graph example in Figure 5 and detailed graph statistics in the Appendix of the Supplementary Material. We hope these supplementary materials may offer helpful context for understanding our HP-KG.

• W2: it is now unclear how robust the generated plans are to real-world settings(uncertainties, or unexpected environmental changes).

Thank you for your comments. We conduct experiments to simulate real-world environmental variations and uncertainties, demonstrating the robustness of our HP-KG.

Specifically, we utilize the Counterfactual Planning split from Actplan-1K[1], which assesses the model's ability to adapt reasoning and planning when task scenarios involve modified conditions or constraints that frequently arise in real-world applications. We present results in the following Table, which demonstrate that our approach successfully leverages procedural commonsense to enable robust LLM planning capabilities even in uncertainty scenarios.

Furthermore, we will simulate more real-world settings (noise, unexpected environments) in future experiments to demonstrate the robustness of our approach.

• Q1: Is it possible to integrate graphical representations and directly input them to the LLM.

Yes, it is possible to directly integrate graphical representations into LLM. Howerver integrating graphical representations and directly inputting them to LLMs requires training the models to adapt to graphical inputs, which incurs high computational costs and may lead to catastrophic forgetting[3] of the LLMs' existing knowledge. Instead, we convert retrieved structured procedural information into textual context for LLM input. This method enables seamless integration without architectural changes, leverages existing reasoning capabilities of LLMs for procedural knowledge application, and maintains interpretability across diverse procedural formats[2]. Due to time constraints, we are unable to fine-tune an LLM for direct graph integration and will compare such methods in future revisions.

• Q2: How does the system handle tasks that require knowledge outside the graph, or tasks that necessitate dynamic replanning.

In our experiments, the HP-KG is constructed from how-to data that is distinct from the benchmark datasets (Actplan-1K[1], RLBench[4]) used for evaluation, which demonstrates the generalization capability of our approach. The HP-KG contains a rich collection of reusable atomic actions that serve as fundamental building blocks. Within the household domain, these action units can be flexibly combined and reconfigured to create diverse procedural sequences, enabling effective generalization across new environments, unseen tasks, and varying contextual conditions. However, it may fail in scenarios that are completely outside the graph.

• Q3: Can the HP-KG framework be extended to other modalities.

Yes, our HP-KG has the potential to be extended to other modalities. MLLMs have demonstrated strong visual understanding capabilities [6], and research has shown that providing additional visual cues can help MLLMs better comprehend the world and enhance their reasoning abilities [7]. We envision incorporating visual imagery and manipulation videos into our HP-KG, which would allow for instruction-based retrieval of relevant task operation diagrams or videos. Providing such visual inputs to MLLMs might improve their understanding of sequential operations and procedural tasks. We plan to explore the integration of visual modalities into our HP-KG in future research.

• L1: The study is limited to household activities

Thank you for your insightful feedback. We have also acknowledged and discussed this limitation in the Limitation Section.

• L2: HP-KG construction might become a bottleneck when generating knowledge graphs covering an extremely wide range of complex manipulation tasks

Thank you for your insight. Our construction framework may not be able to cover extremely complex manipulation tasks.

• L3: Reliance on wikiHow data might introduce biases or limitations due to the specificity of information. Its generalizability to tasks not well-covered by such web data might be limited.

Thank you for your valuable comment. Web data may not be able to ensure good generalizability across diverse tasks. We plan to improve the generalization of our knowledge graph in future work.

[1] ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities. EMNLP 2024.

[2] G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering. Neurips 2024.

[3] Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal. ACL 2024.

[4] RLBench: The Robot Learning Benchmark & Learning Environment. IEEE Robotics Autom. Lett.

[5] Exploring Large Language Models for Knowledge Graph Completion. ICASSP 2025.

[6] Qwen2.5 Technical Report. Arxiv 2412.15115.

[7] Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers. Arxiv 2506.23918.

We sincerely appreciate your acknowledgment. Your thoughtful and detailed feedback has been instrumental in refining our work, and we are deeply grateful for the time and effort you invested in reviewing our paper. Thank you once again for your valuable contribution.

We appreciate your valuable insights and comprehensive review. Your thoughtful comments will greatly enhance our manuscript.

• W1: Validation on complex tasks and performance remain unclear.

we conduct comprehensive experiments on VLABench[1], a more complex benchmark comprising eight distinct manipulation skills (e.g., Insert, Hang, Twist) that extend beyond basic pick-and-place operations. These tasks require comprehensive commonsense knowledge, understanding of physical principles, and sophisticated reasoning capabilities to solve effectively. VLABench can evaluate not only VLAs but also MLLMs. In the MLLM evaluation setting, MLLMs are required to output the skills to be invoked and the corresponding parameters for each skill.

We select four task categories (CommonSense, Complex, M&T, PhysicsLaw), with each category containing 4-8 tasks. CommonSense, M&T, and PhysicsLaw categories evaluate the model's understanding of world procedural knowledge, while Complex categoriy, beyond world knowledge, demand long-horizon planning. We compare the accuracy of MLLMs with and without our HP-KG enhancement, and we run each task for 50 trials. Results are presented in the following table.

Results demonstrate that our HP-KG can enhance MLLMs' planning capabilities by injecting procedural commonsense into the LLM context, thereby providing structured prior knowledge that facilitates robust planning in complex tasks.

Additionally, our HP-KG framework is primarily designed for providing procedural commonsense to LLM context and facilitating more reasonable planning, rather than addressing dexterous manipulation benchmarks such as [3,4]. These benchmarks primarily evaluate models' ability to perform precise finger movements, coordinated multi-finger grasping, and object manipulation. Therefore, our method is not suitable for dexterous manipulations task.

• W2: It should be clearer described in which cases it decreases performance compared to model without it or compared to CoT and why.

1. On the Slide Blocks task, our method with top-5 retrieval yields lower performance compared to the approach without HP-KG. The task's ambiguous instructions result in more irrelevant information being retrieved, thereby degrading planning quality.

2. Our method shows comparable performance to COT on Take_lid and Take_scale tasks. This may be due to the inherent limitations of the RVT model on these two tasks, where neither our method nor COT achieves satisfactory performance.

• W3: the evaluation is comparing different methods in different cases, making it difficult to compare.

Table 1 and Table 2 compare different types of methods. In Table 1, we present a comparison of zero-shot robotic manipulation methods on RLBench. In Table 2, we validate whether our method can enhance pretrained action models. Given that zero-shot methods and pretrained action models appear to be better suited for different types of tasks, we selected slightly different test cases for each comparison. Since our main comparison is between methods with and without our HP-KG enhancement, our experimental setup is fair.

• Q1: p.5 l193 textual results descriptions,feedback - what these mean, where they are collected and how are these utilized?

what these mean: 'textual results descriptions' refers to the consolidated textual description that combines all rules with their respective verification outcomes. 'feedback' represents the LLM's assessment of the given procedure $p$, identifying potential issues.

where they are collected: 'textual results descriptions' are collected by converting each rule verification result along with its original rule text into a structured textual format (e.g., "rule:{rule_check_result}"), and then concatenating all these formatted descriptions to form the final comprehensive textual results. 'feedback' is obtained by feeding these textual results descriptions along with procedure $p$ to an LLM, which then generates the final score and corresponding feedback.

how are these utilized: The 'textual results descriptions' are fed into an LLM along with procedure $p$ to generate the final score and corresponding feedback. 'feedback' is subsequently fed to the Refiner, which modifies procedure $p$ based on the feedback.

• Q2: how do you make sure not to loose some important rule or information by summary agent?

In Table 6, the results show that has been processed by the Summary Agent further enhances the planning capacity of the Base Model, suggesting that our summary agent largely retains important information. Additionally, we selectively summarize procedures with high similarity, which is a common strategy for eliminating redundancy and addressing distributional imbalance, thus reducing the knowledge imbalance[2].

• Q3: why in Table 1/2 top 5 sometimes performs worse and sometimes better than top 3? why CoT drops performance in Table 2 for slide block compared to RVT? why top 3/5 sometimes performs even worse than RVT? are there some typical problems?

1. In our experiments, we also observe this phenomenon. Under the models shown in Tables 1 and 2, retrieving more information (Top-5) may introduce irrelevant information during the model's planning process. This suggests that current models may have limited capacity to effectively filter and utilize extensive retrieved information, thereby degrading planning quality and leading to performance decline with Top-5 retrieval. Our experiment in Table 4 demonstrates that more advanced models can effectively handle Top-5 retrieval without performance degradation.

2. Due to the ambiguous instructions(e.g., 'slide the block to target') in Slide Blocks task, COT may generate suboptimal planning, leading to performance degradation.

3. In Slide blocks task, our method with Top-3 retrieval achieves superior performance compared to the RVT. However, Top-5 retrieval yields lower performance, likely due to irrelevant information introduced by retrieving more content, thereby degrading planning quality.

4. As shown in Table 2, our method with Top-3 retrieval outperforms the baseline RVT model in all cases. Our method with Top-5 retrieval shows decreased performance in only one case, where ambiguous instructions cause the retrieval process to introduce irrelevant information, thereby degrading planning quality.

• Q4: what about more complex tasks? could you show results on these or discuss limitations of your approach?.

We conduct experiments on more complex tasks using VLABench. Detailed experimental results can be found in W1.

• L1: Limitations are very sparse, mentioning only extension outside household tasks.

Thank you for your insightful feedback. Our work may also face limitations in constructing reasonable task planning when dealing with an extremely complex manipulation tasks, due to the limited planning and reasoning capabilities of MLLMs. Furthermore, our HP-KG, which is derived from how-to data, may not generalize well to all complex manipulation scenarios.

[1] VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks. ICCV 2025.

[2] Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI. IEEE Trans. Knowl. Data Eng. 36(11)

[3] DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects. CVPR 2023.

[4] DexH2R: A Benchmark for Dynamic Dexterous Grasping in Human-to-Robot Handover. ICCV 2025.