Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modeling

We deeply appreciate the reviewer for providing valuable feedback. Below are our responses to the raised concerns.

[W1] Training budget considering world model.

• The training cost for the world model is about 30 hours, which is a non-negligible overhead. We have revised Section 4.3 and Appendix B.3 to reflect that. For a more fair comparison, we extend Table 1 to include baselines that also use a world model as feedback. The results are summarized below, where our approach remains advantageous and shows the most significant improvement with world model feedback, confirming its effectiveness in closed-loop robotic planning.

  Method	World	model		Both	novel		Novel	scene		Novel	task
  				SR	GCR		SR	GCR		SR	GCR
  Tree-Planner N=25		x		38.71	63.18		51.58	69.45		40.38	63.75
  Tree-Planner N=25		$\checkmark$		38.71	58.71		51.58	63.94		38.46	57.40
  Tree-Planner N=50		x		38.71	63.50		51.58	69.54		39.74	63.29
  Tree-Planner N=50		$\checkmark$		37.10	57.53		54.74	69.64		39.32	58.84
  Ours		x		33.87	59.98		49.47	66.60		34.62	59.06
  Ours		$\checkmark$		40.32	65.40		65.26	79.47		41.88	62.76

[Q1] Subsequent effects of feedback.

• Following your suggestion, we present iteration-wise results in Figures 11 and 13 in the appendix. As shown in Figure 11, our method improves stably over outer-loop iterations and succeeds with only two feedbacks. In comparison, Tree-Planner can ignore previous feedback and make the same mistake twice ([OPEN] <laptop>) because its tree-backtracking mechanism does not reuse the implicit knowledge in earlier feedback.
• Figure 13 illustrates the continuing effect of feedback on our inner-loop iterations, where each iteration refines the plan while adhering to previous feedback. For example, the second inner loop inserts a critical [WIPE] action while maintaining all objects' relative positions according to the last feedback, leading to later success.

[Q2] Initialization of world model.

• The world model is initialized from Llama 3 8B Instruct and supervised finetuned on all the planner's environmental interactions. This procedure takes place after the equilibrium planner has completed training, so that all of its data can be leveraged at once. The finetuning data is constructed in a format similar to Figure 9 in the appendix, with a different order to predict feedback from the plan. See Appendix B.3 for more details.

[Q3] Switch between real and generated experience.

• Our planner alternates between real experience and generated experience at each outer-loop iteration. This process is intuitively illustrated in Figure 12c in the appendix. We note that it is possible to use generated experience more frequently and reduce the number of environmental interactions if the planner has access to a more accurate world model.

[Q4] Minor typo.

• Thank you for pointing this out, we have corrected the typo and will continue to revise the manuscript.

We deeply appreciate the time and effort you dedicated to reviewing our paper and providing constructive feedback. Below we address the raised concerns one by one.

[W1] Background on equilibrium models and fixed-point iteration.

• Following your suggestion, we have substantially expanded the background introduction with Appendix A, detailing the previous literature on deep equilibrium models and fixed-point iteration. We will continue our efforts to make the draft self-sufficient for readers who are unfamiliar with this line of research.

[W2] Self-refinement with fixed context.

• Even with fixed context, self-refinement is still beneficial in two ways: (1) Vanilla LLM planners are limited by unidirectional dependency, i.e. the previous tokens cannot attend to future tokens, resulting in limited ability to plan ahead. Self-refinement addresses this by allowing each output token to attend to the entire plan (from an older version). (2) Vanilla LLM planners use a fixed forward process with less flexible computation, while self-refinement enables dynamic allocation of inference compute through an iterative process.

• To validate the effectiveness of our method for self-correction with fixed context, it is compared to a self-refinement baseline via prompting. As shown in the table below, our method clearly improves the success rate even without any feedback. We also illustrate the inference traces in Figure 14 in the appendix, where our method is shown to be more effective at steering the output toward a correct plan.

  Method	Both	novel		Novel	scene		Novel	task
  	SR	GCR		SR	GCR		SR	GCR
  Supervised	24.19	32.55		41.05	49.81		26.07	35.53
  SELF-REFINE	32.26	52.29		44.21	61.25		30.98	51.80
  Ours	33.87	59.98		49.47	66.60		34.62	59.06

[W3] Distinction between training and equilibrium solving.

• Our approach consists of two alternating phases: (1) solving for the equilibrium solutions, and (2) training with these equilibrium solutions. The two phases differ in that the first phase involves only forward passes defined by $x_{t+1}=f_\theta(x_t,c)$, while the second phase involves backpropagation of the loss $L(f_\theta(x^*,c),y)$. On the other hand, they are closely connected because the second phase relies on the equilibrium $x^*$ produced by the first phase.

[W4] Connection with LLMs.

• The LLMs correspond to $f_\theta$ in our equilibrium algorithm. Take an example, the equilibrium solving process actually performs an iterative inference of LLM based on the previous output tokens $x_t$ until the new output tokens $x_{t+1}=f_\theta(x_t,c)$ have converged. Following your suggestion, we have revised the main paper to explicitly declare $f_\theta$ as an LLM in Section 3.1.

[W5] Self-refinement within inner loops.

• We illustrate the effect of inner loops in self-refinement with Figure 13 in the appendix. Even without additional feedback, it shows consistent quality improvements during each inner loop. In particular, the first inner loop removes a redundant [GRAB] action, while the second loop adds a [WIPE] action that is crucial for later success, as well as interesting details such as [WAIT] and [RINSE] <sink>. Thus, inner loops are considered useful in our nested refinement framework.
• The working mechanism of inner loops is the same as our previous answer to W2, i.e., introducing bidirectional dependency and using more computation. We also note its similarity to human introspection, both involving little external feedback but a lot of random attempts to improve in new directions (as shown in Figure 13).

[W6] Inputs and assumptions.

• The inputs to our LLM planner are described in Appendix B.3 and Figure 9. It consists of five parts, including a system prompt that asks the LLM to self-refine plans, a task description, an environment description, the latest draft plan, and all previous feedback. Please refer to Appendix B.3 and Figure 9 for more details.
• The assumptions in our proposed method are twofold: (1) LLMs could converge to equilibrium by fixed-point iteration. This has been verified in Figure 6b, which shows that only a small number of iterations (between 2 and 5) is required for convergence. (2) For the implicit function theorem (Theorem 1) to hold, $(I-\frac{\partial f_\theta}{\partial x^*})$ should be invertible, i.e., full rank. This is commonly assumed in deep equilibrium models, including the work [1, 2] similarly based on Transformers. We have noted this assumption in Theorem 1.

Overall, we have made several updates to the draft based on the reviewers' constructive feedback on self-sufficiency, and will continue to revise the draft to reflect all suggestions.

References:

[1] Bai, et al. Deep Equilibrium Models. NeurIPS 2019.

[2] Geng, et al. One-Step Diffusion Distillation via Deep Equilibrium Models. NeurIPS 2023.

We sincerely appreciate the time and effort you dedicated to reviewing our paper and providing valuable feedback. Below are our responses to the raised concerns.

[W1] Justification for approximating $A=(I-\frac{\partial f_\theta}{\partial x^*})^{-1}\approx I$.

• This approximation does not rely on Theorem 3.1 of Fung et al. (2022). Dropping the inverse Jacobian is a common practice in one-step gradient (see Appendix A.2), and has been shown to be effective on Transformer-based LLMs [1]. We thank the reviewer for pointing out the incorrect reference; indeed, it is difficult to get theoretical guarantees for large Transformers. We have revised the draft to remove that reference and to include more relevant evidence [1].

• This approximation does not affect our core contribution in teaching self-refinement via supervised training. In fact, the original Equation (4) before this approximation can already transform self-refinement into a supervised learning problem $\min_{\theta}L(A^\top f_\theta(x^*,c)+(I-A^\top)x^*,y)$ of its equilibrium, bypassing any process supervision or reinforcement learning. This work adopts approximation $A\approx I$ in the trade-off between accuracy and efficiency, opting for the latter. We expect further improvements from more accurate approximation techniques given sufficient computational budgets.

[W2] Limited technical contribution.

• The technical contributions of this paper are twofold: (1) for LLM planning research, it proposes a new supervised training method for self-fining LLM planners; (2) for deep equilibrium model research, it proposes a series of novel designs (nested equilibrium solving, reuse of feedback, etc.) to improve the training and inference efficiency of deep equilibrium models.

[W3/Q2] Ablation of training objective

• To validate the effectiveness of our new training objective, we compare our finetuned LLM with two baselines (the original Llama 3 8B Instruct using our prompts, and self-refinement based on a vanilla supervised fintuned Llama 3 8B) with the setup of Table 2. The results below show that the effectiveness of our method is largely due to the new training objective, while the original or supervised finetuned Llama does not work well.

  Method	Both	novel		Novel	scene		Novel	task
  	SR	GCR		SR	GCR		SR	GCR
  Original Llama + our prompt	0.00	0.00		1.05	1.05		1.07	1.07
  SELF-REFINE	43.55	65.18		54.74	70.24		39.96	62.37
  Ours	56.45	76.63		58.95	87.07		54.91	74.18

[W4/Q1] Fixed-point convergence.

• The following table shows the number of iterations for an LLM to reach its fixed point, aggregated over 10 initial plan for each of 60 random tasks. All LLMs considered, including the original Llama, the supervised finetuned Llama, and our model, can converge to a fixed point within a few iterations, regardless of the initial sequence. This is partly due to our greedy sampling strategy (described in Appendix B.3) that reduces the LLMs' randomness, after which they tend to repeat themselves (Figures 4a, 10a, and 11a) and converge easily.

  Method	Avg	stat
  	Mean	Std	Min	Max
  Original Llama	6.70	2.12	3.22	14.98
  SFT Llama	2.46	0.88	2.02	3.80
  Ours	3.02	1.61	2.32	4.07

[Q3] Dataset partitioning.

• We have surveyed prior papers using VirtualHome in Appendix B.2. There are two recent dataset protocols represented by LLM-MCTS [2] and Tree-Planner [3]. Specifically, LLM-MCTS uses the indoor scenes of VirtualHome and synthesizes its own dataset focusing on object rearrangement tasks. Tree-Planner uses a subset of VirtualHome-Env to evaluate for training-free methods. Since both works have not released their detailed dataset, we simply use the original VirtualHome-Env dataset and partition it accordingly in our experiments.

  Method	Public	#Tasks	#Scenes	Task content	Task splits
  Tree-Planner	x	35	4	household	Novel scene and task
  LLM-MCTS	x	2000	4	rearrangement	Novel scene and task, Novel scene, Novel task, Seen scene and task
  Ours	$\checkmark$	1360	7	household	Novel scene and task, Novel scene, Novel task

References:

[1] Choe, et al. Making Scalable Meta Learning Practical. NeurIPS 2023.

[2] Zhao, et al. Large Language Models as Commonsense Knowledge for Large-Scale Task Planning. NeurIPS 2023.

[3] Hu, et al. Tree-Planner: Efficient Close-loop Task Planning with Large Language Models. ICLR 2024.

Thank you for recognizing our work and providing valuable feedback. Below are our responses to the raised concerns.

[W1.1/Q1] Self-refinement without external feedback or world model.

• In the setting without feedback, our performance is inferior to Tree-Planner due to the different number of plans generated. Tree-Planner always generates 25 or 50 candidate plans at a time before constructing the action tree, while our method refines over a single plan with a few iterations (<10). This gives Tree-Planner a comparative advantage. However, in the other settings that allow feedback (e.g. Tables 2), our method outperforms Tree-Planner by incorporating feedback more flexibly.

• To validate the effectiveness of our method for self-correction without feedback, it is compared to a self-refinement baseline via prompting. As shown in the table below, our method clearly improves the success rate even without any feedback. We also illustrate the inference traces in Figure 14 in the appendix, where our method is shown to be more effective at steering the output toward a correct plan.

  Method	Both	novel		Novel	scene		Novel	task
  	SR	GCR		SR	GCR		SR	GCR
  Supervised	24.19	32.55		41.05	49.81		26.07	35.53
  SELF-REFINE	32.26	52.29		44.21	61.25		30.98	51.80
  Ours	33.87	59.98		49.47	66.60		34.62	59.06

[W1.2] Usage of world model.

• We need the world model as an efficient alternative to the environment. Since environmental interactions can be expensive and sometimes unrecoverable, the world model helps the planner self-correct with fewer environmental interactions (Figure 12), thereby improving the performance under a low interaction budget. This is also confirmed in Table 3 of the main paper, where the world model is shown to be beneficial in both of two experimental settings.

• For a more fair comparison, we extend Table 1 to include baselines that also use a world model as feedback. The results are summarized below, where our approach remains advantageous and shows the most significant improvement with world model feedback, confirming its effectiveness in closed-loop robotic planning.

  Method	World	model		Both	novel		Novel	scene		Novel	task
  				SR	GCR		SR	GCR		SR	GCR
  Tree-Planner N=25		x		38.71	63.18		51.58	69.45		40.38	63.75
  Tree-Planner N=25		$\checkmark$		38.71	58.71		51.58	63.94		38.46	57.40
  Tree-Planner N=50		x		38.71	63.50		51.58	69.54		39.74	63.29
  Tree-Planner N=50		$\checkmark$		37.10	57.53		54.74	69.64		39.32	58.84
  Ours		x		33.87	59.98		49.47	66.60		34.62	59.06
  Ours		$\checkmark$		40.32	65.40		65.26	79.47		41.88	62.76

[W2] Additional benchmark.

• We have briefly surveyed other planning benchmarks (ALFRED, PlanBench, etc.) in the Environment section of Appendix B.1. As shown, VirtualHome remains a more suitable benchmark for closed-loop long-horizon planning, since the other benchmarks focus on either closed-loop or long-horizon aspects. Therefore, this paper only validates the proposed planner on VirtualHome. Nevertheless, we believe that closed-loop long-horizon planning may be implicitly involved in broader applications such as web agents and video generation, and we will continue to explore this direction in future work.

[Q2] Ratio of environmental interactions to world model calls.

• Their ratio is currently set to 1:1, i.e. the planner alternates between using the environmental feedback and the internal feedback from the world model at each outer-loop iteration. It is possible to reduce this ratio and the number of environmental interactions required if the planner has access to a more accurate world model.

[Q3] Dynamic allocation of inference compute

• Our method dynamically allocates the inference budget by performing iterative inference until the output converges. This process is automatic, and we only intervene when the number of iterations reaches an upper bound of 10 (which is rarely reached). Thus, the number of forward passes is mostly determined by the convergence speed of the model itself. Interestingly, Figure 6b shows that this number correlates with the length of the target plan, indicating that the model is able to dynamically allocate more inference compute for more complex plans.

[Q4] Construction of training data.

• The training data is constructed adaptively using the newest equilibrium solutions. Specifically, an equilibrium memory is maintained that buffers all equilibrium solutions, including the newest ones. At each iteration, we curate the training data by weighted sampling from this memory (where the newest solutions are sampled more frequently) and then pairing them with the ground truths. See the Finetuning section in Appendix B.3 for more details.

[Q5] Rationale behind better improvement from feedback.

• Compared to tree-based alternatives, self-refinement is more flexible. Self-refinement takes into account feedback through forward passes of LLMs, allowing arbitrary changes based on the LLMs' knowledge. This approach also enables the model to correct multiple errors in parallel (Figure 10c). In contrast, tree-based alternatives require backtracking in a tree, which is costly when correcting an early mistake and does not fully exploit the verbalized knowledge in feedback.
• Compared to self-refinement via prompting, our training-based approach is more effective. This is because our training objective $L(f_\theta(x^*,c),y)$ directly teaches the LLM to self-refine by mapping a suboptimal equilibrium plan $x^*$ and the feedback $c$ to a better plan $y$. By associating plan refinement with past feedback during training, the LLM learns to take feedback into account and self-refine in a more meaningful direction.

I thank the authors for their response. My score remains the same.