Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modeling

We sincerely appreciate the reviewer for providing valuable feedback. Below are our responses to the raised concerns.

[W1] Existence and uniqueness of equilibrium points

• We empirically confirm that equilibria always exist across different initializations and tasks (see Table 5 in the appendix). The intuition behind this is that LLMs tend to repeat themselves under greedy sampling, leading to easier convergence.
• For the uniqueness of the equilibria, we run 10 seeds per task over 60 random tasks. As shown in the table below, our planner typically converges to 2-4 different equilibrium solutions for the same task. However, these solutions are highly consistent in that they either all succeed (SR$\rightarrow$100) or all fail (SR$\rightarrow$0), further confirming the stability of convergence.

  Task subset	#Equilibrium solutions	Average SR
  I	2.080	97.33
  II	3.885	5.91

[W2] Intuition behind learning to refine

• Training the model to map the equilibrium plan $x^*$ to the ground truth $y$ is essentially mapping a suboptimal solution to a better solution, which involves self-refinement. Meanwhile, $x^*$ provides a good prior so that the training objective does not collapse to direct regression of the ground truth (as in standard supervised learning).
• The generalization of self-refinement is facilitated by our training objective, which avoids directly regressing to the ground truth and therefore overfits less to the training data. As further evidence, in the response to W4 below, we validate the generalizability of our method by evaluating it on ALFRED without retraining.
• The consistency during self-refinement (as shown in Figures 10-14) comes implicitly from the LLM's instruction-following capability and its tendency to repeat itself. We note that it is possible to incorporate explicit rules to generate structured output from LLMs, which may provide further performance improvements.

[W3] Robustness to noisy feedback

• Please see the response to Reviewer Rz3G's W2.

[W4] Generalization to more benchmarks

• Following your suggestion, we test pre-trained VirtualHome planners on ALFRED with updated prompts. For quick evaluation, we consider two planning metrics: task classification accuracy (across 7 task types) and recall of ground-truth action-object pairs in the predicted plan. As shown in the table below, our planner generalizes significantly better than the supervised trained planner.

  VirtualHome -> ALFRED	Task classification acc.	Action-object recall
  SFT Llama	11%	0.50%
  Ours	54%	27.08%

[W5] Missing references

• Thank you for pointing out the recent papers on long-horizon planning. We will revise the draft to include their discussion.

[W6] Distinction between "plan convergence" and "task success"

• "Plan convergence" refers to the convergence of planner output, but this does not necessarily indicate "task success", which requires successful execution of the plan. See Figures 13a, 13b and 14a for failures after plan convergence.

[W7] Visual explanations of baseline failure

• Thank you for your suggestion. We provide more failure cases of the baselines in Figures 10 and 11 in the appendix, where they fail to correct a long plan efficiently. We will add more visualization explanations in the revision.

[Q1] Sensitivity to initialization

• To validate the performance stability w.r.t. initialization, we evaluate over 10 initial seeds for each of 60 random tasks. As shown in the results below, our method has a low standard deviation in SR and GCR and outperforms the strongest baseline Tree-Planner, confirming its stability. For convergence sensitivity w.r.t. initialization, see Table 5 in the appendix.

  Method	SR	GCR
  Tree-Planner (N=50)	38.33	56.95
  Ours	43.50 $\pm$ 5.21	68.88 $\pm$ 5.94

[Q2] Curriculum training

• Following your suggestion, we perform curriculum training on tasks of increasing difficulty. Specifically, the model is trained for six iterations: the first four use subsets of task lengths below 5, 10, 15, and 20, and the last two use the full dataset. As shown in the table below, despite the reduced total data size (by 36%), the model achieves higher SR and comparable GCR in Both novel scenarios. This demonstrates the improved data efficiency of curriculum training.

  Curriculum training	Data size	Both	novel		Novel	scene		Novel	task
  		SR	GCR		SR	GCR		SR	GCR
  x	11469	51.61	75.13		75.79	85.79		56.62	75.53
  $\checkmark$	7411	54.83	73.73		69.47	82.93		52.99	71.31

[Q3] Generalization without retraining

• Please see the response to W4.

[Q4] Limitation of Jacobian-free approximation

• Please see the reponse to Reviewer Rz3G's W5.

[Q5] Practicality on real-world robots

• Please see the response to Reviewer nTaB's W1.

Thank you for the thoughtful and constructive feedback. Below are our responses to the raised concerns.

[W1] Generalization to environment without feedback

• There are two workarounds in the absense of environmental feedback: (1) Use zero-shot LLM feedback during training. Since LLMs have been shown effective as zero-shot reward models in reinforcement learning [1], they could similarly provide guidance in our training setup. (2) Pre-train in environments with feedback. We validate this by evaluating pre-trained VirtualHome planners on the new ALFRED environment, where our planner generalizes well to a completely new environment (see reply to Reviewer Bgnq's W4 for details).

  VirtualHome -> ALFRED	Task classification acc.	Action-object recall
  SFT Llama	11%	0.50%
  Ours	54%	27.08%

[W2] Robustness to noisy feedback

• To evaluate our model under noisy feedback, we randomly replace some environmental feedback with incorrect feedback during inference. As shown in the results below, our model exhibits stable performance under small amounts of noise ($\le$10%), demonstrating its robustness.

  Noise ratio	Both	novel		Novel	scene		Novel	task
  	SR	GCR		SR	GCR		SR	GCR
  0%	51.61	75.13		75.79	85.79		56.62	75.53
  10%	53.23	73.43		74.74	85.84		53.85	73.09
  20%	50.00	73.10		73.68	83.22		54.49	71.80

[W3] Hallucination of planner and world model

• Hallucination is inherent to LLMs and affects both our planner and world model. However, a key strength of our planner is its ability to self-correct based on feedback. As shown in Figures 10-14 in the appendix, the planner may start with an incorrect plan but improves it using feedback from the environment or the world model, leading to significantly better performance than standard LLM planners (see Tables 1 and 2).
• For the hallucination of the world model, we show in the reply to W2 that our method is robust to noisy feedback. Thus, even if the world model occasionally hallucinates, the feedback it provides still helps to improve performance (see Table 3).

[W4] Lack of visual information

• We acknowledge this limitation in Appendix D. While the current method is implemented on LLM planners focused on text input, it could be adapted to video-language planners [2] with visual capabilities. We will continue to investigate this line of planning in future work.

[W5] Impact of Jacobian-free approximation

• Theoretically, due to the complexity of transformers, the Jacobian-free approximation cannot guarantee convergence to the global optimal solution, which may limit performance. Nevertheless, such approximation is a necessary tradeoff for the computational feasibility of large language models, as demonstrated in [3].
• Empirically, we confirm that the model can sometimes get stuck in local optimal equilibrium solutions that lead to task failure (see the respone to Q1 below). While there might be several reaons, we suspect that this is partly due to the limitations of the Jacobian-free approximation. Improving this approximation could lead to better performance.

[Q1] Global optimality of equilibrium solution

• To assess the optimality of equilibrium solutions, we evaluate 10 seeds on each of 60 random tasks. The results show that: (1) our planner can converge to 2-4 different equilibrium solutions for the same task, and the solutions are highly consistent in that they either all succeed (SR→100) or all fail (SR→0), (2) it doesn't always converge to global optimal solutions, as there are still failure cases. This could be due to the limitation of Jacobian-free approximation discussed above.

  Task subset	#Equilibrium solutions	Average SR
  I	2.080	97.33
  II	3.885	5.91

• In practice, we find that the equilibrium solving process always converges (see Table 5 in the appendix). This is largely due to the tendency of LLMs to repeat themselves under greedy sampling, which facilitates convergence.

[Q2] Use of environmental and world model feedback

• Our planner alternates between using environmental feedback and world model feedback at each outer-loop iteration, as illustrated in Figure 12c in the appendix. This allows for better performance within limited environmental interactions. We note that in practical applications, their frequencies could be adjusted according to the accuracy of the world model and the interaction budget.
• Despite the fact that the world model feedback may be skewed, it still provides useful information for self-refinement, resulting in improved performance (see Table 3) and convergence speed (see reply to Reviewer dmhf's W1). This can be explained by the response to W2, where our method is shown to be robust to noisy feedback.

---

[1] Ma, et al. Eureka: Human-level reward design via coding large language models. ICLR 2024.

[2] Du, et al. Video language planning. ICLR 2024.

[3] Choe, et al. Making scalable meta learning practical. NeurIPS 2023.

We deeply appreciate your valuable suggestions, and we would like to address your main concerns as follows:

[W1] Connection to real-world robot control

• Our work focuses on high-level planning that decomposes each task into mid-level actions, as a complementary direction to low-level control. Extending this to real-world robotics with low-level actions presents several key challenges: (1) effective tokenization of actions and visual inputs, requiring progress in vision-language action models, (2) handling latency from the equilibrium solving process. To address the latter, our method includes designs such as reusing equilibrium solutions to reduce overhead.

[W2] Low success rate towards application

• The lower success rate can be attributed to three main factors: (1) limited data in the VirtualHome-Env dataset, with only 1360 action sequence annotations, (2) relatively small model size (Llama 3 8B), which may constrain capability, (3) the absence of multimodal inputs that humans naturally rely on. We expect significant performance gains by increasing model size and multimodal data.

[W3] Clarity of Figure 3 and Section 3.4

• Figure 3 illustrates our training framework. At each step, the planner takes context $c_t$ as input and outputs a plan $x_t$. Only the subset of equilibrium plans interact with the environment and are stored in the equilibrium memory. During training, the planner receives feedback solely from the environment based on these equilibrium plans.

• The training and inference pipelines are detailed separately in Appendix B.3. Both the planner and world model are trained using environmental feedback stored in the equilibrium memory. They then work together during inference, where the world model provides additional feedback to the planner.

[W4] Use of world model and environmental feedback in Table 3

• During training, we use only environmental feedback to train both the planner and the world model. At inference time, we evaluate the following four setups:
  1. No feedback: Only inner-loop refinements without external feedback.
  2. Environmental feedback: Up to 10 rounds, provided after each inner loop converges.
  3. World model feedback: also up to 10 rounds, provided after inner loop convergence.
  4. Both feedback: Alternating between environmental and world model feedback after each inner loop, within a total budget of 10 environmental interactions.

Thank you for the very constructive comments. Below, we first address the raised questions.

[Q1] Advantage of combined feedback

• The world model's feedback offers an alternative to environmental feedback when the interaction budget is limited. By combining both feedbacks, the planner can perform additional self-correction rounds within the same budget. This leads to faster convergence (see response to W1 below) and thus improved performance (see Table 3).

[Q2] Lower executability

• The lower executability of our method is due to format issues in overlength outputs. This is likely caused by complex prompts (including history plans and feedback) or limited memorization of ground truths. However, these errors are minor and can be fixed with simple post-processing, resulting in nearly 100% executable plans in the virtual environment.

[Q3] Iterative training with new dataset

• This is also how we train the model: after each iteration, we update the training dataset with the equilibrium solutions generated by the newly finetuned planner, and use this updated dataset for the next training iteration. As shown in the table below, the overall performance steadily improves over training iterations.

  #Iterations	Both	novel		Novel	scene		Novel	task
  	SR	GCR		SR	GCR		SR	GCR
  0	00.00	00.00		00.00	00.00		00.00	00.00
  2	51.61	74.67		65.26	82.78		60.68	78.40
  4	51.61	73.30		68.42	82.93		53.20	72.81
  6	51.61	75.13		75.79	85.79		56.62	75.53

[Q4] Convergence analysis

• We observe that self-refinement consistently converges across different initializations. The table below summarizes the number of iterations to convergence over 10 runs on 60 random tasks, where all tested LLMs (including two new Qwen models) converge to an equilibrium within a finite number of iterations. The intuition behind this is that LLMs tend to repeat themselves, leading to easier convergence.

  Method	Mean	Std	Min	Max
  Original Llama-3-8B	6.70	2.12	3.22	14.98
  Original Qwen-2.5-0.5B	4.80	2.13	2.69	9.43
  Original Qwen-2.5-7B	7.68	5.03	2.46	16.78
  SFT Llama-3-8B	2.46	0.88	2.02	3.80
  Ours	3.02	1.61	2.32	4.07

[Q5] Dataset generation w.r.t. base model

• Since self-refinement consistently converges in practice, the resulting equilibrium can be reliably paired with ground truth to form meaningful training data. This holds true for both Qwen and Llama base models.

---

We also summarize other weaknesses mentioned in the review and provide our clarifications below.

[W1] Claim of reducing computational cost

• To clarify, we did not claim that the world model reduces computational cost. Instead, its goal is to reduce the number of environmental interactions, as mentioned in Section 3.4. To verify this claim, we compare the evolution of the success rate (SR) with and without the world model in the table below, which shows that the world model allows a similar or higher success rate even with fewer environmental interactions.

  #Interactions	World model	Both novel	Novel scene	Novel task
  0	x	33.87	49.47	34.62
  	$\checkmark$	37.09	65.26	40.17
  1	x	50.00	71.57	53.41
  	$\checkmark$	53.22	75.78	50.85
  2	x	51.61	74.73	55.12
  	$\checkmark$	56.45	76.84	53.63
  3	x	51.61	75.79	56.62
  	$\checkmark$	56.45	77.89	54.91

[W2] Robustness to environmental disturbances

• We simulate environmental disturbances by randomly replacing some environmental feedback with wrong feedback during inference. As shown in the results below, our model exhibits stable performance under small amounts of disturbances ($\le$10%), demonstrating its robustness.

  Disturbances	Both	novel		Novel	scene		Novel	task
  	SR	GCR		SR	GCR		SR	GCR
  0%	51.61	75.13		75.79	85.79		56.62	75.53
  10%	53.23	73.43		74.74	85.84		53.85	73.09
  20%	50.00	73.10		73.68	83.22		54.49	71.80

[W3] Missing references

• Thank you for pointing out the related work on planning with VLMs and LLMs. We will update the draft to include their discussion.

[W4] Description of task planning problem

• Thanks for the valuable suggestion. We will add a brief description of the task planning problem at the beginning of Section 3 to improve clarity.

[W5] Clarity of Figure 3

• Figure 3 illustrates our planning framework during training, and thus includes training-specific components, e.g. the equilibrium memory you noted. The world model does not directly interact with the environment; instead, it is trained on past experiences stored in the equilibrium memory, as shown by the arrow pointing to it. The single arrow from the environment (and none from the world model) indicates that the planner receives feedback solely from the environment during training. We will revise the figure description to make this clearer.