InteractiveCOT: Aligning Dynamic Chain-of-Thought Planning for Embodied Decision-Making

Thank you very much for your thorough review and valuable suggestions. In response to the issues you raised, we provide the following clarifications and revisions:

---

Presentation Clarity of the Paper: Regarding your concern that the paper's presentation is difficult to follow, we assure you that in the revised version, we will meticulously correct all spelling and grammatical errors. Specifically, concerning the subscript notation in the equations, in Equation (9), the subscript $i$ in $a_i$ denotes either a preferred or non-preferred action (i.e. $i =1$ or $2$). We will include additional explanations to clarify this notation.

---

Background Introduction on RL and DPO Algorithms: You pointed out the need for a more comprehensive background on Reinforcement Learning (RL) and Direct Policy Optimization (DPO) algorithms. Due to space constraints, our initial submission focused more on the methodology and experimental sections. In the revised manuscript, we will incorporate a succinct yet thorough introduction to the relevant background, ensuring that readers gain a clear understanding of RL and DPO and their roles in our study.

---

Use of VLM as a Planner: Regarding the use of Vision-Language Models (VLM) as planners, we conceptualize our VLM agent as a single-step planner. Unlike static planning approaches, our VLM generates new analyses and plans at each step. For every output, the VLM first analyzes the instructions and the current context to formulate a plan for the subsequent actions—this constitutes the "thoughts." Based on these thoughts, the VLM produces the necessary text actions to execute. In theory, our approach can support multi-step planning; however, to maintain consistency with baseline settings, we will explore this aspect in future work.

---

Types of Actions and Calculation of Mean Squared Error (MSE): Our actions consist of discrete text actions formatted and output by the VLM. The MSE calculation targets the probability distribution of action tokens between the reference model and the updated model. Our objective is to ensure that the probability of generating actions in the new model remains consistent with the original model. This constraint of consistency ensures that during interactive training, the logical correlation from thoughts to actions is preserved, preventing the loss of this critical relationship.

---

Explanation of Well-Fine-Tuned Model in Line 201: In response to your query about the term "well-fine-tuned model" mentioned on line 201, we provide further clarification. In Section 4 of the Implementation part of the paper, we detail our implementation specifics. Here, the "well-fine-tuned model" refers to a model that has undergone Supervised Fine-Tuning (SFT) on a multimodal decision-making dataset. Our method leverages this SFT model as the foundation for deploying reinforcement learning algorithms, thereby enhancing its generalization and adaptability capabilities.

---

Once again, we sincerely thank you for your insightful suggestions. We are committed to addressing all the mentioned issues by incorporating the necessary additions and revisions to enhance the clarity, comprehensiveness, and overall quality of our paper.

To begin with, we thank the reviewer for the carefully reviewing and insightful advice. Through your suggestions, we gained a series of ideas to improve this work.

---

Some miswords and confusion in the article:

We will correct all word errors in the article, such as donate denote in lines 214, 215, dicision → decision in Fig 1. We will also use the term "preferred action" to denote the preferred action at the moment. In line 222, "traing" will be modified to "training". For weaknesses 4 and 5, we use VLM to output decision-making and COT text. So the output of VLM itself is the strategy of the agent to perform the task in the environment, We use $\pi_{ref}$ to denote the output of a VLM that has not been fine-tuned using trajectories in environments. The whole process is illustrated by "good action" in Fig. 2, where the text output of thought and action is called $\pi_\theta$. In line 186, we will try a more appropriate expression: For every planning instance, planner output a series of planning results.

For Fig. 1, the circle trying to outline in the “preferred action” and “not preferred action” represents from this time-step the thoughts and actions between two trajectories became different, so we use different colors to describe the two trajectories in Fig. 2. From this time-step, we use dpo framework to to optimize the vlm to align with the embodied environment to make decisions. Making the output of the vlm closer to the reasonable better action output at this step. So the pairs are action level.

For weaknesses 12, this paper only discusses the framework of the optimized vlm under the no-reward model. As far as we know, PPO can fine-tune vlm under the framework of reward function or reward model, but often there is no built-in reward function in the environment, and it is often difficult to design the reward function manually and highly depends on experience, such as [1], so the standard adopts the preference optimization framework without reward model. We will also change our description in the contribution.

---

Some implementation details and supplementary material we will provide:

First of all, we will soon provide training details of our model in future versions of the article and in official responses to all authors. At the same time, due to the lack of careful explanation of our pipeline in our work, we will add the sft and dpo example codes in our process to the official response to all authors.

---

Some experiment details:

We use the same reward function as [1]: $r(s_t,a_t,s_{t+1}|g_{task})=50***1**$ s_{t+1}=g_{task} $+**1**$ s_{t+1}=g_{sub-task} $-**1**$ a_{t}\notin A_{adm}(s_t) $$. From this formula, we can see that when the agents in the environment complete the subtask goal, the vlm fine-tuned in a PPO-like pipeline can be rewarded, which is also similar to providing information about the progress of the task completion. In our framework, we do not provide the signal whether the final task is completed and the signal whether the action is a legal action in the current state, so the information related to task completion obtained by our framework is less.

---

[1] Large Language Models as Generalizable Policies for Embodied Tasks

After reading the rebuttal, I will retain my score. The paper needs significant improvements to reach clarity in presentation. I strongly recommend that the authors incorporate into the main paper the improvements they are currently offering to add to the supplementary material.

Specifically, this includes adding appropriate baselines, such as:

• The direct action selection baseline, which would clearly show the advantage of the COT (Chain of Thought) setting.
• A baseline using the same signals (e.g., task progress) for feedback. I disagree with the argument that using task progress constitutes a "less information" setting. On the contrary, it provides a much denser reward signal. Including this baseline would allow for a more rigorous comparison.

Thank you very much for your candid evaluation and valuable suggestions. In response to the issues you raised, we have made the following additions to certain aspects of this paper:

---

The main contribution of this paper extends beyond merely substituting the PPO component with DPO. Our perspective emphasizes that the thoughts generated by the Chain-of-Thought (COT) output are more significant than the final actions taken. Most existing works, including RL4VLM with PPO used, primarily focus on the correctness of final decisions, which can lead to model degradation, poor generalization performance, and other issues. Therefore, our proposed Action Probability Weighting (APW) (see Section 3.2.2) and Action Policy Consistency Constraints (APC) are our key contributions. These are strongly supported by evidence presented in the ablation experiments (Sections 4.2 and 4.3). Additionally, in Equation (3), we enhance step-wise preference construction by modifying the original DPO optimization objective from $r(s, a)$ to $Q(s, a)$ , significantly improving sample efficiency. We will ensure that the revised version of the paper highlights these points.

---

Regarding performance on the Heat & Place and Examine in Light tasks: Each type of task in ALFWorld requires a different average number of execution steps. Most Examine in Light tasks can be completed within a few steps (e.g., 3-4 steps). We initially infer that for simpler tasks, our method more readily acquires planning skills. We will also conduct more fine-grained case analyses in future work to further investigate this observation.

---

Concerning the ablation experiments on COT prompts: Our work primarily investigates whether APW and APC can enhance the model’s performance. Additionally, our reference work RL4VLM has provided detailed data demonstrating that the agent's performance is significantly lower without COT. Therefore, we believe that our experimental results will remain consistent and free from unexpected outcomes. We will ensure that the revised version includes the relevant experimental data to support this.

---

Regarding whether re-planning works better when the action sequence is long: We will incorporate experimental data addressing this aspect in the final revised version to provide comprehensive insights.

---

Once again, thank you for your suggestions. In the final revised version, we will correct all wording errors and clarify the experimental details to ensure the paper’s clarity and accuracy.