Q1 & 2: Human Evaluation and Verification Function

Thank you for this important question regarding quality control. As illustrated in Step 2 of Figure 1, we implement a systematic self-verification mechanism where the model evaluates whether the augmented instructions maintain semantic equivalence with the original instructions. When discrepancies are detected, the system automatically returns to Step 1 for reconstruction. The detailed prompts for this verification process are provided in Supplementary Material C.2. To ensure robustness, we conducted human evaluation on approximately 1,000 randomly sampled instruction pairs, where human annotators reviewed the augmented instructions alongside the complete trajectory observations. The human evaluation confirmed that our automated verification mechanism achieves high precision in maintaining instruction fidelity while successfully enhancing reasoning complexity.

Q3: Reasoning Categories Definition

We sincerely apologize for not clearly defining these key concepts in the original manuscript. This is indeed an oversight in our presentation, and we thank you for pointing out this issue.To more intuitively demonstrate the definitions and effects of different reasoning levels, we provide the following detailed explanations and experimental comparisons:

Basic Reasoning: Uses original simple instructions. Partial Reasoning: Enhances instructions across all dimensions, but provides enhanced reasoning chains only at the first step, with subsequent steps reverting to the basic form. This approach simulates the strategy of providing more guidance at the beginning of complex tasks. Complete Reasoning: Provides fully enhanced reasoning chains at every step throughout the entire task execution process, including comprehensive thinking processes of perception analysis, memory retrieval, goal decomposition, and action planning.

The following table shows comparisons across different settings:

From the experimental results, we can see that complete reasoning achieves a 4.8 percentage point improvement in average performance compared to partial reasoning, while significantly reducing the standard deviation (from 9.3 to 6.8). This indicates that complete reasoning not only improves overall performance but also enhances the model's stability across different task types. This validates the effectiveness of our proposed hierarchical reasoning framework. We will supplement these detailed definitions and explanations in the subsequent version of the paper to ensure that readers can better understand our methodological contributions.

Q4: The LLM-generated reasoning chains are used as supervision signal - did you evaluate the learned reasoning process of the vision-language model separately?

This is an excellent point. While our approach does create inherent alignment between reasoning and action generation (as both are conditioned on the same trajectory history), we conducted systematic evaluation to verify this consistency:

1. We sampled 100 instances and performed human evaluation of reasoning-action consistency, achieving 99% consistency rate.
2. We developed a simple consistency checker that analyzes whether the reasoning chain logically supports the final action, showing 99.7% agreement with human annotations.
3. We also show some examples to show these consistency

Reasoning: Since the task is to put a potato slice in a metal pot on a white table, we need to first ensure the butter knife is put down before proceeding. This prevents any accidental dropping or misplacement of the knife during the process of placing the potato slice in the pot.\nAction: put down the butter knife

Reasoning: Since the soap bar has been put down, the next logical step is to locate the faucet to get water for washing the soap bar.\nAction: find a faucet The key finding is that while the architectural design promotes consistency, empirical validation confirms that the model learns to generate reasoning that genuinely reflects its decision-making process rather than superficial correlations.

Q5: Generalization to unseen Environments

While our primary focus addresses the significant challenge of long-horizon indoor task planning, we acknowledge the importance of domain generalization. To address this concern, we conducted preliminary experiments evaluating our approach on unseen indoor scenarios with different layouts, object configurations, and task contexts.

The results demonstrate that our reasoning-enhanced method maintains its performance advantages in these novel environments, indicating that the learned reasoning capabilities transfer effectively across different environmental contexts. The reasoning mechanisms appear to capture generalizable task decomposition and decision-making patterns that extend beyond the training environments. We plan to include detailed results and analysis of these generalization experiments in our camera-ready version to provide a more comprehensive evaluation of our method's adaptability to diverse scenarios.

Dear Reviewer,

Thank you for your participation in the review process. If you haven't done these steps, please engage in the discussion phase by following these guidelines:

• Read the author rebuttal;
• Engage in discussions;
• Fill out the "Final Justification" text box and update the "Rating" accordingly.

Reviewers must participate in discussions with authors before submitting “Mandatory Acknowledgement”. The deadline is Aug 8, 11.59pm AoE.

Thanks,

AC

We thank the reviewer for the in-depth review of our manuscript. We will address all comments and suggestions in the next revision of the manuscript. Below are our responses to the specific comments:

W1: Concerns about a fair comparison between original dataset and the augmentation strategy.

We appreciate the reviewer's concern regarding the fairness of the comparison between the original dataset and the augmented dataset. We clarify as follows:

Regarding "Fix the training data size for augmented data and original data"

We apologize for the confusion cased. As described in appendix section C.1, both the original and augmented data experiments utilize the same total number of training samples (80,875): the original data is trained for 5 epochs (16k samples × 5), and the augmented data experiment uses the combined set (16k original + 64k augmented) for 1 epoch, ensuring a fair comparison.

To further alleviate your concern, we conducted an extra experiment with a fixed 16k training size (random sampling from augmented data), the results are listed as follows:

We run the experiment 5 times with different random seeds, and the results show that the augmented data consistently outperforms the original data across all evaluation metrics by a significant margin. This further supports the effectiveness of our augmentation strategy.

Regarding "Comparison to other augmentation methods"

See our response to W4&W1_2.

W2: The concern about the possibility of a overlap between training and test data.

We appreciate the reviewer's concern regarding the potential overlap between training and test data. We clarify as follows:

We perform data augmentation and training using the ALFRED training set. EB-ALFRED, consistent with the Lota-Bench protocol, uses the valid seen set for evaluation. We make sure there is no overlap between the training and evaluation data after training set augmentation.

W3: All results are missing standard error.

We apologize for the confusion. We will include the standard error in all tables in the revised manuscript to provide a clearer understanding of the variability in our results. To be more specific, the average and standard error are calculated over 5 runs with different random seeds, and we will clarify this in the revised manuscript. The following table is an example of how we will present the results:

W4 & W1_2: Concerns regarding comparison with other augmentation methods and baseline explanation.

We appreciate the reviewer's insightful comments regarding the comparison with other augmentation methods and the explanation of our baseline.

We organize our response as a table to ensure clarity:

To clarify, in our paper, the baseline refers to the original model, and "basic reasoning" denotes the LLM reasoning over the original instructions.

This should address the concerns regarding the comparison with other data augmentation methods and the clarity Table 1.

W5: Regarding how common it is to not have any environment feedback.

In this context, "environment feedback" refers specifically to messages returned by the simulator, instead of the model's own observations. Most related works (such as Lota-Bench, Vota-Bench, EmbodiedBench, and their successors) incorporate environment feedback as part of the model input. This feedback typically provides information about action validity and possible failure reasons, which can help models make better decisions. However, relying on simulator feedback creates an open-loop setting that does not reflect real-world scenarios, where such guidance is unavailable. Therefore, we believe it is important to evaluate models under closed-loop settings without simulator feedback to better assess their real-world applicability.

W6: Regarding Inference time narration-augmented instructions and concerns about train-test mismatch.

We appreciate the reviewer's concern regarding the inference time narration-augmented instructions and the potential train-test mismatch. We clarify as follows:

At inference time, we do not use narration-augmented instructions. Instead, we use the original ALFRED instructions as input to the model. The narration-augmented instructions are only used during training.

The purpose of WAP is to enhance the model's generalization ability by adding world-aware narratives. The model learns to generate these narratives during training, as it would in inference time. Therefore, there is no train-test mismatch in this regard.

Q1: In Table 3, when going from "Basic Reasoning" to "+ Visual/Spatial Only", why is the performance on Visual tasks and Spatial tasks worse?

Sorry for the confusion, to answer your question, we need first clarify the meaning of "Basic Reasoning" and "+ Visual/Spatial Only" in Table 3.

As shown in the table, for Instruction，“Original” represents original ALFRED instructions，“Enhanced” represents adding WAP enhanced instructions. While for reasoning, “Original” represents reasoning for original instructions，“Enhanced” represents reasoning for enhanced instructions.

In the +Visual/Spatial Only setting, the model is trained with visual/spatial enhanced instructions, but the reasoning component remains unchanged from basic reasoning. This mismatch leads to reduced performance in spatial and visual abilities.

Q2: In equation 5, is the VLM trained to predict both reason and action? I am assuming that it is doing that.

Yes, the model is trained to predict both reasoning and action. A real case example is:

"Reasoning: Since the apple has been sliced, the next step is to put the knife down on the countertop to free up your hands for handling the apple. Action: put down the object"

Q3: What exactly is the difference between "Full Augmentation" and "Curriculum Learning"? I am assuming that "Full Augmentation" directly train on all the data instead of following the 3-staged curriculum.

Yes, you are correct. "Full Augmentation" refers to training on all the data at once, while "Curriculum Learning" involves training in a staged manner, where the model first learns from simpler tasks and gradually progresses to more complex ones.

Conclusion

We would like to thank the reviewer for the constructive feedback and suggestions. We will address all comments and suggestions in the next revision of the manuscript. Specifically, we will clarify the comparison with other augmentation methods, ensure the clarity of Table 1, include standard error in all results, and clarify the training and evaluation procedures. We believe that these changes will enhance the clarity and depth of our work, and we look forward to your feedback on the revised manuscript.

We thank the reviewer for the in-depth review of our manuscript. We will address all comments and suggestions in the next revision of the manuscript. Below are our responses to the specific comments:

W1: Trade-offs between performance and cost regarding observation history and augmented instruction generation.

We appreciate the reviewer's insightful comment on the trade-offs between performance and cost in our approach. In the next revision of the manuscript, we will included a more detailed discussion on how observation history and augmented instruction generation can impact both performance and computational cost.

We summarize the performance and inference time computational cost of our approach with respect to observation history and augmented instruction generation in the following table:

And the data augmentation cost and training cost of our approach is summarized in the following table:

W2: The authors could further elaborate on the sensitivity of the overall system's performance to the quality and specific characteristics of this teacher model.

We appreciate the reviewer's insightful comment and have conducted additional experiments to analyze the sensitivity of our system's performance to the teacher model's characteristics. Specifically, we have conducted experiments with teacher models that are different in quality and size, including Qwen2.5-7 B[1], Qwen2.5-32 B, and Qwen2.5-72 B.

The following table summarizes the performance of different teacher models across various evaluation aspects:

This table demonstrates that larger teacher models generally lead to improved performance across various evaluation dimensions. However, the specific gains can vary depending on the task complexity.

W3: It would be valuable to examine its performance in new, unseen environments and object configurations that differ substantially from those seen during training.

Thank you for the suggestion. We have further conducted additional experiments to evaluate our approach in unseen environments and object configurations that differ from those seen during training. This will be discussed in detail in the revised manuscript. Specifically, in addition to EB-ALFRED (valid-seen), we evaluated our approach on the VOTA-Bench[2] unseen split, which features novel kitchens, living rooms, and object instances that were not present during training:

Q1: What qualitative differences exist in the generated narratives that resulted in the gap between explicitly guided and self-directed enhancement?

Thanks for the insightful question. We have conducted a qualitative analysis over 200 randomly sampled episodes from the generated narratives to understand the differences between explicitly guided and self-directed enhancement. Have the following observations:

1. Explicit augmentation enhances the instructions orthogonally, strengthening the model's ability according to psychological theories. This is evident in how explicit guidance leads to clearer and more structured instructions, which are easier for the model to follow.

2. Self-directed augmentation often results in "lexical sophistication" rather than helpful decomposition. For example, self-directed instructions may use more complex vocabulary without adding visual or spatial cues, which can lead to limited improvement in task performance.

To be specific, an example of self-directed augmentation is:

• Original: "wash the spoon by the sink, put it on the kitchen island"
• Self-augmented: "After rinsing the utensil near the basin, place it atop the central surface where the bread rests."

The use of harder words does not necessarily contribute to better understanding or task execution, as they do not provide additional visual or spatial context, nor do they incorporate world-aware knowledge.

While in explicit augmentation, our orthogonal dimensions (visual, spatial, commonsense, long-horizon) inject task-relevant details:

• WAP-aug: "Wash the large golden ladle by the sink, then place it on the white rectangular kitchen island." (visual)
• Reference: "The spoon, which was used for stirring the morning's oatmeal, is now resting near the sink, glistening under the light from the window. After washing it thoroughly under the running water, place it on the gleaming surface of the kitchen island, where the rest of the utensils are neatly arranged beside the loaf of freshly baked bread. The kitchen island serves as both a workspace and a display area, showcasing the culinary tools and ingredients that bring the family together." (reference)

Q2: Could the authors provide a discussion of common failure modes or scenarios where the WAP-enhanced models still struggle?

Thank you for the question and the opportunity to clarify the limitations of our approach. After investigating the failure cases, we found one common failure mode:

The agent consistently struggles with tasks that require performing tasks that requires counting quantity (e.g., "Would you mind moving two bars of soap to the back of the toilet? It's a clever spot for extra storage and easy access."), as such scenarios are rarely represented in the training data.

We continue to explore ways to improve the model's performance in these scenarios, such as enhancing the training data with more diverse examples that require counting and quantity management. We will include a discussion of these limitations in the revised manuscript.

Conclusion

At last, thank you again for your valuable feedback and suggestions. We believe that addressing these points will significantly enhance the clarity and impact of our paper. We will ensure that the revised manuscript includes a more comprehensive discussion of the trade-offs, sensitivity to teacher model characteristics, performance in unseen environments, qualitative differences in generated narratives, and common failure modes.

References

[1] Qwen Team. "Qwen2.5 Technical Report"

[2] Wang, Siyin et al. "World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning"

We would like to thank the reviewer for the detailed, constructive review of our manuscript. We will address all comments and suggestions in the next revision of the manuscript. Below are our responses to the specific comments:

W1: The augmentation strategy should be tailored for the specific environment and task.

We appreciate the reviewer's insightful comment, and sorry for the confusion brought by Fig. 1. We generally agree that the augmentation strategy should be tailored to the specific environment and task. We agree with the reviewer that, in a scene that contains exactly one lamp, the modifier "cone-shaped lamp" is does not provide additional information for disambiguation.

However, there are cases in our dataset where the scene would require visual understanding to accomplish the task, for example, "Pick up the rectangular object with a blue cover featuring eye patterns and turn on the white cube-shaped lamp with a brown base." In such cases, the visual understanding is crucial for task completion.

EB-ALFRED contains both “sparse” rooms (single lamp) and “dense” rooms (e.g. two plates of different colours, three mugs of different shapes). In the “dense” rooms, the visual context becomes even more important, as there are multiple objects that may need to be distinguished from one another based on their appearance.

For dense rooms, these modifiers are essential for the model to understand the specific objects involved in the task. For sparse rooms, even a modifier is redundant, it helps the model to build an association between the object and its visual appearance, which can be beneficial for the overall tasks.

We will add a detailed discussion on this point in the revised manuscript, emphasizing the importance of visual understanding in task completion and how it varies across different environments.

W2: Performance improvements with Qwen VL and Intern VL models seem promising. Are published results included in this comparison table?

Sorry for the confusion, and thank you for pointing this out. We have indeed included the published results for Qwen VL and Intern VL in Table 1, specifically all open-looped evaluation for Qwen VL[1], Intern VL [2] and other base models are cited from Embodied-Bench[3]. The results for Intern VL3 are conducted by ourselves, as we can't find any publicly evaluated results. We redraw the open-loop part of the table to clarify this point:

$\dagger$ indicates that the results are conducted by ourselves. We will clarify this in the revised manuscript.

W3: The example in sec 5.3 seem like a regular Alfred task and not a surprise to complete successfully. It would be more interesting if the authors can present more non-trivial planning and reasoning behavior.

We appreciate the reviewer's comment, and agree that more non-trivial examples would better demonstrate the model's capabilities. We will compensate this in the revised manuscript by including more complex examples that require advanced planning and reasoning skills in the appendix section.

The sample in section 5.3 was chosen from the long-horizon section of EB-ALFRED, which is considered the "most challenging" section of the dataset. However, we understand that it may not fully showcase the model's capabilities. To alleviate your concern regarding the model's performance, we provide several examples that WAP succeeds while models like Claude or GPT-4o still fails:

• "Place a transparent beverage holder with a cutting tool for butter in it to the sink."
• "Put a bar of cleaned soap on the back of the porcelain seat in the bathroom."

Conclusion

Again, we would like to thank the reviewer for the constructive feedback and suggestions. We will address all comments and suggestions in the next revision of the manuscript. Specifically, we will clarify the augmentation strategy, include more non-trivial examples, and ensure that the performance comparisons are clearly presented with appropriate citations.

We believe that these changes will enhance the clarity and depth of our work, and we look forward to your feedback on the revised manuscript.

References

[1] Qwen Team. "Qwen2.5 Technical Report"

[2] Chen, Zhe et al. "Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling"

[3] Yang, Rui et al. "EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents"