Discriminator-Guided Embodied Planning for LLM Agent

Q2: Groundtruth Noise

R2: Thank you for your insightful question. Labeling ground truth in planning tasks remains a sophisticated and evolving challenge. Approaches such as Monte Carlo search trees (MCST) are often employed to construct outcome-supervised value functions, while others utilize large language models (LLMs) for scoring or fine-tuning to derive reward models.

Although the limited scope of expert data necessitates training smaller models for data augmentation, which may introduce noise, some studies [5,6] have highlighted that even noisy augmented data can contribute to improved generalization and effectiveness. This is particularly evident in reinforcement learning [7,8], where such strategies have been shown to offer significant benefits.

We have conducted an additional evaluation of our discriminator's accuracy, and the results are as follows:

• Acc and OAcc in R1: The discriminator achieved an average score of 9.21 and 9.08 (Groundtruth score: 10).
• Scoring distribution of action lists: We use discriminator to score all the actions in the list.
  • Probability of groundtruth action ranking in the top 3: 76.53%.
  • Probability of groundtruth action ranking in the top 5: 91.26%.
• Average score on successfully planned trajectories: Mean score of 8.632.

Indeed, this question addresses a critical and nuanced point. And We are currently researching heterogeneous methods constructed from a subset of expert and preference data to mitigate these biases. We hope this can address your concern.

Q3: The writing

R3: Thanks for your suggestion. We have revised the manuscript to improve the writing and clearly emphasize the discriminator training and usage as the key contribution.

Q1: The generalization of discriminator

R1: Thanks for your insightful suggestion. We would like to tackle this propose from two perspectives:

1. Versatility of the Discriminator: To demonstrate this, we have conducted experiments on diverse benchmarks, including ALFRED and VH-Long-Task. The detailed results can be found in Appendix E.1 and E.2.

• ALFRED

• VH-LONG-TASK

2. Impact of Data Volume on Discriminator Performance:

we conducted experiments to analyze the model's performance under different conditions:

• Experiments with different types of task combinations:
  • Accuracy (Acc): Average score of expert demonstrations within the training data (GroundTruth: 10).
  • Accuracy varience(Acc Var): Varience of average score of expert demonstrations within the training data.
  • OOD Accuracy (OAcc): Average score of expert demonstrations out of the training data (GroundTruth: 10).
  • OOD Accuracy varience(OAcc Var): Varience of average score of expert demonstrations out of the training data.

The experiments involve varying training data volumes (4x, 2x, 1x, and 0.5x), where 1x corresponds to 20% of expert data. It can be observed that the marginal gains from using additional samples beyond our baseline are limited, while the discriminator demonstrates strong performance on OOD data.

Q2：Compared with search-based methods

R2. Thank you for your valuable feedback. We conducted comparative analyses of the planning with explicitly stored knowledge, search-based methods. And we construct a RAG(retrival augmented) framework. We sincerely hope you to review the detailed results and further comparisons provided in Appendix E.4 as limitation of content here. Specifically, we evaluated:

• RAG Recall Performance:

  • For shorter tasks, RAG demonstrates better performance with higher recall rates.
  • When retrieved samples are closely aligned with the target task, the success rate is significantly higher.

• Performance with RAG Samples in ICL:

  • Shorter tasks benefit more from retrieved samples when integrated into in-context learning (ICL).
  • Planning performance is robust due to data availability in such scenarios, especially with well-matched samples.

• Analysis of Behavior:

  • For short tasks, we observed that in many cases, simply imitating the retrieved accurate task trajectories is sufficient to complete the tasks. However, for long tasks, while the initial steps of imitation tend to be highly precise, the performance of search methods declines once environmental feedback causes deviations from the expected trajectory.
  • The discriminator demonstrates advantages in diverse tasks, especially for long sequence tasks, process evaluation and feedback is effective to ensure stable performance throughout the task trajectory.

We hope these findings demonstrate that while search-based methods can be effective when task-relevant data is available, particularly for short tasks, our approach excels in generalization, especially for long and complex tasks.

Q1: Alfred R1: Thank you for your insightful feedback. The discriminator serves as a modular component during inference. For handling visual and physical information, a feasible approach is either leveraging a VLM or fine-tuning a diffusion model such as RDT[2] or GR-Series[3,4]. Also, prior research has demonstrated the strong alignment between visual and linguistic modalities in semantics[5,6]. This suggests that extending language-based settings to incorporate visual and tactile modalities serves as a valid approach.

On this purpose, we conducted experiments on the ALFRED benchmark, where we retrained the discriminator using ALFRED's text data and evaluated its performance using GPT-3.5, GPT-4o, and Intern-VL2-8B as base models. A comparison with several mainstream baselines is summarized as follows:

• Experimental setup:
  • GPT-3.5 Turbo / GPT-4o: These models acquire environment information and feedback in textual form to perform reasoning.
  • GPT-4o-v / Intern-VL2-8B: These models utilize task instructions (text) and images to gather environment information and feedback for reasoning.

• Results: Detailed information about the experiment is in Appendix E.1. DGAP-4o only uses textual environment information and one-fifth of the expert data for discriminator's training, yet achieves competitive results. In Seen environments, DGAP achieves strong performance (SR: 0.462, GC: 0.507), slightly behind Prompter. However, in Unseen environments, DGAP outperforms Prompter in GC (0.545 vs. 0.537), demonstrating better generalization. On the other hand, DGAP-4o-v, which incorporates RGB images into the planning process, shows a decline in performance. It's likely attributed to two main factors: (1) the prompts and discriminator have not been fully optimized to utilize visual information. (2) the RGB images may introduce ambiguous or redundant information rather than render location information. These observations suggest that the integration of visual data requires more refined strategies to fully unlock its potential benefits.

To fully address this concern, we are conducting additional experiments including fine-tune a vision-language model using image-action pairs as a new data-augmentor and further adjust the discriminator structure to be multimodal. Hopefully we can update it in this period. This updated discriminator is designed to score <instruction, image, action> triplets.

Q1: Scalability to more complex and diverse scenarios

R1. Thank you for your insightful feedback. We address this issue from two perspectives:

i. Multi-Modal Experiments: We supplemented our study with multi-modal experiments conducted on the ALFRED benchmark. The discriminator was retrained using ALFRED's text data and evaluated with GPT-3.5, GPT-4o, and Intern-VL2-8B as base models. A comparison with several mainstream baselines is outlined below:

• Experimental setup:

  • GPT-3.5 Turbo / GPT-4o: These models process environment information and feedback in textual form to perform reasoning.
  • GPT-4o-v / Intern-VL2-8B: These models combine task instructions (text) and images to gather environment information and feedback for reasoning.

• Results: Detailed information about the experiment is in Appendix E.1. DGAP-4o only uses textual environment information and one-fifth of the expert data for discriminator's training, yet achieves competitive results. In Seen environments, DGAP achieves strong performance (SR: 0.462, GC: 0.507), slightly behind Prompter. However, in Unseen environments, DGAP outperforms Prompter in GC (0.545 vs. 0.537), demonstrating better generalization. On the other hand, DGAP-4o-v, which incorporates RGB images into the planning process, shows a decline in performance. It's likely attributed to two main factors: (1) the prompts and discriminator have not been fully optimized to utilize visual information. (2) the RGB images may introduce ambiguous or redundant information rather than render location information. These observations suggest that the integration of visual data requires more refined strategies to fully unlock its potential benefits.

ii. Task Diversity To better reflect high-dimensional continuous tasks encountered in real-life scenarios, which are typically multi-goal, we extended the task settings to include self-constructed long tasks in the VirtualHome environment. By concatenating different task objectives, we constructed tasks exceeding 60 steps, with the task statistics summarized in the table below. The details are shown in Appendix E.2.

Experimental results show that our method significantly outperforms the baseline in success rate, highlighting its superiority in handling high-complexity, long-horizon tasks and demonstrating the generalization capability of the discriminator.

We hope these two additional results demonstrate that our method exhibits adaptability across tasks, regardless of multi-modal inputs, long sequences, or diverse domains. Our method also ensures task-level executability, under which high-level plans can be reliably translated into low-level control signals via a fixed parser.

Q2: Generalize discriminator to visual/physical environment

R2： Thank you for your insightful feedback. The discriminator serves as a modular component during inference. For handling visual and physical information, a feasible approach is either leveraging a VLM or fine-tuning a diffusion model such as RDT[2] or GR-Series[3,4]. Also, prior research has demonstrated the strong alignment between visual and linguistic modalities in semantics[5,6]. This suggests that extending language-based settings to incorporate visual and tactile modalities serves as a valid approach.

On this purpose, we conducted experiments on the ALFRED benchmark, where we retrained the discriminator using ALFRED's text data and evaluated its performance using GPT-3.5, GPT-4o, and Intern-VL2-8B as base models. A comparison with several mainstream baselines is summarized as follows:

• Experimental setup:
  • GPT-3.5 Turbo / GPT-4o: These models acquire environment information and feedback in textual form to perform reasoning.
  • GPT-4o-v / Intern-VL2-8B: These models utilize task instructions (text) and images to gather environment information and feedback for reasoning.

• Results: Detailed information about the experiment is in Appendix E.1. DGAP-4o only uses textual environment information and one-fifth of the expert data for discriminator's training, yet achieves competitive results. In Seen environments, DGAP achieves strong performance (SR: 0.462, GC: 0.507), slightly behind Prompter. However, in Unseen environments, DGAP outperforms Prompter in GC (0.545 vs. 0.537), demonstrating better generalization. On the other hand, DGAP-4o-v, which incorporates RGB images into the planning process, shows a decline in performance. It's likely attributed to two main factors: (1) the prompts and discriminator have not been fully optimized to utilize visual information. (2) the RGB images may introduce ambiguous or redundant information rather than render location information. These observations suggest that the integration of visual data requires more refined strategies to fully unlock its potential benefits.

To fully address this concern, we are conducting additional experiments including fine-tune a vision-language model using image-action pairs as a new data-augmentor and further adjust the discriminator structure to be multimodal. Hopefully we can update it in the period. This updated discriminator is designed to score <instruction, image, action> triplets.

Q1:The novelty and difference

R1: Thanks for the question. We would like begin by introducing the motivation, followed by three typical related works representing distinct grounding approaches, and elaborating on the differences. DGAP serves as a generalized framework designed to address the generalization and efficiency challenges of complex embodied tasks (>25 steps) by designing a discriminator to provide process scores and integrating it with critic optimization in planning.

Then we discuss the differences between our approach and related works in terms of method design, data sources, and model construction:

• SayCan[9]

  • SayCan evaluates actions by integrating high-level and low-level scores.
    • The generative decoding of the LLM produces high-level explanations, which are subsequently integrated with the RL-trained low-level affordance function to evaluate action feasibility.
    • However, this approach demands extensive ground-truth collection and relies on delicate model designs, constraining its generalization capabilities. For example, the affordance model in SayCan is limited to only 15 object categories.
  • In contrast, our method:
    • Leverages the generalization capability of large models (reasoning models) and the domain-specific knowledge of small models (discriminator models).
    • Introduces score labels to bridge these two models.
    • Addresses generalization and efficiency issues under limited expert data.

• LLM Planner[10]

  • The LLM planner employs a retrieval-based approach using few-shot examples to adapt to new environments.
    • However, the static nature of the information introduced during inference restricts the method's adaptability across different tasks.
    • The method is constrained by the availability of expert examples.
  • In contrast, our method:
    • Is less sensitive to the quantity of examples (see Appendix E.3).
    • Augments a limited set of examples to train a discriminator, dynamically providing step-level signals to the LLM.

• ReAd[1]

  • Read is perhaps the most closely related work, as it also provides scores during the process to represent task contributions.
  • However, compared to ReAd:
    • Their method uses an MLP-based structure, while we adopt a RoBERTa with regression head architecture, which offers stronger semantic capture capabilities, particularly for long-text scenarios.
    • ReAd relies on interaction-based data collection, whereas our approach achieves higher sample utilization and greater data collection efficiency by fine-tuning a small model and employing beam search sampling.

Dear Reviewer Em5T,

As the deadline of the discussion period draws near, we would greatly appreciate your attention to our rebuttal. We would like to know whether we have adequately addressed your concerns. Your feedback is crucial to us, and we value the opportunity to address any concerns you have raised. If you have any further questions, we are more than happy to discuss.

Thank you for your time and consideration.

Best regards,

Authors