MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents

While some issues may be challenging to address due to time constraints, I hope these can be explored and resolved in future work. In particular, I would appreciate a more detailed discussion in the main text about the areas I am concerned with:

1. Absence of a detailed ablation study and the potential need for more resources: Future work could benefit from including more detailed analyses on how different model components influence overall performance.

2. Fairness in method comparison: I recommend using a broader range of datasets and feature extraction methods in subsequent studies to demonstrate the advantages and limitations of the proposed method compared to traditional similarity retrieval approaches.

3. Concerns regarding hallucinations and stability: Given the uncertainty of model performance in unexpected scenarios, further validation of model stability when facing out-of-distribution data is advisable.

4. Importance of including diverse tasks: To enhance the robustness and applicability of the method, future work should consider incorporating tasks from different domains, such as webArena, textcraft, or Minecraft.

Despite these points, I find the methodology presented in this paper to be innovative and solidly feasible within its field, representing a valuable contribution. I look forward to seeing these issues addressed.

Q3.a. Could you clarify the meaning of AS (Average Steps) in your evaluation? Does it represent the number of iterations required for the agent to succeed once?

A3.a. Thanks for your valuable question! AS (Average Steps) represents the average number of steps the agent takes to successfully complete a task. However, if the agent fails to complete the task, the task is marked as a failure after reaching the step limit $H$, and the AS value is recorded as the step limit $H$. This design has a notable limitation, as in practice, it is not feasible to allow the agent to operate without a step limit.

Q3.b. Correlation Between AS and SR: Does Average Steps and Success Rate correlate? How do you interpret the results where AS shows minimal difference but SR differs more significantly?

A3.b. In essence, AS (Average Steps) and SR (Success Rate) are correlated: the better the agent's performance, the higher the success rate and the fewer average steps needed to complete tasks. However, due to the aforementioned limitations in the step design, AS fails to accurately reflect agent performance. For instance, assume $H=100$. Method A, with better performance, completes the task in exactly 100 steps, while Method B, with weaker performance, fails to complete the task even with 200 steps. Although these methods show significant differences in success rates, their AS values are identical.

In Table 4, the reason why AS shows minimal differences while SR varies significantly is related to this limitation of average steps. Tasks in AI2-THOR are more challenging than those in LEGENT, requiring more steps to complete.

As a result, methods that barely complete tasks within the step limit appear similar to those that fail to complete tasks within the limit, resulting in closely clustered AS values but significantly larger differences in SR. This phenomenon is observed only in the AI2-THOR environment and not in LEGENT.

Q3.c. Fairness of Using AS as a Metric: Considering the additional computational cost of incorporating an MLLM, would it be more appropriate to use the average inference time of the entire system instead of AS to ensure a fair comparison between methods?

A3.c. Thank you for your valuable suggestion! We acknowledge that AS (Average Steps) is a flawed metric. In contrast, the average inference time per test provides a more equitable basis for comparison. However, it shares the same limitation as AS: when the agent reaches the upper limit $H$ without completing the task, the average inference time is also capped at this point.

We have computed this metric, and the results are shown in the tables below (measured in seconds).

Performance comparison of different methods：

Ablation studies of MART：

Q4. Can You Provide Balanced Case Studies? To ensure a fair comparison, could you provide both the success and failure cases for each method (MLLM Retrieval and similarity retrieval) in your case studies? This would help understand each approach's strengths and weaknesses more comprehensively.

A4. Thanks for your valuable suggestion to improve our case study! We have added balanced case study in Appendix D, encompassing both success and failure cases for each method, along with simplified reasoning processes for clarity.

Q2. How did you overcome and mitigate stability and hallucination issues? Have you verified the effectiveness of your approach across multiple evaluation methods to ensure reliable performance?

A2. Thanks for the valuable suggestion!

How did you overcome and mitigate stability and hallucination issues?

Since we replace the language model head with a Bradley-Terry score head, our retriever model maps the hidden states to scalar scores, diverging from the paradigm of a language generation model. The established definition of hallucination in LLM doesn't exactly suit the case. The hallucination is not a topic to discuss in this scope and is also not included in the prior works listed below:

Sun et al. [1] propose to use LLM for relevance ranking in IR, which use pairwise comparisons and sliding window strategy to choose the most relevant candidate. Their experiments reveal that properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods on popular IR benchmarks.

HELM [2] propose to a pointwise ranking method, which prompt LLMs to output either “Yes” or “No”, and the generation probability is used to determine the relevance of the candidates for a given query.

SELF-RAG [3] trains a LM that adaptively retrieves passages on-demand, and generates and reflects on retrieved passages and its own generations using special tokens, called reflection tokens. Experiments show that SELF-RAG (7B and 13B parameters) significantly outperforms state-of-the-art LLMs and retrieval-augmented models on a diverse set of tasks.

Reference

[1] Is ChatGPT good at search? investigating large language models as re-ranking agents. EMNLP 2023

[2] Holistic evaluation of language models, TMLR 2023

[3] Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, ICLR 2024 Oral

In terms of potential instability, it is indeed a problem that the model may output scores that contradict facts. In our retrieval task, the instability manifests as assigning higher scores to less relevant trajectories, which is a shared problem between both similarity-based methods and our LLM-based method.

Due to the lack of the ground truth of the relevance rank of trajectories, it's hard to assess the instability directly. Considering all the methods share the same procedure in the downstream task execution, the effectiveness of trajectory selection is directly reflected in the success rate. From this aspect, the outperformance of our methods over the retrieval-based baseline reflects the less instability of our method.

Furthermore, while complete mitigation of instability remains challenging, we implement a design to mitigate its impact. In the action planning module, we prompt the agent to analyze the referenced trajectory and evaluate its contribution to the current task. When determining the referenced trajectory is useless, the agent will execute the task with solely task requirements and observations.

To evaluate the effectiveness of this design, we conducted experiments where manually inputting trajectory that had low MART score and is manually verified to be irrelevant to the current task into the agent. Besides, we remove the prompt of mitigating design as an ablation experiment.

The ablation results demonstrated the effectiveness of the mitigating design.

Have you verified the effectiveness of your approach across multiple evaluation methods to ensure reliable performance?

After retrieval, our method and the baseline share identical settings in downstream tasks. Therefore, the success rate in downstream tasks fairly reflects whether the model can select truly effective trajectories. On this metric, our method significantly outperforms the retrieval-based baseline, demonstrating the effectiveness of our approach.

Furthermore, if alternative evaluation methods were available to verify whether the effectiveness score produced by our model is accurate, these methods would show a strong positive correlation with the success rate. This is because the effectiveness score directly represents the framework's capability to achieve task successfully. Thus, we believe the success rate is a sufficient evaluation metric.

Are there specific scenarios where the MLLM retriever significantly outperforms conventional similarity retrieval methods to justify its use?

We evaluated the SR across different task types and observed that our method consistently outperformed similarity-based methods on unseen tasks across diverse environments.

The primary issue with similarity-based retrieval methods lies in their vulnerability to interfering trajectories. These trajectories, although visually or textually similar to the current task instructions or image observations, are irrelevant to the task and do not provide meaningful assistance. This interference can mislead similarity-based methods, resulting in the retrieval of trajectories that are ineffective for task completion.

As demonstrated in Case Study 1, the similarity-based method retrieved a highly similar trajectory, which was irrelevant to the current task and offered minimal assistance, leading to a low success rate. In contrast, our method emphasizes the practical effectiveness of trajectories in completing tasks. This focus allows it to mitigate the impact of interfering trajectories and retrieve trajectories that are truly effective, thereby ensuring higher task success rates.

Figure 2 (line 67) further illustrates that relying solely on similarity is ineffective for retrieving useful trajectories, as similarity is not directly correlated with success rate. The linear fit curve demonstrates that there is virtually no correlation between similarity and the actual success rate. In contrast, the scores produced by our trained retriever model exhibit a clear positive correlation with success rate.

Q1. Whether the benefits of training and using an additional MLLM outweigh the high costs in terms of additional inference time and required training resources? Are there specific scenarios where the MLLM retriever significantly outperforms conventional similarity retrieval methods to justify its use?

A1.

Whether the benefits of training and using an additional MLLM outweigh the high costs in terms of additional inference time and required training resources?

Thanks for pointing out the efficiency issues! We have been carefully considering the issue of efficiency and striving to reduce the cost of training and evaluation. Our method is not expensive during training and inference.

Training Phase:

We replaced the original language model head of LLaVA with a Bradley-Terry score head, enabling our model to generate only one token at a time (i.e., by setting ‘max new tokens’ to 1). Compared to conventional MLLM training, which generates hundreds or thousands of tokens per iteration, this approach is significantly more computationally efficient. Additionally, we utilized LoRA during training to further reduce costs. Specifically, training our method required only 8 hours on 8 A100-40GB GPUs.

Evaluation Phase:

For evaluation, we utilized a single 4090 GPU. To optimize speed, we processed multiple memory trajectories into batches for each query, reducing the complexity to $O(n/b)$, where $n$ represents the total number of memory trajectories and $b$ represents the number of trajectories per batch.

We also compared our inference time with the similarity-based method, and the results are summarized in the table.

However, for the entire pipeline, the main bottleneck is the action planning part. That is, for each task, we only perform one retrieval at the beginning and then perform multiple action planning processes to complete the task. The average time spent on one action planning is 15.06 seconds, and this time has to repeat dozens of times. Although the retrieval stage of MART has a higher time complexity compared to the similarity-based retrieval method, the actual difference is minimal, as the primary time bottleneck lies in action planning.

Considering the comprehensive experiments in three various environments on unseen tasks, MART consistently surpasses baselines by over 10% across different environments. So, we assume the benefits of training additional retriever model outweigh the cost of training and evaluation.

Thank you for your insightful response! Your suggestions are extremely helpful in enhancing the quality and reliability of our paper.

Modifying the original head of LLaVA raises concerns about the additional resources needed for labeling a novel dataset, especially if direct success rate measurements are unavailable to train your MLLM to maintain accurate classification or evaluation performance.

Additionally, it would be beneficial to understand the specific datasets utilized, the labeling process, and whether each application requires training a new model on domain-specific datasets instead of leveraging SOTA LLMs like GPT-4 or even O1.

The first point to clarify is that only the retriever in our method requires training, while the downstream decision-making agent has no models that need training. Although several modules are designed, they all rely on a frozen MLLM and require no further training. The retriever, including its score head, is the only part of our method that requires training. The training data consists exclusively of preference pairs obtained through interaction with the environment, with no other data involved. Preference pairs are induced based on the success rate, which relies on a success signal. However, success detection is a fundamental interface of the simulator. Even without direct setup, it can be inferred through the simulator’s metadata using rule-based methods or MLLM. Implementing a success detector in the real world is not difficult. Several works have used VLMs for success detection. For instance, [1] in real-robot manipulation tasks, MiniGPT4 is fine-tuned to output a binary reward as a success detector. [2] Flamingo is fine-tuned on VQA data involving real-world tasks to determine task completion. Therefore, the effort required to construct the dataset is acceptable.

The absence of a detailed ablation study on planning LLM and MLM retrieval is notable. You cannot guarantee the strict positive correlation between the retrieval and planning processes, but the simple similarity method naturally has this feature. If the planning process is inaccurate, how do you ensure that the success rate used to label data for your retrieval model doesn’t compound errors, potentially reinforcing incorrect planning but correct retrieval scenarios?

Yes, our retriever is designed for the planning agent learning, because we believe that determining whether a trajectory is suitable depends on whether the planning can succeed. We use 5 trials to eliminate the noise from downstream planning.

As we have emphasized previously, similarity does not correlate with effectiveness. A better metric is the retrieved trajectory that results in a higher task success rate. If a trajectory seems relevant and offers useful information to the downstream agent but is difficult for the agent to comprehend—potentially due to its excessive length—leading to a low downstream task success rate, it cannot be considered a suitable trajectory.

In future work, to evaluate the contribution of each component to overall performance, we plan to conduct a more comprehensive ablation study. This will include experiments with other MLLM agents, as well as analyzing various components of the agent, i.e. self-reflection mechanisms, prompt designs tailored to specific functions in action planning, and retrievers based on diverse base models, to assess their impact on overall performance.

Reference

[1] Yang, Jingyun, et al. "Robot fine-tuning made easy: Pre-training rewards and policies for autonomous real-world reinforcement learning." 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024.

[2] Du, Yuqing, et al. "Vision-language models as success detectors." arXiv preprint arXiv:2303.07280 (2023).

We sincerely appreciate your insightful suggestion!

Models such as CLIP and Sentence Transformer serve as feature extractors trained on large-scale datasets; however, their training primarily focuses on relations and does not account for success rates or effectiveness in decision-making tasks.

A potential approach involves selecting high-win-rate trajectories from decision-making task scenarios, aligning them with the current task and observations to construct a domain-specific dataset, and then fine-tuning general feature extractors on these data.

However, this method is both inefficient and significantly more expensive compared to ours. For a given task, sampling $K$ trajectories and interacting with MART can produce nearly $\binom{K}{2}$ sample pairs, whereas the alternative approach generates only one or a few high-win-rate items.

In future work, we aim to further investigate this approach and conduct fair comparisons to highlight the advantages and limitations of our method in comparison to adapted traditional similarity retrieval approaches.

Q1. Lacking innovation comparisons with prior works LLM-Planner and P-RAG.

A1. Thanks for pointing out that! As noted in the appendix B of related works on embodied grounding, LLM-Planner and P-RAG fall under the category of LLM-as-Planner methods. These methods pre-define a low-level controller, where the LLM or MLLM generates sub-goals, and the controller translates these sub-goals into executable action sequences. In contrast, our approach requires the MLLM to directly generate executable actions.

Regarding retrieval methods, both LLM-Planner and P-RAG rely on similarity-based retrieval methods, which do not evaluate the effectiveness of trajectories for task completion. Specifically, LLM-Planner uses a frozen BERT-base model to compute pairwise similarity and retrieve the top-K examples. And P-RAG computes the similarity of both the instruction and observation and sums their scores. In contrast, our method employs a multimodal large language model (MLLM) to predict the utility of trajectories for task completion, allowing the retrieval of the most effective ones.

Q2. Providing an explanation for how Trajectory Abstraction contributes to performance improvements.

A2. Trajectory Abstraction serves two critical purposes. Firstly, it compresses trajectories by removing redundant observations and feedback while retaining key steps and information pertinent to the current task. Secondly, it infers the implicit rules of tasks requiring multi-step operations from imperfect trajectories. These functionalities are demonstrated in both case studies presented in this work.

In the first case study, Trajectory Abstraction effectively compressed a trajectory of 73 steps into only 5 significant milestones while preserving crucial information, such as the location of the target object. This abstraction significantly improved the agent's task performance. Under identical conditions across other modules, the MART framework achieved a success rate of 80% after five validations, compared to a 40% success rate achieved by the similarity-based method.

In the second case study, involving a "heat" task, a 13-step imperfect trajectory obtained through exploration and trial-and-error was input into the Trajectory Abstraction module. The module effectively removed redundant actions and distilled the sequence into 6 essential steps. This refined trajectory enabled the agent to complete the task efficiently. Under identical conditions across other modules, the MART framework again achieved a success rate of 80% after five validations, whereas the similarity-based method failed entirely, with a success rate of 0%.

These results underscore the critical advantages of Trajectory Abstraction in improving task efficiency and robustness in complex scenarios.

Additionally, in the ablation studies, we compared the full MART approach with a version excluding the Trajectory Abstraction module (w/o Abstraction). The results demonstrate that Trajectory Abstraction notably improves the performance of embodied agents across environments.

Q3. If incorrect trajectories are retrieved, could this lead to catastrophic performance loss?

A3. Thanks for pointing out that!

In the action planning module, we prompt the agent to analyze the provided reference trajectory result and evaluate its contribution to the current task. If the agent determines that the trajectory does not provide explicit assistance, it proceeds to complete the task based solely on the current task requirements and observations. In such cases, its success rate is expected to approximate that of a plain agent without reference trajectory input.

To verify this, we conducted experiments where manually inputting trajectory that had low MART score and is manually verified to be irrelevant to the current task into the agent. The results demonstrated that the agent's success rate was comparable to that of the plain agent, confirming that irrelevant trajectories do not result in catastrophic performance loss.

Dear Reviewer QSXR,

We sincerely appreciate your insightful feedback, which plays a crucial role in enhancing the quality of our work.

As the rebuttal deadline approaches, we want to check whether our response addresses your concerns. If there are any additional questions or issues you would like us to address, please let us know. We would be grateful if you could provide stronger comments to support our work after reviewing our response.

Thank you once again for your support and guidance.

Best regards,

Authors of Submission10453

Q2. Lacking key comparisons with prior embodied grounding techniques.

A2. Thanks for your valuable suggestions to expand our baselines!

This suggests that the absence of key comparisons with prior grounding techniques raises concerns regarding the efficacy of the proposed method's performance.

We appreciate the reviewer for highlighting the comparison with prior grounding techniques. As explained earlier, we focus on addressing the embodied grounding of MLLMs, so we newly added previous embodied grounding techniques baselines for comparison, including LLM-as-planner and VLA. However, LLM-as-planner approaches generally require a predefined skill library which is not available in the two embodied environments, so we do not include them in our newly added comparison.

To provide a more comprehensive comparison, we have included the VLA method as a baseline in our evaluation.

The LEGENT paper conducted experiments on the LEGENT environment using the VLA method. VILA-7B [12] was adopted as the backbone model for its ability to handle interleaved inputs. The model was trained on thousands of high-quality trajectory datasets to predict the current action based on task descriptions and the interleaved context of prior observations and actions. The model was evaluated on 100 trajectories spanning all four tasks. The experimental results, compared to our proposed method, are summarized in the table below:

VILA-Sep denotes models fine-tuned separately for each task, whereas VILA-Joint refers to models trained jointly on all tasks. The paper does not specify the amount of data used to train VILA-Joint, but it is likely comparable to or greater than that used for VILA-Sep. In comparison, our method employs just 40 trajectories for training and 32 for testing.

Q3. Providing implementation details including model and training details.

A3. Thanks for pointing out that! We complemented the implementation details about the model, training process and implementation pipeline for the whole framework in Appendix A. We commit that all code, including the model, training process, benchmark tasks, and simulator, will be released upon acceptance.

Reference

[1] Lai, Xin, et al. "Lisa: Reasoning segmentation via large language model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Kazemzadeh, Sahar, et al. "Referitgame: Referring to objects in photographs of natural scenes." Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014.

[3] Nagaraja, Varun K., Vlad I. Morariu, and Larry S. Davis. "Modeling context between objects for referring expression understanding." Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14. Springer International Publishing, 2016.

[4] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).

[5] Haarnoja, Tuomas, et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." International conference on machine learning. PMLR, 2018.

[6] Driess, Danny, et al. "Palm-e: An embodied multimodal language model." arXiv preprint arXiv:2303.03378 (2023).

[7] Brohan, Anthony, et al. "Rt-2: Vision-language-action models transfer web knowledge to robotic control." arXiv preprint arXiv:2307.15818 (2023). [8] Song, Chan Hee, et al. "Llm-planner: Few-shot grounded planning for embodied agents with large language models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[9] Xu, Weiye, et al. "P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Task." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.

[10] Lifshitz, Shalev, et al. "Steve-1: A generative model for text-to-behavior in minecraft." Advances in Neural Information Processing Systems 36 (2024).

[11] Kagaya, Tomoyuki, et al. "Rap: Retrieval-augmented planning with contextual memory for multimodal llm agents." arXiv preprint arXiv:2402.03610 (2024).

[12] Lin, Ji, et al. "Vila: On pre-training for visual language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q1. Overlooking numerous related works concerning embodied grounding approaches.

A1. Thanks for your valuable suggestions!

This work aims to tackle the challenge of grounding in environments. However, the literature review focuses more on embodied agents, memory retrieval in agents, and MLLM.

We would like to clarify that our work does not address a general grounding problem in environments; instead, it focuses on a more specific topic: embodied grounding. For clarification, an embodied agent can be broadly divided into two components: perception (visual grounding[1][2][3]) and decision-making (embodied grounding).

In terms of perception, grounding in environments refers to visual grounding, which involves identifying the most relevant object or region in an image corresponding to a given language query. This aspect of grounding is typically addressed during the pre-training of MLLMs by aligning visual and textual tokens.

In contrast, embodied grounding emphasizes adapting MLLMs to make decisions in downstream embodied environments or real-world scenarios. Specifically, embodied grounding seeks to understand the effects of an agent’s actions on environmental dynamics and how to generate action sequences to achieve a given task. Unlike visual grounding, embodied grounding capabilities cannot be acquired through pre-training due to a significant gap between pre-training datasets and specific execution scenarios, such as safety constraints and object affordances.

Our work focuses on the field of embodied grounding, leveraging a retrieval-augmented generation framework. Therefore, the related work section primarily emphasizes embodied agents and memory retrieval, aligning with the specific focus of our research.

Overlook numerous specific and significant studies concerning how past embodied grounding approaches have tackled this issue.

We sincerely thank the reviewer for pointing out the missing aspects in our related work section. In response, we newly added related works on embodied grounding in Appendix B. Approaches to addressing embodied grounding can be broadly categorized into the following types:

RL: Reinforcement learning (RL) trains an agent’s policy through interaction with the environment, making the agent inherently grounded in the environment[4][5]. However, RL typically requires extensive interaction with environments and often suffers from instability, making it unsuitable for MLLMs.

VLA: These methods focus on fine-tuning vision-language models (VLMs) using expert datasets collected from embodied environments, such as PaLM-E[6] and RT-2[7]. These methods demand a significant amount of high-quality trajectory data for training.

LLM-as-Planner: These methods leverage Large Language Models (LLMs) or Multimodal Large Language Models (MLLMs) to generate high-level plans, which are then translated into executable action sequences by low-level controllers[8][9]. A key limitation of these methods is their reliance on a predefined skill library, which restricts the scope of the agent’s capabilities. Besides, acquiring a skill library might require additional RL or IL training or prior knowledge about the environment[10].

Retrieval-Augmented Agent: This category involves integrating task trajectory data into the prompts provided to MLLMs[11]. These trajectory data, rich in grounding information about the environment, enable agents to perform tasks effectively. Retrieval-augmented methods usually demonstrate greater sample efficiency compared to RL and VLA, thanks to the use of an explicit memory buffer. Our work falls into this category.

Dear Reviewer EV8c,

We sincerely appreciate your constructive feedback, which plays a crucial role in enhancing the quality of our work.

As the rebuttal deadline approaches, we want to check whether our response addresses your concerns. If there are any additional questions or issues you would like us to address, please let us know. We would be grateful if you could provide stronger comments to support our work after reviewing our response.

Thank you once again for your support and guidance.

Best regards,

Authors of Submission10453

Dear Reviewer EV8c,

We sincerely appreciate your constructive feedback, which plays a crucial role in enhancing the quality of our work.

This is a friendly reminder that only half a day remains in the rebuttal period, after which authors and reviewers will no longer be able to communicate. We want to check whether our response addresses your concerns. If there are any additional questions or issues you would like us to address, please let us know. We would be grateful if you could provide stronger comments to support our work after reviewing our response.

Thank you once again for your support and guidance.

Best regards,

Authors of Submission10453

Q1. While the paper demonstrates the effectiveness of MART in several environments, the scope of experiments could be expanded to include more diverse and challenging scenarios to further validate the robustness of the method.

A1. Thanks for your valuable suggestions to expand our experiment scenarios! We validated our approach on the more diverse and challenging ReALFRED [1] environment, where tasks require cross-room navigation and the environment is more realistic. The result of experiments on ReALFRED is shown below, and details are in Appendix C.

Q2. Lacking detailed analysis of computational complexity and resource requirements.

A2. Thanks for pointing this out! In more detail, for the training phase, we use 8 A100s (40GB) to train the retriever model with Lora and only train one epoch to prevent overfitting, and since we have replaced the language model head into a Bradley-Terry score head, our retriever model generates only one new token at a time, which greatly reduce the computation and time cost of both train and evaluation phrases. For the evaluation phase, we just used a computer with a single 4090 GPU to experiment, and for each query, we input multiple memory trajectories as a batch to speed up, so that the complexity is $O(n/b)$, where $n$ means the number of memory trajectories and $b$ means the number of trajectories in batch. We also compared our inference time with the similarity-based method under the same conditions, and the results are summarized in the table.

However, for the entire pipeline, the main bottleneck is the action planning part. That is, for each task, we only perform one retrieval at the beginning and then perform multiple action planning processes to complete the task. The average time spent on one action planning is 15.06 seconds, and this time has to repeat dozens of times. Although the retrieval stage of MART has a higher time complexity compared to the similarity-based retrieval method, the actual difference is minimal, as the primary time bottleneck lies in action planning.

Q3. Lacking collecting and processing details of user interactions.

A3. Thank you for your question. There seems to be a misunderstanding. In the article, "interaction" refers to the agent interacting with the environment to obtain the winning rate as feedback, rather than user interaction. However, we appreciate your suggestion, as it provides valuable direction for future work. We will explore how MART can be integrated into a human-in-the-loop setting.

Q4.1. Providing more detailed implementation guidelines.

A4.1. Thank you for pointing this out! We complemented the implementation details about the model, training process, and implementation pipeline for the whole framework in Appendix A. We guarantee that all code will be released upon acceptance.

Q4.2. Discussing potential limitations and solutions.

A4.2. Thank you for pointing this out! One limitation already discussed in our conclusion is the MLLM's limited context window, restricting input to a single trajectory at a time, similar to one-shot learning. Future work could explore few-shot learning with multiple trajectories, potentially enabling skill splicing and improved performance on complex, longer-horizon tasks. Another potential limitation is the usage of closed-source MLLM (GPT-4o) in our agent framework. Each module in our agent design requires GPT-4o to process a large number of images and text tokens, which requires a long time to wait. We are considering replacing it with an open-source MLLM such as qwen2-vl and deploying it locally, which will reduce the waiting time for communication and make our method available in all regions.

Reference

[1] Kim, Taewoong, et al. "ReALFRED: An Embodied Instruction Following Benchmark in Photo-Realistic Environments." European Conference on Computer Vision. Springer, Cham, 2025.

Dear Reviewer CWhu,

We sincerely appreciate your invaluable feedback, which plays a crucial role in enhancing the quality of our work.

As the rebuttal deadline approaches, we want to check whether our response addresses your concerns. If there are any additional questions or issues you would like us to address, please let us know. We would be grateful if you could provide stronger comments to support our work after reviewing our response.

Thank you once again for your support and guidance.

Best regards,

Authors of Submission10453

Dear Reviewer CWhu,

We sincerely appreciate your invaluable feedback, which plays a crucial role in enhancing the quality of our work.

This is a friendly reminder that only half a day remains in the rebuttal period, after which authors and reviewers will no longer be able to communicate. We want to check whether our response addresses your concerns. If there are any additional questions or issues you would like us to address, please let us know. We would be grateful if you could provide stronger comments to support our work after reviewing our response.

Thank you once again for your support and guidance.

Best regards,

Authors of Submission10453