An Embodied Generalist Agent in 3D World

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

Overall the weaknesses in this work are found concerning the embodied actions tasks.

• In the CLIPort experiment the authors only demonstrate results on 3 out of the 10 tasks...This full table should also be presented in the supplementary information.

We thank you for your comments. We only tested LEO on a subset of the original CLIPort tasks mainly due to storage resource constraints (the whole dataset is substantially larger than the original CLIPort tasks as we introduce object-centric 3D input). Scaling to all 10 tasks is definitely possible, but the storage cost will be huge while the evaluation is still IID. Therefore, per your suggestion, we’re in the process of scaling to more tasks but in a different route. Specifically, we are looking into replacing the current training tasks with open-ended ones using some recent work on LLM-based task generation [1], and performing zero-shot evaluation on the original CLIPort tasks. We believe increasing the diversity of tasks could help with better insight into the generalist agent architecture and training scheme of LEO in terms of zero-shot generalization. We commit to shipping these results ASAP and including them in the final version.

Dear Reviewer bU9n,

We encourage you to view the demo video and animations we have provided at https://generalist-leo-anonymous.github.io. These resources offer a clearer understanding of the tasks we have benchmarked and the design of our model. We also added more visualizations of our collected data regarding at this URL.

As we are approaching the end of the author-reviewer discussion phase, we want to check if our responses address your concerns well. Are there any further clarifications we could provide? We look forward to your feedback. Please feel free to let us know if we have missed any issues.

Thank you again for your valuable feedback!

Best,

Authors

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

• The section of related works should be included in the main text to ensure the completeness of the paper.

Thanks for pointing this out. We have to make a compromise due to the page limit for submission. We will try out best bring it back in the final version.

• The way to achieve vision-language-action simply follows RT-2, thus lacking some novel design and weakening the technical contribution of this work.

RT-2 [1] inspires us on the action tokenization and we adopt a similar framework to RT-2. Nonetheless, we make several key contributions beyond RT-2:

1. We adapt such a framework to 3D scene understanding, and explore to bridge the gap between object-centric point clouds and human language, while GATO [6] and RT-2 [1] are both 2D only, without considering 3D information.
2. We seek to endow the agent with the capabilities of interacting with humans in open scenarios, e.g., dialogue, via 3D instruction tuning. Moreover, we also investigate the agent’s properties, e.g., instruction-tuning data ablation, and scaling law analysis.
3. Instead of training from scratch like GATO [6] or finetuning all parameters as in RT-2 [1], we have ventured into parameter-efficient tuning using LoRA, which has never been explored before in such vision-language-action agents and we believe our conclusions on how this training scheme scale as data and model size grow (see sec. 4.4/4.5) could make valuable contributions to the community.

• The authors need to further polish the writing.

Thanks for your feedback on writing. We put lots of efforts in delivering a well-written paper and have polished our paper for the revised version.

Why is the 3D point cloud data for the third-person global view needed by an embodied agent?

From a high level, providing global 3D input, regardless of being rarely explored in the current embodied AI tasks, is indeed quite popular in robots being deployed to the real world -- in fact, the popularity of mapping techniques like SLAM has demonstrated the need for obtaining 3D information. Therefore, we believe providing such information is useful for embodied agents.

More specifically, in the case of the object navigation of LEO, since LEO has a Markovian policy without a recurrent structure (ex. RNN), obtaining global 3D information is crucial, as exploring the scene and finding objects from 2D input only requires RNN, while with global 3D input (potentially offering map details), LEO will be able to think globally and learn to find the shortest path towards the target object.

For the other tasks, LEO demonstrated (3D dialogue, embodied reasoning with SQA3D [5], etc), all of them are effectively 3D tasks, therefore 3D input is definitely needed.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!

[1] Anthony Brohan, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.

[2] Dave Zhenyu Chen, et al. Scanrefer: 3d object localization in rgb-d scans using natural language. ECCV 2020.

[3] Zhenyu Chen, et al. Scan2cap: Context-aware dense captioning in rgb-d scans. CVPR 2021.

[4] Daichi Azuma, et al. Scanqa: 3d question answering for spatial scene understanding. CVPR 2022.

[5] Xiaojian Ma, et al. Sqa3d: Situated question answering in 3d scenes. ICLR 2023.

[6] Scott Reed, et al. A generalist agent. TMLR 2022.

---

As a side note, we’ve spotted some major issues in our navigation (one task in embodied action) evaluation protocol since the submission of our manuscript. Therefore, we’ve re-worked the whole objnav experiment and that part in both sec. 4.3 and the appendix has been completely renovated. Here are some key insights:

1. We’ve switched to using the standard habitat MP3D objnav validation set (MP3D-val) and the protocol available in habitat-lab (https://github.com/facebookresearch/habitat-lab). Therefore, comparing with baselines like Habitat-web becomes possible. Additionally, we also evaluate LEO on the newly introduced HM3D objnav validation set (HM3D-val) (note: this is “zero-shot” as only MP3D scenes are used for training LEO) and compare it with a strong baseline from CortexBench [17]. For your convenience, we’ve posted the results below.

   	MP3D-val		HM3D-val
   Model	Success (↑)	SPL (↑)	Success (↑)	SPL (↑)
   H.w. (shortest)	4.4	2.2	-	-
   H.w. (70k demo)	35.4	10.2	-	-
   VC-1 (ViT-B)	-	-	57.1	31.4
   LEO	23.1	15.2	23.1	19.1

   We would like to summarize the findings below:

   • First of all, after resolving this evaluation issue, LEO is in fact producing reasonable performances on both standard MP3D-val and even HM3D-val in a zero-shot fashion.
   • Thanks to the 3D branch within LEO that processes 3D input and provides global 3D information(potentially offering coarse map details), LEO is able to attain better SPL than Habitat-web baseline, meaning that LEO is able to take shorter paths by leveraging the map information.
   • Compared to VC-1, an agent trained on HM3D, LEO offers reasonable results given the fact that it has never been trained on HM3D scenes.

2. Per your concern on LEO trained with human demonstrations vs. shortest path, we provide the following results in the appendix:

   	MP3D-seen		MP3D-unseen
   Model	Success (↑)	SPL (↑)	Success (↑)	SPL (↑)
   Habitat-web (shortest)	4.4	2.2	-	-
   Habitat-web (70k demo)	35.4	10.2	-	-
   LEO (shortest)	23.1	15.2	11.1	9.6
   LEO (70k demo)	7.1	5.3	8.9	8.6

   As you can see, training with human demonstrations indeed leads to much worse performances of LEO and we believe the root cause is intuitive:

   • LEO is able to process the extra global 3D scene input (a list of objects in the scene, including their point cloud and bbox information), therefore, learning with shortest path data makes sense — LEO could learn to convert the 3D scene input into map details, then learn to navigate to the target in shortest path by learning from the shortest path navigation data.
   • Further, as in our updated sec. 4.3 and H.2, LEO indeed does not have any recurrent structure when it comes to action prediction, i.e. the only thing relayed from the past is an action history of 4 — LEO effectively can be viewed as a transformer-based feed-forward policy, similar to RT-2 [7]. Therefore learning from human demonstration could be extra difficult, as it includes exploration, which can be hard to learn without a recurrent structure like RNN.
   • Note that, the choice of not having a recurrent structure is a result of the computation efficiency and scalability vs. performance tradeoffs. We commit to exploring more on this in future work.

1. What’s the particular technique to adopt to deal with the 3D input? How is it different from previous methods? Or when the previous method is combined, does it just work like that, or does something non-trivial happen?

We adopt a point cloud encoder to process each object-centric point cloud into a per-object feature (embedding), we call this "tokenization" over 3D input. A subsequent Spatial Transformer [8] applies spatial attention based on the object features and object locations to reason about the spatial relations in the 3D scenes. Different from 3D-LLM [15], which constructs a complicated pipeline that leverages various 2D foundation models to fuse 3D scene representation. We explore to design a simple unified model to feed the 3D scenes into LLMs.

All these details can be found in both sec. 2.2 and appendix D.2. Feel free to let us know if there is anything you would like to learn about about!

2. in this framework, according to the experiments, what ability requires 3D understanding to make it from 0 to 1 or improve a large margin?

Thanks for pointing this out. Here are the abilities:

1. All the 3D-language understanding and reasoning capacity, including 3D dialogue, 3D captioning [2], 3D QA [3], embodied reasoning [4], etc. These tasks not only requires scene-level 3D input (only 2D input won't work), but also are incredibly relevant to building a embodied generalist agent that can perceive, reason and act in a 3D world.

2. Embodied AI capacity, including navigation and manipulation.

   • For the navigation task, in our re-worked ObjNav experiment (please find more details in our additional response in the end), LEO, which effectively adopts a Markvian policy and learn from "shortest path" data, is able to achieve much better performances than baselines that uses 2D only and the same data. The intuition is the global 3D input, which potentially offers map details, allows LEO to think globally and plan shortest path towards the target object without the need for recurrent structure like RNN.

   • For the manipulation tasks, LEO is able to attain better performances, especially on unseen splits of the CLIPort tasks we evaluated, through the use of additional 3D input.

(1) how much data does it need, the key to creating and preparing the dataset?

Thank you for rasing this question. We've explored a "PartialData" configuration with 10% instruction-tuning data in our ablation study in sec. 4.4., which shows significantly worse results than the one tuned with full data. Although it is still hard to tell how much data does it need, but such result could shed some light on the scale of data LEO might needs.

Just to elaborate a bit on data, previous works also indicate that the quality is more important than the quantity for instruction-tuning data [12,13,14]. In preparation of LEO, we've dedicated to developing several techniques on improving the data quality, including O-CoT(sec. 3.3), an pioneering approach to prompt LLM into generating high-quality 3D-language data. We will sure consider how to improve the instruction-tuning data quality further in future work.

(2) does the model parameter size matter at all, if 7B is already saturated (Fig 4-b) how about smaller models?

Thanks for pointing this out. Exploring smaller models is indeed a great direction to explore. Point is, the smallest model that Vicuna (the LLM we adopted) could offer is 7b. But we commit to exploring some other LLM architectures with smaller scale, ex. T5, in the future work.

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

1. On the motivation and coherence: I think there are some word accuracy issues in the motivation. ...First, many works on generalist agents are able to execute in the real world. (e.g. GATO...PALM-E...)...The author should be more precise on what kinds of real-world tasks they cannot execute, and what kind of real-world tasks the paper plans to address.

Thanks for pointing this put. We agree that statements should be delivered carefully. We have clarified the background before this statement and here it is:

Nonetheless, their abilities are primarily demonstrated within 2D domains, thereby limiting the comprehension of the 3D physical environment that envelops humans and other intelligent species.

Our main point here is, prior works lack the capacity for 3D understanding, e.g., reasoning in complex 3D scenes, and performing scene-grounded dialogue with humans. To address these tasks in the real 3D world, we believe the 2D image input is insufficient (evidenced by [1,2,3,4]) and therefore equip the agent with 3D understanding ability. Specifically, all the 3D tasks (ex. 3D dialogue, 3D captioning[2], 3D QA[3], embodied reasoning [4]) LEO is evaluated on requires 3D input, and these tasks are indeed very crucial for embodied agents in the 3D world. Even the manipulation and navigation tasks, which by convention are mostly done with 2D input only, could enjoy some benefits when 3D is also taken into consideration (see the advantages of LEO over baselines on CLIPort tasks).

…Second, about the attainment of general intelligence...the author can narrow the scope of the statement by adding some transition sentences.

Thanks for raising this in-depth question. This paper does not aim at defining what is general intelligence. Rather, our goal is to introduce a generalist agent that can perceive, ground, reason, plan, and act in the 3D world, which might but not necessarily, be an initial step towards more generally capable agents, or general intelligence to some extent.

On the other hand, we never mean that the generalist agent can attain general intelligence as long as it is equipped with 3D understanding ability. Instead, we argue that the 3D understanding ability is a crucial component of embodied generalist agents in the 3D world, our evaluation of tasks including 3D dialogue, 3D captioning[2], 3D QA[3], and embodied reasoning [4], which are all quite relevant and important to embodied agents, has demonstrated exactly that.

We've revised the relevant section to make this more clear as you suggested. Thanks again.

...a) The dataset is fused with existing datasets with high-quality data prompted by the LLMs. How is that challenging?...

Though the assets and LLMs exist, how to harness the LLMs to generate reliable 3D VL data is non-trivial and rarely explored. Text-only instruction-tuning data developed rapidly since Self-Instruct [5] and so did 2D VL instruction-tuning data since LLaVA [6]. In contrast, 3D VL instruction-tuning data lags behind. This is because representing 3D scenes with the LLM-familiar text is troublesome, e.g., tedious descriptions, using 3D bbox may pose great challenges due to excessive numerical reasoning for LLMs. There are also hallucination issues inherited from LLMs. To address the above problems, we propose a novel pipeline including scene-graph-based prompting and refinement procedures, which are non-trivial. Indeed, prompting LLMs into the responses we need for 3D-langauge data is not as easy as it might seem to be. Reviewer ADo8 even raises fundamental concerns on whether this is possible -- just to showcase how divergent our view on what LLMs can do today. The fact is, it is very capable, but not that much, leaving space for us to propose the aforementioned pipeline to make it work better in terms of data creation.

Thanks for your response, especially the results with scaling models.

For this part: I agree with you for the most part. I understand the contribution of filling the gap of one kind format of input-output or task setting, and I recognize the design of the pipeline as "novel" in the Strength part.

But for the method itself, as a unified model for 3D generalist agent, yes, no method has done that before. But let's look at Fig 1, If I just remove the "3D encoder", it is nothing different from a 2D-based generalist agent. And the idea to encode a 3D input, turn it into a feature and concatenate it with the rest of the features, is not different from the 3D referring expression task. There exist no technical contributions but just a new task setting. I won't argue if this new task setting with a static global 3D input is meaningful because different people have different opinions.

That's why, to sum up, I think the method's novelty is limited.

In this case, I will check the contribution on the dataset side, which I also find problematic. (mentioned in the main review and "reply to part1")

We first refine our references in this response. Our reply to your latest concerns comes in the following responses.

[1] Dave Zhenyu Chen, et al. Scanrefer: 3d object localization in rgb-d scans using natural language. ECCV 2020.

[2] Zhenyu Chen, et al. Scan2cap: Context-aware dense captioning in rgb-d scans. CVPR 2021.

[3] Daichi Azuma, et al. Scanqa: 3d question answering for spatial scene understanding. CVPR 2022.

[4] Xiaojian Ma, et al. Sqa3d: Situated question answering in 3d scenes. ICLR 2023.

[5] Yizhong Wang, et al. Self-instruct: Aligning language model with self generated instructions. ACL 2023.

[6] Haotian Liu, et al. Visual instruction tuning. NeurIPS 2023.

[7] Anthony Brohan, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.

[8] Shizhe Chen, et al. Language conditioned spatial relation reasoning for 3d object grounding. NeurIPS 2022.

[9] Ziyu Zhu, et al. 3d-vista: Pre-trained transformer for 3d vision and text alignment. ICCV 2023.

[10] Songyou Peng, et al. Openscene: 3d scene understanding with open vocabularies. CVPR 2023.

[11] Justin Kerr, et al. Lerf: Language embedded radiance fields. ICCV 2023.

[12] Long Ouyang, et al. Training language models to follow instructions with human feedback. NeurIPS 2022.

[13] Rohan Taori, et al. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.

[14] Wei-Lin Chiang, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://lmsys.org/blog/2023-03-30-vicuna, 2023.

[15] Yining Hong, et al. 3d-llm: Injecting the 3d world into large language models. NeurIPS 2023.

[16] Scott Reed, et al. A generalist agent. TMLR 2022.

[17] Arjun Majumdar, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303.18240, 2023.

Dear Reviewer Vn42

It has been a delightful experience to engage in a discussion over the technical details and contributions of our submission. We greatly appreciate your rich experience and insightful feedback in this field. To summarize, we would like to highlight several key points:

1. Dataset Contribution Not Sufficiently Highlighted in the Main Paper Thank you for the acknowledgment. We will polish the paper and emphasize the challenges, efforts, and contributions of building the datasets in a proper way.

2. The Necessity of 3D for an Embodied Generalist Agent and Our Model's Evidence of This Requirement We show the experimental results that demonstrate that 3D input is essential for our 3D tasks. Fundamentally, most of our tasks are 3D based and do not rely on image inputs. We encourage you to view the demo video and animations we have provided at https://generalist-leo-anonymous.github.io. These resources offer a clearer understanding of the tasks we have benchmarked and the design of our model. The question of whether to use images as inputs for constructing 3D representations is indeed an open one. However, it is important to note that this topic falls outside the scope of our current paper. Our focus is on illustrating the essential role of 3D inputs in the functionality of our embodied generalist agent.

3. Concerns Over Limited Technical Contribution of the Method We firmly believe that striving for a more capable embodied generalist represents a long-term aspiration within our field. Our research, hopefully the first in a series, aims to establish a foundational milestone for the development of generalist agents that can effectively comprehend the real 3D world. It is not uncommon for there to be differing opinions regarding the novelty of scientific work, particularly in endeavors that seek to break new ground. However, we wish to emphasize our substantial contributions in this area. Our work involves a new model designed and trained based on a large language model, carefully curating benchmarks, and achieving robust performances across established benchmarks over a large spectrum of 3D and embodied tasks. These contributions are distinctive and play a pivotal role in advancing the community's efforts toward creating more capable generalist agents with an enhanced understanding of the 3D world.

In light of these considerations, we respectfully request the reviewers to reconsider and potentially increase the evaluation score of our work. We deeply value your feedback and hope that our contributions are recognized as significant steps forward in this exciting and evolving field.

We hope the above response can resolve your questions and concerns. Please let us know if there is any further question!

[1] Yizhong Wang, et al. Self-instruct: Aligning language model with self generated instructions. ACL 2023.

[2] Rohan Taori, et al. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.

[3] Baolin Peng, et al. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.

[4] Haotian Liu, et al. Visual instruction tuning. NeurIPS 2023.

[5] Bo Li, et al. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023.

[6] Fuxiao Liu, et al. Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023.

[7] Yining Hong, et al. 3d-llm: Injecting the 3d world into large language models. NeurIPS 2023.

[8] Anthony Brohan, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818, 2023.

[9] Arjun Majumdar, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303.18240, 2023.

4. Results on object-navigation are surprisingly low...Can the authors please clarify the experimental setup for object navigation?

Thank you for pointing this out! We’ve spotted some major issues in our objnav evaluation protocol since the submission of our manuscript. Therefore, we’ve re-worked the whole objnav experiment and that part in both sec. 4.3 and the appendix has been completely renovated. Here are some key insights:

1. We’ve switched to using the standard habitat MP3D objnav validation set (MP3D-val) and the protocol available in habitat-lab (https://github.com/facebookresearch/habitat-lab). Therefore, comparing with baselines like Habitat-web becomes possible. Additionally, we also evaluate LEO on the newly introduced HM3D objnav validation set (HM3D-val) (note: this is “zero-shot” as only MP3D scenes are used for training LEO) and compare it with a strong baseline from cortexbench [9]. For your convenience, we’ve posted the results below.

   	MP3D-val		HM3D-val
   Model	Success (↑)	SPL (↑)	Success (↑)	SPL (↑)
   H.w. (shortest)	4.4	2.2	-	-
   H.w. (70k demo)	35.4	10.2	-	-
   VC-1 (ViT-B)	-	-	57.1	31.4
   LEO	23.1	15.2	23.1	19.1

   We would like to summarize the findings below:

   • First of all, after resolving this evaluation issue, LEO is in fact producing reasonable performances on both standard MP3D-val and even HM3D-val in a zero-shot fashion.
   • Thanks to the 3D branch within LEO that processes 3D input and provides global 3D information(potentially offering coarse map details), LEO is able to attain better SPL than Habitat-web baseline, meaning that LEO is able to take shorter paths by leveraging the map information.
   • Compared to VC-1, an agent trained on HM3D, LEO offers reasonable results given the fact that it has never been trained on HM3D scenes.

2. Per your concern on LEO trained with human demonstrations vs. shortest path, we provide the following results in the appendix:

   	MP3D-seen		MP3D-unseen
   Model	Success (↑)	SPL (↑)	Success (↑)	SPL (↑)
   Habitat-web (shortest)	4.4	2.2	-	-
   Habitat-web (70k demo)	35.4	10.2	-	-
   LEO (shortest)	23.1	15.2	11.1	9.6
   LEO (70k demo)	7.1	5.3	8.9	8.6

   As you can see, training with human demonstrations indeed leads to much worse performances of LEO and we believe the root cause is intuitive:

   • LEO is able to process the extra global 3D scene input (a list of objects in the scene, including their point cloud and bbox information), therefore, learning with shortest path data makes sense — LEO could learn to convert the 3D scene input into map details, then learn to navigate to the target in shortest path by learning from the shortest path navigation data.
   • Further, as in our updated sec. 4.3 and H.2, LEO indeed does not have any recurrent structure when it comes to action prediction, i.e. the only thing relayed from the past is an action history of 4 — LEO effectively can be viewed as a transformer-based feed-forward policy, similar to RT-2 [8]. Therefore learning from human demonstration could be extra difficult, as it includes exploration, which can be hard to learn without a recurrent structure like RNN.
   • Note that the choice of not having a recurrent structure is a result of the computation efficiency and scalability vs. performance tradeoffs. We commit to exploring more on this in future work.

How does the agent handle multi-step trajectory of the agent (observation action pair for multiple time steps)? Is the agent only trained as a markovian policy which doesn’t require encoding the history? Or is the trajectory encoded by simply appending each observations sequentially into the LLM?

Thanks for your comments. As we mentioned in the previous response, LEO is indeed a Markovian policy without any recurrent structure, and the only thing relayed from the past are 4 past actions. Therefore, 3D information (potentially offering map details) is indeed crucial, and learning from human demonstrations could fail as such trajectories include exploration and can be hard to learn without a recurrent structure.

CLIPort numbers in Table 3 are lower than the highest reported numbers in the CLIPort paper. For completeness and accuracy, can the authors update the CLIPort numbers to the highest numbers reported in the paper?

Thanks for raising this. We've checked the original CLIPort paper. We have added the performances of single-task CLIPort in Table 3 and discarded the "multi-attr" CLIPort variant, which utilizes "unseen" tasks (except for the task being evaluated). This never happens in LEO as only seen tasks are used for training. Therefore, comparing with these numbers would be unfair and we choose not to show them.

As you can see, without 3D or 2D information both lead to significant performance drop. Some intuition: since LEO does not have a recurrent policy like RNN (effectively, LEO is similar to RT-2 in terms of the transformer-based feedforward policy), therefore in the navigation task, 2D provide necessary information for avoiding obstacles like walls, while 3D provides global map details making it possible for a feed-forward policy in LEO to learn to navigate to the target in the shorest path. We will include a more comprehensive ablation on 2D/3D in the final version.

For the existing ablations on data presented in Table 18 and Figure 4(a), our take-home messages have been made clear in sec. 4.4:

• The two-stage training (align-then-instruct) is crucial, compared to instruction-tuning only.
• Compositional generalization is challenging. The variant ScanNetOnly, after learning captioning on 3RScan (stage 1) and QA on ScanNet (stage 2), fails to generalize to perform QA on 3RScan.
• More general data leads to weaker task-specific performances. Adding new-domain data slightly harms in-domain performances, e.g., Scan2Cap, (ScanNetOnly vs. NoAct (VL)). Scaling up instruction-tuning data brings significant improvements (PartialData vs. Full (VLA)). The incorporation of embodied acting data weakens VL performances (Full (VLA) vs. NoAct (VL)).

3. ...This is an insufficient way to justify design decisions...I think relying on GPT-4 to subjectively evaluate two captions and checking whether it’s sufficiently grounded to a 3D scene is problematic.

Thanks for raising this. The qualitative examples in our paper are used to illustrate the comparisons. The best way to rigorously assess the quality of such free-form data is human evaluation. Actually, we have manually examined numerous data examples to ensure the reliability of our data and meanwhile polish our refinement procedures. In Appendix A, we present quantitative analysis to justify our refinement procedures. To further quantitatively showcase our superiority over 3D-LLM [7], we analyze the answer accuracy for Object Counting questions based on the 3D-LLM released data and our data (it is hard to compare the data directly due to different source scenes). We present the results here:

Specifically, we consider the questions starting with “How many” and ending with “in the room/bedroom/kitchen”, and check the correctness according to the ground-truth labels/IDs. The accuracy of 3D-LLM released data is close to our raw accuracy (56.45 vs. 57.41), which indicates the similar performance of ChatGPT without advanced prompting technology such as O-CoT. This also shows the necessity of refinement procedures. Since there are no other salient response types such as Object Existence (e.g., “Is there a table in the room?”) in 3D-LLM’s data, we only consider the accuracy of Object Counting as a quantitative indicator.

To sum up, we believe our data construction approach, which is indeed a combination of prompt ChatGPT through our proposed O-CoT and human examination, could provide accurate 3D-language data for training LEO. Moreover, our data quality is considerably better than our counterparts (3D-LLM).

...it’s unclear if the GT can be trusted or not. This puts into question the entire GT dataset.

Thanks for your comments. All the datasets for evaluating LEO (those appear in Table 2), are standard benchmarks introduced by the research community, not collected by us (and we believe they were not produced by ChatGPT). Therefore, the possible errors (which we believe are rare) in the constructed datasets, LEO-align and LEO-instruct should not affect the evaluation of LEO.

We sincerely thank you for your time and constructive comments. Below, we provide detailed replies to your comments and hope we can resolve your major concerns.

I have strong concerns regarding automatic data-collection using LLM and its accuracy...

1. task planning describes...This kind of pre-training / fine-tuning is unnecessary and incorrect.
2. Similarly, the 3D captioning task...This example clearly has a lot of grammatical mistakes, and makes me question the GT data against which the system is evaluated.
3. For scene captioning...Their 3D positions are described incorrectly as well.

Thank you so much for pointing this out! Indeed, generating data using LLMs could be challenging, and that's why we meticulously design several refinement procedures to enhance the data quality (details shown in Appendix A.2, A.3 and A.4). Notably, considering 1) the inherently scarce 3D VL data, 2) the community’s explorations in prompting LLM [1,2,3,4,5,6,7], and 3) our improved scene-graph-based prompting approach, we believe our approach can provide high-quality 3D VL data and greatly contribute to the 3D VL community. We add more visualizations of our data regarding your concerns at this URL.

More specifically:

1. The data example of task planning in Table 21. You mentioned the chair is not adjustable and the robot cannot change the temperature etc. However, we remind that the role of LEO in this task is an AI assistant that first of all, produce plans that are based on the current scene layouts (ex. furnitures in the scenes), then communicate with human and offer suggestions. It never means to actually perform those tasks. Therefore, as long as the produced plan is fully grounded (which it does, the plan in Table 21 clearly includes chairs and tables that are in the scene), and reasonable for humans (that includes adjusting the temperature), it will be viewed as a good plan.

2. The 3D captioning data comes from Scan2Cap (ScanRefer) dataset, one of existing 3D VL datasets, which have some grammatical flaws not caused by us. Compared with these datasets, our own curated data features more natural textual phrases and are mostly grammatically correct thanks to ChatGPT. Check out more comparative examples here.

Dear Reviewer ADo8,

We encourage you to view the demo video and animations we have provided at https://generalist-leo-anonymous.github.io. These resources offer a clearer understanding of the tasks we have benchmarked and the design of our model. We also added more visualizations of our collected data regarding at this URL.

As we are approaching the end of the author-reviewer discussion phase, we want to check if our responses address your concerns well. Are there any further clarifications we could provide? We look forward to your feedback. Please feel free to let us know if we have missed any issues.

Thank you again for your valuable feedback!

Best,

Authors