Task-oriented Sequential Grounding in 3D Scenes

We would like to thank the reviewer for the thoughtful evaluation of our manuscript. We appreciate the constructive feedback provided and will address each of the mentioned limitations point by point.

W1: The experimental evaluation is somewhat unclear, making it difficult to discern whether the model outputs bbx or target IDs. If the output is bounding boxes, the supplementary material lacks the corresponding label data, while if it is target IDs, the origin of the bounding boxes in Figure 9 remains unexplained.

The model outputs are target IDs, since we use ground-truth object masks, as clarified in Line 315. The bounding boxes shown in Figure 9 are calculated from these ground-truth object masks and are included solely for result visualization purposes.

W2.1: The proposed method outputs only target-specific information at each step, without addressing movement between steps, which raises questions about continuity.

Our work focuses on the visual grounding task, aiming to localize the target object described by a sentence within a scene. Consequently, the method outputs target-specific information at each step without addressing movement between steps, as this pertains more to navigation tasks. While handling movement is indeed important, it lies beyond the scope of this study. We acknowledge its significance and plan to explore this aspect in future research.

W2.2: Specifically, is the scene input 𝑆 the same for each step? If so, this would contradict the perspective shifts caused by position changes in real-world applications.

Yes, the scene input 𝑆 is the same for each step. This is consistent with the visual grounding task, which relies on a global scene input to localize target objects. Unlike navigation tasks, which account for perspective shifts using ego-view images from a robot's perspective, our setting involves a static scene without real robots. Thus, perspective shifts are not applicable in our approach.

W3: Additionally, the dataset's purpose seems limited and somewhat misaligned. Fundamentally, it focuses on grounding tasks, where multiple steps appear unnecessary and potentially confusing, with their only purpose seeming to ensure consistency across objects. Without a memory mechanism, ensuring consistency without knowledge of prior steps feels unconvincing. The dataset thus lacks a strong rationale for multi-step grounding, as it does not effectively integrate planning or contextual continuity.

While dual-stream models like 3D-VisTA and query-based models like PQ3D lack memory mechanisms, the 3D LLM LEO incorporates such a mechanism. Specifically, LEO utilizes a grounding token [GRD] to represent each step's prediction, as demonstrated in Figure 6. Because of its autoregressive structure, LEO’s predictions for previous steps directly influence predictions for subsequent steps, thereby enabling consistency across steps and providing a rationale for multi-step grounding.

Q1: Is the scene input 𝑆 the same for each single-step grounding?

Yes, the scene input 𝑆 remains the same for each single-step grounding, as explained in our response to W2.2.

I believe that the motivation stated in the paper does not align cohesively with the proposed dataset and method, failing to form a convincing logical loop. As a result, I will lower the score to 3.

Although line 74 emphasizes that "To solve this task, an agent must understand each step in the context of the whole plan to identify the target object, since a single step alone can be insufficient to distinguish the target from other objects of the same category," the dataset construction and experiment seem to lack sufficient emphasis on leveraging contextual information. This undermines the overall persuasiveness of the approach.

We would like to thank the reviewer for the thoughtful evaluation of our manuscript. We appreciate the constructive feedback provided and will address each of the mentioned limitations point by point.

W1: The definition of the sequential grounding task is limited. The only thing it provides is the links between each step in a sequential task plan and the 3D object involved. And several other factors important for real task planning are not provided.

We acknowledge that our sequential grounding task does not encompass the full process of task planning and execution. However, we believe that 3D object grounding represents a crucial intermediate step. In real-world task planning, an agent must first locate the position it needs to reach or the object it needs to interact with before executing any actions. Research on grounding objects in 3D scenes can enhance an agent's ability to interpret its surroundings and accurately identify target objects, contributing to more effective overall task planning.

W2.1: It's not clear what's the embodiment of the proposed dataset, i.e. what agent is going to perform the task?

Currently, our setting does not involve a real robotic agent performing the tasks.

W2.2: Although the paper mentions "robot" several times, clearly some tasks/steps in the proposed dataset are not designed for robots, but for humans, like "enjoy reading before bed" and "wash your hands thoroughly in the sink". How can these instructions for humans be useful for AI?

We acknowledge that some tasks in the proposed dataset are more natural for humans than for robots. However, these tasks hold value for AI in two key ways:

1. Enhancing human-robot interaction: For robots to effectively communicate and interact with humans, they need to understand a wide range of human daily activities—not just tasks typically designed for robots. This aligns with prior work like VirtualHome [1], which compiled diverse human activities (e.g., "brush teeth" and "go to sleep") to teach robots how to perform everyday tasks. More details are available here.
2. Applications for virtual characters: These tasks can also be leveraged to instruct virtual characters in domains such as augmented reality (AR).

[1] VirtualHome: Simulating Household Activities via Programs, CVPR 2018.

W2.3: Later when it moves to the real-world execution and evaluation stage, how can the proposed dataset help task planning?

An embodied agent must understand its 3D environments and perform tasks specified by human users (like "make me a cup of coffee"). This requires the agent to locate the objects it needs to interact with (e.g., the "coffee machine") in the scene to complete the task. Our work focuses on grounding task-related objects during task planning, a critical preparatory step before executing each action.

W3: This paper does not propose a novel method (a stronger baseline) for the proposed task.

We appreciate the reviewer's concern regarding the lack of a novel method as a stronger baseline. However, the primary goal of this work is to introduce and establish a new benchmark. Developing methods to enhance model performance is a critical direction we plan to explore in future work.

While we have not proposed a new baseline, we adapted several state-of-the-art models to evaluate their performance on our task. Notably, we modified the 3D LLM LEO architecture to enable grounding----an ability it didn't have originally. Despite the overall low performance of the baselines, 3D LLM LEO demonstrated superior results, particularly in task accuracy (t-acc). This finding underscores the potential of 3D LLMs in 3D visual grounding and provides valuable insights for advancing research in this area.

W4: Images in Figure 9 Task 1 are too blurry. I cannot tell whether predictions are reasonable or not.

Thank you for pointing this out. We will replace the images in Figure 9 Task 1 with the higher-resolution version in our revised paper.

We would like to thank the reviewer for the thoughtful evaluation of our manuscript. We appreciate the constructive feedback provided and will address each of the mentioned limitations point by point.

W1: The introduction and abstract note the usefulness of the created dataset for intelligent agents and robots, but the task examples shown in Figure 1 look strange for a robot agent. They may be useful for NPC characters in computer games, but the authors do not mention this in the introduction. In this regard, the authors are asked to adjust the introduction to more accurately explain the motivation for the work done.

Thank you for your suggestion. We will revise the introduction to more accurately convey the motivation behind our work, ensuring it aligns with the potential applications for both intelligent agents and virtual characters.

W2: To demonstrate the quality of modern methods on the created dataset, the authors use only three models 3D-VisTA, PQ3D, and LEO (one model per category: Dual-stream, Query-based, 3D LLM). At the same time, their choice is not sufficiently justified. In Related Works, the authors also note the existence of other models that are not included in the comparison of approaches, which reduces the quality of the constructed benchmark.

We thank the reviewer for their insightful feedback. Evaluating representative models on a newly proposed benchmark is a standard approach to highlighting its attributes, as demonstrated in Multi3DRefer [1]. Our selection of baselines was carefully designed to represent a broad range of model architectures in the 3D vision-language (3D-VL) field.

We chose three representative model types----dual-stream, query-based, and multimodal LLM—following established practices in multimodal research. Specifically, 3D-VisTA, PQ3D, and LEO were selected as the most representative and state-of-the-art (SOTA) models in their respective categories, compared to other approaches discussed in the Related Work section.

Our experiments revealed two key findings: (1) existing models struggle significantly on the SG3D task, and (2) 3D LLMs exhibit markedly higher potential compared to other types of 3D-VL models. While we also benchmarked three additional models (Vil3DRef, ViewRefer, and MiKASA-3DVG), they did not provide additional insights.

To address the reviewer's concern, we will move the metrics for these additional models from the Appendix to the main text in future revisions. Additionally, we are committed to including more baselines in subsequent work to strengthen the robustness and comprehensiveness of our benchmark.

[1] Multi3DRefer: Grounding Text Description to Multiple 3D Objects (ICCV 2023)

W3: The article lacks an ablation study for the selected modifications of the 3D-VisTA, PQ3D, and LEO models, which would also add validity to the results obtained. Moreover, the adaptation of these models is noted as one of the contribution points.

3D-VisTA and PQ3D inherently support grounding, so we did not alter their original configurations. However, we modified the 3D LLM LEO architecture to incorporate grounding, a capability it initially lacked. We conducted two ablation experiments on these modifications to LEO's structure, and the results will be shared soon.

W4: During the benchmark, the authors considered the combination of only pre-trained GPT-4 (with closed code) and 3D object classifier. However, it would be interesting to see the same experiment with other open-source models, the results of which would be easier for researchers to reproduce.

We evaluate the combination of the open-source LLM qwen2-72b-instruct and 3D object classifier, with results summarized below:

This approach achieved an s-acc of 26.8% and a t-acc of 6.3%, which is slightly lower than the "GPT-4 with a 3D object labeler" baseline and significantly underperforms fine-tuned 3D-VL models. These results further support our conclusion in Section 5.2 that fine-tuned 3D-VL models are better suited for this task.

We would like to thank the reviewer for the thoughtful evaluation of our manuscript. We appreciate the constructive feedback provided and will address each of the mentioned limitations point by point.

W1: The main concern is that this benchmark does not fundamentally differ from traditional VG benchmarks, or at least does not justify what new significant challenges this benchmark brings.

We argue that SG3D introduces two significant challenges compared to traditional VG benchmarks:

1. Implicit situation inference across steps: Unlike SQA3D and ScanReason, which provide explicit situational contexts (e.g., “Sitting at the edge of the bed and facing the couch” in SQA3D), SG3D requires models to infer context implicitly from task descriptions and prior steps. For instance, in the "Enjoying reading before bed" task (Figure 1), step 2 ("sit on the mattress on the floor") implies the reading location, guiding the model to select the nearest lamp (lamp-20) in step 3, rather than a farther lamp (lamp-19) at the other bedside.
2. Referring consistency across steps: SG3D emphasizes consistent object references throughout a sequence. For example, in the "Make a cup of coffee and serve it on a plate" task (Figure 1), step 6 ("go back to the table and put down the coffee") requires identifying the same table used in step 3, unlike single-step benchmarks that do not address this consistency.

Q1.1: In this benchmark, could you provide examples of how context is demonstrated in its role? An ideal example would be when there are multiple instances with the same object type (distractors), where the model needs to use information from previous action steps to select the correct one among these

We appreciate the reviewer’s insightful question. Some tasks in Figure 1 indeed include such examples, though the small image sizes might make it challenging to fully illustrate how context plays a role without further explanation. Below, we highlight two specific examples from Figure 1 that demonstrate the model's reliance on prior steps to resolve distractors:

1. Task: Enjoying reading before bed (top-left in Figure 1) Step 3 requires the model to use information from step 2.

• Scene information Two bedside lamps are present—lamp-20 on the left and lamp-19 on the right.
• Step 2: "Sit on the mattress on the floor" specifies the position as the mattress at the left bedside.
• Step 3: "Turn on the lamp to provide light" requires distinguishing between the two lamps. Context from step 2 clarifies that the target is lamp-20 on the left, next to the mattress.

2. Task: Refresh yourself with a beverage (bottom-left in Figure 1) Step 3 requires the model to use information from step 1.

• Scene information: Two desks are visible—a longer desk (desk-10) on the left and a shorter desk (desk-9) on the right. Each desk has a black office chair underneath (chair-12 under desk-10, chair-13 under desk-9).
• Step 1: "Walk to the shorter one of two desks with a monitor" identifies desk-9 (the right desk) as the target.
• Step 3: "Sit on the black office chair under that same desk to enjoy your drink" links "that same desk" to desk-9, requiring the model to identify chair-13 under desk-9, excluding chair-12 under desk-10.

Thanks for the feedback from the authors. The statistics and ablation studies about the role of context in the samples of this benchmark are interesting and address some of my concerns. However, the setting of this task, including the same scene input for each step, the output that only contains the target's ID, and the model that keeps the conventional input/output design, makes the current paper with the current state unable to fully reflect the motivation. Therefore, I encourage the authors to polish the benchmark and methods further to better highlight the challenges in this task, i.e., long context and consistent grounding results during the procedure from the rebuttal, making the paper's study fundamentally different from previous grounding tasks. Given these concerns and other reviews, I would keep my original rating (although it tends to be an upgraded score of 4).