AffordBot: 3D Fine-grained Embodied Reasoning via Multimodal Large Language Models

Thank you for your valuable comments. We appreciate your positive assessment of our method’s performance and analysis. We hope the responses below address your concerns.

W1.1: Clarification on 3D-aware reasoning.

We sincerely appreciate your recognition of our method's effectiveness. However, we respectfully disagree with the concern that our design overlooks deeper 3D spatial or physical reasoning. On the contrary, it preserves 3D spatial information by explicitly associating each element in the 2D rendered views with its precise 3D coordinates. These associations are incorporated into the MLLM input as reasoning-aware cues—each image is overlaid with bounding boxes and paired with textual prompts that enumerate the corresponding 3D attributes. As a result, the model operates not on raw 2D pixels alone, but on spatially grounded visual–textual inputs, enabling effective 3D-aware reasoning throughout the chain-of-thought process.

W1.2&Q1: Comparison to native 3D MLLMs.

Thanks for your valuable comments. The cutting‑edge 2D MLLMs are currently an order of magnitude larger and more capable than 3D‑aware models in both vision-language understanding and long-horizon reasoning, as evidenced by our superior performance. While leveraging 3D-aware MLLMs or directly incorporating point cloud features is promising, such directions currently face several limitations. First, our task requires fine-grained reasoning on small affordances (e.g., knobs, handles) embedded in cluttered 3D scenes, which is particularly challenging. However, most existing 3D MLLMs operate at a coarse object-level granularity and struggle to support our fine-grained task. Moreover, incorporating point cloud features into MLLMs would require architectural modifications and retraining. These scene-level architectures typically require large-scale training data, for instance, 3D-LLM [1] utilizes 300K 3D-text pairs, whereas our current dataset SceneFun3D (230 scenes and 4K instructions) remains quite limited.

W2&Q2: Dependence on segmentation quality.

We appreciate the reviewer’s feedback regarding the dependence on Mask3D segmentation. As acknowledged in Section 4.2 (Lines 290–292), our framework does rely on upstream segmentation to propose candidate regions. However, the subsequent multimodal representation and chain-of-thought reasoning provide a level of robustness that can mitigate some of these errors. This is evident in Table 4, where categories such as "rotate" and "key_press"—despite lower segmentation quality—still yield strong downstream performance. Moreover, compared to the prior state-of-the-art Fun3DU, which also adopts a multi-stage architecture, our method achieves substantially higher AP50 performance, suggesting stronger tolerance to segmentation noise. We also agree that small or occluded affordance elements pose an even greater challenge for segmentation. We plan to explore hybrid segmentation strategies that combine 2D instance cues with 3D geometry, or leverage strong 2D segmentation priors as cross-modal constraints to better localize hard-to-see instances. To better quantify the impact of segmentation quality, we conducted two analyses:

(i) Performance upper bound: We replaced the predicted instance masks with ground-truth segmentation. As shown below, AP50 improves from 20.0 to 39.4, indicating significant headroom and confirming segmentation as a key bottleneck.

(ii) Robustness under noise: We simulate segmentation errors by randomly deleting a percentage of predicted masks. Interestingly, the downstream performance remains stable up to 20% noise. The performance only begins to decline more noticeably when deletion reaches 30%:

These results reflect the strength of the reasoning module, which is capable of partially recovering from upstream failures. This finding outlines a promising direction for future work: employ more robust segmentation strategies or leverage the multimodal reasoning capabilities of subsequent MLLMs to iteratively refine segmentation outputs. We will incorporate these findings into the revised version.

W3&Q3: Additional evaluation of generalization ability.

We appreciate the reviewer’s suggestion regarding the investigation of generalization capability. However, to the best of our knowledge, SceneFun3D is currently the only available scene-level dataset for our task. Preparing new 3D datasets with compatible labels requires extensive manual annotation and is unfortunately infeasible within the rebuttal period. To address the generalization concern, we conducted a test using GPT-rephrased SceneFun3D instructions with more diverse syntax and complex phrasing to evaluate the model’s robustness to out-of-distribution language inputs. Our method maintains comparable performance across original and rephrased instructions, with only a minor drop. In contrast, the baseline Fun3DU suffers a significant performance drop. These results suggest that AffordBot generalizes well to more diverse and complex instruction formulations, validating its robustness to out-of-domain language.

References:

[1] Hong Y, Zhen H, Chen P, et al. 3d-llm: Injecting the 3d world into large language models[J]. Advances in Neural Information Processing Systems, 2023, 36: 20482-20494.

Dear Reviewer HoJn,

Thank you for your insightful comments and thoughtful suggestions, which have been invaluable in improving our paper.

As the author-reviewer discussion period nears its end, we would greatly appreciate it if you could kindly confirm whether our responses have adequately addressed your concerns.

Thank you once again for your time and invaluable guidance.

Best regards,
The Authors

Thank you for your valuable comments. We are pleased that the reviewer recognizes our method as novel, effective, and thoroughly evaluated. Please see our responses below.

W1: Reliance on upstream segmentation.

Thank you for the valuable insights. We agree that the quality of upstream segmentation impacts overall performance. Nonetheless, our two‑stage design already achieves the strongest results. Specifically, AffordBot attains 20.0 AP50 on SceneFun3D, which significantly outperforms the previous SOTA Fun3DU (12.6 AP50). While an end-to-end approach would be elegant theoretically, such alternatives require large-scale training. Given the current dataset scale (230 scenes and 2K instructions), our solution remains the optimal choice. The data scaling and the exploration of end-to-end architectures should be left as future work. We will incorporate these analyses in our revision.

W2: Impact of motion discretization.

We thank the reviewer for the thoughtful feedback. In practice, the vast majority of motion-related objects in SceneFun3D—such as drawers, doors, sockets, and switches—are intentionally aligned along the primary axes, reflecting their real-world physical constraints. Therefore, discretizing the motion axis into a small number of semantic categories is not merely a simplification, but a design choice grounded in the spatial regularity of common indoor affordances. Moreover, the predicted motion axis in our framework serves as a high-level reasoning output in the execution system—analogous to a planning signal that instructs "pull outwards horizontally." This allows subsequent low-level modules or agents to refine and execute the motion with more precision. In this sense, the discretization operates as an abstraction appropriate for reasoning, not for direct actuation. To quantify potential information loss, we analyzed the angular deviation between ground-truth motion axes and their nearest canonical direction. We found that over 58% of all instances fall within 10° of the nearest primary axis, and over 96% fall within 30°. This reflects that most motion directions in our dataset naturally concentrate around the canonical axes we designed.

Q1: In-depth analysis of bottlenecks.

Thank you for your inspiring suggestion! We agree that the overall affordance grounding performance is still relatively low. This reflects the large difficulty of the SceneFun3D dataset and our task. It elevates 3D affordance reasoning to the scale of entire scenes, requiring a system to parse millions of points in indoor environments guided by natural-language instructions to precisely localize small affordance elements such as "knob" or "handle", which obviously showcase the challenging nature of the benchmark. Under such conditions, our approach still improves the previous state-of-the-art AP50 score from 12.6 to 20.0, nearly doubling it, demonstrating that our AffordBot delivers meaningful progress even in such a demanding setting. To dissect the primary bottlenecks, we progressively replaced Mask3D’s predicted masks with ground-truth (GT) masks and further provided an ideal front-view perspective. The results reveal that correcting segmentation alone boosts AP50 by nearly 20 percentage points, indicating that instance segmentation error is currently the primary limiting factor. With perfect segmentation, further providing an optimal viewpoint yields an additional improvement of 2.9 pp, suggesting that active perception still has room for optimization but is not the major bottleneck. Notably, when given both perfect masks and viewpoints, AP25 only reaches 47.4, exposing an additional challenge in cross-modal geometric reasoning—MLLMs may be confused about target localization in some multi-instance or complex spatial relation scenarios. We will add these in-depth analyses in the revised paper, hoping that our method and its thorough dissection will further inspire future work in the community.

Q2: In-depth analysis of Active View Selection.

Thank you again for recognising the contribution of our Active View Selection (AVS) mechanism. AVS leverages MLLM to select the most informative viewpoint based on the observation, which enables the model to perform better. As reported in Table 3, replacing AVS with a simple fixed viewpoint obtained by uniformly sampling video frames lowers AP50 by 8.6 pp, underscoring AVS’s core contribution. Motivated by your insights, we analyzed the failure cases of AVS and found two main patterns: (i) the target part is occluded by objects further ahead, making it absent or blurry in the image; (ii) several instances of the same category and appearance coexist in the scene as distractors. Furthermore, to quantify AVS accuracy, we adopt a strict criterion: a selection is considered successful only if every ground‑truth target element is visible and unobstructed. Under this, AVS achieves a success rate of 56%, which reflects that although our current strategy design has significantly improved compared to previous methods, there is still room for improvement. We will include this analysis in the revised version and, for future work, plan to explore more robust strategies such as multi‑candidate view generation or post-refinement.

Q3: Additional experiments with different MLLMs.

The success of our AffordBot stems from its integrated pipeline, which comprises multimodal scene representation, active view selection, and chain‑of‑thought prompting. It can be easily integrated into different MLLMs and fully utilize their strong reasoning capabilities. To verify this, we replaced Qwen‑2.5‑VL‑72B with several representative alternatives: LLaVA‑v1.6‑34B and the commercial GPT series. These results demonstrate that AffordBot is not tied to a particular model and maintains strong performance across MLLMs. Moreover, the results suggest that the better the MLLM’s vision-language capability, the higher the overall grounding accuracy. We will supplement these findings in the revised paper.

Q4: MLLM capabilities and extended settings.

We fully agree with the reviewer that one of the most exciting promises of an MLLM‑driven pipeline is its capacity for truly long‑horizon reasoning. In our paper, however, we focus on the high-level reasoning as aligned with the existing benchmark. Extending AffordBot to a setting akin to PreAfford, where the agent plans and executes a chain of affordance‑conditioned actions that progressively reshape the environment, would be an appealing next step. Implementing this requires not only multi‑turn CoT prompting but also a closed‑loop simulation that updates the scene after each predicted action, so that subsequent reasoning can take the new state into account. Because such an end‑to‑end simulator integration and experimental campaign lie beyond the time available for the rebuttal period, we plan to explore this in future work.

Thank you for your valuable comments. We are pleased that the reviewer recognizes our method's effectiveness, thorough experiments, and comprehensive analyses. Please see our responses below.

W1: View‑coverage considerations in complex environments.

Thank you for this valuable insight. In our current experiments, we use the SceneFun3D dataset, where annotated affordance elements are densely distributed in the main room. Under this distribution, our surround‑view synthesis strategy can cover almost all labeled elements. We agree that in environments with complex layouts and multiple sub‑rooms, such as those found in ScanNet, a single panoramic sweep may fail to capture occluded areas like restrooms. While our current implementation demonstrates an initial step toward the fine-grained embodied reasoning task, we plan to extend this in future work by developing more comprehensive observation strategies, including incremental observation, to better handle complex scenes.

W2&Q1: Multi‑instance scenarios.

We thank the reviewer for the thoughtful insights. Indeed, in scenarios where multiple affordance instances match the instruction, the language input often includes referential cues (such as "open the top drawer of the right cabinet") to help disambiguate. In our current pipeline, we let the MLLM itself select the most informative view, based on the tailored prompt and the multimodal representation. This design allows the model to leverage spatial references in language to resolve ambiguity among identical objects. During the subsequent reasoning step, the MLLM is required to locate all instances that satisfy the instruction, which may include one or multiple valid instances. Even without explicit disambiguation modules, our method demonstrates stronger understanding than previous work.

W3&Q3: Analysis of view coverage and quantity.

We respectfully disagree. The images shown in Fig. 3 are original RGB frames from the dataset, not our synthesized views. Precisely to address the limited context in such original observations, our surround-view synthesis module (Figure 2, middle) generates comprehensive coverage using only 8 synthesized views, significantly reducing the number of frames required while mitigating the visibility issues shown in Figure 3. To understand the impact of view quantity, we test 2, 4, 8, and 16 views. As shown in the table below, even 4 views offer strong coverage and performance, while 8 views provide complete 360° scanning with peak accuracy. However, increasing to 16 views leads to performance degradation due to redundant overlaps and the limitations of our MLLM (Qwen2.5-VL-72 B), which supports at most 9 images—forcing us to tile views and reduce resolution. Given this circumstance, we adopt 8 views as the best trade-off between efficiency and accuracy and will include this analysis in the revised version.

Q2: Difference from SceneFun3D tasks.

As outlined in Section 1, SceneFun3D provides multiple independent tasks (affordance segmentation, affordance grounding, and motion estimation) with separate benchmarks. However, this pipeline processes subtasks in isolation and predicts motions for all functional parts independently of user instructions, limiting its applicability in real-life scenarios. To overcome these, our task unifies affordance-mask localization, motion type, and motion axis into a structured triplet prediction, enabling joint spatial grounding and interaction reasoning in a single stage. Besides, our task estimates motion parameters only for instruction-relevant affordances, while SceneFun3D operates in an instruction-agnostic manner. These differences in task formulation and output granularity demonstrate that our benchmark extends beyond SceneFun3D and better aligns with real-world embodied task demands.

Thank you for your valuable comments. We are pleased that the reviewer recognizes our contribution, particularly its clear motivation, well-designed method, and thorough experimental analysis. Please see our responses below.

W1: Expressiveness of the motion-axis templates.

The discrete motion-axis templates adopted in our framework serve as a directly executable semantic interface that covers common situations in the real physical world. Specifically, the majority of affordance elements (e.g., "drawer", "plug") are aligned with primary axes. Thus, discretizing motion directions into this set of semantic categories is not merely a simplification, but a design choice grounded in the spatial regularity of real-world environments. Moreover, the predicted motion in our framework functions as a high-level semantic signal—analogous to instructions like "pull outward horizontally". This abstraction enables our high-level system to focus on semantic reasoning, while the execution details will be handled by the downstream low-level action system. As reviewer #m6AC noted, this design provides a practical and interpretable bridge between multimodal LLMs and robotic control, and has also been validated by prior work such as [1]. We appreciate the reviewer’s insight and plan to extend the current design toward finer-grained primitives or continuous vectors in future work.

W2: Influence of segmentation quality.

Thank you for the insightful comment. We agree that upstream segmentation quality affects overall performance. To assess this, we conducted an oracle study using ground-truth masks, which improved AP50 from 20.0 to 39.4, highlighting potential gains from better segmentation. Despite current imperfections, our framework still significantly outperforms the previous SOTA Fun3DU. We plan to explore MLLM-based refinement to further enhance segmentation and will include this analysis in the revised version.

W3: Embodied relevance.

We understand the reviewer's concerns about the degree of "embodiment". We would like to clarify that, by definition, our task inherently requires direct interaction with the real environment, which we believe constitutes a meaningful form of embodiment. Whether in terms of task conditions or application scenarios, the model must perceive and adjust decisions in real time within the physical world, represented by the 3D point cloud of the scene. This key distinction sets our approach apart from typical non-embodied visual question answering. Our task serves as the high-level semantic reasoning component in the robotic execution chain, with outputs directly interfacing with low-level action controllers to form a complete task pipeline. Although some sub-steps seem to be replaceable by non-embodied methods (e.g., OpenMask3D [2], Fun3DU [3]), as shown in Table 1 of the paper, their performance is significantly lower than ours. Our framework exhibits embodiment in several concrete ways. First, the surround-view synthesis and active view selection simulate the perception process of an embodied agent observing a real scene. Second, the Chain-of-Thought reasoning and prompt design operate on multi-modal inputs grounded in 3D space, enabling high-level semantic decisions based on spatial cues. Third, the model outputs target locations, motion types, and axes as interpretable signals that guide downstream action execution. Together, these components form an observation–reasoning–action chain tailored for embodied tasks, rather than passive visual understanding.

Q1: Explanations of geo-sem descriptor injection.

As illustrated in Section 3.2 (Main Paper) and Figure 4 (Supplementary), we first project each 3D element’s bounding box onto the rendered 2D view, producing a corresponding 2D region bounding box. Each 2D box is assigned a unique ID, ensuring a one-to-one correspondence between 3D elements and their 2D projections. We then convert the geometric (e.g., center coordinates, box size) and semantic (e.g., affordance type) information of each element into natural language and append it to the MLLM prompt. Instead of using a separate JSON file, we integrate all geo-sem descriptors as inline text alongside the image, which enables the language model to jointly reason over visual, geometric, and semantic information in a unified manner.

Q2: Clarifications of input modality.

We would like to clarify that our intent of "without 2D input" refers specifically to not relying on any raw 2D input provided by the dataset. AffordBot operates solely on 3D point cloud and generates all 2D views internally via online surround-view rendering. These rendered images serve as intermediate representations, not external inputs. To avoid confusion, we will revise the column header in Table 1 to read "2D raw input" for clarity.

Q3: Minor issues.

Thanks for your feedback, and we will correct this in the revised version.

References:

[1] Zawalski M, Chen W, Pertsch K, et al. Robotic control via embodied chain-of-thought reasoning[J]. arXiv preprint arXiv:2407.08693, 2024.

[2] Takmaz A, Fedele E, Sumner R W, et al. Openmask3d: Open-vocabulary 3d instance segmentation[J]. arXiv preprint arXiv:2306.13631, 2023.

[3] Corsetti J, Giuliari F, Fasoli A, et al. Functionality understanding and segmentation in 3D scenes[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24550-24559.