3D-AffordanceLLM: Harnessing Large Language Models for Open-Vocabulary Affordance Detection in 3D Worlds

1. For organisation of the paper, it would be better if the authors can focus more on the current work rather than using several paragraphs to describe the development of previous works some of which are not closely related.

2. For experiments, it would be better if the authors can explain more about the numbers. For example, in the last row of Table 2, it is a bit strange why mAcc is so low compared with the rest, while the other numbers are the highest. Are these metrics consistent?

3. Both the pre-training and evaluation of the model are conducted on synthetic datasets, but it is an important problem how the model can work in the real world as there can be a large domain gap between the real world scenarios and the synthetic datasets. I am not fully convinced unless the authors can show their training on synthetic datasets can have a good zero-shot capability in real world scenarios. It would be better if the authors can provide some results in real world scenarios.

1. Limited Novelty Regarding the Proposed Task: The authors introduce a task, Instruction Reasoning Affordance Segmentation (IRAS), aiming to address perceived limitations in existing 3D affordance detection approaches. However, IRAS shares substantial overlap with previous work, particularly with the Language-guided Affordance Segmentation on 3D Objects (LASO) task[1]. Both IRAS and LASO involve generating a mask over functional parts of a 3D object point cloud based on a question or instruction. Specifically, each query in both tasks consists of an object type and an affordance type, with the objective of inferring the functional part relevant to the query. To clarify the distinctiveness of IRAS, the authors should elaborate on how their queries are generated, detailing the heuristics and rationale behind their construction, and the specific differences between IRAS and LASO, and any innovations that set IRAS apart as a unique contribution to the field. Without this, the similarities between IRAS and LASO bring into question the novelty of IRAS as a distinct task.
2. Limited Innovation in Methodology: To address these challenges, the authors integrate a large language model (LLM) into a point cloud segmentation framework, using the LLM to process instructions and produce text features along with point cloud features for affordance mask prediction. However, the design involving two separate point encoders is confusing. The authors may claim that incorporating multimodal text and point cloud inputs allows the LLM to capture semantic information better, potentially enhancing segmentation performance. However, given the limited dataset size, it is unclear if sufficient point-text alignment can realistically be achieved. Furthermore, this design lacks supporting experiments to validate its effectiveness. Aside from a part segmentation pretraining step for extracting part-based features, the approach doesn't show a significant difference from LASO, which uses text features from LLM and point cloud features from a point backbone to decode affordance masks. This makes the contribution of pretraining alone insufficient as a distinguishing factor for novelty. The author needs to explain the core difference between your method and LASO.
3. Lack of Rigorous Experimental Validation: though the proposed method shows performance gains over previous approaches, the effectiveness of the proposed design is not convincingly demonstrated due to concerns about the experimental setup and baseline data, as noted in the questions. A key issue is the lack of ablations to demonstrate the extent to which the improvements stem from the pretraining stage, the large-scale pretrained backbone, or the method itself. The following baseline experiments could strengthen the validation. The authors are encouraged to prioritize experiments based on their assessment of each module's importance, conducting analyses to substantiate the effectiveness of different components and explaining which aspects they consider most critical, and why.
   • Basic Baseline A: Use the LLM to encode only text features, inputting them into the decoder along with point features encoded by a standard backbone, like PointNet++. This approach would closely follow LASO’s baseline method.
   • Pretraining Variant B: Use the proposed pretraining task, ROPS, train variant A and then fine-tune A on the IRAS task. This would help isolate the contribution of pretraining compared to A and show its impact on performance. In your current experiments, you simply remove the pretraining step (as shown in Table 6). This makes it difficult to determine whether the improvements over LASO are due to the proposed AFF token or the use of the Point Transformer segmentation backbone. The Without additional experiments, it’s challenging to draw convincing conclusions about the source of the performance gains.
   • Backbone Variant C: Replace the point encoder with the same point segmentation backbone (Point Transformer) used in this work, to determine the extent to which the performance improvement is due to the backbone choice rather than the proposed method.
   • Dual Encoder Variant D: Introduce a second point encoder, initially using PointNet++, which outputs point features to the LLM along with text tokens. The LLM’s output is then fed into the decoder from Variant C. This setup would reveal how much the proposed AFF token contributes to performance.
   • Current Approach E: Test the full model with the proposed AFF token integration and two encoders to measure the combined effectiveness of all components.
   • Comparison with previous approaches F: The fairest comparison to the previous approaches is to remove the pretrain stage and replace your segmentation backbone with PointNet++.
4. Inadequate Clarity and Organization in the Paper: The methodology section lacks clear logical flow, and inconsistent notation makes it challenging to follow. The experimental setup is insufficiently detailed, and numerous errors in descriptions further reduce clarity. Improvements in notation consistency, methodology organization, and experimental description are needed to enhance the overall clarity and coherence of the proposed approach.

[1]Yicong Li, Na Zhao, Junbin Xiao, Chun Feng, Xiang Wang, and Tat-seng Chua. Laso: Language- guided affordance segmentation on 3d object. CVPR 2024

1. While this framework connects the LLM with point encoding and mask decoding and could handle complex text instructions, my major concern is about the way of combining LLM with affordance prediction. In the IRAS fine-tuning stage, the output of the LLM is directly supervised by text targets $y_{txt}$, combined with the fact that the point backbone and mask decoder are pre-trained on text-conditioned part segmentation task, it seems that the current framework is using LLM as just a part classifier and use decoder as a text-conditioned part segmentation network. The visualization also shows that this system tend to predict the whole part as the affordance mask. This could limit the method's application on more fine-grained affordance grounding task, where only a small region of part should be activate. Besides, in this way, the benefit of using LLM is just handling complex instruction and convert it into segmentation text instead of providing detailed prior on affordance (e.g., which region provides a firm grasping). I believe there could be more integrated way of combining LLM to fully leverage the priors from it.
2. The design of the architecture also shows that the current function of LLM is just handling complex text and output text condition for segmentation. For example, in Figure 3, the text condition of the Mask Decoder is the name of part during pre-training, whereas in IRAS fine-tuning it is replace with the output of LLM.
3. The figure 2 and Figure 3 have misalignment on the use of two point cloud networks. I seems that there is a typo in Figure 2 since the Point Backbone is "specifically tailored for segmentation tasks".
4. Minor: The quality of Figure 2 is not good. The fonts of some text seems wrong.

1. Not open-world. The method cannot generalize to unseen 3D object parts, as the current 3D part dataset PartNet covers a limited category of objects with few instances. The proposed method is 3D part data dependent, which is a significant bottleneck. It is too preliminary to claim it works in a 3D open-world scene. In contrast, existing works on open-world part segmentation of 3D point clouds, such as PartSLIP, address this issue.

2. No detailed table on per-class evaluation results. The dataset is unbalanced, and per-class results should be reported.

3. No qualitative results on real-world partial-view point cloud segmentation.

4. No multi-affordance experiments. A single part or region on an object may have multiple reasonable affordances, which is crucial in an open-vocabulary scenario. Related experiments are missing here.

5. Ablation: What if the first stage is also trained on 3D AffordanceNet?

6. No robotics experiments. Affordance is always about interaction, which differentiates it from general part semantics. There are no real-world experiments to show if these better predictions lead to higher robotic manipulation success rates. Many papers introduce real-world robotic experiments to demonstrate learned affordance: SayCan, Where2Explore, OpenAD, SAGE, etc.

My primary concerns are:

• The motivation for using LLMs is not sufficiently justified. The paper argues that traditional methods are limited by fixed label sets and short-text detection, but recent works like CLIP-based approaches have shown success in open-vocabulary detection [1].
• The paper lacks proper discussion of its relationship to similar concurrent work, particularly AffordanceLLM [2]. The architectural similarities and differences should be explicitly addressed. The primary difference is the feature encoder (PointBERF for point cloud vs OWLViT for 2D images).

[1] Gen Li, Deqing Sun, Laura Sevilla-Lara, Varun Jampani. One-Shot Open Affordance Learning with Foundation Models. CVPR 2024. [2] Shengyi Qian, Weifeng Chen, Min Bai, Xiong Zhou, Zhuowen Tu, Li Erran Li. AffordanceLLM: Grounding Affordance from Vision Language Models. CVPRW 2024.

Additional comments which do not need to be addressed

L366 projcetor -> projector