Generated and Pseudo Content guided Prototype Refinement for Few-shot Point Cloud Segmentation

W1. Novelty of using LLM. Specific design.

1. Novelty: FS-3DSeg is more complex than 2D, due to 3D data is complex. No prior work has applied LLMs to FS-3DSeg. Directly extending [1][2] to FS-3DSeg faces challenges: [1] finetunes LLMs and outputs 2D polygon coordinates, unsuitable for 3D objects; [2] lacks 3d-related descriptions. These challenges motivated us to pioneer the use of LLM to solve FS-3DSeg.
2. Our specific design for 3D tasks: ①Four 3D-oriented LLM commands for generating diverse class descriptions covering 3D properties like geometric structures and surface shape. ②3D-specific prompt for generating differentiated descriptions that highlight visual or geometric differences. ③Two Text-To-Prototype Compressors for enriching 3D prototypes with knowledge from LLM.

W2. Novelty of PCPR.

Our PCPR is the first to use pseudo-query context to refine prototypes in FS-3DSeg. Innovations include query-to-prototype compressor, prototype distillation, and pseudo-prediction distillation, improving 3D prototype quality and pseudo mask accuracy.

Difference with other methods: QGPA is used as baseline. [4] only distills prototypes at the last stage, while we distill prototypes at multiple progressive prototype refinement stages to ensure more stable refinement. Beyond that, we also propose pseudo-prediction distillation to improve pseudo masks, which have not been explored before. These designs effectively tackle semantic information bias in FS-3DSeg.

W3. Computational Overhead.

The usage of LLMs does not increase inference time, as descriptions and text features for all classes are generated and stored offline in one phase. During training/testing, we load the stored text features, avoiding redundant computing costs for regenerating them by LLM online. Specifically, the total offline times are 30.41 minutes for S3DIS and 67.45 minutes for S3DIS and ScanNet, mainly due to the text generation process, with negligible feature extraction time (see Table 1 in the rebuttal PDF). During inference, our model better balances time costs and results, achieving superior results with moderate computational cost compared to the SOTA (see Table 2 in the rebuttal PDF).

Q1. Impact of LLMs or prompting strategies.

1. Different LLMs: Using both gpt-3.5-turbo and gpt-4o-mini can achieve superior results compared with SOTA, with 75.74% mIoU and 73.88% mIoU on S3DIS, respectively. (see Table 3 in rebuttal PDF).
2. Prompting Strategies: Differentiated descriptions outperform diverse descriptions by 0.9% mIoU. Combining both strategies yields best performance (see Table 3 in our paper).
3. Number of Descriptions: As shown in Figure 4 (c) of our paper, mIoU increases with more diverse descriptions, stabilizing around 10/command.

Q2. Compare DD loss with other KD.

1. Compared with other KD: We verify the effects of logit distillation, feature distillation, and relational knowledge distillation. Our DD loss performs better by adaptively improving the quality of pseudo masks and prototypes through multi-stage distillation (see Table 4 in rebuttal PDF).
2. Advantage: The complexity of 3D data leads to low-quality prototypes and inaccurate predictions. Our DD loss enables early-stage 3D prototypes or predictions to gain insights from their deeper optimal counterparts, facilitating bidirectional information exchange, and resulting in accurate segmentation results.

Q3. Effects of pseudo mask errors.

1. Robustness: Our method is robust to pseudo mask errors, since PCPR and DD loss enable prototypes and prediction to mutually benefit each other, adaptively improving pseudo mask quality.
2. Impact of thresholds: Experimental results in Table 5 in rebuttal PDF file reveal:

• Threshold = 0: Optimal performance without filtering operation (reported in our paper).
• Threshold <= 0.1: Comparable to Threshold=0, indicating effective integration of pseudo query context.
• Threshold > 0.1: Performance decreases as most query features are filtered out.

Q4. Why performance gain is larger on ScanNet?

1. ScanNet has more classes, so the limited 3D support set is insufficient to distinguish classes. Our GCPR integrates rich semantic knowledge from LLM into 3D prototypes to improve performance.
2. ScanNet’s complex scenarios lead to higher class information bias. Our PCPR integrates reliable pseudo query context to refine prototypes, boosting performance.

Q5. Computational complexity of GPCPR.

GCPR and PCPR take 12.84 ms and 11.01 ms, respectively. We achieve the best performance with moderate total computing cost (see Table 2 in rebuttal PDF).

L1. Scalability.

In the offline stage for generating differentiated descriptions, it takes 20 minutes and 50 minutes for S3DIS and ScanNet. To address generation complexity for more classes, we provide the following alternatives:

1. We can divide all classes into several groups and generate differentiated text in parallel. This greatly reduces the time cost and maintains our superior performance.
2. We can remove the differentiated description generation process and only use diverse descriptions. By this, mIoU drops slightly from 75.74% (reported in the paper) to 73.84 % but still far exceeds SOTA (DPA 70.19%).

L2. Dependency on LLMs.

Descriptions and text features of all classes are generated and stored in offline stage. During training/testing, we load stored text features without regeneration, thus ensuring availability and consistency across different settings or time periods. Besides, we do not rely on a specific LLM, using other LLMs (e.g., gpt-4o-mini) in GCPR module still far surpasses SOTA (see Table 3 in rebuttal PDF). In addition, even if no LLM is available, using alternative texts such as Word2Vec or CLIP Text still exceeds the SOTA due to the effective cooperation of our GCPR, PCPR, and DD losses (see Figure 4(b) of our paper).

For L3-L5, please see our response to W3, Q1 and Q3, respectively.

Dear Reviewer FCL4,

We thank you for taking the time to review our manuscript and offer detailed and constructive comments. We appreciate your positive reception of our work and carefully considered each of your points and would like to address your comments as follows:

1. Computing cost and specific metrics.

Thank you for your helpful suggestion. In our approach, generating descriptions with LLM and extracting text features through CLIP for all classes in the dataset are performed offline before the training and testing phases. Once text features are stored through offline operations, we can directly load the stored text features without regenerating by LLM online, avoiding bringing redundant computing costs during training and testing. We will add the analysis of computing cost to our paper, as detailed below:

(1) Offline computation cost: The table below shows the time cost of extracting text features for all classes in the entire dataset. We analyze that the total offline time is primarily determined by the description generation process, with feature extraction time being negligible. And the time for generating descriptions from LLM varies on different datasets, depending on the number of classes.

(2) Specific metrics: The table below compares the computational costs and experimental results of our model with SOTA methods under the 2-way 1-shot setting. Our approach effectively balances computational cost and performance, achieving the highest performance with moderate computational cost.

2. Distillate knowledge from text to prototype directly.

Thank you for your insightful comment. [1][2] follow the paradigm of 3D-CLIP contrastive pre-training and aim to distill knowledge from pre-trained CLIP to align 3D, 2D and text representations. Following them, we conduct more experiments to directly distill knowledge from text to 3D prototypes. Specifically, we remove the proposed Text-to-Prototype Compressors and Prototype Distillation Loss in the GCPR module and incorporate a contrastive loss to align 3D prototypes and text features, where the prototype is treated as a query, with text features from the same class as positive samples and those from different classes as negative samples. As you noted, this avoids the need for LLM and CLIP during inference. The experimental results are shown in the table below.

We find that while 3D-text contrastive training can distill text knowledge into 3D prototypes to some extent, it is not as effective as our approach. We analyze that the success of [1][2] depends on the use of large-scale 3D datasets with a large number of classes for pre-training, whereas our FS-3DSeg task involves limited labeled data and non-overlapping training and testing classes, making such distillation methods less effective for this task. In contrast, our approach proposes two Text-to-Prototype Compressors in the GCPR module, which directly aggregate diverse and differentiated text features to enhance the quality of 3D prototypes. Additionally, the prototype distillation loss facilitates effective information transfer between prototypes at different stages, improving the prototype refinement process and leading to superior performance in the FS-3DSeg task.

Thank you once again for your positive feedback, and constructive and insightful suggestions. Your feedback is crucial in helping us refine our paper.

Dear Reviewer ABMp,

We would like to express our gratitude for taking the time to review our manuscript and providing detailed and constructive feedback. We appreciate your positive reception of our work and carefully considered each of your points and would like to address your comments as follows:

1. Explain symbols in Figure 1.

Thank you for your valuable suggestion. We will add legends to Figure 1 to clearly explain the symbols. Specifically, $\mathbb{P}$ denotes the original 3D prototype set. $\dot{\mathbb{T}}$, $\ddot{\mathbb{T}}$, $\dot{\mathbb{P}}$ and $\ddot{\mathbb{P}}$ represent prototype sets at different refinement stages. Among them, $\dot{\mathbb{T}}$ and $\ddot{\mathbb{T}}$ represent prototype sets refined by diverse and differentiated descriptions, $\dot{\mathbb{P}}$ and $\ddot{\mathbb{P}}$ indicate prototype sets refined by QGPA module and Query-to-Prototype Compressor, respectively. $S$ denotes cosine similarity, and $\odot$ represents matrix dot product.

2. Experiments under the new setting of [1].

Thank you for your valuable comments. Based on the new experimental settings corrected by CVPR 2024 paper [1], we conduct more experiments under the 2-way 1-shot setting on S3DIS dataset. The results are presented in the table below. Our method achieves slightly higher performance than COSeg but far exceeds attMPTI and QGPA, demonstrating the proposed GCPR module, PCPR module and DD-loss can collaborate to improve the prototype quality by integrating comprehensive text descriptions and reliable pseudo query context. We will include these experimental results and analysis in the final version of our paper to provide a more convincing comparison with SOTA under the new setting.

3. In Table 3, the PCPR module seems to only yield a marginal improvement. Please provide some explanations.

The impact of the PCPR module depends on the scale of the dataset, the number of classes, and the complexity of scenes.

(1) In Table 3, on the S3DIS dataset, without GCPR (Row 2 vs. Row 1), PCPR significantly improves the baseline by 8.68% mIoU due to its effective handling of class information bias between support and query sets. With GCPR (Row 6 vs. Row 5), PCPR only further yields a 0.64% improvement because GCPR already addresses the semantic information constraints of support set in S3DIS dataset with less classes and simpler scenes.

(2) On the ScanNet dataset, removing PCPR causes a 5.33% drop in mIoU under 2-way 1-shot setting (68.60% without PCPR vs. 73.93% with PCPR), highlighting the importance of PCPR in reducing class information bias in complex scenes with more classes.

In summary, PCPR is more impactful on large-scale dataset with greater complexity and more classes, such as ScanNet. We will explain this more clearly in the paper.

4. Which classes perform best/worst? Is there any connection between this finding and LLMs?

Thank you for your insightful question.

(1) The class performance of our model is shown as below:

• S3DIS dataset: Best on "chair" (or "sofa") and worst on "ceiling" (or "wall") under $S^0$ (or $S^1$) setting.

• ScanNet dataset: Best on " curtain " (or "sofa") and worst on "floor" (or "picture") under $S^0$ (or $S^1$) setting.

For example, under the 2-way 1-shot setting, the class performance are as follows:

We observe that our model excels in classes with complex geometric structures, unique shapes, and rich colors, but performs not as well in classes with flat geometric structures. Notably, compared to the baseline—which performs best on "chair" (or "sofa") and worst on "beam" (or "table") on S3DIS dataset under the $S^0$ (or $S^1$) setting—our model consistently shows improvements in both per-class IoU and mIoU. This is particularly evident in classes with complex structures, such as "beam" and "table", where performance increases from 32.06% to 63.50% and from 53.22% to 72.39%, respectively.

(2) Connections with LLM: The superior performance of our model on complex structures can be attributed to the introduction of LLM, which can generate more diverse and distinguishable visual or geometric descriptions for classes with complex structures, resulting in higher performance. Conversely, for classes with flat structures, LLM produces less rich descriptions that struggle to capture subtle differences, resulting in not as good performance.

Once again, we thank you for your positive feedback, constructive criticism, and thoughtful suggestions. Your feedback plays an integral role in refining our work.