GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model

Q1. working mechanism of Point Prompt

As for working mechanism of Point Prompt, it can be analyzed through Equation 18 and 19 in paper. Previous prompting methods operate at token level, which corresponds to local patches, failing to adjust exact points within patches.

In contrast, our Point Prompt directly prompts within patches, operating at point-level grains. When adapting to downstream, it is optimized via task loss to focus on critical points that encode subtle yet essential geometric information.

As for visualization, the learned Point Prompts are already present in Figures 5 and 6. However, as there are only 20 prompts within 2048 points, may difficult to see. In final version, we will explicitly highlight learned Point Prompts in visualization for better clarity.

W1. ablation study on Point Prompt in Table 3

We supplement additional ablation results on "GAPrompt without Point Prompt" as shown below. Incorporating the learnable Point Prompt yields an 1.02% gain in accuracy.

Q2,W2. comparison with PPT

Thanks for your advice, and we will add it to main table in final version. The primary reason we did not initially include this comparison is that PPT hasn't been officially published and only available on arXiv. Even though, we compare on 4 different representative backbones across 4 datasets, and our GAPrompt achieves 14 higher performances in 16 experiments with fewer trainable parameters(0.6M), due to our lightweight geometry-aware prompting design.

W3,Q3. few-shot and part segmentation experiments

We supplement few-shot experiments on ModelNet40 and segmentation experiments on ShapeNetPart below. We compare our method with other SOTA works based on pre-trained Point-FEMAE. It can be seen that our method performs well in few-shot setting and can generalize to segmentation tasks well, verifying the efficacy of GAPrompt.

Few-shot on ModelNet40

Result in ShapeNetPart

Q4. difference before and after Self-Attn in Fig. 4

The difference before and after Self-Attention arises from distinct inference orders. This is similar to choosing between "Attn→FFN" or "FFN→Attn" in Transformer models. The Prompt Propagation mechanism enhances spatial information, further facilitating interactions between prompts and point tokens. When applied after Self-Attention, it provides stronger performance improvements.

Q5a. determination of Point Prompt number

The number of Point Prompts is set proportionally to the input point cloud resolution, typically 0.5%-1% of the dataset resolution. For clean datasets (e.g., ModelNet40, OBJ_ONLY), 0.5% yields slightly better results (+0.1%), corresponding to 5 and 10 prompts for 1024 and 2048 points, respectively. While for noisier datasets (e.g., OBJ_BG, PB_RS_T50), 1% provides a small gain (+0.2%), resulting in 20 prompts for 2048 points.

Q5b. setting of $L_p$ and $p_i^{\prime}$

$L_p$ is set to 10, as shown in the ablation study below. When $L_p$ is too small, the prompting effect is insufficient, leading to suboptimal performance. Conversely, an excessive $L_p$ does not yield further improvements while incurring additional parameter overhead. As for $p_i^{\prime}$, we use Kaiming Uniform initialization.

Thanks for your appreciation and valuable advice!

Q1. inference time and FLOPs comparison

We test the inference time and FLOPs of our method and other SOTA methods, including IDPT[ICCV23], DAPT[CVPR24] and Point-PEFT[AAAI24]. All experiments are conducted on a RTX 4090.

As table shown below, our method attains the fastest inference time with only 11.1 millisecond per sample and fewest FLOPs with 5.0G MACs among parameter-efficient works. Notably, our method merely introduces 4% computational overhead but brings over 97% reduction in trainable parameters.

Q2. number of the enhanced prompt tokens

The number of enhanced prompt tokens in our experiments is set to 10 as a hyperparameter. We provide additional ablation experiment results below and will add it into final version for better interpretability.

We find that our method produces peak performance when adopting 10 enhanced prompt tokens. A smaller number leads to suboptimal prompting effects due to insufficient guidance, while an excessive number does not yield further improvements but incurring additional computational cost.

C1. citation to PointGST and PPT

Thanks for your reminder and we will cite them in final version. Notably, at the time of our submission and even at present, both works remain as preprints on arXiv and have not been officially published. However, our method still excels both in terms of accuracy and efficiency.

We compare with them on four dataset based on four representative backbones, as shown below. GAPrompt achieve 12 SOTA results while PPT and PointGST get 2 SOTA results respectively. And our method has only 0.6M trainable parameters, attaining highest parameter efficiency, attributed to our geometry-aware point-level prompting design.

Q1,W1. computational cost of multi-resolution grouping

Multi-resolution grouping of Point Shift Prompter can hardly become a bottleneck for real-time applications.

Although multi-resolution FPS/KNN operation has $O(N^2)$ complexity, its overhead remains minimal compared to $O(N^2)$ attention mechanism and expensive MLPs, because the feature size of FPS/KNN is 3, far less than the 384-dimensional model features.

We supplement a breakdown of computational costs. As seen, our three modules collectively account for less than 2% of total computational cost, with majority stemming from encoder and attention mechanism.

Q2. pretraining dependency

In addition to backbones already introduced (mask-based Point-MAE and contrastive pre-trained ReCon), we supplement additional Point-BERT. Notably, in Table 1 of the paper, ReCon is exactly pre-trained via contrastive learning and our method surpasses full fine-tuning with only 1.38% trainable parameters.

Based on Point-BERT, GAPrompt still attains highest performance with lowest trainable parameters, verifying its generalizability to pre-trained backbones.

Q3. density variations

We supplement experiments in extremely sparse inputs condition, at 128 input resolution on ModelNet40. While all methods faces a performance drop, GAPrompt consistently outperforms other PEFT methods, demonstrating robustness to varying input densities.

Q4. latency analysis

We measure inference time on ScanObjectNN using an RTX 4090, as shown below. GAPrompt incurs only 0.9ms additional latency, accounting for less than 9% of the base 10.2ms required by Point-MAE. Furthermore, our inference time is less than other PEFT methods, benefiting from lightweight point-level prompting design.

W2. task specificity

We supplement additional results on part segmentation dataset ShapeNetPart and semantic segmentation dataset S3DIS. Even though such dense prediction tasks are challenging, our method excels previous SOTA methods including DAPT[CVPR24] and PointPEFT[AAAI24] in both efficiency and efficacy.

Results in ShapeNetPart

Results in S3DIS

W3. initialization sensitivity

Our point prompt initialization exhibits robustness on non-uniform LiDAR data. In our Table 1 experiments, the three ScanObjectNN variants originate from real-world LiDAR scans, which inherently contain non-uniform distributions and background fragments. Despite this, GAPrompt still achieves SOTA results, even surpassing full fine-tuning, using less than 2% trainable parameters.

S1. potential method enhancements

Thanks for your insightful suggestions!

As classic algorithms with $O(N^2)$ complexity, replacing FPS/KNN with sparse point cloud operations can further enhance efficiency.

As for adaptive weighting between shape features and prompts, current hyperparameter setting relies human expertise. We agree adaptive weighting would be a better choice, deserving future exploration.

S2. theoretical analysis advice

Thanks for your valuable advice. We will provide additional discussions on robustness of point shifts against adversarial perturbations and Point Prompt initialization strategies in final version.

R1. comparison with PPT and PointGST

Although PPT and PointGST are currently available only as preprints on arXiv and have not been officially published, we supplement comparison experiments as shown below.

The results across four datasets and four representative backbones illustrate that GAPrompt achieves 12 SOTA results while PPT and PointGST each attain 2. Furthermore, our approach requires only 0.6M trainable parameters, the highest parameter efficiency.

W1. innovation and difference against PPT and PointGST

Our GAPrompt differs from these methods in two key aspects.

1. Finer-grained prompting: PPT operates on position encodings of point tokens, as a coarse-grained token-level prompting approach. In contrast, GAPrompt introduces fine-grained point-level prompting via the Point Prompt and Point Shift Prompter, allowing more precise and adaptive feature modulation at the individual point level.

2. Interpretability and geometric awareness: PointGST introduces a Spectral Adapter, transforming point tokens from spatial domain to spectral domain, which belongs to adapter methods. But GAPrompt belongs to prompt methods, avoiding obscure spectral domain transformations and ensuring stronger interpretability and geometric awareness.

W2. part or semantic segmentation tasks

We supplement additional segmentation results on both ShapeNetPart and S3DIS datasets.

It can be seen that our GAPrompt excels other methods including the arXiv twos. We achieve six SOTA metrics across 2 datasets and 2 backbones with highest parameter efficiency, only 0.3M additional parameters beyond 5.2M of downstream head.

Results in ShapeNetPart

Results in S3DIS

W3. a disguised way of increasing dataset size, data augment, unfair

Our Point Prompt is not data augmentation and the setting is absolutely fair, cause the inputs are same for all methods. The reason is three-folded:

First, the Point Prompts are randomly initialized, rather than additional sample points from each instance. So, no training nor testing data leakage.

Second, Point Prompts are fixed after trained, adapting to a specific domain and intensifying discriminant information, not varying for each instance.

Finally, comparing to the 1024 points input, 20 prompts are less than 2%. We provide results on ModelNet40 at 1044 resolution. Nothing changes from 1024 points.

W4. point prompt number as zero in Figure 7

We add more ablation on point prompt number $P$ as below. When $P$ is zero, it equals to drop Point Prompt module. However, we still attains 89.65% with only other two modules, surpassing DAPT's 88.51% and Point-PEFT's 89.35%. This is attributed to the point-level adaptation of Point Shift Prompter and prompt enhancement of Prompt Propagation, while further gain can be achieved with incorporation of Point Prompt.

S1. data symbols

Thanks for advice, we will detail the symbols in Sec. 3.2 and 3.3 and check out other symbols.