Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes

Q1: Architecture details for reproduction and code for clarification.

Thanks for your valuable suggestions. In Appendix D.2 (pp. 19), we provided additional architectural details of our framework. Specifically, following AGILE3D, the point encoder in Figure 1 consists of a backbone—Minkowski Res16UNet34C—along with an attention module for click-scene interaction. The architectural components related to Neural Processes (NPs) are depicted in Figure 1 and detailed in Sec. 4.2–4.4, including details about scene-level and object-level aggregators (Sec.4.2), probabilistic prototype modulator (Sec.4.3), and training objectives (Sec.4.4).

To better illustrate the hierarchical NP structure, we further provided a graphical model at the following anonymous link: https://github.com/anonusers2025/rebuttal_figure/blob/main/graphical_model.pdf. This figure offers a more visual and accessible explanation of how our model integrates scene-level and object-level latent variables for probabilistic interactive 3D segmentation.

We will further improve the description about architecture details in the final version to facilitate reproduction and understanding. We will also release our code to facilitate implementation and future research.

Q2: Discuss and analyze the reasons behind minor improvements on the ScanNet.

We thank the reviewer for the comment. Our model is exclusively trained on ScanNet, and we observe relatively modest improvements over AGILE3D on this benchmark. This is expected for several reasons.

First, all baseline models and ours are trained on ScanNet, and these models have already seen a large number of structurally similar indoor scenes during training, the segmentation task becomes relatively easy in this in-domain setting. As a result, strong baselines like AGILE3D already perform very well with only a few user clicks, e.g., reaching 82.3% IoU after 5 clicks, and the performance quickly saturates. Under such conditions, the benefits of our probabilistic design are less pronounced, and the potential for further improvement is naturally limited.

Second, our method is specifically designed to handle uncertainty and model generalization through a probabilistic framework with hierarchical latent variables. These advantages are less evident in in-domain settings such as ScanNet, where the model benefits from exposure to similar scenes during training. In contrast, they become significantly more valuable when applied to challenging, unseen, and out-of-domain scenarios. For example, on KITTI-360, which features unstructured outdoor LiDAR scans with large domain gaps, NPISeg3D achieves +10.9% and +9.8% mIoU improvement over AGILE3D under single-object and multi-object settings, respectively. Similar trends are observed on S3DIS and Replica.

These results highlight that while our method shows modest gains in in-domain settings like ScanNet, it delivers substantial benefits under more realistic, ambiguous, and out-of-domain conditions—where generalization and uncertainty modeling are crucial.

Q1: Computational efficiency and Reliability of uncertainty estimation.

Thank you for your valuable comments. Below, we address each aspect in turn.

Computational efficiency. Our NPISeg3D introduces negligible extra parameters introduced by our neural process module which enhances generalization and enables reliable uncertainty modeling. As shown in Table 11 of the Appendix (pp. 14), NPISeg3D has 40.0MB of parameters, compared to 39.3MB in the previous SoTA AGILE3D. This amounts to only a 1.8% increase, which is minimal. Despite this slight overhead, NPISeg3D delivers significantly improved segmentation performance. For instance, It surpasses AGILE3D by 11.3% mIoU with 5 clicks under the single-object setting on KITTI-360, demonstrating that the modest computational cost is well justified by the substantial performance gains.

Reliability of uncertainty estimation. To evaluate the reliability of our model’s uncertainty estimation, we provide extensive qualitative results in Figures 6 and 7 of the supplementary material. In particular, the first two rows of Figure 6 show that after a single click, the uncertainty map effectively highlights erroneous regions and object boundaries—areas that are inherently ambiguous in interactive segmentation. These results demonstrate that our method produces meaningful and reliable uncertainty estimates even with very few clicks. Moreover, as the number of clicks increases, we observe a clear reduction in uncertainty alongside improved segmentation quality, further validating the robustness and interpretability of the predicted uncertainty.

Q2: Extension to other 3D representations, such as 3DGS. And potential integration with open-vocabulary 3DGS semantic segmentation methods, such as GOI and Chatsplat.

We appreciate the reviewer’s insightful comment. Our current method is primary designed for point cloud inputs, which provide a simple and efficient representation that generalizes well across diverse scenes without requiring scene-specific optimization.

Meanwhile, we also agree that extending interaction to other 3D representations, such as 3D Gaussian Splatting (3DGS), is a highly promising direction. However, direct interactive segmentation in 3DGS is non-trivial: individual Gaussians lack explicit semantic meaning and are optimized per scene, making object-level interaction challenging. Recent works, such as Click-Gaussian [1], GaussianCut [2], and ISRF [3], have explored propagating 2D user interactions into 3D via dense multi-view supervision. These approaches typically rely on 2D-view-based interactions and project multi-view masks into 3D space.

Inspired by these methods, our method could be extended to such settings by first generating 2D masks from user clicks using our probabilistic framework, i.e., hierarchical neural process structure, then lifting these masks into 3DGS space using known camera poses or depth maps. This could enable selecting or reweighting Gaussians based on segmented regions, facilitating interactive segmentation without requiring per-Gaussian semantic labels or retraining.

Regarding integration with open-vocabulary 3DGS segmentation, methods such as GOI and ChatSplat incorporate language-guided semantics into 3DGS. We believe combining interactive inputs (e.g., clicks or referring expressions) with open-vocabulary reasoning is a compelling future direction. For instance, user interactions could guide the adaptation of semantic hyperplanes or modulate Gaussian importance weights during inference.

Although 3DGS and other 3D representations are not the main focus of our current work, we will include the above discussions in the final version to provide a broader perspective. We also find it an interesting direction to explore how our probabilistic interactive framework could be adapted to 3DGS pipelines to support language-driven 3D interaction in future work.

[1] Choi, Seokhun, et al. "Click-gaussian: Interactive segmentation to any 3d gaussians." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[2] Jain, Umangi, Ashkan Mirzaei, and Igor Gilitschenski. "GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting." The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.

[3] Goel, Rahul, et al. "Interactive segmentation of radiance fields." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Q1: Limitations in Out-of-Domain Generalization. (1) Out-of-domain performance falls short in-domain. (2) The paper does not explore domain adaptation techniques.

Thank you for your insightful comments. Similar to previous methods like AGILE3D, our model is trained solely on ScanNet and evaluated on out-of-domain datasets such as KITTI-360 to assess its generalization capability. Due to the significant domain gap between ScanNet (indoor RGB-D scenes) and KITTI-360 (outdoor LiDAR scenes), segmentation on KITTI-360 remains inherently challenging. As a result, even SOTA method like AGILE3D achieves only 44.4% mIoU after 5 clicks under the single-object setting.

Nevertheless, NPISeg3D achieves substantial improvements over previous state-of-the-art (SoTA) methods on KITTI-360. Specifically, it improves mIoU by 10.9% and 9.8% over AGILE3D under the single-object and multi-object settings, respectively, demonstrating the robustness of our approach even under large distribution shifts.

To further enhance out-of-domain performance, one feasible solution is to consider domain adaption techniques, e.g, domain-specific fine-tuning. As shown in the table below, our model—when fine-tuned on KITTI-360—achieves significantly better performance across all metrics, approaching levels that are practical for downstream tasks. For example, mIoU@5 improves from 44.0% to 79.2%, and the number of clicks (NoC@90) required to reach 90% IoU decreases from 17.6 to 12.3. These results demonstrate the strong adaptability of our method to specific domains when fine-tuning data is available.

We agree that domain adaptation is a valuable and complementary direction for interactive 3D segmentation, and will consider more advanced domain adaptation techniques such as parameter-efficient fine-tuning and test-time adaption in future research.

Q2: Computational Complexity & Scalability.

As shown in Table 11 of the Appendix, our method introduces only marginal computational overhead due to its probabilistic modeling. The parameter size increases by just 1.8% (40.00MB vs. 39.30MB), and the FLOPs remain comparable (4.94G vs. 4.73G). Although Monte Carlo sampling with 5 samples adds some overhead, both sampling and decoding are fully parallelized. As a result, our method achieves an inference speed of 65 ms per forward pass, closely matching AGILE3D’s 60 ms and supporting real-time interaction.

Moreover, our iterative and random training strategy accelerates training by approximately 2.5× compared to AGILE3D, further highlighting the scalability of our approach for large-scale datasets.

Despite this slight overhead, our method delivers significant performance improvements. For instance, on the high-resolution LiDAR dataset KITTI-360, it surpasses AGILE3D by 11.3% mIoU with just 5 clicks under the single-object setting, demonstrating that the modest computational cost is well justified by the substantial performance gains.

Q3: Clarity & Accessibility of Technical Concepts. (1) Rigorous math of Neural Processes may limit accessibility for non-experts. (2). The explanation of the hierarchical NP structure could be improved.

Thank you for the helpful suggestion. We fully agree that it is important to make the mathematical formulation more intuitive and accessible, especially for readers who are less familiar with Neural Processes (NPs). In the final version, we will revise the exposition to improve clarity and incorporate more intuitive explanations where appropriate.

To better illustrate the hierarchical NP structure, we further provided a graphical model at the following anonymous link: https://github.com/anonusers2025/rebuttal_figure/blob/main/graphical_model.pdf. This figure offers a more visual and accessible explanation of how our model integrates scene-level and object-level latent variables for probabilistic interactive 3D segmentation. We will further improve and refine this part in the final version to enhance readability and understanding.

Q1: Include discussion of some interactive 3D segmentation methods that translate SAM features into 3D such as SAM3D and SA3D.

Thank you for suggesting these relevant interactive 3D segmentation works, SAM3D and SA3D, both of which explore translating 2D SAM features into 3D. Specifically, SAM3D leverages the pretrained SAM to automatically generate 2D masks in RGB images, and then maps these masks into 3D point cloud using pixel-wise depth from RGB-D images—without requiring additional training or fine-tuning. Similarly, SA3D utilizes radiance fields as an off-the-shelf prior to bridge multi-view 2D images and 3D space. It first generates a 2D mask using SAM on a single view, and then performs mask inverse rendering and cross-view self-prompting to iteratively refine the 3D mask of the target object across multiple views.

In contrast, our approach directly tackles native 3D interactive segmentation by taking 3D interactive clicks and point cloud inputs without relying on 2D projections. We view SAM3D, SA3D, and our method as complementary contributions that collectively advance 3D interactive segmentation.

We will incorporate a discussion of these works in the final version to provide a more comprehensive perspective on related approaches.

Q2: Supplementary materials issues.

We kindly clarify that the Appendix (supplementary materials) was included in the submission (pp. 12–20). In Appendix, we provided the ELBO derivation (Sec. A), additional quantitative and qualitative results (Sec. B) to show the effectiveness of our method across different settings, user study details (Sec. C) to demonstrate the usability in practical scenarios, and implementation details (Sec. D) such as model structure and click simulation strategy.

Q3: More qualitative results on diverse inputs.

Thanks for this valuable comment. In Appendix (pp. 12–20), we provided extensive qualitative comparisons in Figures 4–7, showcasing diverse input scans and click prompts. Specifically, Figure 4 illustrates segmentation masks generated with varying numbers of clicks in the single-object setting, while Figure 5 presents results for multi-object segmentation. Figures 6 and 7 further display segmentation masks and uncertainty maps across different click counts for single- and multi-object cases. These examples cover challenging scenarios involving varying object complexities and sparse user inputs, highlighting the robustness and generalization capability of our method.

Q4: Performance falls short on challenging out-of-domain datasets like KITTI-360.

Thank you for your feedback. Similar to prior methods such as AGILE3D [1] and Inter3D [2], our model is trained exclusively on ScanNet and evaluated on out-of-domain datasets like KITTI-360 to evaluate generalization. The substantial domain gap between ScanNet (indoor RGB-D scenes) and KITTI-360 (outdoor LiDAR scenes) inevitably leads to suboptimal performance on KITTI-360. As a result, even SOTA method like AGILE3D achieves only 44.4% mIoU after 5 clicks under the single-object setting.

Nevertheless, our method demonstrates significant improvements over previous SOTA approaches. Specifically, our NPISeg3D improves mIoU by 10.9% and 9.8% over the previous SoTA AGILE3D on KITTI-360 under the single-object and multi-object settings, respectively, highlighting its effectiveness even under large distribution shifts.

To enable effective deployment in downstream tasks, one feasible solution is to consider domain adaption techniques, e.g, domain-specific fine-tuning. As shown in the table above, our model—when fine-tuned on KITTI-360—achieves significantly better performance across all metrics, approaching levels that are practical for downstream tasks. For example, mIoU@5 improves from 44.0% to 79.2%, and the number of clicks (NoC@90) required to reach 90% IoU decreases from 17.6 to 12.3. These results demonstrate the strong adaptability of our method to specific domains when fine-tuning data is available.