Towards 3D Objectness Learning in an Open World

(Question 1) Clarification on the use of "prompt-free"

We thank the reviewer for clarifying this. We will revise the paper to explicitly state that our method is semantic prompt-free, i.e., it requires no category labels, text prompts, or semantic priors at inference time.

While we do utilize point-grid prompts (e.g., 64x64 uniform sampling) to interact with SAM, these prompts are purely spatial and entirely semantic-agnostic, meaning they carry no category or textual information. This design intentionally avoids biasing the model toward any specific object class and aligns with our class-agnostic manner. We will revise the manuscript to explicitly clarify that our approach is semantic prompt-free.

(Weakness 1 and Question 2) Is the 3D Object Discovery phase offline or online?

The entire 3D object discovery phase, including SAM inference, multi-scale point sampling, and LDET refinement, is performed offline as a one-time pre-processing step before training. This design avoids any runtime overhead during inference, and does not impact inference latency or memory usage. In practice, we process 5,285 training samples using 4 RTX 3090 GPUs in parallel, each applying one 3D proximity threshold. On average, each GPU consumes 7,877 MB of memory, and each sample takes approximately 3.98 seconds to process. This step is decoupled from the detection pipeline. We will clarify this in the revised version. It can be seen that the inference time is not long, and measures that support parallelization can further accelerate the process, making this part of the offline algorithm relatively easy to apply to large-scale scenarios.

(Question 3) Detailed qualitative or quantitative analysis of the multi-modal router

We thank the reviewer for the insightful suggestion. To better understand the behavior of our cross-modal Mixture-of-Experts (MoE) router, we conducted a quantitative analysis of expert weights across different object and scene types. The router assigns dynamic soft weights to the semantic (RGB), geometric (point cloud), and fused experts based on local reliability. We observe that in high-texture regions (e.g., books, monitors), the semantic expert is predominantly selected (averaging 62% of the weight), while in sparse or textureless scenes (e.g., boxes, outdoor structures), the geometric expert dominates (58%). In cluttered or occluded indoor environments, where neither modality is fully reliable, the fused expert contributes the highest share (47%), enabling the model to integrate complementary cues. These findings validate the router's ability to adaptively leverage different modalities under varying 3D conditions. We will include visualizations and detailed statistics in the supplementary material.

(Weakness 2) Impact of 2D-to-3D Projection Error

We acknowledge the reliance on camera parameters for 2D-to-3D projection. Inaccuracies caused by sensor noise or imperfect calibration can introduce projection errors, potentially leading to mislabeling background points as foreground and generating fragmented or incomplete 3D supervision signals. To address this, we apply frustum-based filtering followed by clustering to remove outliers and suppress background points. Moreover, our cross-modal MoE module adaptively balances semantic (2D) and geometric (3D) cues, enabling the model to down-weight unreliable 3D inputs and mitigate the impact of projection noise during training.

In general, our method successfully localizes a wide range of objects, including those with challenging spatial arrangements such as closely placed or partially occluded instances. This is enabled by the joint use of 2D semantic priors and 3D geometric cues. Nonetheless, we observe occasional misses in cases involving objects with deformable surfaces, low color and texture contrast. This is likely caused by occasional failures of SAM and the 2D detector. We will include analyses of such cases in the revised version.

(Question 1) Defining 3D Objectness and distinguishing true objects from background clutter

In our work, 3D objectness denotes the likelihood that a spatial region corresponds to a physically discrete object, distinguishable from background structures or noise, regardless of semantic category. Specifically, denote the input features as $F_{input}$, a learnable model that maps input features to an objectness confidence score as $\phi(\cdot)$, this process can be denoted as: $I \left[ \phi(F_{input}) > \tau \right]$, where $\tau$ is the threshold for foreground-background separation; I$[\cdot]$ is the indicator function (1 for foreground, 0 for background). Our model $\phi$ is trained to approximate this decision function, assigning high scores to foreground objects and low scores to background or clutter.

In autonomous driving or robotic applications, it is true that certain complex scenarios make distinguishing foreground objects more challenging. However, our OP3Det method is designed to mitigate this issue as much as possible in multiple ways. On the one hand, for obstacle avoidance in autonomous driving, rare and novel categories are inevitable in real-world scenarios, making it difficult to collect and label all object classes during training. In this context, the ability of OP3Det to detect new classes in an open-world setting can greatly assist the detector in improving obstacle avoidance, leading to a more robust and reliable system. To this end, we conducted experiments on nuScenes, a dataset focused on autonomous driving, treating "car, trailer, construction vehicle, motorcycle, bicycle" as seen classes and the remaining five as unseen classes. We measured AP for both novel classes and all classes, with the results presented in the table below. As can be seen, our method outperforms previous state-of-the-art closed-vocabulary and open-vocabulary detectors, effectively demonstrating its potential in autonomous driving applications.

On the other hand, in the case of sparse KITTI point clouds, the multimodal fusion and the MoE architecture enable the $F_{input}$ to adaptively select information from different modalities (point cloud, RGB images, and multimodal data), allowing downstream tasks such as obstacle avoidance to be minimally affected by issues in any single modality. With these designs, the learned model $\phi(\cdot)$ can be more robust, making $I \left[ \phi(F_{input}) > \tau \right]$ more accurate.

(Weaknesses 1) On the Coherent Integration and Novelty of Our Pipeline Design

(1) Task motivation:
Our goal is to detect all objects in a 3D scene, including unseen categories. This shifts the challenge from category recognition to learning a generalized notion of “3D objectness.” Achieving this is non-trivial, especially under data scarcity and incomplete 3D information. While 2D data benefits from abundant labels and well-established foundation models, 3D data lacks both. This motivates us to exploit 2D semantic richness to support 3D generalization.

(2) Methodological design:
Rather than a patchwork, each module is purposefully chosen to solve a core sub-problem toward the above objective. SAM provides class-agnostic 2D priors, which we project and refine using a multi-scale point sampling strategy to robustly localize 3D candidates, resulting in a 3.6% improvement in AR. To address the inherent sparsity and occlusion in point clouds, we introduce multi-modal fusion. However, naive fusion methods (e.g., addition or concatenation) yield suboptimal results. Our Cross-Modal MoE module dynamically routes information between modalities based on local reliability, outperforming static fusion by +13.3% $AR_{novel}$.

Each design decision is systematically motivated by the goal of learning robust, category-agnostic 3D objectness—not merely combining components, but integrating them to overcome the unique challenges of 3D open-world detection.

(Weakness 2 and Question 2) Threshold $\tau$ Selection and Generalization Across 3D Scenes

(1) Choice of sampling thresholds.
Our multi-scale point sampling is designed to improve robustness across diverse 3D scenes. By using a set of 3D proximity thresholds ($\tau$ = {0.2, 0.5, 1, 2}), we avoid overfitting to a particular point cloud density. This design ensures that both small, dense objects (common in indoor environments) and large, sparse structures (typical in outdoor scenes) can be effectively sampled. We conducted ablation studies varying $\tau$ and found that this particular set yields the best, regardless of whether for indoor or outdoor scenes, as shown in the table below. We also found that the performance is relatively robust to the choice of $\tau$. This demonstrates that our multi-scale design is not arbitrarily chosen, but tuned for generality for both indoor and outdoor scenes.

(2) KITTI performance and cross-dataset generalization.
We agree that KITTI poses unique challenges—its limited category diversity and the significant modality gap between RGB and LiDAR reduce the utility of 2D priors. Despite this, benefiting from the 3D object discovery process, together with the generalizable sampling thresholds, our method still generalizes well when transferred to other outdoor datasets. Specifically, when trained on KITTI and evaluated on nuScenes, OP3Det continues to achieve higher AP$_{all}$ compared to standard baselines, validating the robustness of our sampling and fusion strategy across domains.

(3) Multi-scale point sampling algorithm pseudocode.
We include the pseudocode for our multi-scale point sampling process in the supplementary material.

(Weaknesses 3) On the Role of SAM

Due to SAM's strong robustness, we chose to use it. Issues such as fragmented masks are common problems for segmentation models like SAM because they do not consider the concept of “objects” or objectness during training. Therefore, the problem we address is a fundamental and general one, rather than being closely related to scene complexity. To tackle this, we employ sampling and a 2D class-agnostic detector trained on large-scale data. Such 2D detectors typically exhibit good robustness as well, enabling them to fundamentally filter out these fragmented masks. As can be seen, since both the problem and the solution we consider are fundamental rather than specific to failures in particular scenarios, our method demonstrates generality and strong generalization ability in complex environments.

(Weaknesses 4) The Adaptive Behavior of the MoE Module

While our MoE adopts a lightweight architecture—attention followed by a softmax-based routing layer—this simplicity does not undermine its effectiveness. Crucially, its strength lies in context-aware expert selection across modalities. To assess the rationality of the gating mechanism, we conducted a quantitative analysis of the expert weight distribution across diverse scenes and object types. Results demonstrate clear and meaningful patterns: the semantic expert is favored in high-texture scenes (averaging 62% of the weight), the geometric expert is prioritized in sparse or textureless regions (58%), and the fused expert dominates in cluttered indoor environments (47%). These trends are consistent with intuition and confirm that the router does not apply a static weighting scheme, but rather dynamically adjusts based on the local reliability of each modality. This empirical evidence substantiates the MoE's adaptive behavior. We sincerely thank the reviewer for raising this insightful concern and we will include this analysis in the revised version of the paper.

(Weakness 5 and Question 3) Fairness and Integrity of Baseline Adaptations in Class-Agnostic 3D Detection

We have already carefully modified existing methods like VoteNet, CoDA, and OV-Uni3DETR to align with our proposed open-world, class-agnostic detection setting. Specifically, for closed-vocabulary baselines like VoteNet, we retrained by replacing the original classification heads with a class-agnostic head. For open-vocabulary baselines like OV-Uni3DETR, we tested various generic textual prompts (e.g., “object”, “thing”, “entity”) and also utilized a class-agnostic head.

We also compare our method with other class-agnostic 3D detection approaches. The mentioned OpenScene[2] and 3D-OVS[3] are both semantic segmentation methods, which are difficult to adapt to object detection. Therefore, we compare with CuTR[1], a recent method for category-agnostic 3D object detection, which aligns closely with our setting. We re-implement this method and list the comparative results in the table below. As can be seen, our method significantly outperforms CuTR across all metrics. This highlights the superiority of our approach in discovering diverse, novel 3D objects without category labels, validating the effectiveness of our objectness-based design.

Reference

[1] Cubify Anything: Scaling Indoor 3D Object Detection, CVPR 2025

[2] OpenScene: 3D Scene Understanding with Open Vocabularies, CVPR 2023

[3] Weakly Supervised 3D Open-vocabulary Segmentation, NeurIPS 2023

(Weakness 1) Task Definition & Contextualization for "Open-World Prompt-free 3D Detection"

(1) "Class-agnostic detection" refers to the task of detecting object instances without predicting their specific category labels. It simply determines whether a region contains an object. "Open-vocabulary detection" assumes access to category names or prompts and aims to recognize objects from an open set, often including unseen classes. "Open-world detection" emphasizes generalization to unknown categories. Our proposed task, open-world prompt-free 3D detection, is a class-agnostic formulation that avoids any use of semantic labels or prompts. Instead, it focuses on learning 3D objectness with the goal of detecting all object instances, especially novel ones. Grounded in 3D detection, it contributes to the broader vision of open-world learning through a semantic-free, discovery-driven perspective.

(2) Learning 3D objectness poses fundamentally different challenges compared to 2D objectness. Unlike in 2D vision, where object boundaries are often well-defined in dense pixel grids, 3D point clouds are sparse, noisy, and incomplete due to occlusions and sensor limitations, making it significantly harder to infer objectness. Furthermore, 3D datasets are scarce, manually annotated, and limited in category diversity, in contrast to the abundance of large-scale image-text datasets available for 2D. These fundamental differences make the direct transfer of 2D open-world methods like Detic or OWL-ViT to 3D neither straightforward nor effective. We will clarify these in the revised version.

(Weakness 2) Superficial Treatment of 3D-Specific Challenges

While our original submission emphasized data scarcity and modality fusion, our model was in fact designed with these 3D-specific challenges, aiming to better learn 3D objectness. For the sparsity problem, we utilize a voxel-based architecture that supports learning from sparse point clouds. For occlusion and foreground-background ambiguity, we incorporate RGB images that anchor dense 2D semantics into 3D space. The multi-modal paradigm allows the model to effectively resolve incomplete geometry, missing parts, and ambiguous object boundaries, which fundamentally motivates our choice of a multi-modal framework.

We also emphasize that transferring objectness from 2D to 3D is non-trivial. Unlike dense RGB pixels, 3D data is incomplete, lacks uniform resolution, and contains structural noise, making direct application of 2D paradigms inadequate without careful cross-modal alignment and fusion.

(Weakness 3) Insufficient Baseline Comparisons

To further demonstrate the effectiveness of our OP3Det, we re-implement more recent methods as baselines and compare them, as shown in the table below. The 2D open-world method DetPro[1], when combined with 2D-to-3D projection, underperforms compared to our OP3Det (72.4% vs. 78.8%). This is because the projection of 2D masks into 3D introduces information loss and geometric errors, and such methods fail to exploit the complementarity of RGB and point cloud modalities. Multi-view 3D detectors such as ImVoxelNet[2] and NeRF-Det[3] also yield much lower performance (34.2% and 37.5%, respectively), as they rely solely on 2D images. These methods do not utilize 3D point clouds, where abundant 3D information exists, making them ill-suited for precise 3D object detection under open-world conditions. We also include PointContras[4], a representative 3D self-supervised approach, which achieves AR$_{novel}$ of 73.3% but is still outperformed by OP3Det. This is because PointContrast is designed for representation learning, not open-world object detection, and lacks both explicit 3D objectness modeling and localization knowledge. Together, these results demonstrate that OP3Det is not merely “SAM + projection,” but a carefully designed framework that addresses the core challenges of 3D open-world detection over multi-modal inputs.

(Weakness 4) Failure to Explore MLLM-Based Alternatives

(1) Existing MLLMs such as LLaVA-3D[6] and 3D-LLM[7] are not directly applicable baselines for our task. Despite recent progress, most of the existing 3D MLLMs still cannot accept raw point clouds as input. Instead, they rely on intermediaries such as multi-view images or depth images to process 3D information. Therefore, in terms of input modality, these methods are difficult to apply directly to our scenario.

(2) We argue that current MLLMs are typically not suitable tools for generating dense region proposals in our setting. These models are inherently prompt-driven and operate over 2D image inputs. Given that a user query is conditioned on language, they typically output only a small number of semantic-level bounding boxes or region captions and cannot generate dense region proposals with confidence scores required for 3D object discovery. In comparison, our method targets prompt-free, class-agnostic 3D object discovery by generating hundreds to thousands of candidate regions per scene to achieve high recall, a capability that current MLLMs are not equipped for.

(3) While MLLMs excel at vision-language tasks, they lack the ability to perform fine-grained, token-level fusion of RGB and point cloud features. Our MoE module is specifically designed for this purpose, enabling spatially adaptive fusion based on scene context. It dynamically adjusts modality weights, which is crucial for accurate, prompt-free 3D detection.

(Weakness 5) Inadequate Ablation Studies

(1) Models beyond SAM. While our pipeline is built on SAM due to its strong generalization ability and wide adoption, we agree that evaluating alternative or improved segmentation backbones can provide additional insight. To this end, we conducted an experiment replacing the default SAM with HQ-SAM[5], a recent enhanced variant that introduces special tokens and global-local feature fusion for improved mask quality. All other components of the pipeline were kept unchanged. These findings validate the flexibility of our pipeline and indicate that stronger 2D priors can translate into better 3D detection performance.

(2) Necessity of the 3D point cloud modality. To assess the indispensability of 3D input, we ablate the point cloud modality and only use RGB images as inputs. As shown in the Table, removing 3D leads to a significant performance drop (e.g., AR$_{novel}$ drops from 74.4% to 38.4%), especially on novel categories. This confirms that 2D alone lacks sufficient spatial and structural information to model 3D objectness in open-world 3D scenes, where distinguishing foreground from cluttered backgrounds is particularly difficult.

(3) Dynamic fusion (MoE) vs. static fusion. To validate our fusion design, we compare our Cross-Modal Mixture-of-Experts (CM-MoE) with static strategies like addition and concatenation. As shown in the Table, CM-MoE consistently outperforms the baselines (e.g., AR$_{novel}$: 74.4% vs. 66.0%/65.4%), demonstrating its ability to dynamically adjust to modality-specific reliability.

(4) 2D detector post-processing. As shown in the Table, adding the 2D class-agnostic detector (LDET)[8] as a post-processing step to SAM + multi-scale point sampling yields clear performance gains:
AR$_{novel}$ improves from 61.9% to 66.1%, AR$_{all}$ from 54.2% to 59.2%, and AP from 7.6% to 10.0%.

Reference

[1] Learning To Prompt for Open-Vocabulary Object Detection With Vision-Language Model, CVPR 2022

[2] ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection, WACV 2022

[3] NeRF-Det: Learning Geometry-Aware Volumetric Representation for Multi-View 3D Object Detection, ICCV 2023

[4] PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding, ECCV 2020

[5] Segment Anything in High Quality, NeurIPS 2023

[6] LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness, 2024

[7] 3D-LLM: Injecting the 3D World into Large Language Models, NeurIPS 2023

[8] Learning to Detect Every Thing in an Open World, ECCV 2022

Dear Reviewer,

Thank you for your very helpful and constructive review. We hope that our rebuttal has addressed your concerns. We are happy to answer any follow-up questions you might have. If your major concerns have been addressed properly, we would kindly appreciate it if you could consider updating your score accordingly.

Sincerely,

Authors.

(Weakness 1.1) Quantifying object candidates in AR computation

We retain 1,000 predicted objects per scene when computing AR, following common practice in 3D detection benchmarks. This ensures a fair comparison across methods. We will clarify such evaluation details in the final version.

(Weakness 1.2) On the role of confidence thresholds in AR evaluation

For evaluation, we follow prior open-vocabulary 3D detection works (e.g., OV-Uni3DETR) and adopt a standardized post-processing strategy where a fixed number of top-K predictions (1000 per scene) and the same confidence threshold (0.01) are retained for AR computation.

Therefore, the lower AR observed in closed-world or open-vocabulary detectors is not due to high thresholds filtering out correct predictions, but rather reflects the actual limitations of these models in generating high-recall proposals for novel categories. We will clarify this in the final version.

(Weakness 2.1 and Question 2) Adaptability of 3D object discovery & Necessity of a new detector

Yes, our 3D object discovery pipeline can be leveraged to expand training data, especially for previously unseen object categories. With a broader range of annotated objects, the discovered 3D regions can be used to fine-tune existing closed-world or open-vocabulary detectors.

To demonstrate this, we conducted experiments where we used our discovered objects to augment training data and fine-tune existing detectors such as FCAF3D[1] and Tr3D[2]. As shown in the table below, incorporating discovered objects leads to substantial improvements in AR$_{\text{novel}}$:

Although existing detectors benefit from our discovery pipeline, OP3Det further outperforms them. It is worth noting that both FCAF3D and TR3D are multi-modal models, showing that a newly designed detector is necessary for effective multi-modal fusion in an open world.

(Weakness 2.2) Evaluating the compatibility of our point-sampling strategy

We agree that combining SAM3D[3] with our proposed point-sampling strategy could be a promising direction. To validate this, we conducted an ablation study where we integrated the SAM3D strategy with our multi-scale point sampling method before 3D box generation.

As shown in the table above: SAM3D alone performs worse due to unfiltered noisy masks containing clutter; Adding our point-sampling strategy improves SAM3D performance (from 58.6% to 61.7%); Our full pipeline still outperforms (62.8%) due to additional refinement from the 2D class-agnostic detector, which helps filter non-object regions before 3D lifting.

These findings support the generalizability and effectiveness of our point-sampling method for robust 3D box generation.

Reference

[1] FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection, ECCV 2022

[2] TR3D: Towards Real-Time Indoor 3D Object Detection, ICIP 2023

[3] SAM3D: Segment Anything in 3D Scenes, ICCV 2023 Workshop