LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D

We appreciate the reviewer for the thorough reading of our manuscript and valuable comments. Below we address each point raised in detail

Clarification regarding our LX3D

We noted the reviewer mentions our dataset being automatically generated, we'd like to clarify that our dataset is human-collected. Annotators curated and validated every sample. We invite the reviewer to read more details about our annotation setup in Appendix D.

Runtime analysis and assumptions [L029-L033]

Indeed, extracting 2D features and lifting them to 3D is computationally expensive. We address this by caching the environment’s 2D features for each view, as well as the featurized point cloud. For ScanNet experiments, we compute this cache offline; and for robot experiments we compute it while doing the initial environment exploration phase. With this feature cache, a forward pass of our model takes ~1 second for a scene with ~100k feature points and utilizes ~8 GB of VRAM on an A100 GPU.

We can utilize such caching because our benchmarks operate under static (ScanNet) or quasi-static (robot) environments. Extending our approach to dynamic scenes would require real-time 2D feature computation and continuous updates to the featurized environment. We believe the former is a matter of engineering, while the latter is an active research area, explored by methods like Lifelong LERF (A. Rashid, 2024). We will include these assumptions in our limitations section.

[L086-089] rewording

We propose rewording this claim to more accurately state: "Locate 3D does not require ground-truth region proposals, high-quality mesh sampled pointclouds, or surface normals at inference time, making it suitable for real-world deployment."

Clarification on positioning

We propose revising [L011-L014, right col] to clarify: "Many existing 3D RefExp models are evaluated on benchmark datasets with pointclouds sampled from high-quality post-processed 3D meshes or object instance segmentations already available. In contrast, Locate 3D operates directly on raw sensor observation streams without requiring intermediate processing steps like mesh reconstruction, instance segmentation, or pre-trained object detectors at inference time."

We want to highlight that most previous work has been trained and evaluated utilizing mesh pointclouds, while our approach leverages pointclouds obtained directly from RGB-D sensors (please see our response to reviewer C37t on this topic).

Discussion of ConcreteNet

We will add ConcreteNet to the related works section and baselines in Table 1. We note that this method operates on ScanNet's mesh-sampled pointclouds rather than sensor pointclouds. While ConcreteNet achieves 56.12% Acc@25 on ScanRefer, our approach demonstrates superior performance (59.9%) under the more challenging setting.

3D-JEPA exposition

We appreciate the observation. In our revision, we will provide a clearer introduction to this concept in section 2.2 explaining (1) The core principles of the JEPA framework (2) Why adapting this approach to 3D pointclouds (3) Main deltas to the original framework.

Discussion on Limitations

We will add a limitations section discussing (1) our current approach's constraints with dynamic environments (2) our method being limited to few-room environments (potentially addressed by exploring RoPE positional embeddings or similar techniques).

Typos

We appreciate the careful proofreading, we will correct all of these in the revised manuscript.

Release of LX3D dataset

We will publicly release the LX3D dataset, trained Locate 3D model, and 3D-JEPA backbone upon publication.

Generalizability (novel-object localization)

We believe both questions can be boiled down to how well CLIP features are preserved through 3D-JEPA pre-training. Our linear probing experiments in Section 4.2 address this, particularly on the "noun correctness" task 3D-JEPA achieves 73% accuracy compared to ConceptFusion's 66%. These results demonstrate that 3D-JEPA not only preserves but enhances the semantic understanding from foundation models. This pattern mirrors developments in language modeling, where contextualized embeddings (like BERT) consistently outperform non-contextualized word embeddings for both common and rare words.

Mask and patch feature aggregation

For 3D points observed in multiple views, we aggregate features by weighted averaging. We voxelize the pointcloud (5 cm voxel size) and compute a single feature per voxel by weighted averaging all contained features. Weights are calculated using trilinear interpolation based on distance to voxel boundaries. This weighted averaging approach is common practice in both our baselines (e.g., ConceptFusion) and traditional RGB-D mapping literature (Real-time 3D Reconstruction in Dynamic Scenes using Point-based Fusion - ICCV 2013). We explored alternative aggregation methods (max pooling, simple average, summation) before selecting weighted averaging.

We thank the reviewer for their detailed feedback. Their comments have helped us identify areas where we can better explain and justify our technical contributions. We address each comment in detail below.

Deriving better 3D features from 2D features; 3D-JEPA

The 3D nature of our approach comes from the architecture and learning objective rather than the individual pointwise inputs. Our model processes the entire featurized pointcloud using explicit 3D coordinates to produce per-point features. 3D-JEPA's objective forces the model to learn additional information beyond 2D features through masked prediction in 3D space, requiring it to infer features for occluded regions using contextual information from the surrounding visible 3D scene. In short, the "3D" in 3D-JEPA refers to the domain in which the representation learning occurs.

Our approach has precedent in prior work like OpenScene, which demonstrated that distilling 2D vision-language features to 3D via contrastive learning results in 3D models that outperform the original 2D features.

Empirical evidence shows 3D-JEPA features encode valuable information beyond lifted 2D features:

1. 3D-JEPA features demonstrate superior performance in linear probing experiments (Section 4.2) when compared to ConceptFusion features (34%→39% and 66%→73%)
2. Frozen 3D-JEPA features provide ~5% improvement over ConceptFusion inputs to the decoder (Table 2)
3. ConceptFusion (direct 2D-to-3D feature lifting) achieves 20% success on our robot evaluations compared to Locate 3D's 80% (Table 9)

Aggregating multi-view features

We voxelize the pointcloud at 5 cm resolution and aggregate multi-view point features within each voxel using weighted averaging based on trilinear interpolation distances. This aligns with standard practice in prior works (e.g., ConceptFusion); given space constraints, please also refer to our response to reviewer gKhM.

Farthest point sampling for masking

Yes, we tried farthest point sampling but did not see improvements. We additionally experimented with several other masking strategies as discussed in Appendix A.1. However, none of the methods improved over random masking (similar to the findings in Assran et. al., 2023). Thus, we use random masks for their computational efficiency.

Fairness of VLM baselines and usage of depth information

Yes, the two VLM baselines use depth information. In fact, they use the exact same inputs as Locate 3D (RGB-D + camera pose and intrinsics) to ensure fair comparisons. The VLM baselines are detailed in Appendix F.

At a high level, these baselines use a modular pipeline that consists of first selecting an RGB frame using a VLM, then selecting an object in the 2D frame using GroundingDINO, SAM-2, and a VLM, and finally determining a 3D bounding box using depth and camera extrinsic and intrinsic information.

Comparison with 3D-LLM and LEO on ScanRefer

Locate 3D significantly outperforms 3D-LLM [1]. Specifically, 3D-LLM reports 30.3% Acc@0.25 on ScanRefer, while Locate 3D achieves 59.9% (Table 1). We will add results from 3D-LLM [1] to Table 1. LEO [2] is evaluated on ScanQA but not on ScanRefer (note that ScanQA assumes text outputs, while ScanRefer requires bounding boxes or instance masks). We will discuss LEO in related work.

Comparison with other 3D semantic alignment and scene-level features

Please refer to our previous response for a comparison of our approach to 3D-LLM, which trains a model to consume the entire scene. We are also happy to discuss additional related work if you might have concrete suggestions.

PTV3: Scratch vs pre-trained in Table 2

In our approach, PTV3 is pre-trained with 3D-JEPA. In Table 2, we study alternatives including “random” initialization (i.e., PTV3 is initialized from scratch), and initializing from “PonderV2” weights. In all three cases, the encoder is finetuned end-to-end for referential grounding. We will add these details to the Table 2 caption.

Table 2: Random initialization vs. 3D-JEPA (Frozen)

The better performance of random initialization compared to frozen 3D-JEPA can be attributed to the significant number of extra trainable parameters (~250M) that are optimized specifically for the referential grounding task. When we freeze the 3D-JEPA encoder, we only train the decoder and significantly limit the model's ability to adapt the encoder parameters to the task at hand. However, it's important to note that fine-tuning the 3D-JEPA encoder (our full model) provides the best performance, showing that while trainable model capacity is important, the pre-trained weights offer valuable initialization that leads to better results when allowed to adapt to the target task.

Table 1 clarifications (metrics, setting, VLM)

The VLM baselines (LLaMA and GPT-4o) use the exact same metrics and settings as other methods (such as Locate 3D) in Table 1. Specifically, the VLM baselines return 3D bounding boxes as detailed in Appendix F.

We thank the reviewer for their thorough feedback. Below we address each point in detail

Clarification on mesh PC vs sensor PC

By "Mesh PC" we refer to pointclouds sampled from carefully reconstructed 3D meshes that undergo extensive post-processing. Most prior works use this format as it’s ScanNet’s default.

In contrast, "Sensor PC" refers to pointclouds obtained directly from RGB-D sensors. These contain raw depth measurements including noise, missing regions, and registration errors. This format better represents real-world deployment scenarios.

Methods originally reported on Mesh PC suffer significant performance drops (8-10%) when trained and evaluated on sensor data (Table 1), 3D-VisTA drops from 53.1% to 45.9% Acc@25 and BUTD-DETR from 50.28% to 40.7%. This gap was also observed by Odin (Jain et al, 2024). Under the (more rigorous) evaluation setting comprising sensor pointclouds, Locate 3D achieves SoTA (61.7% Acc@25) performance. We will clarify this terminology in the paper.

For the mesh pc methods in Table 1, the reviewer believes most of them are using pointcloud as input, not mesh.

The reviewer is correct – while these methods use pointclouds as input, we use the term "Mesh pointcloud" to indicate that these pointclouds are sampled from cleaned, post-processed meshes.

3D-VisTA evaluation clarification

3D-VisTA was originally designed for mesh pointclouds with a two-stage architecture: first using Mask3D to generate object proposals, then processing these proposals with the pointcloud for grounding. Due to implementation constraints and pre-computed proposal dependencies, we report results using sensor PC for 3D-VisTA while maintaining the original Mask3D proposals from mesh PC. We note that this represents an upper bound for 3D-VisTA's performance on sensor data, as using sensor PC for proposal generation would only decrease performance further. Even with this advantage, 3D-VisTA achieves 45.9% Acc@25, while Locate 3D significantly outperforms it at 61.7%.

Question about query selection

For ScanNet benchmarks we select the query with the highest “token probability” for the target noun. In our robot experiments, we use an off-the-shelf LLM to parse the referring expression and identify the target noun before applying the same selection mechanism.

Authors mention that 3D grounding approaches [...] are prone to failures'. The reviewer is curious about the differences.

These approaches, such as 3D-VisTA (Zhu et al., 2023) and PQ3D (Zhu et al., 2024), rely on external object detectors to generate proposals without considering the language query and then train their model to select one. If the object proposal misses, the entire model is bound to fail. Single-stage approaches like ours and BUTD-DETR (Jain et al., 2022), directly predict the bounding box by jointly considering the scene and language tokens. We will incorporate this discussion into our paper.

LX3D out-of-domain evaluation clarification (ScanNet vs ScanNet++)

We see key differences in scanning hardware, setting (partial scene vs. full scene), scene size, and distribution of queries between the train and eval settings. Further, our eval set contains both ARKitScenes and ScanNet++. While we agree that some aspects of the visual distribution remain constant (e.g. common object classes), we believe that the evaluation faithfully represents the domain gap present during a new deployment of our model to indoor scenes.

As for your latter point (Locate 3D not being the best in multiple subsets) – we believe the VLM approach is strongest on ScanNet++ because each input contains only 5 frames, and Locate 3D not achieving maximum performance on FRE is an artifact of the small eval set size (~250 annotations on 1 scene).

Better perf on Acc@25 but lower on Acc@50

This discrepancy arises because PQ3D operates on mesh pointclouds, while our method works on sensor pointclouds. At lower IoU thresholds (Acc@25), our approach effectively predicts the bounding box despite sensor data imperfections. However, at higher thresholds (Acc@50), achieving precise alignment becomes more challenging, resulting in a more significant performance drop for sensor-based methods. In short, PQ3D is evaluated in a more relaxed setting than ours, and is at an advantage.

Despite our best efforts, we could not re-train PQ3D with sensor pointclouds due to their use of multiple backbones and multi-stage training strategies, making a direct comparison difficult.

CLIP and DINO backbones in Table 3

Whenever we use CLIP features, we first obtain masks using SAM and then extract CLIP embeddings for these masks following Conceptfusion. This approach is necessary because CLIP produces global image embeddings, not patch or pixel-level features. For DINO, we directly use its dense patch-level features and map them to individual points without requiring a segmentation step. We will add this explanation to Section 2.1.

We thank the reviewer for their thorough review and constructive feedback. Below we address each point in detail.

Clarification about AR application

Indeed, we do not demonstrate deployment on an AR device. Our intended claim is that Locate 3D's input format is directly compatible with data streams from AR devices and robots - it operates on raw RGBD and camera parameters, which is the standard output format of these platforms. We demonstrate this compatibility through evaluation on diverse sensor-captured datasets (ScanNet, ScanNet++, ARKitScenes) as well as robot deployment. We will revise our phrasing to clarify that we're referring to input compatibility rather than explicit AR deployment.

Inference speed, memory usage, computational cost

We will add this discussion to the paper. Our inference pipeline operates in two phases. First, we compute and cache 2D features and a featurized pointcloud of the environment. This can be done offline for static environments like ScanNet, or during an initial exploration phase for robots. With this cache in place, our model performs inference in approximately one second for a scene with ~100,000 feature points with an average peak VRAM of ~8.1 GB on one A100.

LX3D dataset analysis; data volume vs diversity

We ran an additional experiment, in which we find that scene diversity is a key factor in LX3D improvements. Specifically, we compare two conditions: (1) ScanNet training data + 30K LX3D samples also from ScanNet and (2) ScanNet data + 30K LX3D samples from ScanNet++ (i.e., same quantity, but better quality). We find that training on better quality scenes (2) outperforms (1) by ~2%. We will include this additional experiment in a revision.

The annotation quality of every sample LX3D was validated by human annotators. Specifically, only samples marked as unambiguously correct are included in the final dataset to ensure reliability, more details about our annotation setup and dataset can be found in Appendix D.

Furthermore, we will publicly release the LX3D dataset upon publication, providing a high-quality asset to support future research in 3D referential grounding.

Combining RGB, CLIP, DINO features

The features are combined by concatenating CLIP and DINO features with a harmonically embedded representation of the RGB features. Specifically, CLIP features are generated for SAM masks, with a learnable token used when mask features are unavailable. Full details are provided in Appendix A.2 and we will add a summary to the main paper. For more details, we invite the reviewer to read the response to a similar question posed by reviewer gKhM.

Impact of different 2D image features

We provide experimental analysis in Table 3. We observe that features extracted from larger models (CLIP-L, SAM-H) consistently outperform those extracted from models with fewer parameters (CLIP-B, MobileSAM), with CLIP-L achieving 59.2% Acc@25 compared to CLIP-B's 53.7%. Additionally, concatenating features from CLIP and DINOv2 significantly improves results compared to using either of these in isolation, reaching 61.7% Acc@25, indicating that each of the features provide complementary information that benefits 3D object localization.

Discussing related work that performs lifting of 2D features (OpenScene, Lift3D)

We appreciate the suggestion. Approaches such as OpenScene (CVPR 2023), Lift3D (CVPR 2024), and prior ideas proposed in distilled feature fields (Neurips 2022), similarly lift 2D features to 3D via neural rendering or contrastive learning. However, we incorporate two key advances in addition to lifting 2D features. (1) our self-supervised 3D pretraining (3D-JEPA) approach contextualizes the lifted 3D features, (2) our specialized language-conditioned decoder enables precise referential object localization. Both (1) and (2) contribute significantly to our performance. As shown in Table 9, Locate 3D+ (8/10) substantially outperforms Concept Fusion (2/10), a zero-shot feature lifting method, on complex referential tasks. We'll add this discussion to the related work section.

Multi-view inconsistency of SAM masks

Correct, SAM masks are not multi-view consistent. This limits the performance of prior methods that use 2D features lifted to 3D in a zero-shot fashion. However, our SoTA results (in Table 1) demonstrate that 3D-JEPA effectively learns to deal with such noise. Qualitatively, we find that 3D-JEPA features are smoother and more consistent than 2D lifted features (as shown in Figure 1), demonstrating (again) that our method is robust to such issues.