From Thousands to Billions: 3D Visual Language Grounding via Render-Supervised Distillation from 2D VLMs

We appreciate that the reviewer likes our results and data scaling performance. We answer the questions below and will improve the writing given the valuable suggestions.

---

Model architecture and its operational flow…

Encoder Backbone: is a SparseConv UNet [1] (Sec. 3.4), following PonderV2 implementation.

Gaussian Head is a lightweight MLP (2 layer, 128 dim) mapping each point from the Encoder Backbone to Gaussian parameters (L182-L190), including scaling, color, etc.

The Mask Decoder Transformer follows the MaskFormer pipeline [2], using m learnable tokens (mask proposal tokens in Figure 4) to predict potential 3D masks. Each token corresponds to a binary 3D instance mask over the Gaussians.

The correspondence matrix C (m x |Q|) grounds the m instance masks into the Q language tokens. Each element indicates the probability that the mask corresponds to that particular language token.

The mask decoder is a Transformer decoder, where the visual and language tokens serve as the Key and Value, while the proposed mask tokens act as the Query inputs to the decoder.

To further aid clarity and reproducibility, we will release the code and improve the final version. Lastly, we note that LIFT-GS is model-agnostic, imposing minimal architectural constraints and being readily adaptable to other architectures (Lines 224–230).

Pretraining Data

We used the training scenes from ScanNet and ScanNet++ for pretraining, except for the ablation study in Table 6. For images, we sampled frames from the original video trajectories with a frame skip of 30 (i.e., at 1 Hz). Each selected RGB-D frame was unprojected using the provided camera intrinsics and poses, then voxel-pooled at a 5 cm resolution, so the total point cloud number is controlled.

Claims

Using rendered features/images as supervision is not new in 3D VL understanding... This paradigm is the first in the 3D referential domain, to the best of my knowledge.

We agree and will make the claim more precise by specifying that we mean grounding with complex language (3D referential grounding).

The claim "render supervised framework…" is quite bold to make. For example, extending... or to 4D dynamic scenes is inherently non-trivial.

We appreciate the reviewer’s suggestion and will revise it to make it more accurate. We appreciate any recommendations, and suggest the following: “The render-supervised framework provides a general and extensible design. LIFT-GS shows how to use it for highly structured tasks, such as 3D referential grounding and object detection”.

We genuinely believe that our pipeline presents a general and extensible design. While adapting it to new tasks may involve non-trivial effort, we argue that such extensions are both feasible and conceptually straightforward.

For image inputs, one could use methods like Dust3R [3] to regress point maps from images as point clouds. For dynamic scenes, it is possible to regress motion basis coefficients for each point, as Shape-of-Motion[4]. These examples illustrate our belief that the proposed pipeline can be extended to support a wide range of applications beyond the current scope.

Experimental analysis

Role of L_{RGB} loss

We keep L_{RGB} loss because it is necessary to supervise the reconstruction of the 3D Gaussian fields. We agree that this loss may not significantly benefit downstream tasks, as it provides limited semantic information.

Improvement of adding Scannet++ in Table6?

Excellent point. In Tab. 5, increasing the finetuning data by 100% yielded about a 5% Acc@0.5 improvement. Increasing the pretraining data by 30% yielded a 1% improvement.

Based on the curve in Figure 6 (we show the real and estimated values based on the curve below), the performance gain from adding ScanNet++ data (30% more) for pretraining is roughly equivalent to adding 15% more ScanNet finetuning data with 3D annotations. It shows that the effective transfer ratio is roughly 1 / 2; i.e., collecting twice the number of raw videos alone can yield improvements comparable to building a fully annotated 3D dataset.

Therefore, pretraining on ScanNet++ is highly efficient and cost-effective, considering that annotating 3D referential grounding data requires orders of magnitude more effort than collecting raw video alone.

[1] https://github.com/facebookresearch/SparseConvNet/blob/main/examples/ScanNet/unet.py

[2] MaskFormer: Per-Pixel Classification is Not All You Need for Semantic Segmentation

[3] DUSt3R: Geometric 3D Vision Made Easy

[4] Shape of Motion: 4D Reconstruction from a Single Video

We sincerely appreciate the reviewer’s recognition of LIFG-GS, including “the method is well presented and builds upon existing building blocks”, and believing “As such the proposed 3d supervised pre-training task is valuable to the wider community.”. We answer the question below and will make them more clear in the final version.

---

How are the different scales of maks that are generated by SAM handled in the loss? Segmentations in different images that see the same object will contradict such that a consistent 3d segmentation is not easily possible. He. et al. "View-Consistent Hierarchical 3D Segmentation Using Ultrametric Feature Fields", ECCV'24 addresses this problem via a hierarchical segmentation.

That’s an excellent question, and we appreciate the opportunity to elaborate on this point. A key advantage of our approach is that it does not require 3D view-consistent masks, as our method is fully learning-based and relies solely on 2D supervision.

In contrast, existing methods that leverage SAM often rely on hand-crafted hierarchical structures or empirically designed mask-merging heuristics to improve consistency across views. These procedures are typically complex and heuristic-driven, introducing additional sources of noise and making the pipeline less robust.

Our framework takes a different route: the 3D model is queried using 2D CLIP embeddings, and training is guided only by corresponding 2D mask and grounding losses. As a result, there is no requirement for the 2D masks to be consistent across views. Instead, we encourage the Transformer decoder to learn how to produce coherent 3D masks that align with these independently supervised 2D views, in a fully data-driven manner—without relying on manually designed post-processing steps.

To extract masks from SAM, we sample a grid of points across each image and generate masks accordingly. We apply Non-Maximum Suppression (NMS) to reduce overlapping masks and discard extremely small ones. Aside from these minimal filtering steps, we do not apply any further merging or heuristic post-processing.

---

Which model is used to compute the ground truth feature map F_2D?

We use the same pipeline as in Figure 3 to compute the ground-truth 2D feature map F_2D​. Specifically, we apply the CLIP image encoder to extract features for each segmented region, assigning the resulting feature vector to all pixels within that region. For segmentation, we adopt the SAM-H model, and we use the CLIP-L model to extract features. This pipeline is flexible and can also accommodate alternative backbones, such as DINO-v2.

---

A comparison between PQ3D and the proposed approach is missing on the very same input data…. However, the novelty of the approach lies within the generic (pre)training from 2d ground truth rather than beating the top performer on any benchmark.

We sincerely appreciate the reviewer’s recognition that the novelty of our work lies in proposing a generic (pre)training framework using 2D supervision, rather than merely achieving state-of-the-art results on specific benchmarks.

Regarding PQ3D, we note that the reason why our method does not use exactly the same input data—i.e., mesh point clouds—as PQ3D, is that mesh point clouds are impractical in real-world applications due to the need for time-consuming meshing and human annotations (L298-303). To ensure a fair comparison, we made every effort to retrain all baselines under our unified setting for an apples-to-apples comparison. However, we were unable to successfully reproduce PQ3D due to its complex, multi-stage training pipeline that spans across multiple datasets.

We greatly appreciate the reviewer’s suggestions and agree that more rigorous, apples-to-apples evaluations—using identical input settings as prior works—would further improve the clarity and fairness of our comparisons. While we are actively working on this, the time and computational constraints during the rebuttal period prevent us from including those results here. We plan to incorporate them in a future revision of the paper.

---

Additional Reference

Thanks for pointing out, and we will include them in the final version.

We thank the reviewer for their thoughtful feedback and appreciate the recognition that using 2D supervision is “very useful for scaling up the training data,” “the claim… is supported by the experiments.”, “cross-scene render-supervision is innovative…”. We address the questions below and will incorporate improvements in the final version.

---

Compared to baselines, which output language embedding in 3D Gaussians with additional features, like Langsplat.

Thanks for the suggestion — we will clarify it in the final version. Both LIFT-GS and LangSplat use render-supervised distillation. However, LangSplat's per-scene optimization method is primarily designed for object segmentation (nouns only), while LIFT-GS is a cross-scene method supporting 3D referential grounding, involving more complex language.

LangSplat and its variants rely on language-vision feature dot products for open-vocabulary segmentation: reconstructing 3D RGB and CLIP feature fields and computing dot products with text embedding for segmentation (Encoder only). However, contrastive language-vision models (e.g., CLIP) tend to behave like bag-of-words models [2], making them struggle with handling slightly long language expressions with relational structure, such as spatial relationships, key in referential grounding.

In contrast, LIFT-GS employs structured supervision from MDETR, training a transformer decoder over visual and language tokens to directly predict 3D masks and groundings through learned attention and referential loss. We illustrate the failure modes of dot-product-based methods on both 3D and 2D with language inputs in Fig https://postimg.cc/bd6QKTYk, https://postimg.cc/47Y9ZCxc.

We compare LIFT-GS and LangSplat variants on 3D referential grounding benchmark ScanRefer. As LangSplat operates on rendered 2D images instead of 3D, we use a variant Semantic Gaussians [1] that reports much higher performance and directly segments 3D; as well as LERF (which both papers compare to).

---

Comparison and discussion about SceneVerse [5], D-LISA [6].

Thanks for your suggestions, and we will include comparisons and discussions in the final version. The submission incudes comparisons to PQ3D (ECCV 2024), which reports stronger performance than both SceneVerse and D-LISA. For open-vocabulary segmentation, we compare against the numbers reported in PQ3D and show that LIFT-GS outperforms it.

For the 3D referential grounding task, we tried our best efforts to retrain baselines under the Sensor Point Cloud setting for a fair comparison; adopting PQ3D, 3D VISTA, and BUTD-DeTr as our main baselines. Despite our best efforts, we weren't able to train PQ3D. However, we believe that 3D-VISTA is the appropriate comparison here, since the requested references perform comparably to that method on mesh point clouds. We compare their reported results as published below.

Finally, we emphasize that LIFT-GS is both model- and data-agnostic, making it orthogonal to advances in data (e.g., SceneVerse) or architecture (e.g., D-LISA). Stronger models or larger 3D datasets can be seamlessly incorporated into our framework for better performance, as indicated in Figure 6.

---

Computation Resource Comparison.

The difference in computational cost primarily arises from the type of point cloud data used during training. We follow the setting of BUTD-DeTR, using sensor point clouds—unprojected from posed RGB-D frames (frame skip of 30, voxel size 0.05 cm)—resulting in ~30k points per scene. This setup reflects real-world use cases more closely (L305).

In contrast, most other methods, including SceneVerse, use ScanNet mesh segments, which are pre-annotated via face clustering, reducing scenes to ~300–1500 segments (∼100× fewer elements). This significantly lowers computational cost.

However, meshing is not viable for real-world or real-time applications—it is computationally expensive and also contains human-annotated information. That said, for fairness, LIFT-GS can also operate on mesh segments, and under this setting, training requires comparable compute (8×A100 GPUs for 2–3 days).

---

Typos:

We greatly appreciate these catches and will fix them in the final version.

[1] Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting

[2] When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?

First, we would like to thank the reviewer for taking the time and effort to engage with the paper, and we look forward to a productive discussion. Before addressing the individual questions, we would like to clarify the core focus of our work.

While our method involves 3D scene reconstruction as an intermediate step, the primary goal of this paper is 3D vision-language grounding—specifically, localizing 3D instances (i.e., point cloud masks) based on complex language queries. 3D spatial understanding is a known weakness of existing VLMs ([1; Fig. 5]), but is a critical capability for applications across robotics and embodied AI. For example, it can identify pick-and-place locations in long-horizon manipulation and rearrangement tasks.

The key challenges and contributions of our work lie in developing an effective and scalable pipeline for (pre-)training large 3D visual-language grounding models (the Decoder), and the contribution is to do this using differentiable rendering using 2D supervision (Line 97).

---

similar ideas have been explored via … feedforward transformers (Large Spatial Model, SAB3R)

Could the reviewer kindly provide a citation or reference for SAB3R? Despite our best efforts, we have been unable to find a corresponding paper or preprint. On the project website, we found [3], the paper link on the project website returns a 404. The only reference we found is on a personal website [2], which does not link to any publication or arXiv submission. Moreover, none of the other referenced papers evaluate on any of the standard 3D visual-language grounding benchmarks (ScanRefer, SR3D, NR3D).

---

Why point cloud inputs?... why focusing on point clouds is beneficial

As mentioned in the paper, the losses used in LIFT-GS are largely agnostic to inputs and could work with multi-view image inputs. The fact that the introduced problem formulation is independent of the input type highlights the primary focus of our paper, which is a new method for training 3D visual-language grounding models (especially Decoder), and we believe this is a strength of the method.

We chose to focus primarily on point cloud inputs, since they are currently one of the most widely adopted 3D representations in robotics, and the return type of SLAM and SfM. Point clouds offer flexibility and modularity, as they decouple the 3D reconstruction process from the grounding task, allowing us to leverage a wide range of input sources—including single-view, sparse-view, or long RGB(-D) videos, as well as LiDAR scans or other sensors. This makes point cloud-based methods especially well-suited to diverse real-world conditions.

Since pointcloud reconstructions can be incrementally updated and are largely independent of the specific sensor package (camera, depth sensor, IMU, etc). These properties make point clouds a practical choice for real-world robotic systems. For better or worse, they have become a de facto representation for 3D language grounding. We will make this point much clearer in the final version.

---

Technical Novelty and compared to prior works. The method sections (3.1–3.4) describe a task formulation, standard 2D-supervised training for 3D reconstruction, and a conventional network for lifting point clouds into 3D Gaussians. The contributions in terms of problem and model novelty are not sufficiently distinguished from prior work. A stronger claim on what specific challenges are uniquely solved by focusing on point clouds and 3D Gaussian representations is needed.

Reconstructing 3D Gaussian splats from point clouds is neither the focus nor a claimed contribution of our work. Our primary contribution lies in (pre-)training a large Transformer-based mask decoder for 3D visual-language grounding without requiring 3D labels, made possible through the use of differentiable rendering.

In contrast, the mentioned prior methods—whether based on per-scene optimization or feedforward models—focus on reconstructing the RGB and feature fields of the 3D scene and perform language grounding by computing dot products between language CLIP features and 3D feature fields. These approaches primarily consist of encoder-only architectures, whereas our work emphasizes the use of a powerful Transformer-based decoder for mask prediction. We discuss it in L96-109, L153-164.

Due to the limitations of CLIP embeddings, dot-product-based grounding methods struggle with handling slightly complex language expressions and cannot reliably reason about relative spatial relationships. This restricts their effectiveness in realistic grounding tasks. Because of character limits, we provide quantitative comparisons and discussions in the reply for Reviewer eYEH.

[1] https://open-eqa.github.io/

[2] https://tianx-ia.github.io/

[3] https://web.archive.org/web/20250501000000*/https://uva-computer-vision-lab.github.io/sab3r/static/pdf/Semantic_Augmented_3D_Foundation_Models__Writing.pdf