ReferSplat: Referring Segmentation in 3D Gaussian Splatting

We sincerely appreciate your positive feedback on our work: clever, reasonable, effective PCMI and GTCL design, and comprehensive ablation studies.

Q1&Q2: Generalization ability

A1: The experiments in our main paper follow a per-scene optimization setup, which naturally limits direct generalization to unseen scenes. To address this, we explore a generalized training paradigm, as shown in Tab. 11 of the Appendix. In this setting, the referring feature is no longer per-scene initialized; instead, it is predicted from other Gaussian attributes such as color, opacity, and position, following a feed-forward paradigm. This design enables generalization beyond per-scene optimization. To further validate the generalization capability of our method, we conduct additional experiments on the larger-scale and more diverse ScanNet dataset. We select 30 scenes from the official training split for joint training and 5 scenes from the validation split for evaluation. Language expressions are sourced from ScanRefer. As shown in the table below, our method achieves strong performance across diverse and unseen environments, highlighting its robustness and generalization. We expect that extending our framework with scalable architectures (e.g., MVSplat [a]) will lead to further performance gains and improved applicability in real-world scenarios.

[a] MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images, ECCV 2024.

Q3: GOI result

A3: Thank you for this insightful suggestion. To further validate the effectiveness of our method, we have conducted additional experiments comparing our approach with GOI [b] on the Ref-LERF dataset. The results demonstrate that although GOI outperforms LangSplat, its performance remains notably lower than that of ReferSplat. This highlights that, although both methods utilizing 2D pseudo-mask supervision, the key difference lies in how effectively the connection between 3D Gaussian points and natural language expressions is established within the 3D scene. We will revise the main paper to discuss GOI [b] and update Tab. 5 with the new comparison results shown below.

[b] GOI: Find 3D Gaussians of Interest with an Optimizable Open-vocabulary Semantic-space Hyperplane, ACM MM 2024.

Q4: Position information

A4: Thank you for your insightful question. Incorporating positional information into the referring feature (Eq. 7) enhances the model’s ability to understand spatial relationships described in the referring expressions while also enriches the referring feature with geometric context from the 3D scene. Instead of degrading performance, this geometric context helps ensure that segmentation masks accurately cover the complete target object, resulting in improved open-vocabulary segmentation performance.

We sincerely appreciate your positive feedback on our work: meaningful, reasonable, and thorough.

Q1: Claim 1

A1: In established 2D/3D referring expression segmentation (RES) tasks, referring segmentation involves segmenting target objects based on free-form natural language expressions that often include spatial relationships or object attributes. In contrast, open-vocabulary segmentation typically focuses on identifying all objects of a given category, usually specified by a single category name. As shown in Fig. 4 of our paper, the average sentence length in the LERF-OVS dataset is approximately 1.5 words, with the vast majority of queries being category names (e.g., “cup”) and very few containing descriptive or relational information (e.g., “tea in a glass”). This highlights a clear difference in the nature of the language inputs between the two tasks. To avoid confusion and better reflect the novelty of our task, we have revised the sentence to: “a new task that focuses on segmenting target objects in a 3D Gaussian scene based on natural language descriptions that often contain spatial relationships or object attributes.”

Q2: Claim 2

A2: While the elephant is clearly visible in some training views, it is not directly visible from the novel camera viewpoint during inference in Fig. 1. We highlight this to emphasize a key difference from 2D RES, which relies solely on single-view information and struggles to handle such invisible scenarios. In contrast, the proposed ReferSplat leverages multi-view information to construct a holistic 3D scene representation, enabling robust reasoning even when target objects are occluded or not directly visible. For further clarification, please refer to L128–137 (second column). We will revise the corresponding description to make this point clearer.

Q3: Claim 3

A3: We agree with your comment and have revised the sentence for better clarity. The updated version is: “Existing methods rely on matching the text query with 2D rendered features instead of performing localization directly in 3D space, which limits their performance in complex scenarios.”

Q4: Related work

A4: Thank you for the suggestion. We will expand the related work to provide more discussion on datasets like ScanRefer and MultiRefer.

Q5: Referring expression sentences generation

A5: The referring expressions are manually annotated by human annotators prior to training and are included in our Ref-LERF dataset.

Q6: Generalize to new referring expressions

A6: For generalization to new expression, our method uses BERT embeddings for their strong language understanding and generalization, gained via pre-training on diverse text. By aligning 3D Gaussian features with BERT representations via Gaussian-Text modeling, the model can interpret and generalize to unseen object relationship descriptors. We will clarify this in the revision. For generalization to new scenes and new datasets, please refer to our responses to Reviewer 2Vdk Q4 and Reviewer t3nW Q5, respectively.

Q7: Overall similarity

A7: Thank you for the insightful question. Our method aggregates per-word similarities, allowing the model to account for the influence of individual terms, including spatial relations like “far away.” Visualizations show that such words activate relevant regions, suggesting the model captures their importance in segmentation decisions. Additionally, we incorporate Gaussian-Text Contrastive Learning on global sentence-level features to enhance holistic comprehension of the sentence context, including complex spatial relationships. This combination of local (word-level) and global (sentence-level) cues enables the model to better interpret both positive and negative spatial terms. We will clarify this aspect in the revision.

Q8: Mask usage

A8: Our method does employ pseudo masks generated by Grounded-SAM during training. The intended point is to emphasize that our method eliminates the need for manually annotated ground-truth masks, which are often costly and impractical. We will revise any ambiguous statements in the paper to more accurately reflect this and avoid confusion.

Q9: CLIP result

A9: As suggested, we conduct experiments comparing BERT and CLIP embeddings for language features in R3DGS. Results show that BERT consistently outperforms CLIP. This is likely because CLIP focuses more on noun categories, while referring expressions often involve spatial and attribute-based descriptions. We will include more discussion and results in the revision.

Q10: Typos

A10: Thank you! We will carefully proofread and correct any typos or ambiguous phrasing in the revision.

Q11: Impact statement

A11: We will add impact statement in the revision.

We sincerely appreciate your positive feedback on our work: novel, well-motivated, effective PCMI and GTCL design, and well-written.

Q1: Diversity comparison to existing datasets

A1: The comparision to ScanRefer and Multi3DRefer is shown in the table below. Due to the limited availability of large-scale 3DGS datasets, our Ref-LERF is built upon the widely used LERF dataset, extended with referring sentence annotations to support the R3DGS task. While the number of scenes is limited, Tab. 5 of the main paper shows that our method achieves strong performance, demonstrating its effectiveness. Furthermore, as discussed in our response to Q4, our method also generalizes well to larger and more diverse datasets, e.g., ScanNet, under the generalized training setting.

Q2: Evaluation metric

A2: To provide a more comprehensive evaluation, we report mAcc@0.25, which measures the percentage of predictions with IoU > 0.25 and is commonly used in 3D point cloud referring segmentation tasks. This metric reflects performance in practical scenarios, including occlusions. As shown in the table below, ReferSplat achieves a mean mAcc. of 38.4 on Ref-LERF, significantly outperforming LangSplat and demonstrating our method’s robustness. We will include this metric and the results in the revision.

Q3: Dataset splits

A3: The dataset comprises 772 training images and 22 testing images, with 236 language descriptions used for training and 59 for testing, totaling 295 descriptions. We will add the detailed dataset split information in the revision.

Q4: Generalization to unseen scenes

A4: The experiments in our main paper follow a per-scene optimization setup, which naturally limits direct generalization to unseen scenes. To address this, we explore a generalized training paradigm, as shown in Tab. 11 of the Appendix. In this setting, the referring feature is no longer per-scene initialized; instead, it is predicted from other Gaussian attributes such as color, opacity, and position, following a feed-forward paradigm. This design enables generalization beyond per-scene optimization. To further validate the generalization capability of our method, we conduct additional experiments on the larger-scale and more diverse ScanNet dataset. We select 30 scenes from the official training split for joint training and 5 scenes from the validation split for evaluation. Language expressions are sourced from ScanRefer. As shown in the table below, our method achieves strong performance across diverse and unseen environments, highlighting its robustness and generalization. We expect that extending our framework with scalable architectures (e.g., MVSplat [a]) will lead to further performance gains and improved applicability in real-world scenarios.

[a] MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images, ECCV 2024.

Q5&Q6: Computational costs

A5: We conduct experiments on the ramen scene from the Ref-LERF dataset using the same NVIDIA A6000 GPU to compare the computational cost of our ReferSplat against SOTA methods. Results show that ReferSplat achieves significantly lower computational complexity and faster inference speed than LangSplat. While GS-Grouping excels in storage and FPS, ReferSplat outperforms all methods in segmentation performance. ReferSplat also has the shortest training time, thanks to a lightweight preprocessing pipeline that avoids costly operations like language feature compression (LangSplat) or mask association with video tracking methods (GS-Grouping). These results demonstrate that ReferSplat’s compact, efficient design is well-suited for real-world and large-scale 3D applications. We will include these comparisons in the revision.

We sincerely appreciate your positive feedback on our work: dataset contribution, SOTA result, relationship modeling, and exhaustive ablations.

Q1&Q7: Detailed evidence like video

A1: We have provided additional qualitative video results at ReferSplat.mp4, which include:

• Cases involving ambiguous language queries
• Cases with incomplete language input
• Performance under different novel views
• Performance on significant viewpoint and perspective shift

These videos demonstrate our method’s ability to handle diverse and challenging referring expressions, showcasing its robustness to ambiguity and incomplete descriptions. The qualitative results further validate the effectiveness of our method and its potential for real-world applications.

Q2: Computational costs

A2: We conduct experiments on the ramen scene from the Ref-LERF dataset using the same NVIDIA A6000 GPU to compare the computational cost of our ReferSplat against SOTA methods. Results show that ReferSplat achieves significantly lower computational complexity and faster inference speed than LangSplat. While GS-Grouping excels in storage and FPS, ReferSplat outperforms all methods in segmentation performance. ReferSplat also has the shortest training time, thanks to a lightweight preprocessing pipeline that avoids costly operations like language feature compression (LangSplat) or mask association with video tracking methods (GS-Grouping). These results demonstrate that ReferSplat’s compact, efficient design is well-suited for real-world and large-scale 3D applications. We will include these comparisons in the revision.

Q3: Difficult with sudden viewpoint changes

A3: This is a broader challenge in the field, not specific to our method. Baselines like LangSplat and GS-Grouping also show performance degradation under significant viewpoint changes and perspective shifts, as shown in the ReferSplat.mp4. Future work will focus on developing robust multimodal representations and improving global scene understanding to enhance robustness under extreme viewpoint variations.

Q4: Elephant visible?

A4: While the elephant is clearly visible in some training views, it is not directly visible from the novel camera viewpoint during inference in Fig. 1. We highlight this to emphasize a key difference from 2D referring expression segmentation (RES), which relies solely on single-view information and struggles to handle such invisible scenarios. In contrast, the proposed ReferSplat leverages multi-view information to construct a holistic 3D scene representation, enabling robust reasoning even when target objects are occluded or not directly visible. For further clarification, please refer to L128–137 (second column). We will revise the corresponding description to make this point clearer.

Q5: Messy Rooms dataset evaluation

A5: Drawing from experience in 2D/3D RES tasks, models trained on sufficiently diverse datasets can generalize well and exhibit strong zero-shot capabilities, as exemplified by models like Grounded-SAM. As shown in Appendix Tab. 11, our model shows promising generalization results under the generalized training setting. In particular, it generalizes well to unseen scenes on the ScanNet dataset, which contains numerous objects and diverse scene layouts (see Reviewer 2Vdk Q4). Building upon this foundation, joint training on large-scale, diverse datasets such as ScanRefer, Ref-LERF, and MultiRefer presents a viable path toward adapting our model to more complex environments like Messy Rooms, where manual annotation and detailed descriptions are impractical. The above analysis highlights the scalability and cross-dataset generalization potential of our approach in real-world applications.

Q6: Pseudo mask error

A6: To reduce the impact of errors from pseudo masks, we employ a two-stage optimization strategy (as described in L310-316), iteratively refining mask predictions during training. As shown in Tab. 1 of the main paper, this two-stage strategy outperforms the one-stage pipeline, demonstrating its effectiveness in mitigating the impact of noisy pseudo masks and ensuring robust overall performance. It is worth noting that the default results are based on the one-stage pipeline for a fair comparison with previous methods.