Open-Ended 3D Metric-Semantic Representation Learning via Semantic-Embedded Gaussian Splatting

(somewhat ordered from most to least important)

W1. No SOTA Review/Comparison w.r.t. Prior CLIP Dimensionality Reduction Methods

The first key contribution claimed by the authors relates to tackling the high dimensionality of CLIP language features for their integration into 3DGS representations. However, this topic has already been covered in various way by prior art. I.e., all existing language-embedded 3D Gaussian models have to deal one way or another with the prohibitively large size of CLIP features. Some solutions leverage the lower-dimensional latent space of scene-specific autoencoders [Qin et al., 2024], others apply PCA [Xu et al., 2024], etc., and some already implemented codebook-based strategies similar to the one proposed here [Qu et al., 2024; Shi et al., 2024, Liao et al., 2024].

The fact that the authors neither compare to these existing CLIP-dimension-reduction techniques, nor even mention them, is problematic, showing a lack of contextualization relative to prior art and questioning the actual scope of their technical contributions.

W2. No SOTA Review/Comparison w.r.t. 3D Language Consistency/Stability Methods

The authors also failed to disclose prior art tackling 3D semantic consistency in language-embedded 3DGS representations. For example, CLIP-GS [Liao et al., 2024] seemingly applies a similar regularization of 3D language features over consecutive frames, as well as voting-based regularization of CLIP features over segmented regions, similar to the proposed "semantic stability guidance". While the proposed contrastive-learning approach may be considered somewhat novel, it is also a relatively low-hanging extension of CLIP-GS' solution.

Similar to the previous point W1, the key issue here is the lack of contextualization/comparison done by the authors w.r.t. their technical claims. How different is their consistency/stability schemes from CLIP-GS ones? Which ones are actually better? — Here again, the readers are left in the dark.

W3. SLAM Framework Heavily Based on Existing Solutions

While not claimed as a contribution per se, the authors similarly poorly contextualized their SLAM framework, with no references provided for the parts that they borrowed.

E.g., their tracking step—composed of a linear forward projection to initialize the camera parameters (Equation 7), followed by a multi-modal rendering-based optimization (Equation 8)—appears heavily based on prior work, e.g., SplaTAM [Keetha et al, 2024] and SGS-SLAM [Li et al, 2024c].

W4. Minor Remarks

• The authors do not provide any insight on possible limitations / failure cases / societal impact.
• The wrapping of text around some large tables (Tables 2 and 3 especially) and the reduced margins makes some paragraphs hard to read.
• Equation 2: Symbols $\mu^{2D}$ and $r^{2D}$ are introduced there but never used elsewhere.
• Equations 3, 4, and 6: the 2D projection of $g_i(p)$ should be used, rather than the 3D position.
• $\mathcal{M}_{SG}$ could be formally defined for clarity/reproducibility (i.e., no equation is currently provided for this similarity function).

Additional References:

• Xu et al. "TIGER: Text-Instructed 3D Gaussian Retrieval and Coherent Editing." arXiv preprint arXiv:2405.14455 (2024).
• Liao et al. "CLIP-GS: CLIP-Informed Gaussian Splatting for Real-time and View-consistent 3D Semantic Understanding." arXiv preprint arXiv:2404.14249 (2024).

1.The paper claims that the proposed SLAM method outperforms existing NeRF-based and 3DGS-based approaches. However, in camera pose estimation, the accuracy of CG-SLAM [ECCV'24] and RTG-SLAM [SIGGRAPH'24] is much higher than your method. Their average accuracy is 0.27 and 0.18 respectively, while the accuracy of this paper is 0.30 in Replica dataset (Table 2). For reconstruction performance, SGS-SLAM [ECCV'24], which is also a semantic SLAM system based on 3D Gaussian Splatting, has higher reconstruction accuracy than this paper. For rendering performance, other Gaussian SLAM methods, such as Gaussian-SLAM and MonoGS[CVPR'24], outperform your method. The results of this paper fail to prove the effectiveness of your method. 2.In SLAM, real-time performance is very important. This paper does not show the running speed of the system. 3.What’s the memory usage of storing the entire scene? 4.This paper does not compare visualization results with existing Gaussian SLAM methods, such as SplaTAM [CVPR'24] and MonoGS [CVPR'24], which are all open source. 5.In terms of innovation, the concept of contrastive learning has been widely used in semantic Gaussians, and the contrastive learning innovation in this paper is just constructing a loss. The innovation of the paper is relatively poor.

• Although this application is new, it is not novel to apply distillation from a 2D-trained model into 3D, and it seems this paper addresses a similar problem in feature distillation as seen in methods such as LangSplat and Feature-3DGS.

• I have significant concerns about the writing: Section 2.3 should be moved to the preliminary part. Figure 1 and Figure 7 does not convey much new information compared to earlier literature.

• The experiments lack clarity: it is unclear why feature distillation improves both tracking and rendering accuracy, as SLAM uses a dense RGB-D sequence, which is sufficient for mapping, and there is no new method for the interaction between mapping and features (Tab. 5). Additionally, CLIP only predicts image-level embeddings, so how is it possible to integrate the CLIP model into 3D-GS directly (e.g., pixel-aligned CLIP in line 883)?

• It is crucial to upload a supplementary video to visualize the effects of distillation consistency and the quality of 3D segmentation and related tasks, whereas they are missing.

1. In line 231, How to determine the "well-reconstructed parts of the map"? It would be good to provide more detailed explanations? One possible way I think, as suggested in SplaTam based on rendering errors of different image parts.
2. In the “Intra-Inter Semantic Consistency Objective” module, the underlying insight for intra-frame consistency is that pixels belonging to the same object should share identical semantic features. However, the method appears to assume that the segmentation of each frame is consistently accurate. In practice, pixels on the edges of a 2D mask often exhibit high uncertainty, which may require more careful processing to ensure robustness.