3D Vision-Language Gaussian Splatting

(we copy some of the new relevant tables form the Appendix below, to facilitate the reviewer's task.)

Table 19. Ablation results on different levels of disentanglement between the per-modality Gaussian parameters, evaluated on the downstream open-vocabulary semantic-segmentation task on LERF scenes.

Q1. Per-scene training of fusion module

The reviewer is correct that the fusion module is trained for each scene. We followed the protocol from prior works (LangSplat, GOI) in optimizing our framework separately for each scene. We agree with the reviewer that the generalization of language-embedded Gaussian representations, or at least some of their components (e.g., fusion module), is an interesting open challenge; but we believe that it falls beyond the scope of our current work.

E.g., an obstacle towards generalizing our fusion module across scenes is its dependency on the feature-compression model (since the fusion module receives compressed language features). This compression model is itself specific to each new scene (in LangSplat and our work, we use an auto-encoder trained on the subset of scene-specific language embeddings; but other works also use scene-specific compression techniques, such as codebooks in GOI). Generalizing the compression model would come with a significant computational cost for the whole framework: current compression techniques are extremely effective because they are scene-specific. E.g., our auto-encoder can reduce CLIP vectors from 768 dimensions to only 3 simply because the number of unique language concepts relevant to a single scene is relatively small.

A less effective compression would mean a higher dimensionality for the Gaussians (c.f. larger 3D language vector), impacting everything from inference time to model optimization. Generalizing components of language-embedded Gaussian-based frameworks is, therefore, a challenging task, deserving of its own research.

Q2. View interpolation

To be candid, we were also pleasantly surprised by the effectiveness of the proposed view-blending augmentation, confirming our intuition w.r.t. the impact of cross-view regularization over the sparse language modality. For an additional discussion on augmentation strategies, we invite the reviewer to check our response W2 to Reviewer Awye.

As mentioned there, we could use the color results of the 3DGS model to guide the augmentation process, either in an online or offline manner. Similarly, we could also, indeed, extend the 3DGS model to render depth images, to help reproject/wrap language maps to new viewpoints. However, such a depth-based strategy comes with similar downsides as the one proposed by Reviewer Awye:

• As an online augmentation:
  • Depth-based image wrapping is a computationally-heavy operation compared to 2D interpolation, so its use as online augmentation may be unrealistic for many setups.
  • Adding the online rendering of depth maps by the Gaussian model would further slow down the training.
• As an offline augmentation:
  • It would require to pre-train a vision-only 3DGS model, splitting the overall training back into two phases and thus slowing it down (see Table 8).
  • The benefits of offline augmentation are usually limited compared to online augmentation (the former only adds a fixed/limited number of images to the training set, whereas our augmentation scheme provides new randomized language maps at every training iteration). Please refer to the new Table 15 [Reviewer Awye] to see a comparison of our online mix-up strategy with a more realistic offline solution.

Q3. Further disentangling visual and language Gaussian representations

We thank the reviewer for another thoughtful question. The main reason why prior 3DGS-based or NeRF-based language-embedded scene representations subordinate the semantic modality to the visual one is that the 2D semantic maps are too sparse to regress the underlying 3D scene geometry from them. The visual/shading information provided by the corresponding color images is required for the 3D representation to properly converge. Moreover, for some downstream tasks such as 3D editing, it is necessary to maintain the same underlying geometry across modalities. For these reasons, we decided to keep all spatial properties of the 3D Gaussians (i.e., mean position and covariance) common to the two modalities, and to disentangle only the modality-specific parameters (radiance vs. language vectors, opacity vs. semantic indicator).

We substantiate the above with a new experiment, shared in Table 19, Appendix A.3.5. We evaluate multiple variations of our solution over the open-vocabulary semantic segmentation task, with different levels of disentanglement between the represented modalities. As expected, the model with completely separate Gaussians for each modality (similar to training a semantic-only 3DGS model for the segmentation task) fails to effectively learn the underlying 3D information and converges poorly, impacting the downstream results on unseen views. Our solution performs the best out of the four.

Q1. Notations $F^W$ and $\mathcal{L}_{\text{sem}}$

While the 3D language embeddings $f^i$ (i.e., language vectors attached to each 3D Gaussian $i$) indeed do not change with the view angle, the rasterized 2D semantic features $F^W$ do depend on the view $W$. For a given pixel location $v$ (e.g., coordinates (100, 100) in the image), the language embedding $F(v)$ assigned to it will change depending on where the camera is looking (e.g., if the camera is placed in front of cup, then pixel (100, 100) will be assigned a cup-related language vector, but this will change if the camera faces a different object). We apologize for not making it clearer in the paper and we will try to improve.

Regarding $\mathcal{L}_{\text{sem}}$, we agree that an example of a loss function typically used in prior works could be provided. We have edited the manuscript accordingly. Thank you for the suggestion!

Q2. Distribution symmetry in Figure 3

We would like to kindly point out that the distribution of $l^i - o^i$ for the ramen scene, shown in Figure 3, is actually not symmetric. E.g., the peak near 1.0 is higher than that near -1.0; the values around -0.10 are higher than those around +0.10; etc. It should also be noted that the overall opacity and semantic-indicator values of the 3D Gaussians depend on the content of each scene.

With the example shared in Figure 3, our goal is simply to highlight that there are scenes/objects that would greatly benefit from disentangling opacity and semantic indicator. I.e., for these scenes/objects, their Gaussian representation is parametrized by significantly different $l^i$ and $o^i$.

Q3. Efficiency analysis on all scenes

We provide the results on all the other scenes of the LERF dataset in Appendix A.5.2, c.f. new Tables 21-24. These results confirm that our method outperforms prior art in terms of computational and memory efficiency, with an average training time of $73.6$ min per scene and an average FPS of $76.2$.

Q4. HUGS reference

We thank the reviewer for their suggestion, and we have added a reference to HUGS into the Related Work section of our paper.

Q5. Grammar error

We thank the reviewer for their appreciation of our writing and paper structure. We have run a grammar check on L356-357, but it did not return any error. Did the reviewer mean another line? Note that we will perform a grammar and spell check of the manuscript once more when preparing the final version.

(we copy some of the new relevant tables form the Appendix below, to facilitate the reviewer's task.)

Tables 21-24. Efficiency and image-quality analysis on LERF’s data, averaged over all scenes.

W1. Evaluation of contributions on objects sharing similar colors

We thank the reviewer for their remark. We agree that our submission would benefit from providing more insight w.r.t. the impact of our contributions on specific challenges, such as scenes containing objects with similar appearances but different language embeddings.

Therefore, we added two figures to our manuscript, analyzing the impact of the cross-view regularization brought by our augmentation technique on objects with similar colors/textures, as well as on objects composed of arts with inconsistent appearance:

• Figure 18 shows examples of multiple objects sharing the same colors. E.g. in the first scene, the scissors and spatula share the same color, leading part of the scissors to be misclassified as spatula when camera-view blending is not applied. Similarly, in the second scene, the pot and table share the same wooden texture, causing part of the pot to be misidentified as table by the model without camera-view blending.
• Figure 19 shows examples of objects composed of multiple colors/textures. In the first example, the camera and the spatula are composed of parts with different colors, and some of these parts end up misclassified by the model without camera-view blending. The same can be observed in the second example: the bike is composed of parts with varied appearances, and the model optimized without our augmentation misses many of them; whereas the model optimized with correctly identifies the whole object.

W2. Rendering speed

Among our contributions, only the cross-modality fusion operation and semantic operator are applied at inference time, and both have a negligible computational impact:

• Our fusion module is simply composed of a self-attention operation and barely takes $\sim$5-7 milliseconds.
• Our semantic indicator replaces the opacity value passed to the rasterization function during language-embedding rendering. Therefore, it has no computational/dimensional impact on the inference (this additional parameter only impacts the memory footprint of the model).

The faster inference speed compared to other methods comes from the integration of RGB image rasterization and language-feature rasterization into a single stage in our custom CUDA implementation (instead of the sequential RGB-then-language rasterization performed in prior works). This integration explains why our method, along with GS-Grouping (Ye et al., 2024), outperforms other approaches significantly on the FPS metric in Table 8. Note that the reason why our method is faster than GS-Grouping probably lies in the lower number of Gaussians that our model needs to represent the scene, as rendering time scales with that number.

W3. Succinct title

We appreciate the reviewer's suggestion. We agree that our title may be a bit short and overly catchy. As implicitly mentioned in the abstract and introduction, our justification is that, unlike prior works that subordinates the language modality to the appearance one, we are the first to focus on striking a balance between the two modalities; and thus the first to propose an actual "vision-language Gaussian splatting model", instead of a "vision model extended to language". We are brainstorming a more explicit title, maybe along the lines of "3D Vision-Language Gaussian Splatting: Restoring the Balance between Modalities" (?).

Q1. Impact of semantic learning on rendering quality

We have added PSNR and SSIM results to Table 25 in Appendix A.5.2, i.e., for various scenes of the LERF dataset. While this work focuses on semantic accuracy, our solution maintains competitive performance w.r.t. image quality, compared to the state-of-the-art, highlighting that our effort to strike a balance between the two modalities did not penalize the visual one. Moreover, comparing to the results from color-only 3DGS (same as LangSplat as this method fixes all 3DGS parameters after its pre-training), we observe that semantic learning does not have an obvious impact on color-modality metrics (i.e., image quality).

We thank the reviewer for giving us the opportunity to present these results and to improve our submission accordingly.

W6. Detailed statistics for translucent and reflective objects

We thank again the reviewer for their constructive suggestion. We agree that our paper could benefit from showing results for each object category separately, to highlight how objects with translucent/reflective properties benefit the most from our semantic-indicator contribution. We have performed this evaluation on various scenes, and we have added the corresponding results and discussion to Appendix A.5.3 (see Tables 26-30).

As shown in Figure 3, translucent and reflective objects are mostly represented by 3D Gaussians with low opacity values (in order to model their complex light transport). Using these low opacity values to rasterize the language features cause unwanted artifacts. This is alleviated by our semantic indicator. Our model correctly learns higher indicator values for Gaussians representing the above-mentioned object categories (e.g., see large $l - o$ results for the translucent glass or reflective bottle/cup in Figure 3). The improvement on downstream semantic tasks, e.g., open-vocabulary semantic segmentation, is thus especially large for these categories. E.g., the segmentation of glass is improved by $14.5\%$ and sake cup by $14.3\%$, whereas the segmentation of non-translucent/non-reflective is improved by $3.6\%$ on average, c.f . Table 26.

Q1. Self-attention vs. cross-attention

Our framework leverages cross-modality self-attention, where query, key, and value are all $c^i \oplus f^i$ as shown in Equation 5. In the ablation on cross-modal rasterizer (Section 4.3), the cross-attention fusion tested in Table 4 is performed by using $f^i$ as query and $c^i$ as both key and value.

Q2. Dimension reduction

For the extraction and compression of the language features, we follow LangSplat's pipeline (Qin et al., 2024), c.f. L157-161. I.e., an auto-encoder is trained to compress the scene-relevant CLIP features from their original size (768) to a much more compact $d_f$ target (with $d_f=3$ in LangSplat and our work).

Afterwards, both the Gaussian splatting model and the downstream tasks only rely on the compressed feature space. E.g., for open-vocabulary downstream tasks, the input queries are first encoded by the same auto-encoder network, before being compared to language features generated by the Gaussian splatting model.

We hope that this answers your question. We will further clarify in the manuscript.

(we copy some of the new relevant tables form the Appendix below, to facilitate the reviewer's task.)

Table 15. Comparison between data-augmentation strategies, over downstream open-vocabulary semantic segmentation (mIoU) on LERF scenes. Note that the original training sets contain between 124 and 297 images per scene. For the offline augmentation using 3DGS-rendered images and off-the-shelf open-vocabulary models (SAM/CLIP), 120 novel views are added, with their viewpoints sampled via interpolation.

Table 20. Evaluation on 3D-OVS dataset extended to additional scenes.

Table 26. Per-category mIoU results for the open-vocabulary semantic segmentation task on LERF's ramen scene. Categories with a "*" are object with translucent/reflective properties.

(Tables 27-30 in the Appendix provide similar results on other LERF scenes.)

W1. Role of $u^i$

The notation $u^i$ represents the fused features, i.e., the concatenated color features and language features passed to the rasterizer.

W2. Mix-up vs. 3DGS+CLIP-based data augmentation

We agree with the reviewer that one could apply the selected off-the-shelf language-feature extractor (e.g., CLIP) to generate new pseudo-ground-truth features. This could be done either:

• online, i.e., using the model's predicted color image as input to the language-feature extractor in order to generate the target language map to evaluate the one rendered by the model.
• offline, i.e., using a pre-trained color-only 3DGS model of the scene to generate novel views, pass these views to the feature extractor, and use all the resulting data to train the language-embedded representation.

However, the computational cost of running the selected language-feature extractor (CLIP) is too heavy for online usage, and the offline strategy also comes with several downsides. First, one of our contributions is our joint CUDA-based rasterization function, which enables the fast rendering of both modalities (color and language). While most prior works (Qin et al., 2024) train their model in two phases (a first phase to train a color-only 3DGS model, then a second phase to extend it with language information), we leverage our custom rasterizer to efficiently optimize our vision-language model in a single training phase (see Table 8 highlighting our training and rendering efficiency). The downside of this faster training is that, unlike prior solutions, we do not have access to a pre-trained 3DGS model of the target scene. Adding a pre-training phase for the color-only model would have a significant computational overhead. Second, the aforementioned strategy (i.e., rendering additional views using a pre-trained 3DGS) only adds a fixed/limited number of images to the training set (c.f. offline rendering before the training starts), whereas our augmentation scheme provides new randomized language maps at every training iteration (c.f. online augmentation).

We quantitatively compared the suggested off-the-shelf augmentation strategy to ours in a new experiment, which confirms that our augmentation technique does indeed result in a more robust representation w.r.t. downstream semantic tasks. Please refer to the new Table 15 and corresponding discussion added to Appendix A.3.4. Note that in future work, it would be interesting to investigate how both strategies could be optimally interleaved (e.g., by pretraining our model with mix-up augmentation; then, after some iterations, augmenting the training data with 3DGS/CLIP-rendered pairs).

W3. Global ablation study

We thank the reviewer for the suggestion. We have added a global ablation study on two datasets (LERF and 3D-OVS) presented in Tables 11-12 of Appendix A.3.1. The results illustrate that each of the three components plays a crucial role in improving the performance.

W4. Training efficiency of vision-language 3DGS models

The provided time values are high since they include pre-training phases, such as the extraction of language features (i.e., applying CLIP to input images) and the training and inference of the auto-encoder to compress the said CLIP features.

It should be highlighted that the training times shown in Table 8 are actually in the expected range for Gaussian splatting model covering both RGB and language modalities (i.e., much slower than RGB-only models due to higher dimensionality, dual rasterization, etc.). Related works share similar training times as the ones that we measured — see Table 3 in the GOI paper (Qu et al., 2024) for example.

W5. 3D-OVS scene subset

We follow the protocol adopted in LangSplat (Qin et al., 2024), where experiments are performed on the five most relevant scenes. While the 3D-OVS indeed contains a total of ten scenes, even the dataset authors only use six scenes in their paper presenting 3D-OVS. While we believe that it is thus reasonable to evaluate on 5 scenes, we agree with the reviewer that the more results, the better.

We thus provide new results on two extra scenes, table and snacks in Table 20 of Appendix A.5.1. The results shared there confirm the effectiveness of our solution in terms of open-vocabulary semantic segmentation.

W1. Differences and relationships between color and semantic representations

We thank the reviewer for their suggestion. Additional experiments and discussions have been added to the Appendix:

• In new Tables 26-30, we analyze the impact of disentangling appearance and language rasterization on translucent and reflective objects (in terms of mIoU for the open-vocabulary segmentation task). As discussed with Reviewer Awye and in Appendix A.5.3, in conjunction with Figure 3, we observe that translucent and reflective objects tend to be represented with Gaussians having low opacity values. After introducing our semantic indicator to replace the opacity during language rasterization, we note a large difference in the opacity values (low) and semantic-indicator values (high) learned by these Gaussians (e.g., see large $l - o$ results for the translucent glass or reflective bottle/cup in Figure 3). The improvement on downstream semantic tasks, e.g., open-vocabulary semantic segmentation, is thus especially large for these categories. E.g., the segmentation of glass is improved by $14.5\%$ and sake cup by $14.3\%$, whereas the segmentation of non-translucent/non-reflective is improved by $3.6\%$ on average, c.f . Table 26.
• In new Figures 18-19 in Appendix A.4.4, we qualitatively demonstrate the impact of color over-fitting over specific scenes. As discussed with Reviewer Jaap:
  • Figure 18 shows examples of multiple objects sharing the same colors. E.g. in the first scene, the scissors and spatula share the same color, leading part of the scissors to be misclassified as spatula when camera-view blending is not applied. Similarly, in the second scene, the pot and table share the same wooden texture, causing part of the pot to be misidentified as table by the model without camera-view blending.
  • Figure 19 shows examples of objects composed of multiple colors/textures. In the first example, the camera and the spatula are composed of parts with different colors, and some of these parts end up misclassified by the model without camera-view blending. The same can be observed in the second example: the bike is composed of parts with varied appearances, and the model optimized without our augmentation misses many of them; whereas the model optimized with correctly identifies the whole object. The proposed technique enhances semantic consistency across diverse views through regularization.

W2. Clearer Figure 2

Following the reviewer's remark, we have added Figure 9 to Appendix A.1 (implementation details), depicting parts of the original Figure 2 with larger fonts and spacing.

Q1. Scene content benefiting from our contributions

We thank the reviewer for their interesting question. As mentioned in our above W1 response, our introduction of the semantic indicator especially benefits object categories with complex light interactions (e.g., translucent or reflective materials), since prior work suffer from re-using the Gaussians' opacity values to rasterize the language maps (see Figure 3 and Tables 26-30 for further discussions). Furthermore, the cross-view regularization introduced by our view-blending augmentation technique appears to benefit object categories sharing similar colors, as well as object categories composed of multiple parts with different colors/textures, as empirically shown in Figures 18-19 and discussed in Appendix A.4.4.

We would like to add that it would be interesting to measure the performance of existing vision-language Gaussian splatting models as a function of the complexity of the scenes (in terms of increasing number of depicted objects and/or texture complexity), but the literature lacks such a benchmark. Collecting and annotating 3D scenes with increasing complexity to build such a benchmark would greatly benefit the community, we believe (though it goes beyond the scope of our current work).

(we copy some of the new relevant tables form the Appendix below, to facilitate the reviewer's task.)

Table 26. Per-category mIoU results for the open-vocabulary semantic segmentation task on LERF's ramen scene. Categories with a "*" are object with translucent/reflective properties.

(Tables 27-30 in the Appendix provide similar results on other LERF scenes.)