Tackling View-Dependent Semantics in 3D Language Gaussian Splatting

We sincerely thank you for your constructive comments. We hope our response can help address your concern.

Weaknesses

W1: The claimed innovation in 3D Scene Decomposition shows notable similarities with the previous work on openGaussian. I recommend that the authors explicitly discuss these similarities and clarify how their approach differs from and improves upon the prior work.

Thanks for the insightful suggestion. We discuss the relationship between OpenGaussian and LaGa below:

Similarity: Both OpenGaussian and LaGa first decompose a 3D scene into objects before associating the scene with semantic information.

Key Differences in Motivation: The two methods are built upon fundamentally different motivations. OpenGaussian addresses the inconsistency between 3D point-level features and 2D pixel-level CLIP features, attributing it to (1) inaccurate 2D–3D associations introduced by alpha-blending during differentiable rendering, and (2) limited expressiveness from 3D feature compression for efficinet rendering. In contrast, LaGa focuses on tackling the semantic discrepancy among multi-view observations of the same 3D object, which naturally arises in open-vocabulary 3D understanding. This shift in perspective leads to substantially different methodologies.

Key Differences in Methodology: To mitigate 2D–3D feature inconsistency, OpenGaussian decomposes the 3D scene so that it can directly assign a 2D CLIP feature to each Gaussian of an object. It employs a rule-based representative view selection strategy for this assignment. While straightforward, this strategy neglects rich information embedded in multi-view semantics and lead to suboptimal performance.

In contrast, LaGa extracts semantic descriptors across views using adaptive clustering and then adopts the weighted relevance aggregation to suppress noisy semantics and enhance robustness. This design enables LaGa to construct a more comprehensive and robust 3D semantic representation. Notably, LaGa does not involve any 3D vision-language feature training, and thus is also unaffected by the 2D–3D inconsistency problem that OpenGaussian aims to address. The design of LaGa enables it to tackle complex 3D objects with various multi-view semantics, evidenced by its significant +18.7% mIoU improvement over OpenGaussian.

On Scene Decomposition: We acknowledge that our scene decomposition strategy is not the core innovation of LaGa, which is inspired by prior contrastive-learning-based approaches. However, compared to OpenGaussian’s two-stage codebook-based decompsition pipeline, which requires manual tuning of two codebook sizes and thus incurs higher computational cost, LaGa adopts a more lightweight solution, i.e., using HDBSCAN to automatically determine the number of decomposed objects based on feature density. This results in comparable decomposition accuracy while significantly reducing system complexity.

We will incorporate this clarification and discussion in the revised paper for improved clarity and completeness.

W2: In addition to the quantitative comparisons for 3D open-vocabulary localization, I would also like to see 2D query results, as well as qualitative results on the mipnerf360 dataset— for example, for “Room” and “Garden”.

Thank you for the valuable suggestion. We provide several 2D segmentation masks in Figure 8, which demonstrate the 2D projections of LaGa’s 3D query results. Since LaGa does not involve any 2D CLIP feature learning, it is not intended for direct querying on the 2D image plane. Nevertheless, we believe that our 3D-centric paradigm can serve as a more flexible alternative to 2D methods, as the 3D query result can be rendered from arbitrary viewpoints without requiring repeated queries for each individual image.

In addition, we have conducted qualitative experiments on the MIP-360 dataset, including scenes such as Room and Garden, and observed that LaGa generalizes well to these scenes. The corresponding visualizations will be included in the revised paper.

We thank you for your effort in reviewing our paper. We hope the following response would address your concerns.

Weaknesses:

W1: ... does not support multi-scale segmentation like LangSplat ...

Thank you for the comment. LaGa supports multi-scale segmentation, though not sufficiently emphasized in the main paper. Some details are provided in Appendix B under the implementation section. Specifically, during the scene decomposition, LaGa maintains three affinity features for each 3D Gaussian, which are supervised using SAM-extracted segmentation masks at the subpart, part, and whole levels, respectively. During inference, the final result is produced by averaging the responses from these three levels, enabling LaGa to capture semantic concepts across varying granularities.

This multi-scale capability is further reflected in the visualization results. For example, in Ramen scene shown in Fig. 6, OpenGaussian merges the 'kamaboko' with the surrounding noodles due to its lack of multi-scale segmentation ability, whereas LaGa successfully distinguishes it. Similarly, in Fig. 7, LaGa accurately segments the pirate hat from the rubber duck. Moreover, we verify that when queried with 'rubber duck', LaGa retrieves both the duck and its pirate hat, demonstrating its effectiveness in capturing hierarchical part-whole semantics.

We will incorporate the visual results of 'rubber duck' into the revised paper, along with more specifically designed visualizations to better illustrate the multi-scale segmentation ability of LaGa.

W2: ... engineering-driven and does not provide enough knowledge advancement. The idea of addressing the feature inconsistency has been proposed in OpenGaussian.

Thank you for the thoughtful comment. While our method includes practical design choices, we believe it also contributes meaningful insights to open-vocabulary 3D scene understanding. In particular:

1. Identification and analysis of view-dependent semantics: We are the first to systematically identify and quantitatively analyze the phenomenon of view-dependent semantics, as illustrated in Figures 3, 4, and 12. This issue, long overlooked by prior work including OpenGaussian, plays a critical role in open-vocabulary 3D understanding and poses unique challenges for multi-view semantic aggregation.
2. An effective approach for semantic aggregation across views: We propose an effective approach, LaGa, which is able to robustly suppress noisy or inconsistent semantics while preserving informative signals, without requiring manual rules or task-specific heuristics. We believe the simplicity, generalizability, and effectiveness of our approach can serve as a good foundation for future research in this area.

Comparison with OpenGaussian:

While both LaGa and OpenGaussian aim to address feature inconsistency, their motivations and methodologies differ fundamentally.

OpenGaussian focuses on the inconsistency between 3D point-level features and 2D pixel-level CLIP features, attributing the problem to (1) inaccurate 2D–3D feature association caused by the alpha-blending in differentiable rendering, and (2) limited expressiveness due to 3D feature compression for rendering efficiency. To address this, OpenGaussian decomposes the 3D scene, so that it can adopt a hand-crafted strategy to directly assign a 2D CLIP feature from a specific view to all 3D Gaussians within an object. However, this hard assignment discards rich multi-view semantic cues, resulting in substantial information loss and ultimately sub-optimal performance.

In contrast, LaGa recognizes feature inconsistency as the semantic discrepancy among multi-view observations of the same 3D object, which arises naturally in 3D scene understanding. To address this, LaGa proposes to aggregate view-specific 2D semantics into comprehensive and robust 3D semantic representations through adaptive semantic descriptor extraction and descriptor reweighting. Note that LaGa does not involve any 3D vision-language feature training, and thus is also unaffected by the 2D–3D inconsistency problem that OpenGaussian aims to address.

The revelation of the critical view-dependent semantics issue and the corresponding improvements help LaGa achieve a significant performance gain: LaGa outperforms OpenGaussian by +18.7% mIoU under the same experimental setting, validating the effectiveness of our approach.

Questions:

Q1: Multi-scale segmentation?

Yes. Please refer to our response to W1.

Q2: Runtime statistics of this method (e.g. how long and how much memory does it take)?

LaGa is highly efficient. We evaluate its runtime performance on the LERF-OVS dataset using a single NVIDIA RTX 3090 GPU. The inference time per query ranges from approximately 80 ms to 200 ms, with peak GPU memory usage between 5 GiB and 13 GiB. Note that compared with 2D methods like LangSplat, which takes about 250 ms per query on the rendered feature map, LaGa directly deliver 3D point-wise understanding.

Thanks for careful evaluation.

Weaknesses

W1: Core part similar to SAGA, GARField ... seem crude and not impactful...

This is a misunderstanding about our method's core. Rather than decomposition, its core lies in the view-aggregated representation, which we believe is not crude.

It consists of two novel modules:

1. Cross-view descriptor extraction adaptively captures multi-view semantics for objects.
2. Weighted descriptor relevance aggregation refines the representation by assigning weights to descriptors. Unlike hard filtering with information loss, all descriptors are preserved.

LaGa improves 18.7% mIoU over SOTA, and ablation shows +10.6% over baseline. This is clearly impactful.

W2: ... require careful tuning ...

The view-aggregated representation needs no sensitive tuning. Its K_max is fixed to 20 across all scenes/datasets. It is stable across K_max from 5–30.

W3: HDBSCAN ... how to obtain number of clusters N_p ...

HDBSCAN automatically infers N_p. Its epsilon is fixed to 0.3 without per-scene tuning. As shown below, LaGa has stable performance across a wide range (0-0.3). Too large epsilons lead to object merging.

W4: Sensitive to SAM masks

Boundary regions mis-segmented in one view often appear as interior in others. Through multi-view aggregation, these inconsistencies can be eliminated in 3D, similar to prior work (SemanticNeRF, SA3D).

W5: ... reason for k-means ... why not FPS

We adopt k-means to reduce noise by aggregating features locally in the semantic space. In contrast, FPS may preserve more outliers.

For the concern about averaging distant features, we add an denoising to discard features far from their centroids (brings a minimal +0.2% mIoU). The FPS performs worse:

W6: ... evaluations in 2D ...

No standard 3D benchmark exists for open-vocabulary 3D-GS. We think a sparse set of well-chosen 2D annotated views is a proxy, as neighboring views have redundant information. We check LERF-OVS annotations and find them sufficient in multi-view coverage. This is why most works use it as the main evaluation.

W7: Ablation W7-1: ... -DW vs +DW on F. and T.? W7-3: T., Avg pooling ...

The performance fluctuations in F. and T. (Table 4) stem from the sensitivity of fixed k-means clustering. Manually set K may be too small to capture semantic diversity or too large causing outliers. Both are harmful for hard cases.

In Figurines, querying "Pikachu" targets a plastic bag with a Pikachu print. Even K=5 splits the "Pikachu bag" semantics from "bag". By +DW, they are down-weighted as outliers. In Teatime, CLIP sometimes misclassifies a plate as "apple". Avg. pooling or small K merge such descriptors, while K=10 offers better separation. K=20 introduces additional noise. The avg. pooling performs well in Teatime because its objects have more consistent multi-view semantics than other scenes. Avg. pooling is independent of K, simply averaging all multi-view features per object.

Although corner cases are rare, they affect performance (Pikachu: 97% to 31%; Apple: 87% to 21%) and bring instablity. Our adaptive strategy successfully handles them and enables consistent gains with +DW.

W7-2: Waldo kitchen...

Thanks for finding this. It is an error caused by swapped DW^c and DW^d results between the F. and W. columns. The correct results are

W7-4: ...adaptive scheme not clear winner

This is a misunderstanding. The adaptive scheme is designed to deliver consistently strong and robust performance across diverse scenes without per-scene manually tuning.

W8 (reference): SAGA, GARField ...

They are designed for class-agnostic segmentation, rather than our task. To our knowledge, no prior work evaluates them on LERF-OVS. For comparison, we adapt the released SAGA code for 3D-OVS to deliver the following results:

Questions

Q1: Binarized gaussians with 2D GT masks.

During evaluation, for each query, binarized gaussians are projected to 2D via 3D-GS rendering. The resulting 2D masks are compared with GT without any associating.

Q2: Cross-View Descriptor Extraction as ablation.

Disabling it is infeasible, as 3D objects require at least one descriptor. Instead, we evaluate it by comparing adaptive scheme with alternatives (Sec. 6.4). '-DW' (Table 4) show results of this module work alone.

Q3: Larger clusters L2-norm > 1.

This won't happen since all features are L2-normlized before clustering (L. 192).

Q4: Train Langsplat?

No. We do not train LangSplat (neither CLIP features for Gaussians) in LaGa. L632-633 indicates we train three affinity features for each 3D Gaussian to maintain the multi-level SAM segmentation. We will clarify.

We sincerely thank you for instructive comments. We hope our response can help clear your concerns.

Weaknesses

W1: ... failure cases and analysis.

Thanks for the suggestion. We summarize key failure cases and provide representative examples:

1. Bag-of-Words Effect in CLIP: When prompted with phrases like 'pirate hat on the rubber duck' or 'cookies in the plate,' CLIP often activates on individual nouns rather than the intended composite concept. This also affects LaGa. Future work may use large language models for better compositional grounding.

2. Lack of Context in 2D Semantics: LaGa extracts 2D object-level semantics by feeding segmented regions into CLIP. While this reduces distraction from unrelated objects, it also removes necessary context. For instance, in Teatime, hooves are unrecognizable without the full sheep, but with the full object, only 'sheep' is detected since missing part-level cues.

These cases reflect the challenges of real-world open-vocabulary perception and the gap between model and human understanding. We will include this discussion and illustrative examples in the paper.

W2: ... understanding of the weighted descriptor relevance aggregation ... why works needs clarification ...

We hope the following clarification can help address your concern:

Since LaGa does not manually control the segmentation (by SAM) or subsequent semantic feature extraction (by CLIP), the extracted 2D features may contain incorrect or noisy semantics. For example, the spine of a book may look like a 'knife' from certain views. To mitigate this issue, we adaptively weight each semantic descriptor based on two criteria:

1. Directional Consistency: This metric measures the cosine similarity between each individual descriptor and the global semantics of the object. Descriptors inconsistent with the global semantics receive lower weights. For instance, in the case of the book, descriptors representing a 'passport' align closely with the global semantics 'book,' resulting in higher weights, whereas descriptors resembling a 'knife' are suppressed due to semantic inconsistency.
2. Internal Compactness: In addition to clearly incorrect semantics, some descriptors result from ambiguous or noisy segmentation (e.g., SAM oversegmentation). They may lack clear meaning yet distort the global feature and compromise directional consistency. To address this, we introduce Internal Compactness, defined as the L2 norm of a descriptor. If the features in the cluster are semantically consistent and have similar directions, their average will have a relatively large L2 norm (close to 1). In contrast, if the features are inconsistent with diverse directions, their vector average will cancel out, resulting in a lower norm. Thus, the norm serves as a confidence measure for semantic reliability.

Together, these criteria help LaGa emphasize descriptors that are both globally consistent and semantically coherent. We will clarify this design further in the paper.

W3: typos.

Thanks. We will fix them.

Questions

Q1: (1) ... varying segmentation granularity across views? (2) ... inconsistent segmentations affect the final result? Further discussion

Thank you for the insightful suggestion.

(1) To address varying segmentation granularity, we adopt a multi-level modeling strategy (Appendix B). For each 3D Gaussian, we learn three affinity features corresponding to SAM's subpart, part, and whole-level masks, and construct parallel view-aggregated representations. Predictions from all levels are averaged during inference.

(2) When segmentation inconsistency occurs within a level, LaGa resolves it in a statistial learning manner: For a 3D region observed from multiple views with inconsistent segmentation, the dominant label (i.e., the granularity supported by the majority of views) is reinforced through training, enabling convergence toward a stable 3D prediction.

During decomposition, in each level, all masks for the same 3D object are grouped together regardless of granularity. In the cross-view descriptor extraction, masks of different granularities will be assigned to separate semantic descriptors if they exhibit semantic discrepancies. This avoids semantic conflicts during aggregation.

This design also enables multi-granularity semantic retrieval: when queried with part-level prompts, the whole-level may exhibit a high response (e.g., 0.5) if it contains the semantic descriptor of that part. At the part level, the queried part will yield a high response (e.g., 0.6), while the remaining regions produce lower responses (e.g., 0). By averaging predictions, the model produces a high response (0.5 + 0.6) for the queried part, a moderately high response (0.5 + 0) for the whole object, and low responses (0 + 0) for unrelated objects. This hierarchical behavior aligns with human cognition and reflects real-world compositional semantics.

We will add the above discussion to our paper to help understanding.

We sincerely thank you for your time and effort dedicated to evaluating our work. We greatly appreciate your recognition of our motivation, methodological design, and the clarity of our writing. We find the remaining concerns mainly locates in the 'Experimental Designs Or Analyses' section. We hope our responses below will address them.

(Experimental Designs Or Analyses) However, potential dataset bias, limited qualitative analysis, an untested recall threshol d (0.75), and narrow ablation scope (e.g., omitting decomposition details) are issues that could weaken generalizability and transparency if not addressed.

Potential dataset bias: Thank you for the comment. As you mentioned, our evaluation involves diverse benchmarks: LERF-OVS (complex 360° scenes), 3D-OVS (forward-facing views with diverse objects), and ScanNet (real-world scans). We believe this coverage is sufficient to demonstrate the effectiveness of our method. We are not entirely sure which specific bias is being referred to and would be happy to further discuss this concern.

Limited qualitative analysis: Please kindly refer to our appendix for more visualization results including more visual comparisons, multi-view visualizations, and results on the 3D-OVS dataset. To further strengthen our qualitative analysis, we will incorporate additional visualization results about multi-granularity segmentation cases and representative failure cases to the paper.

Untested recall threshold (0.75): The threshold of 0.75 was chosen based on analysis of the precision–recall trade-off under different cosine similarity values. Here, precision refers to the proportion of retrieved 3D Gaussians that actually belong to the corresponding 3D object. Low precision indicates that unrelated Gaussians are being retrieved for a given 2D mask.

As shown in the table below, lowering the similarity threshold (e.g., to 0.7) increases the proportion of samples with high recall (>0.9), but at the cost of significantly reduced precision. For example, at 0.7, the average precision of high-recall samples drops to just 13.4%.

From these observations, we find that thresholds in the [0.75, 0.8] range strike a better balance. We conservatively select 0.75, where 50.2% of samples exhibit low recall, clearly demonstrating the challenge of view-dependent semantics. A threshold of 0.8 also yields valid results, with fewer high-recall samples but higher precision.

Note that the average precision does not reach 100%, as CLIP features operate at the semantic level, not the instance level. Therefore, 3D Gaussians with similar semantics but belonging to different objects may also be retrieved, even with a high threshold.

Narrow ablation scope:

Thanks for the suggestion. To further demonstrate the effectiveness of our design choices, we ablate two core components of the decomposition process:

1. An ablation study on the data resampling strategy used for training the affinity features.
2. A hyperparameter analysis of the HDBSCAN decomposition algorithm.

Effect of Resampling Strategy:

As shown above, removing the resampling strategy leads to consistent performance drops across all scenes, highlighting its importance for stable learning of affinity features.

Effect of HDBSCAN Epsilon:

We empirically set epsilon = 0.3 for all experiments. As the table shows, LaGa achieves strong performance within a broad and reasonable range ($\epsilon \in [0, 0.3]$). Performance degradation only occurs when $\epsilon$ becomes too large, potentially causing unintended merging of semantically distinct objects. This analysis shows that LaGa is not sensitive to this hyperparameter.