GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting

We thank the reviewer for their thoughtful feedback and kind words about our work.

Computational cost is not real time.

The reviewer is right in noting that the graph cut algorithm does not run in real-time. However, once segmented into foreground and background Gausisans, it can be rendered in real-time. We kindly refer the reviewer to Table 2 (global response) to see a detailed analysis of pre-processing, training, and segmentation time for our method. Since our method does not perform changes to the 3DGS optimization, the overall time is much less than the baselines. Exploring real-time segmentation is an exciting future direction for this project.

Mapping user input to gaussians could be better described.

In 3DGS, we can keep track of the Gaussians splatting to a particular pixel in 2D while rasterizing from that viewpoint. So for a particular Gaussian, we can compute the ratio of pixels splatted in the foreground (as per the 2D mask) and the total number of pixels splatted by that Gaussian. This gives us the coarse estimate. Thank you for highlighting this. We will add an equation and an illustration to make this more clear to the reader.

Intuition for the cluster similarity term

The intuition behind adding a cluster similarity term is to improve the accuracy of foreground identification. If a Gaussian is in a region with other Gaussians that are likely foreground, it might also be foreground. We start with a coarse estimate of each Gaussian's likelihood of being foreground or background, derived from coarse splatting. However, this initial estimate can have errors due to inaccuracies in the 2D segmentation masks. Nodes with high confidence (weights $w_g$ close to 1) of belonging in the foreground can serve as prototypes for other similar nodes. Directly finding the closest high-confidence node for each Gaussian is computationally expensive. Therefore, we cluster these high-confidence nodes to reduce the computational load. We also present an ablation study on the number of clusters in Table 8. After experimenting with various heuristics, this clustering approach proved to be quite effective.

Without the mask refinement method (just from user scribbles), by how much the performance go down?

We kindly refer the reviewer to Figure 12 which shows this effect qualitatively. If we do not use any 2D mask (just rely on the user scribbles), we only get the area around the scribble from coarse splatting. Applying graphcut has much more significant benefits in this case. We show the quantitative results on five scenes from the LLFF dataset (scribbles following NVOS bnchmark). The numbers in the table are IoU, classification accuracy.

The performance drop is natural as the default implementation uses masks from multiple views but graph cut on 3D Gaussians can still retrieve major parts of the objects even with just the scribbles. We can include more qualitative results in the appendix to show this.

We thank the reviewer for accessing our work and providing valuable feedback. We provide clarifications to the concerns and questions they raise.

Justify the tech novelty. Graph cut from 2d pixels to 3D Gaussians seems very straightforward.

Extending graph cut from 2D pixels to 3D Gaussians involves several non-trivial design choices and findings. We kindly refer the reviewer to the global response.

Comparison with LangSplat and Gaussian Grouping and detailed running speed comparison

We kindly refer the reviewer to Table 1 (global response) for comparison with LangSplat and Gaussian Grouping, and a detailed speed comparison in Table 2 (global response). For the 3D-OVS dataset, we provided a comparison with these two baselines (Table 10). We would like to clarify that the last line of Table 10 is the average over the Lawn scene (not the overall average). Our overall average is much higher than other baselines as shown below. A reason behind this is that Gaussian Grouping and LangSpalt optimize features for all the Gaussians and it therefore limits interactivity with specific objects. Interactivity here refers to the choices made when selecting the 2D supervision mask. Once a supervision mask is chosen, the underlying feature field is optimized based on that. On the other hand, for GaussianCut, the 2D masks are chosen based on the user input.

Overall average on the 3D-OVS dataset. The reported metric is IoU.

Since the code for CoSSegGaussians is not public yet, we can’t provide a comparison against that.

Dependence on video segmentation model

We use video segmentation models for better performance, but our method can work with just one mask as well (Table 4 and Figure 1 in the rebuttal pdf). While our model performance does suffer when using a single mask, it can still perform reasonable segmentation. Langsplat, Gaussian Grouping, CoSSegGaussians, however, require a segmentation mask for all training views.

Limited improvement from proposed n-links and t-links

Results in Table 5 show the average performance on 7 scenes. While the effect of each individual component might not seem a lot on average, the effect can be significant depending on the scene. We show two scenes from the SPIn-NeRF dataset where removing the n-links can have a significant effect. GaussianCut also retrieves fine details (as shown in Figure 4 and 10) which are otherwise missed when just using the semantic maps. The numbers in the table are IoU / classification accuracy.

Comparison on more datasets like Replica and Lerf-mask

We thank the reviewer for suggesting additional datasets. We show quantitative results on four datasets: LLFF from NVOS, SPIn-NeRF, 3D-OVS, and Shiny, and have also compared performance against SA3D, ISRF, SAGA, SAGD, LangSplat, Gaussian Grouping. In addition, we take scenes from mip-NeRF and LERF (Figure 8) to show qualitative results. If needed, we can provide quantitative results on more benchmarks for the revised paper.

How will GaussianCut perform when video segmentation/track fails due to large motion? which may lead the coarse segmentation containing large portions of errors.

We kindly refer the reviewer to Figure 5-7. We have also added extreme case (no video segmentation model) qualitative result in Figure 1 (rebuttal pdf). The quantitative results (IoU / classification accuracy) are shown below:

How many Gaussians are considered during the graph construction?

We consider all the Gaussians to construct the graph. The number is same as base 3DGS model, ranging from ~850k to 4M for all of our scenes (time taken between 40 sec to 4 mins).

We thank the reviewer for their constructive review. We would like to respond to the questions and concerns that they have posed.

Key contributions clarification

We kindly refer the reviewer to the global response which provides a detailed explanation of our key contributions.

Details of associating 3D Gaussians with 2D masks

Gaussians are assigned coarse estimates based on their weights during alpha-rendering at rasterization, i.e., for the Gaussians splatted onto one pixel, it is assigned a different weight just for this pixel. This weight is based on its contribution at alpha blending. However, we had experimented with binary weights as well and we did not observe any major performance difference. The binary weights also give a similar performance in our experiments. We will include these details about the mapping of masks to the Gaussians in the revised paper. Thank you for the feedback.

Time cost analysis

For a detailed speed analysis, we kindly refer the reviewer to Table 2 (global response).

We thank the reviewer for the follow-up.

Coarse splatting

We elaborate on the coarse splatting in further detail below:

Consider an optimized 3DGS model for a scene, $\mathcal{G}$. For $n$ viewpoints, we obtain masks $\mathcal{M}$ := {$M^j$}$_{j=1}^n$

from a video segmentation model corresponding to $n$ viewpoints $\mathcal{I}$ := {$I^j$}$_{j=1}^n$ . $M^j$ indicates the set of foreground pixels in the viewpoint $I^j$.

For each Gaussian $g \in \mathcal{G}$, we maintain a weight, $w_g$, that indicates the likelihood of this Gaussian belonging to the foreground. To obtain the likelihood term $w_{g}^j$ pertaining to mask $j$ for Gaussian $g$, we unproject the posed image $I^j$ back to the Gaussians using inverse rendering and utilizing the mask information,

$$w_g^j = \frac{\sum_{**p** \in M^j } \sigma_g(**p**)T_g^j(**p**) }{\sum_{**p** \in {I}^j} \sigma_g(**p**)T_g^j(**p**)}$$

where $\sigma_g(**p**)$ and $T_g^j(**p**)$ denote the opacity and transmittance from pixel $**p**$ for Gaussian $g$. If $g$ does not contribute to $**p**$, the transmittance is taken to be $0$. Combining over all the masks,

$$w_g = \frac{\sum_{j}\sum_{**p** \in M^j } \sigma_g(**p**)T_g^j(**p**)}{\sum_j\sum_{**p** \in {I}^j} \sigma_g(**p**)T_g^j(**p**) }$$

As mentioned in the paper, we use the same formulation as proposed by GaussianEditor [a] and kindly refer the reviewer to their paper for further details.

For binary weights (hard assignment), we simply keep a count of the number of foreground and background pixels the Gaussian $g$ splats to in $I^j$.

$$w_g = \frac{\sum_{j}\sum_{**p** \in M^j } \mathbb{I}(T_g^j > 0) }{\sum_j\sum_{**p** \in {I}^j} \mathbb{I}(T_g^j > 0)}$$

This $w_g$ is used directly in Equation 3 in the paper. We show the IoU on several scenes comparing soft assignments and hard assignments. Since soft assignment has marginally better performance, it is our default implementation.

Novelty concerns

Regarding the novelty, the graph construction proposed in our method is significantly different from the point cloud literature. Our proposed method is much simpler and both the approaches mentioned by the reviewer involve training networks to obtain the segments.

• GrowSP: Unsupervised Semantic Segmentation of 3D Point Clouds: This method does not employ graph cut for segmentation. It learns per-point features, extracts superpoints, progressively grows those superpoints, and performs clustering for segmentation.
• SAM-guided Graph Cut for 3D Instance Segmentation: While this method does use graph cut on point clouds, their formulation differs significantly from our approach. They apply graph cut on superpoints which are obtained from another pre-segmentation model. Therefore, when an object to be selected is part of a superpoint, this technique is unable to segment it out. Our approach, on the other hand, provides finer user control as we do not rely on any point cloud pre-segmentation module. Their primary method involves training a graph neural network (GNN) using high-quality pseudo labels whereas our approach is training-free. Their non-GNN baseline also requires masks from multiple views to assign edge weights. Our proposed approach can work even with a single mask. Moreover, since this method stores SAM features, adapting it for 3DGS would require a much higher memory footprint than our approach (please see discussion section).

More generally, while some advancements in point cloud segmentation can be tranferrable for 3DGS, our coarse splatting (section 3.3) and terminal links (section 3.4) are heavily tailored for 3DGS.

[a] GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting, CVPR 2024

We thank the reviewer for their thoughtful and constructive feedback. We have provided clarifications throughout and added additional experiments to address the concerns they raised. We appreciate the reviewers’ attention to detail and will incorporate all the suggestions made by them.

Runtimes of proposed method and baselines

We have added all the runtimes suggested by the reviewer and we kindly refer them to Table 2 (global response).

Inconsistent baselines

We ran all the baselines on the SPIn-NeRF dataset (10 scenes). For ISRF, we get OOM issues at 1008 resolution for 360-degree inward scenes so we resize it to 1/4th resolution.

For the qualitative results in Figure 4, since MVSeg (part of SPIn-NeRF) is designed for inpainting task, extracting the segmentation module was challenging. For SAGA, we run out of memory for the garden scene in RTX 4090 GPU. It's because the scene has considerably more images (185 images, ~4.4 million Gaussians) and SAGA stores a 32-dimensional feature for each Gaussian. Therefore, we were not able to include SAGA in Figure 4. We will also include all the baselines on all the datasets in the final version.

Visualizations of the energy terms

We will be happy to provide visualizations that can improve the clarity of our method. However, it is not clear to us how energy function can be visualized. The weights are assigned on edges, not nodes, which makes visualization tougher. Also, since the energy function is not optimized over time (it is minimized using min-cut algorithms), we can not plot the overall energy either. We would like to request the reviewer to provide suggestions on how meaningful visualizations can be included.

I was confused about what is meant by "decomposing the boundary Gaussians."

SAGD[a] also does not require any segmentation-aware training given an optimized 3DGS model. They decompose Gaussians (split a single Gaussian into two) that may lie at the boundary of an object

The single mask ablation (line 298) should probably be included in Table 5.

We provide the ablation with a single mask for the NVOS benchmark below. We will also include this in Table 5 in the revised paper.

"Gaussians that map to the same object would be closer spatially." assumption justification

Our proposed method is scene agnostic, i.e., it works regardless of the distribution of the underlying Gaussians. With the recent advances in point-based and Gaussian-based techniques, these artifacts might be mitigated and since our model works on a pre-trained 3DGS, it can adapt to such advances. This assumption worked well for the all scenes we tested.

Sensitivity to hyper-parameters

We share a default setting in the paper which performs reasonably on all our datasets. The sensitivity of each parameter can be very scene-dependent. For instance, in a scene where parts of an object have different colors, a very high weight on the color similarity can affect adversely. We show the effect of $\lambda$ (controls the pull of neighboring vs terminal edges) and $\sigma$ (decay constant of the similarity function) on two scenes. The reported metric is IoU.

Coarse splatting baseline with well-chosen hyperparameters

For the four 360-degree inward scenes in the SPIn-NeRF benchmark, we show a sweep of the threshold (default is 0.3). GaussianCut outperforms all the thresholds considered for coarse splatting. It is worth noting that adjusting the threshold of coarse splatting also improves the quality of graph cut (as we directly use these for terminal links weights).

How are the frames ordered before they are passed to the video segmentation model? How sensitive is the method to this ordering?

We run the camera on a fixed trajectory to obtain rendering from different viewpoints (spiral trajectory for front-facing and round for 360-degree inward scene). We limit the number of frames to 30 for front-facing and 40 for 360-degree scenes. However, all the training images can also be used for coarse splatting (Table 6). Although it might not be preferred for scenes that have a large number of training images. SAM-Track is quite good for unordered multi-frame images as well. The results in Table 6 were obtained after directly giving all training images to SAM-Track.

To what extent does the proposed method's performance rely on SAM-Track's quality? Do any baselines rely on different video segmentation models, and if so, how do the metrics change when those are updated to use SAM-Track?

All our experiments are based on SAM-Track and we did not experiment with another model. Our method can work on any video segmentation model and its quality can affect the final performance. Although we start with SAM-Track, our mask quality is improved significantly (Figure 11).

[a] SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition, arXiV, 2024

We appreciate the reviewer for their feedback on our paper. We thank the reviewer’s suggestion for helping to improve the clarity of our work

Clearer motivation for the introduction

Our work is motivated by leveraging the explicit nature of 3DGS representation. With user inputs and the underlying geometry of the scene captured by the Gaussians, objects can be segmented from a scene, without requiring any additional optimization of the 3DGS model. We provide a clearer motivation below and we will include the changes in the revised version of the paper.

3D Gaussian Splatting represents a scene using a set of Gaussians, thereby offering an explicit representation of the scene. Prior works in 3DGS scene segmentation involve augmenting the set of Gaussian with a per-Gaussian feature, that is optimized during the Gaussian fitting, supervised by 2D features. These features provide semantic information and can be used for segmentation. Since 3DGS stores the parameters of each Gaussian explicitly, the size of the feature embedding exacerbates the already high memory footprint of the method. Therefore, such methods have relied on learning low-dimensional features per Gaussian. While this enables a 3D consistent segmentation, optimizing a per-Gaussian feature significantly increases the fitting time of the Gaussians to the scene. Our proposed methodology is a post hoc technique that operates directly on an optimized 3DGS field without requiring any segmentation-aware fitting. We directly tap into the representation and map each Gaussian to its corresponding object(s). We do this by formulating the Gaussians in a scene as an undirected graph and partitioning the set of Gaussians allowing for the extraction of subsets that represent specific objects.

Is the method training-free for segmentation when provided with a pretrained GS model?

Yes, our approach is training-free when provided with a pre-trained GS model. We have highlighted this in the abstract and we kindly refer the reviewer to Figure 1 which shows that GaussianCut operates directly on a pre-trained GS model and user inputs. We will also modify our writing to explain this more clearly. To summarize our implementation, we take an already optimized GS model with user inputs on any one image. The user input is processed into dense masks and the masks are also propagated to multiple images using a video segmentation model. We do a "coarse splatting", which assigns each Gaussian a likelihood ratio, which is the fraction of pixels splatted in the foreground (as per the 2D mask) and the total number of pixels splatted by that Gaussian. This step does not require any further GS optimization. It is computed by simply rasterizing each view that has a corresponding 2D segmentation mask. Finally, we formulate a graph from the Gaussians that uses this additional likelihood term and the inherent properties of the Gaussians already learned in the initial fitting.

As mentioned by the reviewer, this is indeed significant as it provides segmentation without requiring any changes to the fitting process. As noted in Table 2 (global response), it also saves substantial time compared to optimizing features.

Comparison with other 3DGS and per-Gaussian feature baselines

We thank the reviewer for suggesting additional baselines. We would like to clarify that we do provide 3DGS-based baselines. We provide SAGA[a] and SAGD[b] results on the LLFF and SPIn-NeRF datasets (Tables 1, 3, 9), both of which are based on 3DGS. We also compare against Gaussian Grouping, LangSplat, and CGC (all of which are based on 3DGS) in Table 10 (appendix). We had not compared against them on all datasets because our method is focused more on interactive segmentation and these methods propose learning a feature field for all the scene elements (a more detailed distinction between these baselines and our method is provided in the global response). Based on the feedback, we have reported Gaussian Grouping and LangSplat comparison on the NVOS benchmark in Table 1 (global response).

3D evaluations missing

We completely agree with the reviewer that 3D GT is indeed a better metric to evaluate 3DGS segmentation. As noted by the reviewer, most datasets and approaches do not provide such ground-truth and benchmarks for this evaluation. SPIn-NeRF, Shiny, and 3D-OVS, while not 3D consistent, provide masks for multiple views to show the efficacy of methods to some extent. Obtaining the ScanNet dataset requires approval from the authors, which could take up to a week. Since we did not have that much time and the 3DGS baselines (SAGA, LangSplat, Gaussian Grouping) have a larger runtime, especially for scenes with a higher number of images, we could not include the quantitative results for the rebuttal. We will include results from Replica or ScanNet in the final paper.

[a] Segment Any 3D Gaussians. arXiV, 2023
[b] SAGD: Boundary-Enhanced Segment Anything in 3D Gaussian via Gaussian Decomposition, arXiV, 2024