WildGaussians: 3D Gaussian Splatting In the Wild

We thank the reviewer for the constructive feedback. We appreciate that you find that “the performance in removing occluders is impressive”, and would like to raise the score if we can justify half-image test-time optimization. We will address all comments below and will adjust the paper accordingly.

W1: Novelty

We consider SWAG (published on arXiv on March 15th, latest revision: April 5th) as concurrent work and have developed our approach independent of the paper. While SWAG also used appearance embeddings, there are important technical differences.

Similar to our approach, the authors of SWAG were motivated by Urban Radiance Fields to try predicting affine transformation parameters for the SH coefficients to model changes in appearance. The poor performance of this approach (see Tab. 8 in the SWAG paper) led the authors to conclude that “affine color transformations cannot model all appearance changes” and motivated the SWAG approach, which uses a feature grid with an MLP for color prediction. This MLP has to be evaluated for all Gaussians for each rendered view. As a result, SWAG has “10 times longer inference time per frame [5] compared to 3DGS”. Our approach shows that the conclusion drawn by the authors of SWAG regarding the expressiveness of affine transformations is wrong. In addition, our approach is both significantly faster to render and achieves better results (see PDF). We believe that this is interesting to the community.

While our approach for handling transient objects is inspired by NeRF On-the-go, Fig. 3 in the main paper shows that the original formulation from NeRF On-the-go is not directly applicable to our problem setting. This is due to the formulation not being robust to appearance changes (which are not considered in NeRF on-the-go), see L186-195. We believe that showing how to adapt this approach to the case of changing appearance is interesting for the community.

W2: Fairness of the comparison

Thank you for raising this concern. For Photo Tourism, we actually follow the NeRF-W evaluation protocol (use half of the image for test-time optimization and only evaluate on the other half), but did not make this clear in the paper. For the NeRF On-the-go dataset, there is no test-time optimization needed. We will clarify this in the paper. Looking at the source code of GS-W (released after the NeurIPS submission deadline), we noticed that GS-W actually computes appearance information from the full test image. This indeed significantly boosts their performance. When using the NeRF-W protocol, our approach clearly outperforms GS-W. See the PDF for details.

W3: Appearance embedding applied to other viewpoints

We agree showing the consistency of the appearance modeling would benefit the paper. Unfortunately, we did not manage to complete the video in time. For the rebuttal, we include images sampled from our video in the PDF. As can be seen, the scene’s appearance is correctly maintained under viewpoint changes.

Question: GPU memory and training time per scene

Below, we provide the peek GPU memory and training times for individual scenes:

Limitations: Incapable of recovering with too many occlusions

We wanted to express the following: If there are not enough observations of a part of the scene, e.g., because it is occluded in nearly all training images, our approach will struggle to correctly reconstruct the region, even if it filters out all occluders. The same limitation applies to other works in this field, e.g., NeRF On-the-go, GS-W, SWAG, NeRF-W, etc.: It is only possible to reconstruct areas that have been sufficiently observed. Still, as shown in our experiments, our approach works well without access to complete ground truth (we estimate occlusion masks during training rather than using ground truth masks) in realistic conditions. We will clarify this in the paper.

Thank you very much for the positive and constructive feedback! We highly appreciate that you find our approach “commendable, with detailed methods and experiments” and that the paper “demonstrate[s] that WildGaussians can handle complex scenarios involving dynamic objects and varying appearances, setting a new benchmark for real-time 3D rendering and modeling”. We will address all concerns below, will fix the typos, and will adjust the paper accordingly.

Large-scale dynamic objects

Indeed, our method should be able to handle very large dynamic objects. However, please note that some of the objects removed are quite large. As we report in Table 3, for some scenes the occluders occupy on average 26% of the entire image. Moreover, in some scenes in the NeRF On-the-go dataset, occluders can even be buses or trams, which we consider “large occluders”. We will add examples to the paper. For the dataset visualization, please refer to the website of NeRF On-the-go.

Is PSNR computed after masking?

No masks were used for the PSNR computation (nor for the other metrics). The test images of both Photo Tourism and NeRF On-the-go do not have occlusions. We will clarify this in the paper.

Comparisons to SWAG and GS-W

We consider both SWAG (published on arXiv on March 15th, latest revision: April 5th) and GS-W (published on arXiv on March 23, latest revision: July 14th) as concurrent work. We developed our work independently from them. All three approaches (SWAG, GS-W, ours) consider the same problem (modeling scenes with changing appearances from images containing occluders) and closely follow NeRF-based approaches proposed for the problem: they model appearance per Gaussian using appearance features stored per Gaussian and handle dynamic objects/occluders during training. They differ in the way both these parts are implemented:

• To model appearance changes, SWAG uses a feature grid with an MLP for color prediction. This MLP has to be evaluated for all Gaussians for each rendered view. As a result, SWAG has “10 times longer inference time per frame [5] compared to 3DGS”. In contrast, our approach simply uses a shallow MLP to predict an affine transformation of the SH coefficients to obtain the final color. This approach is much more efficient and is “backward compatible” with 3DGS (see L144-147 in the paper), i.e., for a fixed target appearance it is not necessary to evaluate an MLP during rendering. To handle transient objects, SWAG introduces a trainable, image-dependent occupancy term per Gaussian. In contrast, our approach uses pre-trained DINO features to predict which training image regions contain static scene parts respectively dynamic objects.
• To model appearance changes, GS-W combines per-Gaussian appearance features with features selected (per Gaussian) from a reference image to predict colors via MLPs. Similar to our approach, this approach to appearance modeling is significantly more efficient than the one used by SWAG. To model transient objects, GS-W trains a Unet-based model to predict a visibility map from a given training image, while our approach predicts uncertainties by comparing features from rendered and actual images. As can be seen, all three approaches differ significantly in the way they implement both stages. Additionally, our approach outperforms both SWAG and GS-W. We will extend our discussion in L86-91 to more clearly highlight the differences and add the results for SWAG and GS-W to the paper.

The attached PDF compares our approach to both SWAG and GS-W. Please note that the results reported for GS-W differ from those reported by the authors of GS-W. The GS-W code, released about a month ago (i.e., after the NeurIPS submission deadline), uses the full test images for computing appearance information. The results reported for GS-W in the PDF were obtained by adjusting the code to follow the common test protocol used by all other approaches (including ours), which use one half of each test image to compute appearance information and the other half for evaluation. As can be seen, our approach outperforms both SWAG and GS-W. We will add the results to the paper.

I have raised my score to accept. Thanks for your efforts during the rebuttal and discussions. I hope the authors can include the comparisons and discussions in the revision to make the paper more solid and convincing.

Thank you very much for the positive and constructive feedback! We really appreciate that you consider our work “highly-relevant to the research community”, and that our “contributions are clear and simple” while our method “significantly outperforms baselines”. We will address all concerns below, will fix the typos, and will adjust the paper accordingly.

W1: Limited novelty

While we are indeed inspired by both Urban Radiance Fields (URF) and NeRF on-the-go, there are important technical differences in applying these ideas in the context of 3DGS “in the wild”. We believe that they are interesting enough for the community to warrant acceptance.

• Differences to URF: In order to handle exposure changes, URF only models a global affine transformation per image. In contrast, our per-Gaussian embedding enables handling local appearance changes in various parts of an image. The entry “w/o Gaussian embeddings [24]” in Tab. 4 corresponds to using the approach from URF. As can be seen, not using the per-Gaussian embeddings significantly reduces performance. The authors of SWAG report a URF-inspired approach for the same problem that performs significantly worse than their SWAG method (this observation is used to motivate the significantly more expensive SWAG approach, see our reply to reviewer nPTW for details). They conclude that “affine color transformations cannot model all appearance changes”. Our work disproves this statement as our method outperforms SWAG, as shown in the attached PDF. We believe that this is interesting to the community.
• Differences to NeRF On-the-go: While our approach for handling transient objects is clearly inspired by NeRF on-the-go, Fig. 3 shows that the original formulation from NeRF on-the-go is not directly applicable to our problem setting. This is due to the formulation not being robust to appearance changes (which are not considered in NeRF on-the-go), see L186-195. We believe that showing how to adapt this approach to the case of changing appearance is interesting for the community.

W2: Missing related work

We will add Robust Gaussian Splatting and other recent concurrent works (SWAG, GS-W, etc) to the related work section, as well as comparisons to relevant methods (see the results shown in the attached PDF).

W3: More illustrations for the justification of a binary uncertainty mask Thanks for the great suggestions. We promise to add more illustrations and the choices for threshold.

Q1: Why are baselines different

Photo Tourism and NeRF on-the-go pose different challenges (strong appearance changes and moderate occlusions vs. limited appearance changes and strong occlusions). Consequently, methods developed for one scenario are typically not evaluated in the other and vice versa (in addition, NeRF on-the-go was released only very recently). We use both datasets to showcase the robustness of our approach to both appearance changes and strong occlusions. For the rebuttal, we ran multiple baselines on both datasets (see PDF), leading to more shared baselines.

Q2: Why WildGaussians fares worse with low occlusion in Table 1, and how to mitigate it?

While NeRF on-the-go outperforms WildGaussians for low occlusions, the difference is marginal for PSNR (20.63 vs. 20.62) and SSIM (0.661 vs. 0.658). Without appearance modeling, WildGaussians outperforms NeRF on-the-go in both metrics (see Tab. 3). WildGaussians’ approach to handling transient objects is designed to be robust to appearance changes, and this added level of robustness compared to NeRF on-the-go seems to hurt the LPIPS performance of WildGaussians. The fact that 3DGS, Gaussian Occupancy Fields, and Mip-Splatting all achieve a similar LPIPS as NeRF on-the-go, suggests that occlusion handling is not very important for such a low level of occlusion.

A potential way to mitigate performance degradation is to try to automatically detect which components (appearance modeling, handling transient objects) are necessary for a given scene.

Thank you very much for the positive and constructive feedback! Below, we address the concerns raised in the review. We will adjust the paper accordingly and fix the typos.

W1: The ideas in the paper are not uncommon, similar to SWAG, GS-W, and Scaffold-GS

To our understanding, Scaffold-GS (CVPR 2024) neither uses appearance encoding to model appearance changes nor pre-trained models, and it is not evaluated on datasets with significant appearance changes or moving occluders.

We consider SWAG and GS-W as concurrent work. Our approach, developed independently, addresses the same problem (modeling scenes with changing appearances and occluders) and follows NeRF-based methods but differs in implementation:

• SWAG: Uses a feature grid with an MLP for color prediction, leading to slower inference. We use a shallow MLP for an affine transformation of SH coefficients, making it more efficient. For transient objects, SWAG uses a trainable, image-dependent occupancy term per Gaussian, but we use pre-trained DINO features to predict regions with dynamic objects.
• GS-W: Combines per-Gaussian appearance features with features from a reference image via MLPs. It uses a Unet model to predict visibility maps, but we predict uncertainties by comparing features from rendered and actual images.

All three approaches differ significantly in implementation. Additionally, our approach outperforms both SWAG and GS-W (cf. PDF). We will extend discussion, and add results for SWAG and GS-W.

W1: This work seem like a combination of existing methods

As detailed above, our approach differs significantly from existing approaches in technical details.

• Appearance modeling: SWAG shows that our approach is non-trivial: it reports that authors tried to predict an affine transformation of the SH coefficients, but observed significantly worse results than with their approach (cf. Tab. 8 in 2403.10427v2). They conclude that “affine color transformations cannot model all appearance changes”. We show that this conclusion is false, as we outperform SWAG.
• Transient objects: Ours is the first work in the context of 3DGS that uses a pre-trained foundation model for uncertainty modeling.

W2: Comparisons over stronger baselines

Similar to SWAG and GS-W, we compare to standard 3DGS in Tab. 1, 2. Tab. 4 contains a comparison with a stronger baseline: w/o appearance&uncertainty corresponds to Mip-Splatting+AbsGaussian(GOF) + our sky modeling. This baseline is not substantially better than 3DGS on Photo Tourism (PSNR: 18.48 vs. 18.13). We also add a comparison on NeRF On-the-go dataset in the PDF, where we clearly outperforms both the baselines and GS-W, especially with lots of occlusions. We will add the results and baselines to Tab. 1, 2.

W3: Disagreement with the claim to “handle changes in day-to-night transitions without sacrificing fine details”

Indeed, being able to handle appearance changes seems to come at the price of a (slight) loss in detail. We will soften the claim to “Compared to 3DGS, we can adeptly handle changes in appearance such as day-to-night transitions at the cost of a slight blurring in fine details.”

Q1: Comparison to SWAG and GS-W

GS-W only released code about a month ago and SWAG has not released code. Looking at the code of GS-W, it uses the full test image for computing the appearance (see our reply to BAXf for details), and fixing the code to use NeRF-W evaluation protocol significantly reduces performance (see PDF). Our approach outperforms both SWAG and GS-W.

Q2: Why not predicting color directly like Scaffold-GS?

We still use SH coefficients so that after fixing the appearance embedding, we can bake our representation back to 3DGS for better portability and faster runtime. In our experience, predicting offsets from a base color stored in Gaussians leads to more stable training.

Q2: Use of DSSIM and L1

We follow VastGaussian and use L1 to ground the appearance (taking the final color into account) and DSSIM to ground the structure. Using the renderings without appearance modeling, which increases the performance slightly in our experiments. We will clarify this in the final version.

Q2: Eq. (11) entire loss function?

The loss is sum of Eq. 11 and Eq 9. We will clarify this.

Q2: When initializing sky Gaussians, why remove points not visible in any camera when 3DGS will prune them automatically?

We remove these points as they will never be seen in any view and, therefore, there is no point adding them and slowing down the training (3DGS will never prune points not visible in any camera).

Q3: Only two methods compared on NeRF On-the-go dataset

The dataset was released right before the submission deadline, and we were only able to compare to 3DGS and NeRF On-the-go in time. Above, we show results for GS-W, Mip-Splatting, and Gaussian Opacity Fields. We will include the results in Tab. 1.

Q3: Unfair to compare with 3DGS when you use Mip-Splatting and AbsGaussian (GOF)

In Table 4, we also report the numbers on our model with app. modeling and uncertainty modeling disabled. This is essentially Mip-Splatting+AbsGaussian(GOF) with our sky modeling.

Q3: Why is 3DGS slower than WildGaussians?

3DGS cannot explain the transient objects and it grows Gaussians to account for that. The excessive Gaussians (floaters) slow it down. We will make this clearer in the final version.

Q3: Ablation study for sky handling

We will add an ablation study to Table 4. Unfortunately, we did not have time to do this for the rebuttal.

Q3: Why adding appearance modeling to datasets without appearance changes decreases the performance?

Indeed, enabling appearance modeling for datasets with no appearance changes reduces performance slightly. The same happens with NeRFs, e.g., see Mip-NeRF 360 paper. The added (unnecessary) degrees of freedom, in our case appearance embeddings, make the optimization problem more difficult.