3D Gaussian Splatting as Markov Chain Monte Carlo

We thank the reviewer for their constructive comments and suggestions. Here are our responses to your comments:

Technical contribution

We respectfully disagree, as our method cannot be summarized as a mere resampling strategy. It is a completely new take on the 3D Gaussian Splatting optimization that removes ALL the non-differentiable heuristics (and inherent training instability) that were introduced by the original authors: “opacity reset”, “densification”, "splitting", and “pruning”. We have further shown that this is non-trivial to achieve, as one must carefully design components in a way that obeys the basic principles of hybrid MCMC. Thanks to our re-formulation, future works could integrate various advancements in perturbed Gradient Descent, as reviewer Cpa7 suggests.

Additional compute cost

Please see the global response for the training time. In short, considering only the compute time for each Gaussian the added time is negligible. Ours does run slower if trained with a high opacity regularizer due to more transparent Gaussians, but it ultimately gives higher quality results, and it also can be tuned to train faster while still outperforming 3DGS.

Noise to other parameters being harmful

There have been a number of recent papers demonstrating that the locations of the Gaussians suffer heavily from local optima, which leads to their reliance on good initializations [7, 8]. Hence, adding noise to these parameters helps escape these local optima, and Gaussians to “explore” the scene. Other degrees of freedom (rotation, scale, opacity) do not suffer as heavily, hence the noise simply slows down convergence unnecessarily. Please see additional details showing more experiments on the noise design in our answer to reviewer Cpa7 if the reviewer is interested.

We thank the reviewer for the positive assessment of our paper. Please let us know if you have any concerns or questions and we will try our best to respond.

We thank the reviewer for sharing the enthusiasm that we have for our method. Here is our response to your comments:

Training time between SfM points version vs Random points version

SfM has faster convergence, but to obtain the best PSNR performances both were run for about the same amount of wall-clock time. However, note our primary objective was not to improve convergence, but to reduce reliance on good initialization and also remove the heuristics. We believe the latter two to be critical in situations where initialization could be difficult, such as 3D generative modeling or dynamic scene reconstruction.

ScanNet++

Indeed this would be very interesting. We were not able to run these experiments due to the resource crunch during the rebuttal period, but we hope to be able to add these results for the camera ready.

Training time

Please see the global response for the training time. In short, 90ms was milliseconds, not minutes and considering only the compute time for each Gaussian the added time is negligible. Ours does run slower if trained with a high opacity regularizer due to more transparent Gaussians, but it ultimately gives higher quality results, and it also can be tuned to train faster while still outperforming 3DGS.

L40--L41 Suboptimal placements leading to wasted compute and poor quality renderings

Our existing experiments show that existing relocation heuristics place Gaussians at worse locations than ours, leading to lower rendering quality (this is especially true when random initialization is used). In regards to wasted compute, we wanted to highlight the fact that Gaussian splitting heuristics becomes unnecessary, but we realize that this can be misinterpreted and will remove this comment.

Surfaces

This is a great suggestion. While we were not able to finish this during the short rebuttal period as it involves setting up a new evaluation pipeline, we will investigate this in the future.

We thank the reviewer for their constructive comments and suggestions.

Relevance of the hybrid MCMC framework

Our work is indeed related to gradient descent methods with noise and perturbations as suggested. In our case, our motivation for opting to interpret our framework as a hybrid MCMC method is due to the necessity of “jump” moves, and because we simply are much more familiar with the MCMC literature than what you pointed us to. As you suggest, we will extend our related works with the suggested literature in a revision. However, note that because much of the modeling space (the scene) is empty and Gaussians should only be located near surfaces, it is critical that Gaussians are moved toward these surfaces quickly. Under the MCMC formulation, they can be naturally integrated as “jump” moves or “resampling” moves [A, B], which we leverage.

Regarding the noise in the momentum component, this is clearly an interesting direction to explore. We are excited that our reformulation seems to be opening doors to these various design choices which were originally not possible due to the heuristics that were involved in the optimization, such as abrupt reset of Gaussian opacities or manual splitting of large Gaussians. These heuristics clearly break any hope to formally study the convergence characteristics of the method, and in our work we remove all of them, which will hopefully lead to more work on this topic.

[A] Lindsey et. al, “Ensemble Markov chain Monte Carlo with teleporting walkers”, SIAM/ASA Journal on Uncertainty Quantification. 2022

[B] Green, “Reversible jump Markov chain Monte Carlo computation and Bayesian model determination”, Biometrika, 1995

Design of the noise term

In Table 3 we already had a brief comparison of an alternative noise strategy where we add noise also to other Gaussian parameters. We have further tested without using the covariance term and the opacity term for the Tank and Temples dataset (we chose this dataset due to the short amount of time available for the rebuttal). We report the results in the table below:

Having both covariance and opacity is important to achieve the best performance:

• Without the covariance term, the noise is difficult to “undo” with gradients for narrow Gaussians, which then cause the Gaussians to randomly walk about and no longer represent the desired distribution.
• Without the opacity term, opaque Gaussians that are significantly contributing to the reconstruction loss would not be able to converge/stabilize, leading to a loss of high-frequency details.

Further, note that in the table above we had to use $\lambda_{noise}=5\times10^2$ for “No opacity*”, as if we use the same hyper-parameters as in the paper, the model does not train at all (PSNR<7).

Noise term scheduler

As suggested, we also tried the scheduling from [2], and we additionally tested a simple linear scheduler. Again, we ran our experiment on the Tank and Temples dataset. As shown in the table below, our exponential decay scheduler (which uses the same scheduler as in the original 3DGS paper) works the best:

Derivation of Eq. 9

The detailed derivation of Eq. 9 is provided in Appendix. A. The main approximation is the approximation of the integral that integrates the squared difference between the two distributions. We instead minimize the difference between the individual integrals, projected into each view, which is inspired by sliced Wasserstein methods [17]. As the color values that we integrate over are always non-negative, there are no underlying assumptions that could break.