Variational Bayes Gaussian Splatting

Dear Reviewer,

We appreciate your thoughtful feedback and have revised our paper to accommodate the identified weaknesses:

Introduction and background

We updated the introduction to better motivate our work and highlight the research gap and contributions.

Unclear Novelty and Elimination of Replay Buffers

The elimination of replay buffers is a direct result of VBGS’s variational Bayesian formulation, which enables parameter updates that integrate new data while retaining prior information through a variational posterior. Unlike gradient-based methods, which require access to old data to mitigate forgetting, VBGS maintains a probabilistic representation of parameters that naturally accumulates evidence over time. We updated section 3.3 to better highlight this mechanism.

Computational Efficiency Analysis

We agree that computational efficiency is a critical aspect of practical usability. We updated the discussion with a paragraph on memory usage and computational efficiency of VBGS compared to 3DGS. We also added Appendix E with a more in-depth discussion on computational cost.

Baseline Selection and Limited Comparison

The choice of the "Gradient" baseline was motivated by the direct relevance of 3DGS as a closely related state-of-the-art method for Gaussian mixture-based scene representation. Given our focus on the continual learning use case without replay buffer, we did not find any other baselines suited for an apples-to-apples comparison. However, we did expand our benchmarks to include the Replica dataset (Straub et al, 2019), as employed in the SLAM benchmark of Keetha et al., where we feed camera frames from a particular SLAM trajectory in-order and do continual updates on VBGS.

Evaluation metrics

Thanks for pointing that out. We updated the manuscript to introduce PSNR as the metric for reconstruction quality when it first appears in the evaluation section.

We hope these revisions address the main concerns raised.

Dear Reviewer,

First of all, we would like to thank you for the detailed feedback and for identifying areas where our work could be clarified or strengthened.

Limited Improvement in Table 1

We acknowledge that the PSNR values in Table 1 show comparable performance to 3DGS in the static dataset setting. However, our main focus is on scenarios involving continual learning, where VBGS provides significant advantages over 3DGS due to its resilience against catastrophic forgetting. This advantage is most evident in our continual learning experiments, where 3DGS performance degrades as new data is introduced, whereas VBGS maintains consistent performance.

Reliance on RGBD Data

We recognize that reliance on RGBD data is a limitation compared to 3DGS, which directly optimizes over RGB projections. However, for the use cases we envision, i.e. robotics, 3D data is often available through RGBD or lidar sensors. In the absence of depth data, one could resort to a depth estimation method in combination with VBGS. To demonstrate this, we added an experiment to the appendix where depth information is not available, and is estimated through a pretrained neural network (using metric DepthAnythingV2).

Limited Baseline Comparisons and Datasets

We chose to limit our comparison to Gaussian splatting approaches as our method specifically proposes an alternative method for optimizing Gaussian Mixture Models with benefits for continual learning problems. This way, we are able to effectively compare gradient based Gaussian splatting with VBGS for a mixture model with N components, while for NeRFs this kind of parameterizations are hard to define.

Given our focus on continual learning without a replay buffer, we did not find any other baselines suited for fair comparison. However, we did expand our benchmarks to include the Replica dataset (Straub et al, 2019), as employed in the SLAM benchmark of Keetha et al., where we feed camera frames from a particular SLAM trajectory in-order and do continual updates on VBGS.

Specific answers to questions:

Why does 3DGS struggle with catastrophic forgetting?

Catastrophic forgetting arises in 3DGS because it relies on gradient-based optimization, which updates parameters for new data without explicitly preserving information about prior data. In streaming data settings, older data is often inaccessible or not stored in memory, leading to overwriting of previously learned features. Mitigation strategies, such as replay buffers, can alleviate this issue but require significant memory and computational resources. VBGS avoids catastrophic forgetting by maintaining a variational posterior over parameters, allowing updates that integrate new information while preserving knowledge of previous observations. We expanded our discussion of these mechanisms in the introduction to better contextualize this advantage.

Numerical Results on the Habitat Rooms Dataset

The numerical results on the Habitat rooms dataset can be found in Table 6 of Appendix D.3.

Dear Authors,

Thank you for your detailed rebuttal. After carefully considering the technical quality, presentation, and novelty of your work, I have decided to maintain my current score. I will continue discussions with the other reviewers to ensure a fair and thorough evaluation.

Dear Reviewer,

Thank you for your valuable feedback on this manuscript.

Limited Number of Gaussians in Experiments, Memory Requirements and Computational Demand

It is true that the current state-of-the-art 3DGS methods have millions of components and we are only comparing in a range of thousands. This has a couple of reasons. First and foremost our focus is on continual learning without a replay buffer, and not so much on achieving better reconstruction quality. We believe that this is important for a lot of practical applications (e.g. robot navigation). Second, our variational update scheme naturally trades off accuracy and complexity. Hence, depending on the prior, it will favor less components if the price to pay in accuracy is low. See also point 3. Third, to balance the experimentation time and compute cost, we did not focus on large-scale datasets such as MipNeRF-360.

Instead of optimizing a Gaussian Mixture Model (GMM), VBGS in fact represents a distribution over GMMs and does variational inference to get a posterior distribution. This means that instead of representing each Gaussian component with its color, mean, scale, rotation and opacity (=14 parameters per component) as done in 3DGS, we represent all parameters of the Normal-Inverse-Wishart likelihood for color, position and Dirichlet prior (= 29 parameters per component), which does indeed increase the memory requirements by a factor of 2 for the same number of Gaussians.

This representation allows us to turn “training” into a coordinate ascent variational inference (CAVI) algorithm, which can be decomposed in two steps: computing model assignments which has complexity O(nm) with n the number of data points and m the number of Gaussian components, and computing the parameter updates with complexity O(n). Note that we can parallelize these computations over both the number of components and the data using accelerated hardware.

In terms of compute time it is hard to do an apples-to-apples comparison with 3DGS, as although 1 CAVI update step is more consuming than 1 gradient update step, you typically need way less CAVI update steps to converge. For the image experiments, we measured the average compute time for both CAVI and gradient descent to converge to the same loss, and found that VBGS was significantly faster. However, for the larger 3D experiments, we had to split the data in several batches because of limited GPU memory on our machine, which resulted in a worse overall runtime for VBGS. An interesting venue for future work would be to actively sample data points and only compute the update for these points.

We added a section in the Appendix E highlighting this aspect.

Comparison with Gradient-Based Methods (Table 6)

With respect to Table 6 in Appendix D.3., we would like to emphasize that in the case of random initialization VBGS (both with and without reassign) performs better than Gradient for “Skokloster Castle” and “Apartment 1”, while Gradient performs better for the “Van Gogh Room”. In the case of initialization on data, the Gradient approach performs better on the “Van Gogh Room” and “Apartment 1”. However, note that all these values are within the standard deviation of each other. We added a section in the appendix to describe this table in more detail.

Lack of Dynamic Gaussian Pool Resizing

We conducted the suggested experiment where we initialize VBGS with approximately the same number of components as the resulting size of a dynamically growing 3DGS. We observe that the performance of VBGS does not improve over a model with 100K components. This has two particular reasons: 1) the prior size of the components is set inversely proportional to the number of components (i.e. if more components are used, they should each be smaller). However we notice that when they are too small this . 2) The objective function (the evidence lower bound) naturally trades off complexity with accuracy and prefers models of smaller size. The updated table indicates the amount of components that are used for VBGS.

Specific answers to the questions:

Figure 2.a Missing "Gradient (Data Init)" Component

Thank you for pointing this out. We will set up the data init experiment for the gradient baseline and update Fig 2.a in the coming days.

Applicability to Scenes in Motion

In its current form this model does not handle scenes in motion. To deal with moving objects, one would need to add a dynamics model that iteratively predicts the next spatial location of each of the components. An approach that could be used in combination with VBGS to deal with changing scenes is to work with a forgetting rate, where the update of new data is weighed more than the prior statistics, and hence the model can overwrite the changed parts of the scene with the new observation.

We thank the reviewer for their flexibility and look forward to further discussions during the rebuttal phase.

Dear Reviewer,

First of all, we would like to thank you for these valuable insights and suggestions.

W1: Lack of comparison against recent SLAM methods or datasets

Thank you for raising this point. Our goal was to demonstrate VBGS as a general-purpose solution for continual learning in Gaussian splatting. To ensure focus, we selected benchmarks aligned with 2D image / 3D scene representation and online learning exploiting the conjugate properties of Gaussian Mixture Models. That said, SLAM-specific benchmarks would indeed provide additional insights into VBGS’s performance in real-world navigation tasks.

We would like to emphasize that we validated our approach on 3 rooms of the Habitat dataset (Manolis, 2019). However, as the data collected is not in a continual trajectory, we now added three rooms from the Replica dataset (Straub et al, 2019), as employed in the suggested work of Keetha et al. Concretely, we feed camera frames from a particular SLAM trajectory in-order and do continual updates on VBGS. We did not integrate with camera tracking, as Keetha et al backpropagate gradients w.r.t the pose to track camera pose. Developing a pose update rule using a VBGS model is left as future work.

W2: Significance of the mean-field approximation

The mean-field approximation was adopted to ensure computational tractability while still maintaining a variational posterior over Gaussian parameters. This approximation is crucial as this allows to update each variable independently as a separate step in the coordinate ascent algorithm. We added a sentence in the main text to emphasize this point.

In our experiments, we observed that VBGS achieved competitive results compared to gradient-based methods like 3DGS, indicating that the mean-field assumption does not significantly hinder performance in the evaluated settings. However, we agree that the conditions under which this approximation might fail warrant further exploration.

W3: Reliance on depth maps

In line with your suggestion, we added an experiment to the appendix where depth information is not available, and is estimated through a pretrained neural network (using metric DepthAnythingV2). We show that we are still able to learn the room structure at the expense of a drop in PSNR (from 20.46 to 16.26). Note that the quality of the resulting VBGS will depend heavily on the accuracy of the predicted depth.

Specific answers to questions:

Q1: As the main focus of this paper is not to propose a SLAM system (we currently do not have camera tracking) we evaluated on other datasets. However, we now added three rooms of the Replica dataset.

Q2: The mean-field approximation assumes independence between parameters in the posterior, which simplifies updates but may limit flexibility when parameter dependencies are critical. We expect the performance of VBGS to degrade in scenarios where: The true posterior exhibits significant correlations between parameters, as these cannot be captured by the mean-field approximation. Highly multimodal posteriors are present, where mean-field assumptions can lead to overly simplified solutions. In our current experiments, the mean-field approximation performed well, likely due to the Gaussian mixture model structure, where dependencies between parameters are less pronounced, and the mixture can represent multimodal distributions. We are grateful for the reviewer’s thoughtful comments, and we hope to have satisfied all the reviewer's concerns and we look forward to receiving updated feedback.