Metropolis-Hastings Sampling for 3D Gaussian Reconstruction

We sincerely appreciate your insightful comments and questions. We have tried to respond to all of your concerns as follows. Moreover, thank you for pointing out the typo; we will revise it for our final manuscript.

[W1] Misleading Clarification

We acknowledge the reviewer's concern about our positioning. While we follow a similar overall pipeline (relocation and loss formulation) and address the same fundamental problem of replacing heuristic density control, we believe our characterization is accurate, as we propose a fundamentally different solution approach to the same problem. Specifically, 3DGS-MCMC employs SGLD (Stochastic Gradient Langevin Dynamics) that perturbs all Gaussian parameters with noise and accepts every move, whereas our method uses Metropolis-Hastings sampling that targets specific locations based on multi-view error aggregation and employs principled accept/reject decisions. We will revise our final manuscript to better acknowledge 3DGS-MCMC's pioneering role while more clearly stating how our MH-based approach with multi-view importance scoring represents a distinct methodological contribution to the same problem space. Our intent was not to diminish prior work, but rather to demonstrate an alternative probabilistic framework that achieves faster convergence.

[W2] Lack of Related Work

Efficient Reconstruction We have included the following works on efficient reconstruction in Sec. 2.2 of our paper [9, 22, 36, 38, 48], but we will further add additional more recent works on efficient reconstruction [e1, e2, e3] to Sec. 2.2. These works focus on deterministic geometric analysis or post-hoc uncertainty pruning on converged models. Unlike these methods, our probabilistic framework reduces the number of Gaussians through principled Bayesian sampling that dynamically adapts to scene-specific reconstruction signals.

[e1] Papantonakis, Panagiotis, et al. "Reducing the memory footprint of 3d gaussian splatting." Proceedings of the ACM on Computer Graphics and Interactive Techniques 7.1 (2024): 1-1

[e2] Hanson, Alex, et al. "Speedy-splat: Fast 3d gaussian splatting with sparse pixels and sparse primitives." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[e3] Hanson, Alex, et al. "Pup 3d-gs: Principled uncertainty pruning for 3d gaussian splatting." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

Densification We have included related work regarding densification in Sec. 2.3 [4, 20, 30, 52], but as densification was a rapidly growing field of interest during the past year, we may have unintentionally overlooked recent studies. We will add recent densification works using neural prediction [d1] or deterministic geometric splitting [d2]. These methods can face limitations, such as generative approaches may not generalize well to novel scene types [d1], and geometric methods use fixed rules that may not adapt to diverse scene complexity [d2]. In contrast, our framework provides theoretical convergence guarantees via stochastic exploration of the solution space.

[d1] Nam, Seungtae, et al. "Generative Densification: Learning to Densify Gaussians for High-Fidelity Generalizable 3D Reconstruction." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[d2] Deng, Xiaobin, et al. "Efficient density control for 3d gaussian splatting." arXiv preprint arXiv:2411.10133 (2024).

[W3] Preliminary

We will move the introduction of 3DGS-MCMC and Metropolis-Hastings Sampling to the preliminary in the final manuscript.

[W4] Confusing Description

In the context of our paper, a voxel is the axis-aligned cube of edge length v, which is used purely for counting how many Gaussians occupy each bit of world space. To construct the voxel grid, we first lay a regular cubic lattice of size v across the scene’s coordinate frame. After that, the centre of every Gaussian is mapped to one of these voxels by dividing its 3D position by v and taking the integer part. We then record only how many Gaussians fall inside each voxel, $c_\Theta(v)$. These occupancy counts feed the sparsity voxel-wise prior as denoted in Eq. 4, which penalizes voxels that hold more than one or two Gaussians. Note that as we only use the voxels for counts, and never for rendering, we don’t need to store a full 3D grid.

[W5/Q1] Results in Figure 1.

As we are not allowed to provide images, we will provide the statistics regarding timing. Our method shows faster convergence for both Mip-NeRF360 and Tanks and Temples datasets, and shows faster convergence even when measured by the average of all the datasets. We provide the statistics of the convergence time when averaged across all 13 scenes below, and include the "98% of Final PSNR" as another reference.

We also provide a “time-to-equal-PSNR” metric, as suggested by reviewer U3jc, below.

Our method consistently shows faster convergence compared to 3DGS-MCMC. We will update the figure to reflect all datasets in our final manuscript.

[W6] Limited Improvement

Our method shows mixed quantitative results on MipNeRF-360 and Tanks & Temples, which is a point we acknowledge in our paper's limitations section.

Qualitatively, our method is on par with or even superior to baselines in capturing the fine details of the primary subjects, as can be seen in Fig. 2 and our supplementary video. However, as discussed in our limitations, the lower quantitative metrics are predominantly caused by our model's difficulty in representing vast, low-frequency backgrounds, such as the sky. Since these metrics are averaged over the entire image, imperfections in the large background regions can penalize the overall score.

The high-quality results of reconstructing indoor scenes demonstrate our method's strong potential when scale variance is more manageable. On these scenes, our method achieves superior metrics across all measures, solidifying that the core algorithmic contributions are sound. This trade-off stems from the extreme scale variance (exceeding 170,000x) inherent to unbounded outdoor scenes, where our model struggles to simultaneously optimize for the compact, detailed Gaussians of the foreground and the massive, smooth Gaussians required for the background. As we state in our limitations, adapting our model to robustly handle both ends of this scale spectrum is a clear direction for future work, and this can be addressed by exploring techniques regarding multi-scale representation.

[W7/Q2] Varying Number of Gaussians

We provide results of our method, 3DGS-MCMC and 3DGS, with varying numbers of Gaussians across all scenes. Note that the performance of models is similar across varying numbers of Gaussians.

Average Number of Gaussians across all scenes = 0.53M

Average Number of Gaussians across all scenes = 1.40M

[W8/Q3] Ablation Study

Thank you for the valuable feedback. To directly compare Metropolis-Hastings (MH) Sampling with SGLD, we conducted an ablation study where both methods had only their key algorithm implemented. We removed our design components (voxel-wise prior, importance score) and also took out opacity and scaling regularizers from both methods. This setup creates the most direct comparison between our MH sampling and the SGLD approach.

The results demonstrate that our Metropolis-Hastings sampling strategy provides consistent improvements over 3DGS-MCMC's SGLD approach. This result indicates that the MH sampling strategy provides a reliable advantage in reconstruction quality.

We sincerely appreciate your insightful comments and questions. We have tried to respond to all of your concerns as follows.

[W1/Q1] Performance - Significance of MH and Failure Cases

In order to directly compare our methodology with 3DGS-MCMC, we conducted an ablation study where both methods had only their key algorithm implemented. We removed our design components (voxel-wise prior, importance score) and also took out opacity and scaling regularizers from both methods. We believe that this setup creates the most direct comparison between our MH sampling and the SGLD approach.

The results demonstrate that our Metropolis-Hastings sampling strategy itself provides consistent improvements over 3DGS-MCMC's SGLD approach. Furthermore, this result indicates that the MH sampling strategy provides a reliable advantage in reconstruction quality.

Moreover, beyond the significance highlighted in our paper, key significances of our MH framework are as follows:

1. Automatic scene-complexity adaptation. As mentioned in the paper, 3DGS-MCMC must commit to a fixed budget of Gaussians before training, which can under-fit simple scenes or waste memory on complex ones. By contrast, our accept/reject chain keeps inserting points only while the posterior still yields photometric gain, so the optimization automatically stops at the scene-appropriate count with no prior complexity.

2. Dimension-selective updates that accelerate convergence. 3DGS-MCMC injects Langevin noise into every parameter of every Gaussian each step, which inflates stochastic gradient variance and forces smaller learning rates. On the other hand, we restrict stochastic moves to positions only. This trims the update dimensionality and lets each accepted proposal deliver a larger effective step. Thus, the PSNR curve for our method is smoother and achieves convergence earlier than 3DGS-MCMC (Fig. 1), confirming that the dimensionality cut translates into faster practical convergence.

However, as mentioned in our paper, in sparse outdoor scenes such as Flowers, Stump, etc. (Mip-NeRF 360) or Train, Truck (T&T), our PSNR trails that of 3DGS-MCMC. These gaps arise not from missed high-frequency details, which we capture well as can be seen in Fig. 2 and the supplementary video, but from faint background strata that carry little image gradient yet contribute to pixel-wise error. Our MH sampler weighs proposals by multi-view photometric discrepancy; once high-error foreground patches are saturated, the residual background error is dispersed across many low-density voxels, so their individual acceptance probabilities fall below the threshold, leaving some regions undersampled. To improve these failure cases, adopting adaptive temperature annealing to MH acceptance or adopting multi-scale representations or dual-stream foreground/background samplers could be viable solutions, and open grounds for future work.

Dear Reviewer,

I would like to invite you to the discussions with the authors. At least, please carefully read the others' reviews and authors' responses, and mention if the rebuttals addressed the concerns or not.

To facilitate discussions, the Author-Reviewer discussion phase has been extended by 48h till Aug 8, 11:59 pm AoE; but to have enough time to exchange opinions, please respond as quickly as possible.

Thanks,

Your AC

We sincerely appreciate your insightful comments and questions. We have tried to respond to all of your concerns as follows.

[W1] Multiple-Views for Importance Score - Training Cost

While our method does require multi-view evaluation for importance scoring, hence leading to an increase in the total training cost, the faster convergence shown in Fig. 1 demonstrates that this overhead is worthwhile in practice. Our method reaches 95% of the final PSNR faster than 3DGS-MCMC in both iteration count and wall-clock time, meaning the improved sampling efficiency more than compensates for the additional per-iteration cost. The key insight is that spending slightly more time per iteration to make better sampling decisions results in needing fewer total iterations to converge, and this is achieved even with our current unoptimized implementation.

Additionally, our approach uses only a camera subset (annealed from broad to focused coverage) rather than the full training set, which helps control the computational burden. Moreover, our method achieves comparable or better reconstruction quality while using dramatically fewer Gaussians (as shown in Table 4), leading to significant memory savings. Nevertheless, we hope to explore more efficient importance score computation or adaptive view selection strategies to further reduce this overhead.

[W2] Discussion on the surrogate

Computing the exact loss reduction ($-\Delta\mathcal L$) for each candidate Gaussian would require re-rendering the entire scene from all viewpoints in the selected camera subset for every proposed splat. This would increase computational complexity from O(N) to O(N×K×M), where N is the number of candidates, K is the number of cameras, and M is the scene complexity, making the method computationally impractical. To solve this problem, we derived a surrogate by analyzing the first-order multivariate Taylor expansion based on inserting a small Gaussian in the scene. (Mentioned in our paper in Appendix C)

Indeed, the best way to measure the quality impact would be a direct comparison, but a direct empirical comparison between the surrogate and exact $-\Delta\mathcal L$ would require computing full multi-view rendering errors for thousands of candidate Gaussians, which was precisely the computational burden our surrogate is designed to avoid. This creates a paradox where validating the surrogate's accuracy would negate its computational benefits.

Nevertheless, we believe that the quality impact of selecting a surrogate to calculate $-\Delta\mathcal L$ is minimal, as the surrogate maintains the relative ordering of candidate importance. Since our surrogate is derived from a rigorous first-order approximation, and the logistic function $\sigma(z)$ is strictly monotonic, the relative ordering is mathematically preserved. For example, if candidate A truly provides greater improvement than candidate B, it would be $I(A) > I(B)$ in our surrogate. This ordering preservation is sufficient for quality maintenance because the MH acceptance test ensures that better candidates receive higher acceptance probabilities, guiding the sampler toward high-impact regions. Over many iterations, this statistical bias toward high-importance areas drives convergence to optimal Gaussian distributions, as evidenced by our results showing maintained or improved reconstruction quality.

[Q1] Model's effectiveness coming from sampling or importance score

We conducted an ablation study by removing the multi-view importance score and other components in our method. We also reported the average number of Gaussians across all scenes in the parentheses.

Based on the study, our method without the multi-view importance scores still achieves strong performance, outperforming 3DGS-MCMC on all three metrics on Deep Blending and LPIPS in Tanks & Temples. This shows that replacing heuristic-based densification with principled Metropolis-Hastings sampling is effective.

To further clarify, our multi-view component is strategically integrated into the sampling framework, not as the primary driver of performance, but as an intelligent guidance mechanism that makes the probabilistic sampling more informed and consistent. Overall, our sampling methodology demonstrates that principled sampling with robust guidance is effective.

We sincerely appreciate your insightful comments and questions. We have tried to respond to all of your concerns as follows.

[W1] Weakness in quantitative results

We would like to first highlight that our core Metropolis-Hastings (MH) sampling strategy demonstrates improvements over 3DGS-MCMC's SGLD approach in a direct comparison (please see our detailed analysis in Q1/W3). However, we acknowledge that our overall method shows mixed quantitative results on MipNeRF-360 and Tanks & Temples, which is a point we also acknowledge in our paper's limitations section.

Qualitatively, our method is on par with or even superior to baselines in capturing the intricate geometry and fine details of the primary subjects, as can be seen in Fig. 2. However, the low quantitative metrics are predominantly caused by our model's difficulty in representing vast, low-frequency backgrounds, such as the sky. Since these metrics are averaged over the entire images, background imperfections can disproportionately penalize scores despite the high-quality foreground reconstruction.

The results on the indoor scenes with manageable scale variance demonstrate our method's strong potential. Our method outperforms the baselines in these scenes, suggesting that core algorithmic contributions are sound. This challenge stems from the extreme scale variance (exceeding 170,000x) inherent to unbounded outdoor scenes, where our model struggles to simultaneously optimize for the compact, detailed foreground Gaussians and the massive, smooth background Gaussians. Adapting our model to robustly handle both ends of this scale spectrum is a clear direction for future work, and this can be addressed by exploring techniques such as multi-scale representation.

[W2] Clarity on performance gains on timing

In Fig. 1, the horizontal axis indeed counts training iterations. We imposed vertical markers that correspond to actual wall-clock time at which each method reaches a chosen quality threshold. We intended to give a single, compact view of how fast along the iteration trajectory each method would be experienced in real time.

Regarding the reviewer's concern with the "95% of final PSNR" metric, we would like to clarify that we were not intending to demonstrate our method in a good light; we were rather unaware of the typical “time to equal PSNR” metric. We initially chose “95% of the final PSNR” as the metric, as a confidence-interval-like view of near-final quality that is directly comparable across methods, especially as absolute PSNR ceilings differ across scenes. Moreover, using 95% is adopted as a practical convergence metric in recent vision works; for instance, [1] uses 95% to compare optimization speed.

Following the reviewer's suggestion, we provide the “time to equal PSNR” metrics averaged across all scenes for both our method and 3DGS-MCMC. As the approximate highest average PSNR value was 30 dB, we selected our final target PSNR as 30 dB. The results show that our method converges approximately 2.18 minutes faster than 3DGS-MCMC. We also show lower target PSNR values to show that our method consistently converges faster. We will include these metrics in our final manuscript.

Note that we used the official implementation provided by 3DGS-MCMC and conducted experiments on NVIDIA GeForce RTX 3090 for both methods to ensure a fair comparison.

[1] Zhang, Yunxiang, et al. "Image-gs: Content-adaptive image representation via 2d gaussians." arXiv preprint arXiv:2407.01866 (2024)

[Q1/W3] Direct Comparison of MH & 3DGS-MCMC and impact of design heuristics

Thank you for this important question. The concern about disentangling the gains from our core Metropolis-Hastings (MH) framework versus our other design choices is crucial. To address this directly, we have conducted experiments to measure the contribution of each component.

1. Quantifying the Contribution of Metropolis-Hastings (MH) in an "Apples-to-Apples" Comparison with 3DGS-MCMC

To isolate the exact contribution of our core MH sampling strategy, we ran experiments with both our method and 3DGS-MCMC, both methods with only their key algorithmic methodology implemented. We believe that this setup creates the most direct "apples-to-apples" comparison between our Metropolis-Hastings (MH) proposal strategy and 3DGS-MCMC’s Stochastic Gradient Langevin Dynamics (SGLD) approach.

The results demonstrate that our Metropolis-Hastings sampling strategy provides consistent improvements over 3DGS-MCMC's SGLD approach. This result indicates that the MH sampling strategy provides a reliable advantage in reconstruction quality.

2. Disentangling Design Heuristics

To address the reviewers' concern regarding this, we wish to clarify that our model’s components are theoretically grounded designs that function as interdependent parts of our Metropolis-Hastings framework. Our paper theoretically derived how these components must work together. As mentioned in Eq. 18, the first term $\sigma\bigl(I(i)\bigr)$ represents the photometric gain of adding a new Gaussian, which is calculated from Eq. 10 and the second term $D(v’)$ represents the sparsity penalty, calculated from Eq. 4. A proposal is accepted only if the product of these two terms is high, and this ensures that our method prioritizes Gaussians that are photometrically useful and spatially efficient.

To further disentangle our design components, we present a detailed ablation study breaking down our method. We start with pure MH Sampling, which is the MH sampling algorithm with the regularizers. We incrementally added our key design components, and all the experiments were conducted in the same setting as our original experiments, with no separate parameter tuning for individual components. The results are summarized in the table below.

The results show that naively combining our design components without the full, synergistic framework (e.g., Pure MH + Eq.4, Eq.10) leads to an inefficient model. While it achieves a high PSNR, it does so by nearly doubling the number of Gaussians to 1.45M. This demonstrates that the components, when not properly balanced by the complete acceptance rule, can lead to uncontrolled growth, chasing marginal gains at a significant computational cost.

While Pure MH achieves a marginally higher PSNR on the Mip-NeRF360 dataset, our final MH-3DGS model achieves this competitive result using 12% fewer Gaussians (0.75M vs. 0.85M). With all the design components together, we are able to create a much more compact and robust performance across all scenes.

[Q2] Equation 9

To clarify the notation, when we refer to pixel p in Eq. 9, we mean the pixel coordinate where a specific Gaussian projects in each camera's image plane. Since the same 3D Gaussian will project to different pixel coordinates in different camera views, the aggregation is performed per-Gaussian across its projected pixel locations in multiple views, not at the same pixel coordinate across different images.

The process works as follows: we compute residuals locally for each camera $c$ at the pixel location where each Gaussian projects, then average these residuals across the $k_t$ cameras in $C_t$. This aggregation pools error statistics across multiple perspectives of the scene for stable error estimates. The per-Gaussian importance is then evaluated based on where each Gaussian projects in the aggregated error maps.

We agree with the reviewer that the current notation can be misleading, and we will revise the notation for the final manuscript. Thank you for pointing out this important issue.

Dear Reviewer,

I would like to invite you to the discussions with the authors. At least, please carefully read the others' reviews and authors' responses, and mention if the rebuttals addressed the concerns or not.

To facilitate discussions, the Author-Reviewer discussion phase has been extended by 48h till Aug 8, 11:59 pm AoE; but to have enough time to exchange opinions, please respond as quickly as possible.

Thanks,

Your AC

Dear reviewers, Thanks so much for reviewing the paper. The discussion period ends soon. To ensure enough time to discuss this with the authors, please actively engage in the discussions with them if you have not done so.