Geometrically Consistent Generalizable Splatting

Novelty and losses

We believe that the reviewer missed the central premise of our work, which challenges the status quo in the important field of learning to generate splats from images. The view synthesis loss, used prevalently in the field, is shown to be ill-posed in estimating over-parameterised splats and needs to be amended. In the absence of the insights from this paper, the community will continue to throw more data onto the problem or explore the next set of architectural changes that provide minor improvements in terms of avoiding structural degeneracies in self-supervised splatting.

The reviewer cites Neuralangelo’s regularisation, which penalises estimated curvature from the SDF alongside an eikonal constraint, to establish that the proposed orientation loss is not novel. We argue that splats are not trivial to convert to SDF. Further, our regularisation does not estimate or penalise curvature at all. We thus wonder if the reviewer misinterpreted critical aspects of the presented approach.

To clarify, $L_{orient}$ is a consistency term between two different normal estimates coming directly from Gaussians in a rendering-free manner. The local normal of the surface is estimated (i) from Gaussian means using image neighbourhoods, and these are forced to align with (ii) the orientation of the Gaussians coming from covariance matrices as defined in 2DGS. This simple construct (that the authors have not seen used in Gaussian-splatting literature at all) provides a necessary signal for orientation supervision for the first time.

$L_{orient}$ is even simpler than the reviewer assumed. It follows the most natural way of estimating local surface normals from a depth map, as done in creating the NYUD-v2 dataset. Our submission is that this simple loss is stable and crucial for learning orientations. The well-celebrated loss introduced in 2DGS is complex in design, involves alpha composition of orientations, is slower to compute, and has been shown to be unstable. We believe that unnecessary complexity should not be encouraged for the sake of being novel.

$L_{align}$ enforces a bundle-adjustment-style loss that the reviewer alludes to—though it’s minimised only at context-view locations for which Gaussian means are predicted and is computationally efficient. (Note: similar to all generalizable splatting approaches, pre-estimated intrinsics and extrinsics are used during training, so no PnP is solved in this loss.) DUST3R did not require such a loss as it had full 3D ground truth for training. Our work shows that when ground-truth depths are not available for learning, this natural drop-in replacement becomes essential in adapting DUST3R-like frameworks self-supervised—a trick NoPoSplat and all other variants of DUST3R have missed to date.

These contributions are by no means trivial or insignificant. Our paper isolates some critical oversights by several works that extended depth and point-cloud estimation frameworks to splat predictions. Ignoring the degeneracies explained leads to meaningless estimations of two-thirds of the splat parameters by all splat predictors, as shown in Figure 2. We believe these findings are crucial for meaningful development in the field of generalizable splatting.

Pose estimation comparison with RoMa and DUST3R

While Re10K results evaluate in-domain performance of our approach and out-of-domain performance for RoMa and DUST3R, the converse is true on ScanNet, on which our approach was not trained but baselines were. ACID remains out of domain for all methods. Note that methods like DUST3R also use more supervision (ground-truth depth maps) and more data, facilitating generalisation. Despite this, our approach outperforms them in several aspects.

We agree that Table 1 places a lot of cognitive load on the reader. However, this pose-evaluation setup (and presentation) is standard practice in the field. Our Table 1 mimics NoPoSplat and several other works. As foundation models have evolved, several well-known models are trained on very large and sometimes publicly unavailable datasets. This paper aims to provide sensible ablations justifying the core contribution of the work (risking being called a NoPoSplat variant) while facilitating comparison with the state of the art as fairly as possible.

Training time

We want to highlight that our inference is marginally faster than NoPoSplat (estimating one fewer scale); losses are also computed at inference.

During training, Align and Orient losses add negligible overhead compared to rendering and backprop; both losses are local and fairly easy to estimate requiring projections of Gaussians means to image place. Fixing training hyperparameters, hardware and the rasterizer (2DGS), total training times for the models are:

Variations are statistically insignificant. We will add these in supplementary material. Note that we use A100 GPUs (40GB), which only has half the GPU memory compared to hardware used in NoPoSplat. This has implications on the total number of GPUs used to have large batch size (similar to NoPoSplat) and requires some changes in training hyper-parameters, which are reported. We reproduced NoPoSplat results with our hyperparameters and hardware to ensure fair comparison and reported the same in supplementary material. The discrepancy in training time or number of GPU is only due to the hardware restrictions.

Novel view synthesis results

The focus of this work is not to improve novel view synthesis, and no explicit design choices provide any meaningful change in view-synthesis quality. Changes in view synthesis due to different loss functions are statistically insignificant and should not be given much emphasis at all in our opinion. However, view-synthesis experiments were included in the supplementary document for completeness, ensuring geometric consistency does not come at the cost of view-synthesis capability as is the case for some splatting methods like 2DGS.

Coordinate conventions

Please note that MVSplat/pixelSplat does not prescribe any surface-normal estimation procedure or conventions. The so-called “Normals” for all baselines are visualised by us following the surface-normal definition in the 2DGS/PGRS papers: the minimal-elongation eigenvector (corresponding to the lowest eigenvalue) at every context-image pixel. These visualisations are used to highlight that baseline orientations lack any coherent structure and that estimated scales do not conform to underlying geometry. Alternatively, 3D Gaussians can also be visualised as larger opaque blobs, but we believe our image-like visualisations are more effective and easier to interpret (Figure 2). These visualization choices, however, do not affect the reported baseline results in this paper.

Alternative visualization and quality of results

As suggested, we can share PLY-file visualizations on the project page; however, these are highly sensitive to viewpoint choice. Rendered Gaussians provide a qualitative evaluation of view-synthesis performance, and many geometric errors and inconsistencies get disguised in such visualizations. We choose a mesh visualization (and standard depth visualization) which highlights geometric accuracy/consistency, or lack thereof.

Lack of sharpness

We refer the reviewer to our earlier responses. While a geometric edge-aware regularisation can be trivially used to improve the situation to some extent, our current results are qualitatively and quantitatively a significant improvement over the baselines without these modifications.

FLARE comparison

FLARE is trained on multiple large RGB–D datasets (including ScanNet), uses ground-truth depths for Gaussian means, and adds rendered-depth supervision via AnyDepth. This strong geometric supervision and extra data give FLARE several advantages over our method, which relies only on Re10K and no explicit depth supervision, making direct comparisons inherently unfair and grossly disadvantageous for us and other baselines.

Additionally, FLARE is very sensitive to test-time image resolution, uses five frames during testing, and reports different pose and depth evaluation metrics, which further complicates direct comparison.

That said, we evaluated pose estimation against FLARE using the same number of input views (2), the same metrics (AUC@[5^\\circ,10^\\circ,20^\\circ]), and identical PnP-RANSAC hyperparameters to ensure a fair comparison.

On the ScanNet test set, our method (using a lower resolution) still outperforms FLARE. Note that we have not trained our model on ScanNet, while FLARE has.

Pixel-wise Gaussian

The association of Gaussians with context-view pixels is a construct that all generalizable splatting algorithms we study adopt. This constraint comes from the fact that many splat predictors append DPT-like decoders to estimate Gaussian parameters on existing depth- or image-aligned 3D point-cloud predictors. The scalability of per-pixel Gaussian prediction is an important open problem and perhaps requires architectural changes. This work restricts itself to making prevalent pixel-aligned generalizable splatting geometrically accurate. We agree that $L_{align}$ assumes pixel alignment and might need to be adapted to suit novel non-pixel-aligned Gaussian representations of the future. Findings from this work will provide a good foundation for such adaptations.

We reiterate that the paper highlights important degeneracies posed in predicting 3D Gaussians using feed-forward networks. Despite several efforts, these degeneracies persist in the literature and are explained to be due to the insufficiency of view-synthesis losses. We present a thorough analysis of how existing frameworks converge to predict meaningless Gaussian scales and orientations, motivate and implement geometric regularisations to estimate meaningful surfel-like 2D Gaussians via feed-forward neural networks. The contributions of the presented approach are adaptable to any existing pose-free or pose-aware feed-forward splat predictors trained with or without depth supervision. In our view, this work would generate immediate impact and increase the applicability of splat predictors beyond the view-synthesis application. We encourage the reviewer to read the comments to Reviewer S4Ch and Reviewer ugeC, where the novelty and potential impact of our framework are explained in detail. We attempt to address other specific questions of the reviewer here:

$L_{orient}$ , $L_{align}$ and full rank 3D Gaussian

We want to clarify that $L_{orient}$ or $L_{align}$ loss can be applied to 3D Gaussians with full-rank covariance as well. $L_{align}$ operates solely on the 3D Gaussian means, so it can be used as is if the predictions are full 3D covariance matrices. $L_{orient}$ merely enforces two different estimates of local surface normals to be consistent. One local-normal estimate relies on image-based neighbourhood and Gaussian means, imposing no restriction on the rank of the covariance matrix. The alternative local normal is defined by the eigenvector of the covariance matrix corresponding to the smallest eigenvalue—they are estimable for both degenerate and full-rank Gaussian covariance.

In fact, the non-degenerate covariance matrices and the normals estimated from them as prescribed above are used in the work PGSR: Planar-based Gaussian Splatting for Efficient and High-Fidelity Surface Reconstruction. This work advocates replacing the hard surfel-like constraints used in 2DGS with a soft penalty term that encourages low-rank Gaussian covariance where possible. This is very similar to the “mixed rank” strategy the reviewer mentioned and, in fact, was considered during the conception of the paper but rejected based on empirical evidence.

In the original 2DGS work, rank-deficient covariance matrices are shown to improve geometric accuracy of the estimated Gaussians, although this comes at the cost of view-synthesis performance degradation, i.e., 2DGS provides more accurate meshes but trails behind 3DGS in view-synthesis accuracy. PGSR proposes to take a middle ground by penalising the rank of the Gaussian covariance to improve view synthesis. It thus can be argued that the expressiveness of full-rank Gaussians is beneficial, at least for view synthesis.

However, contrary to findings of 2DGS and PGRS, it was found in our experiments that, in the context of learning splat predictors, 2DGS outperforms 3DGS representation on all grounds, including accurate view synthesis. This is perhaps due to the fact that overly expressive 3DGS can easily overfit to the texture of individual scenes—leading to better novel-view synthesis for one scene. However, these improvements are scene-specific and unstructured when generalised to large datasets. In the context of learning, 3DGS’s added expressiveness makes the loss function susceptible to local minima. Full ablations of pose (Table 1), depth (Table 2), and view-synthesis experiments (supplementary material) suggest that the expressiveness of full covariance is always harmful in our setup.

We agree that imposing hard rank deficiency reduces theoretical expressiveness. However, more expressiveness does not guarantee better results. On the contrary, over-expressive representations are sometimes the source of degeneracies, invoking the need for strong regularisations. Full-rank/mixed-rank Gaussians were rejected based on our experimental results, and we will explain this in the paper more clearly. However, we are happy to include an ablation for 3DGS with our loss for completeness and will add the relevant discussion.

Other application / evaluation of expressibility

We do not understand what the reviewer means by the 2DGS’s “inability to model volume, thickness, or multi-normal regions”. We are unaware of the inability of 2DGS to model thickness or volume. Perhaps the reviewer means transparent regions when they mention fuzzy areas and normal discontinuities when they mention multi-normal regions? Note that the rendered depths were evaluated on all pixels of context as well as virtual views on the ScanNet dataset (might these pixels include such fuzzy and multi-normal areas). We are happy to evaluate the trained models with the reviewer’s prescribed metrics on selected pixels to address the concern. We will be grateful if the reviewer will be kind enough to elaborate and suggest a concrete evaluation protocol that answers these concerns.

Relighting and other applications of 3D Gaussian splats are out of the scope of this paper and are not discussed in any feed-forward splats either. We inherit all the limitations and advantages from the 2DGS and generalizable splatting in terms of deploying the predicted splats to such applications. The limitations inherited from literature will be listed in the paper.

Training stability due to 2DGS

We are unaware of any claim the paper makes regarding training stability due to 2DGS. 3DGS variants (NoPoSplat) are trained successfully without any stability issues in this work. We, however, claim geometric accuracy when deploying 2DGS. We also mentioned that the 2DGS paper’s original normal loss was found unstable during training and $L_{orient}$ was introduced. Any text suggesting that 2DGS brings training stability should be removed from the paper, and we appreciate the reviewer for identifying such a mistake.

View synthesis ablations

The paper’s focus is on accurate geometric estimation, and it does not discuss a minor reduction in the number of parameters (it trivially estimates two instead of three scales, keeping all other architecture and prediction heads of NoPoSplat intact). Predicting one less scalar provides a minor efficiency boost that, in our view, is insignificant to discuss. We fail to see a trade-off in using 2DGS over 3DGS. 2DGS-parameterisation is theoretically sound, leads to minor efficiency gains, and provides boost in accuracy in all aspects of evaluation that we performed over 3DGS counterpart—including view synthesis.

We thank the reviewer for their valuable time and effort in reviewing our paper. The typos will be fixed, and write-up suggestions will be incorporated.

We disagree with the reviewer on the lack of novelty and contribution in our work. It should be noted that feed-forward multi-view splat prediction is an important problem. This paper highlights that all existing approaches that claim to solve this problem effectively suffer from a lack of supervision, due to which they fail to learn the orientation and scale of Gaussians. Ours is the first framework to highlight these important oversights and fix them in a systematic, generalisable manner.

Of course, relations between point clouds, mesh, and volumetric density fields have been well studied in the literature for decades. We readily admit that our regularisations are inspired—or even “lifted”—from this rich literature. They are simple but effective in achieving our goals. Note that 2DGS itself inherits its surfel-like splat formulation as well as geometric regularisation from this very literature, using them in the context of multi-view splat inference (without learning). We do the same for generalisable splatting (with learning).

Technical adjustment of 2DGS ($L_{orient}$)

We request the reviewer to consider that the design and deployment of meaningful regularisations in the context of a specific problem is non-trivial and requires care. For example, our work shows that using regularisation from 2DGS as is does not work (see supplementary material). The predicted splats from such regularisation fail to capture detailed scene geometry, and training diverges to trivial planar reconstructions. This is likely due to the extremely sparse views used for training in our case; the original 2DGS algorithm was never tried under such conditions. The proposed loss, despite its simplicity, is more stable and effective than the well-celebrated 2DGS normal-consistency loss and should not be ignored as a “technical adjustment.” The 2DGS normal-consistency term is relatively complex, requires gradients to be propagated via the rendering engine, and is novel in its formulation—but leads to optimisation instability. Our solution is shown to be empirically better (and essential) and should not be penalised for its simplicity.

Similarly, our alignment loss supplements the inherent structure–pose ambiguity that DUST3R-like frameworks suffer from when depth supervision is unavailable; multiple point-cloud/camera-pose combinations can produce the same rendering. It is obvious in hindsight that alignment loss must be used alongside view-synthesis loss to make the learning objective well posed. However, all generalisable splatting algorithms that attempt to “splatify” DUST3R crucially missed this requirement. Not using it is a critical design flaw of some prevalent frameworks in the literature. We agree that more sophisticated regularisation terms can be invented, and some might outperform our proposed alternative. However, acknowledging that geometric degeneracies exist in the absence of regularisers is the first step toward making meaningful progress.

Tiny improvements and blurry depths

We want to highlight that duplicating texture to the geometry is neither a minor nor a single issue of existing frameworks. Geometry from existing generalizable splatting methods is unusable for most practical applications, such as navigation, manipulation, or AR/VR. Consider, for example, trying to augment a toy duck on top of the quilt in Figure 4 (last row). Local planes are not readily estimable from such depths for successful augmentation. Even after correctly placing the duck on a local plane on a quilt, the entire quilt texture (stripes) will be assumed to occlude the small toy duck based on the estimated geometry. Alternatively, try to imagine a toy copter attempting to fly through the paintings and walls due to the big holes predicted by baseline approaches. Our “blurrier” geometric edges do not have quite the same adverse effect when used for these tasks.

We do admit that the results are blurred around true geometric edges more than desired. One reason is that $L_{orient}$ was minimised at all the input image pixels. However, the underlying normal-estimation procedure from Gaussian means is undefined at scene edges and should be ignored at discontinuities. An edge-aware variant of discarding $L_{orient}$ at depth discontinuities has since been adapted, and it provides relatively sharper reconstructions. It should be noted that such a change is trivial and does not make the approach any more novel. We maintain that the reconstructions provided in the original manuscript are significantly better than baselines on many grounds, and these improvements are overlooked.

Pose-free vs pose-aware methods

The reviewer misunderstood our categorisation of the pose-free generalisable splatting approach. We clarify that all methods discussed require ground truth relative poses between input views (and virtual views) during training. The comparisons are fair, as all approaches use ground truth camera poses during training. The term “pose-free” here means that NoPoSplat, Splatt3R, and our approach do not need relative poses at inference; MVSplat and pixelSplat require this information at inference and are thus called pose-aware.

Regarding Splatt3R, the framework uniquely does not modify the registered point-cloud estimation of DUST3R at all. Thus, the point-cloud estimation and PnP-RANSAC–based pose-estimation accuracy of DUST3R remain intact in Splatt3R.

Literature shows that Splatt3R’s view-synthesis performance is inferior to almost all other baselines. We also evaluated its rendered depth, which exhibits the same geometric inconsistencies. Splatt3R’s rendered depths have an average AbsRel error of 0.148 (ours: 0.100) on the ScanNet test set. Note that Splatt3R was trained on ScanNet. We can include these results in the paper upon acceptance.

We thank the reviewer for their time and effort in reviewing this submission. The reviewer agrees with the motivation and claims made. They also agree that these claims are substantiated through experiments and appropriate ablations. They, however, choose to recommend rejection of the paper, citing the simplicity of the ideas that this paper introduced and underestimating the impact of our work. We provide a detailed thematic response to address the reviewer’s concerns here.

Novelty and “Merely Regularising NoPoSplat”

We respectfully disagree with the reviewer’s assessment that incorporating multiple well-thought-out regularizations in solving an important learning problem is limiting and not novel enough to be accepted at NeurIPS. On the contrary, this paper challenges the status quo, isolating geometric degeneracies that all existing solutions in the literature suffer from and overlook. The paper proposes to fix these degeneracies systematically with remedies that are trivially adoptable in nearly all state-of-the-art splat predictors—creating a large impact on the field. If these are not traits of a good NeurIPS paper, what will be?

Since the introduction of generalised splatting in pixelSplat (CVPR 24), several attempts have been made in the past two years to solve the problem. These include MVSplat (ECCV 24 oral), Splatt3R (CoRR), NoPoSplat (ICLR 25 oral), and DepthSplat (CVPR 25). Much like the reviewer did for our work, many of these works can be trivialised. MVSplat simply replaces the pixelSplat's epipolar-attention transformers with the well-studied cost-volume-based architecture of UniMatch; NoPoSplat plugs in the DUST3R architecture in MVSplat; and Splatt3R adds a DPT decoder head to a frozen pre-trained DUST3R for estimating additional 3D Gaussian parameters. We disagree with such trivialisation of these important works, including our own. Note, however, that all these highly-received papers claim to learn the orientation, scale, and opacity of 3D splats, they use existing network architectures and standard well-studied view-synthesis losses. All of these frameworks fail to do what they claimed effectively, as shown in this paper.

This paper demonstrates degeneracies in predicting splat parameters using view-synthesis loss—this loss is used by the entire community without exception. So the issues mentioned in the paper are not NoPoSplat-specific. These traits are shared amongst all splat-prediction frameworks, including the recent adaptation of VGG-NT for splatting in AnySplat. We want to reiterate that NoPoSplat was merely chosen as the baseline, given its state-of-the-art performance at the time and its ability to work without requiring a relative pose between images at inference time. The losses used in the paper provide a “simple recipe” to facilitate necessary supervision for Gaussian orientations and scales for the first time. Findings of this paper have the potential to change the landscape of feed-forward splatting and should not be viewed as a minor adjustment to NoPoSplat alone.

Authors are unaware of any attempt made to isolate and discuss degeneracies in 3D splat predictions in the last two years, let alone address them by deploying suitable regularizations that the reviewer considers “common”. This alone showcases that the ideas presented in this paper are non-trivial. The simplicity of our frameworks is its core strength and should not be used to trivialise our work’s contributions or potential impact.

Marginal improvement over SOTA (2%)

The 2% improvement comment is a gross generalisation of our experimental findings and is unfounded. We strongly contest that the quantitative improvements of our approach are marginal.

• Pose estimation improvements: The regularizations improve pose estimation with PnP + RANSAC by 9.6% (from 57.2% to 62.7% AUC@$5^\circ$ error threshold over the baseline on Re10K. The improvement is nearly 73% (7.8% vs 13.5%) on ScanNet. These results showcase the effectiveness of the proposed regularizations and justify our claims of making predicting geometrically meaningful and accurate raw splats. Even with pose optimisation with view synthesis loss minimisation, our method predicts accurate poses (with AUC@$5^\circ$ error threshold) for 15.6% cases against 10.9% of NoPoSplat on ScanNet — well above 2% the reviewer claims. Note that NoPoSplat, on the other hand, can not retain DUST3R's performance with PnP-RANSAC on ScanNet, let alone improve on it. It is evident from experiments that “splatifying” DUST3R naively does not work.

• Structure estimation improvements: We would like to point out that the paper provides a boost in relative depth accuracy ($\delta_1 < 1.10\uparrow$) of 22.7% without (20.6% with) view synthesis-based relative pose optimisation over the baseline of NoPoSplat on ScanNet. These improvements are in line with NoPoSplat (30% without and 33% with pose refinement over MVSplat) and are not minor.

Most importantly, we would like to highlight meaningless Gaussian orientations, unstructured scales and grossly erroneous view-inconsistent depths with texture-sensitive protrusions that “every” studied approach predicts. These degeneracies are unacceptable, and ours is the only approach that fixes them. We encourage the reviewer to read our response to Reviewer ugeC, highlighting the qualitative depth prediction improvement that is attained by our approach as well. We are sure that upon a closer inspection of the qualitative results, both the reviewer and ACs will agree that the current state-of-the-art feed-forward networks do not provide splats with accurate enough geometry to be used in applications like robotic navigation and manipulation. The proposed approach makes large strides in attaining this.

Pose estimation and photometric optimisation

The reviewer asks why view-synthesis loss minimisation does not yield a similar boost as observed in NoPoSplat. Note that view-synthesis loss minimisation is tailored to NoPoSplat: large alignment errors in RAW splat predictions are fixed with this optimisation. The optimisation is cumbersome, involving multiple iterations of time-consuming rendering steps. We use it as is for completeness and fair comparison with NoPoSplat’s full pose-estimation pipeline. Possible reasons for limited gain:

1. View-synthesis loss is not the sole loss minimised during training, making it non‑optimal for pose optimisation. A combination of PnP and view-synthesis loss might be more suitable. Moreover, our approach uniquely produces accurate surface normals as Gaussian orientations; these orientations can inform relative rotation estimation (e.g. by minimising point-to-plane geometric errors for aligning rigid surfaces) but are not used in the current pose-estimation pipeline.

2. The pose-optimisation routine assumes 3D covariance matrices for Gaussians. Our hard, surfel-like Gaussians must be converted to full 3D Gaussians by adding a small third scale, which may introduce inefficiencies.

Selecting or designing an optimal pose-estimation routine is not the focus of this paper. Our work highlights and removes geometric inconsistencies in splat predictions. Pose-estimation heuristics vary widely in generalizable splatting (and other neural-geometric estimation) literature and continue to evolve (e.g. modern frameworks like VGG-NT drop PnP-RANSAC and rendering-based refinement in favour of direct network-based pose prediction). Our geometrically consistent splats are compatible with these frameworks and could improve them. We use PnP-RANSAC as a proxy for measuring alignment error between two sets of raw splats predicted by any pixel-aligned method. Other heuristics from surfel-registration literature could yield faster or more accurate inference but are out of this paper’s scope.

Outdoor scene and object-centric reconstructions

The reviewer in the summary suggests that our $L_{orient}$ encourages orientation of the neighbouring Gaussians to be similar. We want to clarify that this is not correct and might have led the reviewer to assume that higher curvature poses a problem for our regularisation. Please note that the L_orient only says that the means of the neighbouring Gaussians should provide appropriate local surface normals. These surface normals should align with the Gaussian orientations that the network predicts. Curvatures are neither estimated not penalised. It can be seen that true surface discontinuities are captured in the estimated normals by our approach (see Figures 1,2 of the main paper and supplementary material). Additionally, both Re10K and ScanNet have some objects (and humans) with reasonable curvature. Our approach estimates these curvatures correctly too. We will include some of these visualisations in the paper alongside CO3D results for all baselines upon acceptance.