SplatFormer: Point Transformer for Robust 3D Gaussian Splatting

We thank the reviewer for recognizing the strengths of our submission, including the motivation behind the target task, the novelty and soundness of adapting a 3D point transformer for 3DGS refinement, the comprehensive experimental results, and the clarity of the paper. Below, we address the reviewer’s question concerning the performance of SplatFormer when trained and evaluated with varying input and out-of-distribution test views.

Question. Varying input and test views

First, while SplatFormer is trained with supervision only from top-down out-of-distribution (OOD) views with elevations near $90^\circ$ and the input views with elevations $<=15^\circ$, it can refine the input 3DGS during inference to enhance renderings from a wide range of viewpoints across the entire upper hemisphere. In our original submission, Fig. 2 shows that SplatFormer consistently outperforms 3DGS at elevation angles between $20^\circ$ and $90^\circ$. Additionally, SplatFormer improves renderings at varying distances. To demonstrate this, we reduce the camera-to-origin distance by a specified factor and render zoomed-in, close-up views. Below, we report the PSNRs for different elevation angles and camera radii ($R$) on the GSO-OOD dataset.

The results demonstrate that SplatFormer’s refinement improves not only the top-down OOD views used during training but also renderings across a wide range of elevations and camera positions. This is achievable as long as the input captures provide full angular coverage along one axis (e.g., azimuth) while having limited angular coverage along another axis (e.g., elevation).

Notably, when $R \leq 0.6$ and Elevation $\leq 30^\circ$, the test views and input views share similar elevations but differ in their distances to the object. This corresponds to a case mentioned by the reviewer, and in such instances, our trained SplatFormer consistently enhances 3DGS performance.

However, we acknowledge a limitation of SplatFormer’s current simplified training procedure, which focuses on a specific input view distribution. It struggles to enhance quality when faced with significantly different input view distributions from those used to optimize the input 3DGS. For example, if only top-down, high-elevation views (Elevation $\geq 50^\circ$) are used for training 3DGS, our current SplatFormer model, trained on the setup with low-elevation views as input views and top-down views as target views, provides limited improvement in quality for low-elevation views (Elevation $\leq 10^\circ$):

We attribute this limitation to two factors. First, high-elevation input views capture only a small portion of the scene, leaving much of the lower regions unobserved - areas typically covered by low-elevation views. Since SplatFormer does not learn to generate (or 'hallucinate') completely unseen parts of the scene, it cannot correct artifacts in these unobserved regions. Second, variations in the distribution of input views produce different types of 3DGS artifacts, some of which differ significantly from those encountered during training. We believe this limitation can be addressed by creating a more diverse OOD synthetic training dataset with a wider range of input view trajectories.

Despite this limitation, the main contribution of our work lies in introducing the important problem of OOD-NVS and proposing a novel architecture and learning paradigm to address it. In future work, we plan to expand and diversify both the training dataset and the applications of SplatFormer, enhancing its robustness to arbitrary shifts in train-test camera distributions and its ability to handle unbounded, large-scale scene processing.

We thank the reviewer for raising this valuable question. We will include a geometry comparison experiment in our revised manuscript, as presented in Figure G.3 of the updated appendix.

To quantitatively evaluate performance, we measure the mean absolute error (MAE) between the ground-truth depth and normal maps, and the corresponding rendered depth and normal maps, under out-of-distribution (OOD) views for 3DGS [8] and our method. Specifically, the depth maps for 3DGS and our method are rendered as the weighted average depth of Gaussian primitives, which is a common way to derive depth maps from 3DGS as used in the gsplat toolbox [9] and other 3DGS-related work [10, 16]. We then follow 2DGS [17] to compute the normal maps using finite differences on the estimated surface derived from the depth maps.

The results show that, in addition to improving rendering quality, our method enhances the accuracy of rendered depth and normal maps:

We also provide a visualization of the rendered depth map in Figure G.3 of the updated appendix. In our revised manuscript, we will include more comprehensive qualitative and quantitative results on geometry.

Additionally, since 2DGS [17] produces more regularized depth and normal maps than 3DGS, our method could potentially further improve geometry results when applied to 2DGS refinement, as discussed in Lines 1128–1133 of the Appendix.

We thank the reviewer for their thoughtful feedback and for recognizing the strengths of our work, including the importance of addressing the OOD-NVS problem, the extensive experiments validating our approach, and the state-of-the-art performance achieved on various object-centric datasets.

Below, we address the reviewer’s discussion points: (W1) Non-Object-Centric Scenes: Exploring the applicability of our method to real-world, non-object-centric scenes with more complex foregrounds and backgrounds; (W2) Geometry Comparisons: Providing additional comparisons on reconstructed geometry, including metrics such as depth and surface normals. We hope these responses address the reviewer’s questions and further clarify the contributions of our work.

Weakness 1/2. Applicability to Non-Object-Centric Scenes

We appreciate the reviewer’s recognition of our experiments on real-world objects. We believe that strong performance in object-centric scenes can contribute to addressing complex scenes, particularly if a compositional approach is used to decompose the foreground and background, allowing SplatFormer to process individual objects separately.

Additionally, we would like to direct the reviewer’s attention to Section F, "Limitations and Future Directions" (Lines 1162-1185) of the Appendix, where we discussed SplatFormer’s potential for real-world unbounded scenes. Specifically, we conducted an experiment using the MVImgNet [5] dataset, a large-scale collection of multi-view images. For each capture in MVImgNet, we split the views into frontal views and side views, treating one as input views and the other as OOD test views. We trained SplatFormer on a curated MVImgNet-OOD dataset and tested it on OOD views of held-out scenes. More experimental details are provided in Appendix F. For reference, we include the test results from Table F.1 below:

We provided visual results in Figure F.1 and added additional results in Figure G.2 of the appendix in our updated manuscript, demonstrating that SplatFormer reduces floater artifacts and improves geometry in certain cases. However, we acknowledge its limitations in enhancing high-frequency details, which may be due to the need for a larger-capacity point transformer or a novel multi-scale hierarchical design to handle large-scale, irregularly spaced point clouds.

Regarding the challenge of acquiring real-world datasets, we speculate that incorporating a diverse range of synthetic datasets could reduce reliance on massive-scale real-world datasets. Notably, training our model exclusively with synthetic data has already demonstrated promising results on real-world test scenes (Table 2, Figure 5, and Figure F.5).

Finally, unbounded scene reconstruction from limited observations remains an open challenge, with most existing prior-enhanced NVS methods [1, 2, 3, 4] primarily focused on object-centric scenes. In contrast, SplatFormer not only excels in these settings but also demonstrates promising potential for unbounded scenes. Enhancing the network architecture and incorporating more synthetic and real-world training data will be key areas of focus in our future work.

We appreciate the reviewer’s insightful suggestion. While leveraging pseudo-OOD views from diffusion-based novel view synthesis (NVS) baselines is an interesting approach, we observed that such views often lack plausibility and introduce artifacts. Incorporating these erroneous pseudo views into the 3DGS training pipeline can actually degrade the final reconstruction quality compared to 3DGS [8].

Experiment Details:

To illustrate this, we conducted experiments using state-of-the-art open-source diffusion-based NVS models: SV3D [2] and EscherNet [3]. As detailed in our response to Reviewer K5mf, EscherNet outperformed other baselines, including SyncDreamer [1] and SV3D [2], due to its ability to process multiple input views, whereas other diffusion-based methods are limited to a single input view. By combining the pseudo-OOD views generated by SV3D or EscherNet with the ground truth input views, we trained 3DGS and obtained the following results:

A visual comparison is included in Figure G.1 and the second uploaded video in the supplementary material (file name: compare_with_diffusion-sparse-baselines.mp4).

Discussion:

Our results indicate that pseudo-OOD views generated by diffusion-based methods are often suboptimal, introducing errors that propagate through the training pipeline and degrade performance. This is reflected in the lower PSNR, SSIM, and LPIPS values for models trained with pseudo views compared to our method.

In contrast, our approach directly refines 3DGS representations using a point transformer, enabling robust generalization to challenging OOD poses without relying on pseudo views. This avoids the pitfalls of hallucinated artifacts and ensures that reconstructions maintain high fidelity.

Further discussion on the limitations of sparse-view and diffusion-based NVS methods is provided in our response to Reviewer K5mf ("Weakness 1: Generative Diffusion Priors for OOD-NVS").

We thank the reviewer for their thoughtful feedback and for recognizing the strengths of our work, including the meaningful contribution of a generalizable transformer for refining 3D Gaussian splats, the construction of datasets tailored to the OOD problem, and the extensive experiments validating the effectiveness of our method.

In the following, we address the identified weakness (W1) concerning comparisons with LaRa under unfavorable conditions and respond to three discussion points: (Q1) utilizing pseudo-OOD views for view synthesis, (Q2) the training setup and OOD view selection, and (Q3) the analysis of SplatFormer’s resource efficiency.

The reference list for the cited papers is provided in our general response.

Weakness. Additional Comparison with LaRa on a 4-View Setup

We appreciate the reviewer’s comment and suggestion. Since LaRa is computationally expensive and limited to a maximum of four input views due to memory constraints on standard GPUs (e.g., RTX-4090, 24 GB), we can only provide four input views to LaRa for our OOD-NVS tasks, which involve 32 input views. However, following the reviewer’s suggestion, we are happy to perform an additional comparison under the four-view setup by applying our framework to the Gaussian splats generated by LaRa [4].

Experiment Details:

We used LaRa's released checkpoint to predict 2DGS from four input views for our Objaverse training scenes. Subsequently, we trained our SplatFormer to refine LaRa’s outputs using the same procedure described in the experiments section of our paper. Even in this constrained four-view setup, SplatFormer achieved a noticeable improvement over LaRa’s baseline performance on the Objaverse-OOD and GSO-OOD test sets.

Discussion:

It is important to note that LaRa’s 2DGS encodes information from only four input views, which often results in suboptimal visual quality with significant blur artifacts, particularly when viewed from OOD perspectives. While SplatFormer enhances these outputs, fully reconstructing OOD views from such an extremely sparse input-view setup remains highly challenging. This limitation arises from the inherently ill-posed nature of the reconstruction task under these conditions.

Our method, by contrast, is specifically designed for scenarios with denser input views, where more comprehensive information is available for reconstructing high-fidelity outputs. This design aligns with practical applications, such as immersive navigation or surgical visualization (L:048), where capturing dense, calibrated views is both feasible and necessary. Accordingly, in our main paper (Table 1), we focus on comparisons in the dense-input setup, emphasizing LaRa’s limitations in scaling to scenarios with dense input views.

We thank the reviewer for their insightful question regarding the efficiency of SplatFormer. In the final manuscript, we will highlight SplatFormer's resource efficiency and include a detailed analysis and comparison to emphasize its scalability and practicality.

Memory and Inference Time Analysis

To assess SplatFormer's efficiency, we measured its memory usage and inference time across varying numbers of Gaussian splats. The results are summarized as follows:

In the GSO test set, the average number of input splats is 50k, with 80% of inputs containing fewer than 70k splats - an amount sufficient to capture high-frequency details. These results demonstrate that SplatFormer scales efficiently with increasing numbers of Gaussian splats, with memory usage and inference time remaining within practical bounds.

Comparison with Other NVS Methods

To provide additional context, we benchmarked SplatFormer's performance against other state-of-the-art NVS methods, including the diffusion-based EscherNet [3] and LaRa [4], which specifically highlights efficiency as a key advantage. The results are as follows:

We measured LaRa’s inference time and memory usage for generating 2D Gaussian splats (2DGS) from four input views, and EscherNet’s for producing a single output view using the same four input views.

Notably, another advantage of our method is that it generates 3DGS splats in a fast single pass, enabling real-time rendering into any view. This is significantly more efficient than approaches like SV3D [2] and EscherNet [3], which rely on 2D diffusion generative models and require multiple time-consuming diffusion steps to produce only a single frame.

The results demonstrate that SplatFormer achieves significantly better inference time and memory efficiency while utilizing a more lightweight architecture, making it a more scalable model.

We will include a discussion on efficiency in the revised manuscript.

We thank the reviewer for raising this insightful question. In the revised manuscript, we will include an ablation study to analyze the impact of the ratio of OOD views to input views during SplatFormer’s training on the final model performance.

As noted in Lines 860–861 of Appendix D in our manuscript, our training setup involves rendering four target images per scene, with a default setting of 70% OOD views and 30% input views.

Experiment Details

To investigate the effect of varying the OOD view ratio, we conducted an ablation study. Specifically, we trained four SplatFormers on the Objaverse-OOD training dataset, using different ratios of OOD views and input views for the photometric loss. We then evaluated the models' performance on various test views. Then we evaluate the models’ performance on different test views. Due to computational constraints, this study was performed using a fraction of the computational resources allocated for our full model (4x RTX-4090 GPUs with 200k steps and the gradient accumulation step set to 2).

Metrics were evaluated on Objaverse-OOD test scenes across three elevation ranges: low (20°–30°), mid (40°–60°), and high (70°–90°).

Discussion

Our results indicate that increasing the ratio of OOD views during training improves performance at extreme viewpoints (e.g., high elevations), though it slightly compromises rendering fidelity at lower elevations. We selected the default 70% OOD ratio as it provides a balance between enhancing OOD views and preserving high fidelity for in-distribution views.

We plan to complete this study using the full computational budget and include it in the revised manuscript to provide a more comprehensive analysis of the impact of OOD and input view ratios.

Weakness 2/3. Scene-level Refinement, Unbounded Scenes:

Recognizing the importance of addressing OOD-NVS in complex scenes, we discussed SplatFormer’s potential for real-world unbounded scenarios in Section F: Limitations and Future Directions (Lines 1162-1185) of the originally submitted Appendix. To evaluate its performance, we applied our proposed framework to the MVImgNet dataset [5]. On the test set, our method outperforms 3DGS:

We present the visual comparison in Figure F.1 and Figure G.2, showing that SplatFormer reduces floater artifacts and improves geometry in many cases. However, we acknowledge its limitations in enhancing high-frequency details, likely due to the current network's insufficient capacity for processing large-scale, irregularly distributed point clouds. Future improvements could involve designing a novel multi-scale hierarchical point transformer architecture to handle large scenes and incorporating real-world training data alongside synthetic data. Additionally, since SplatFormer demonstrates strong generalization on real-world objects (Table 2, Figure 5, and Figure F.5), it may be feasible to decompose the scene and process individual objects separately.

Unbounded scene reconstruction from limited observations remains an open challenge, with a majority of existing prior-enhanced NVS methods [1,2,3,4] also focusing on object-centric scenes. In contrast, SplatFormer not only excels in such settings but also shows promising potential for unbounded scenes. Improving the training strategy and network architecture will be our focus in future work.

Weakness 3/3. The Underperformance of Sparse-view Reconstruction Methods:

In our earlier response (Weakness 1), we addressed state-of-the-art open-source diffusion-based single-image-to-3D models, SyncDreamer [1] and SV3D [2]. Evidence shows that relying on a single input view introduces significant ambiguity, leading the model to hallucinate novel views that are misaligned with the input capture.

To further extend the discussion, we compare our approach to LaRa [4], a state-of-the-art method that predicts 2D Gaussian splats from four input views. While LaRa achieves impressive results in its original paper, we demonstrate that it struggles to produce high-quality OOD views.

In the original submission, we evaluated a fine-tuned version of the LaRa framework on the Objaverse-OOD and ShapeNet-OOD datasets. Here, we extend this evaluation by testing two LaRa models on the GSO-OOD test set: the released checkpoint model and the model fine-tuned on our Objaverse-OOD dataset. To highlight LaRa's degradation in OOD views, we report metrics in both in-distribution views (elevation<=10$\degree$) and out-of-distribution views (elevation>=70$\degree$).

$\dagger$: LaRa reports a PSNR of 29.15 on the GSO test set used in their paper. However, we observe that their test images primarily feature empty backgrounds, which inflates the PSNR score. In contrast, our test set includes extensive foreground regions, presenting a more challenging evaluation scenario.

We include a visual comparison between the fine-tuned LaRa and ours in Figure G.1, and the two supplementary videos (compare_with_major-baselines.mp4, compare_with_diffusion-sparse-baselines.mp4), demonstrating that LaRa's outputs suffer from noticeable blurriness and performance degradation in OOD views.

The results of SyncDreamer [1], SV3D [2], and LaRa [4] indicate that existing single/sparse-to-3D methods struggle to produce accurate OOD views due to their inability to utilize more than four input views to resolve ambiguity. In contrast, our method is agnostic to the number of input views, as it operates directly on the initial 3DGS set, which can be generated from an arbitrary number of views.

Question 1. Video Comparisons:

We direct the reviewer to the original supplementary materials, which include qualitative video comparisons (file name: compare_with_major-baselines.mp4) of our method against the major baselines.

To enhance the discussion and facilitate a more comprehensive visual assessment, we provide another video (file name:compare_with_diffusion-sparse-baselines.mp4) comparing our method against the diffusion-based and sparse-view baselines (Weakness 1/3 and Weakness 3/3).

We thank the reviewer for their valuable feedback and for recognizing several positive aspects of our submission, including addressing a significant challenge in neural rendering and proposing an efficient and well-founded approach using Point Transformer 3DGS refinement.

Below, we address the main discussion points raised by the reviewer: (W1) the utilization of generative priors for out-of-distribution novel view synthesis tasks, (W2) the method's capability for scene-level reconstruction and handling unbounded scenes, (W3) the reasons behind the underperformance of single-image-to-3D models, and (Q1) the request for video comparisons.

Weakness 1/3. Generative Diffusion Priors for OOD-NVS (part-1)

Generative models, particularly diffusion models, hold significant potential for improving novel view synthesis. We provide a discussion of these methods in the original submission (lines 184-186) and perform a comprehensive quantitative evaluation in the OOD-NVS setup against representative baselines, including SyncDreamer [1], SSDNeRF [12] (Table 1, Figure F.2, Figure F.3), and DiffBIR [13] (Table 3, Figure C.1). The selected methods encompass diverse approaches to integrating diffusion-based priors, including single-image-to-3D techniques leveraging 2D diffusion priors (SyncDreamer), methods utilizing diffusion-based 3D priors (SSDNeRF), and pipelines focused on diffusion-based image enhancement (DiffBIR).

Our analysis highlights the following limitations of these methods within the considered OOD-NVS scenario:

• Erroneous Hallucination under OOD Views: Generative models may hallucinate content in OOD views that is absent from the input views. In real-world scenarios requiring accurate reconstruction, such as surgical scene reconstruction, this behavior presents significant challenges to the models' applicability.
• Limited Capacity for Handling Multiple Input Views: Most models are restricted in the number of input views they can effectively process and fail to leverage the dense image sets that are often available in the considered OOD-NVS scenario.
• Multi-view Inconsistency and Flickering Artifacts: These methods struggle to maintain temporal and spatial coherence across generated views.

To further validate our findings, we evaluated additional state-of-the-art open-source diffusion-based methods and extended the comparison to include some previously discussed approaches. The evaluated methods are as follows:

• SyncDreamer (ICLR 2024) [1]: A single-image-to-3D method leveraging Stable Diffusion [6]. Please note that the method was already evaluated in the original submission.
• SV3D (ECCV 2024) [2]: A single-to-3D method utilizing Stable Video Diffusion [7].
• EscherNet (CVPR 2024) [3]: To the best of our knowledge, the only open-source diffusion-based method capable of processing more than 10 input views.

For SyncDreamer and SV3D, we use the first frame of the input view set as the input condition. For EscherNet, we use all of the 32 input views as input conditions.

For a fair comparison with our model, we fine-tuned the released checkpoints of SyncDreamer and EscherNet using the same Objaverse-OOD training set and computational resources employed for SplatFormer. Fine-tuning was necessary for SyncDreamer, as the released model is limited to generating its predefined 16 novel views, and it also enhanced EscherNet’s performance on our OOD test set. Since the concurrent work SV3D does not provide its training script, we tested its released checkpoint in a zero-shot manner.

To address the multi-view inconsistency in the per-frame generated results of these diffusion models, we also attempted to optimize 3DGS using the output of the diffusion models. Specifically, we employed the fine-tuned EscherNet and the released SV3D to generate 9 OOD views. These generated novel views, combined with 32 ground-truth input views, were used to train a 3DGS. We then evaluated the OOD renderings produced by the distilled 3DGS, denoting these experiments as EscherNet$\rightarrow$3DGS and SV3D$\rightarrow$3DGS. Notably, these experiments also directly address a question raised by Reviewer MA5d (Question 1/3: Pseudo-OOD Views for 3DGS training).

Please refer to the following sections for the experimental results.

We report metrics on the GSO-OOD dataset:

We have included a video comparison in our updated supplementary material (file name: compare_with_diffusion-sparse-baselines.mp4). We encourage the reviewer to watch the video, as it visually demonstrates the limitations of diffusion-based methods.

The following observations can be made from the presented quantitative and qualitative results:

• Although diffusion-based baselines generally produce visually plausible OOD views, they often hallucinate scene elements that are inconsistent with the input views. This issue also affects EscherNet, even when provided with all 32 input views. For example, when processing Adidas (c) shoes with the iconic three parallel stripes, EscherNet mistakenly hallucinates a fourth stripe in its OOD view.
• Likewise, the hallucination error propagates into the distilled 3DGS, making it incapable of addressing the artifacts in OOD views.
• These issues are also reflected in the quantitative results of all the baselines, which show a PSNR more than 6 points lower than our method (EscherNet$\rightarrow$3DGS: 18.88 vs. Ours: 25.01).

In contrast, our method addresses the limitations of diffusion-based NVS through holistic reasoning over a set of 3DGS primitives. Another key advantage of our approach lies in its computational efficiency: the input splats are refined in a single pass through the 3D point transformer, eliminating the need for the computationally expensive denoising process required to generate each image in diffusion-based approaches.

Nonetheless, we recognize the impressive capability of diffusion-based methods to generate complex, photorealistic scenes with minimal input. Exploring ways to combine their strengths with the 3D consistency, efficiency, and robustness of our method offers a compelling direction for future research.