IBGS: Image-Based Gaussian Splatting

We appreciate the reviewer for the constructive comments that are valuable to improve our method. We address the reviewer's concerns below.

---

1. How the multi-view color consistency loss of IBGS leads to more precise Gaussian parameters with high-opacity concentrating around the true surfaces

To verify IBGS leading to higher opacity for Gaussianss that are close to the true surface, we conduct the following experiment. First, for each pixel in the image, we find $K=8$ Gaussian-ray intersections that are closest to the ground truth surface points. This process is done for both IBGS and 3DGS, resulting in 8 opacity maps for each method. For each opacity map, we compute the average opacity across all pixels and compare the average values produced by IBGS and that produced 3DGS. As a result, on the Ignatius scene in the Tanks and Temples dataset, the average opacity values of IBGS are approximately 26% higher than that of 3DGS. This result demonstrates that IBGS have Gaussians with higher opacity locating near the ground-truth surface compared to 3DGS.

Furthermore, we evaluate the rendered depth map of IBGS and 3DGS by using the $K$ median Gaussians (Eq. 5, main paper). Intuitively, it can be seen from the Eq. 5 that, in order to obtain accurate depth value, either of the two following conditions must be satisfied: (1) All the intersection points $x_k$ concentrate around the ground-truth surfaces and higher weights $w_i$ which are related to the Gaussian opacity or (2) there may exist intersection points $x_k$ that are far from the surfaces; however, their weights $w_i$ are relatively smaller than those of the points closer to the surfaces. Therefore, methods that yield higher depth accuracy better satisfy the two conditions, and thus having more high-opacity and surface-aligned Gaussians. The depth error in the table below shows that IBGS significantly outperforms 3DGS, achieving an average error reduction of 63%. This result is a strong indication that IBGS produces more high-opacity Gaussians concentrated near the true surfaces compared to 3DGS. In our training pipeline, the weights $w_i$ are optimized via the multi-view color consistency loss $L_{photo}$. Therefore, this loss is the primary component driving the formation of high-opacity, surface-aligned Gaussians in IBGS.

(We report the absolute relative depth error in the table below)

Relation to the limitations of low-degree spherical harmonics: Due to the low-degree spherical harmonics, 3DGS struggles to handle complex view-dependent color. In contrast, IBGS relies on color information from multiple source views to infer the view-dependent color at a target viewpoint. Given the high-opacity Gaussians concentrating the true surfaces, IBGS can accurately represent the scene geometry, yielding more precise pixel correspondences between views which are essential for obtaining information regarding the view-dependent effect in the neighboring images.

---

2. Clarification for the exposure correction module

We agree with the reviewer that the ultimate goal of the exposure correction module is to encourage exposure consistency across views and improve fidelity. We would like to further clarify that the proposed module helps correct the exposure of a rendered image by mimicking the exposure of the nearest neighboring views. This exposure correction can lead to either an increase or a decrease in the brightness of a rendered image. We thank the reviewer for the valuable suggestion. In the revised version, we will provide more clarification for this module, as well as include a visualization of a case where the exposure correction module helps reduce overexposure.

---

3. Why exposure correction is only applied for TNT dataset

Following [1], we only apply the exposure correction to the TNT dataset since the camera auto-exposure setting is enabled while capturing images in this dataset, as discussed in [2]. In practice, the exposure correction module can be activated by default without concern for performance degradation, as it has minimal impact when there is no exposure variation across the training views. To validate this, we applied the exposure correction module to the MipNeRF-360 dataset and present the results below. It can be observed that the method’s performance remains largely unchanged when exposure correction is incorporated.

---

4. Comparison to more baselines, including Spec-Gaussians

In Tab. 4 (supp. mat.), we present a comparison with additional baseline methods, including Spec-Gaussians. The results indicate that our method consistently achieves the best SSIM across all datasets. Compared to Spec-Gaussians, IBGS demonstrates superior image quality in all evaluated metrics. Unlike Spec-Gaussians, which focuses solely on modeling the specular component of scene appearance, IBGS is not limited to modeling specular color and can also capture various types of view-dependent effects, such as reflection and diffraction. To further demonstrate the superiority of IBGS in modeling view-dependent color, we compare IBGS and Spec-Gaussians on three scenes with challenging view-dependent color from the Shiny dataset. The results in the table below show that IBGS achieves significantly better performance, outperforming Spec-Gaussians by at least 4.52 dB in PSNR.

(We report the image metrics as PNSR/SSIM/LPIPS in the table above)

---

5. Comparison to more baselines, including SpecTRe-GS

Unfortunately, we were unable to compare with SpecTRe-GS, as it was published at CVPR 2025, which occurred after the NeurIPS 2025 submission deadline. Moreover, the official implementation of this method has not yet been released.

---

6. Comparison in terms of training time and rendering FPS

We provide a rendering speed comparison between IBGS and prior methods in Tab. 3 and Tab. 6 (supplementary material). While IBGS introduces additional computations compared to 3DGS, resulting in lower rendering FPS, it still achieves real-time performance of over 30 FPS in most cases, as shown in both tables. Furthermore, Tab. 6 presents a study of the trade-off between rendering speed and image quality by varying the resolution of the predicted color residual map. By reducing the resolution by half, IBGS achieves a rendering speed of at least 58 FPS across all datasets while still delivering the better image quality compared to 3DGS and SuperGaussian.

Regarding the training time, the table below shows the comparison between our method, 3DGS and SuperGaussian. Due to the additional computation, the training time of our method is longer than 3DGS's. On the other hand, compared to SuperGaussian which learn Gaussian spatially-varying texture, the training time of our method is faster in all datasets. We will include the training time comparision in the revised version.

(We report the training in minutes, average across all scene in a dataset)

---

7. Scalability of IBGS in case of large scenes

We agree with the reviewer that IBGS could encounter memory bottlenecks when handling large scenes or a substantial number of images. Specifically, IBGS requires pre-loading the source images into GPU VRAM to avoid the overhead associated with on-the-fly memory allocation for these images. However, this approach leads to higher VRAM usage of IBGS compared to that of 3DGS and SuperGaussian, as illustrated in the table below. A feasible solution to mitigate this limitation is to load each image into VRAM only when it is selected as a neighboring view needed for rendering. Nevertheless, this approach introduces a trade-off between speed and memory usage, as dynamic VRAM allocation can adversely affect the rendering speed of IBGS. We thank the reviewer for this valuable insight and we will include a discussion of this speed-and-memory trade-offs in the revised version.

(The VRAM memory is reported in GB)

---

8. More discussion for the method limitations

We thank the reviewer for the suggestion and will expand the discussion of our method’s limitations, including (1) requiring additional computations, leading to slower training times/rendering speed compared to 3DGS, and (2) the trade-off between speed and memory usage when dealing with large scenes.

---

[1] Chen, Danpeng, et al. "Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction." IEEE Transactions on Visualization and Computer Graphics (2024).

[2] Barron, Jonathan T., et al. "Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

We thank the reviewer for spending time reviewing our paper, as well as acknowledging the strength and novelty of our method. We address the reviewer's concern as below.

---

1. Comparison to AbsGS

We include a comparison with AbsGS in Tab. 4 (supp. mat.). Results indicate that our method consistently achieves better performance over AbsGS across all datasets. Note that AbsGS focuses on recovering high-frequency details through a novel homodirectional gradient-based densification strategy. Therefore, the contribution of AbsGS can be considered orthogonal and complementary to that of our method, and its densification approach can be integrated into ours to yield further performance improvements. This is confirmed by the results in Tab. 4 (see "Ours + [20]") where we incorporate the densification strategy proposed in AbsGS into our training pipeline and further enhance the performance of our method.

---

2. Comparison to Octree-GS

As suggested, we present the comparison with Octree-GS in the table below. The results indicate that IBGS significantly outperforms Octree-GS on the Mip-NeRF 360 and TNT datasets, while showing comparable performance on the Deep Blending dataset. We thank the reviewer for the suggestion and will include the comparison with Octree-GS in the revised version.

(We report the image metrics as PNSR/SSIM/LPIPS in the table above)

---

3. Training costs of IBGS

We shows the average training time (in minutes) of our method, 3DGS and SuperGaussian across three datasets in the table below. All methods are trained using a single RTX 4090 GPU. On average, the training time of IBGS is faster than that of SuperGauss which learns per-Gaussians spatially-varying texture. However, our method requires longer training time compare to 3DGS due to additional computations for color residual predictions. We will include the discussion for the training time in the revised version.

(Training time is reported in minutes, average across all scenes within a dataset.)

---

We sincerely appreciate the reviewer for spending time reviewing our paper. The provided comments are very constructive and valuable to improve our paper. We address the reviewer's concerns as below.

---

1. Reduced rendering speed

While IBGS introduces additional computations for predicting color residuals, resulting in slower rendering speed compared to 3DGS (as shown in Tab. 3 and Tab. 6, supp. mat.), it consistently maintains rendering speeds above 30 FPS in most cases. This demonstrates that our method can still achieve real-time rendering, making it suitable for online applications. Furthermore, as demonstrated in Tab. 6 (supp. mat.), predicting the color residual at half resolution allows IBGS to nearly double its rendering speed while still achieving superior image quality compared to 3DGS and SuperGaussians. To the best of our knowledge, we are the first image-based Gaussian splatting method capable of producing photorealistic images while preserving the fast rendering speed advantage of the Gaussian-splatting approach.

---

2. Training stability at early stage

We agree with the reviewer that the rendered depths and normal could be inaccurate especially in the early training stage and could lead to an unstable training if we apply the losses involving the rendered depth and normal. To cope with this, we deactivate the color residual module during early training stage, and only activate it when the 3D Gaussian primitives can reliably represent the scene geometry. Specifically, we train our method for a total of 30,000 iterations following the training strategy outlined below:

• First 7,000 iterations: We only optimize the 3D Gaussians using the color rendering loss $L_{rgb}$. At this stage, the geometric losses and the color residual prediction modules are deactivated by setting $\lambda_1 = \lambda_2 = 0$ (Eq. 10, main paper) and $\gamma = 1.0$ (Eq. 11, main paper).
• Starting from the 7,000th iteration: We activate the geometric losses, including the multi-view color consistency loss $L_{photo}$ and the normal consistency loss ${L}_{normal}$, to accurately model the scene geometry. This is done by setting $\lambda_1 = 0.3$ and $\lambda_2 = 0.03$.
• After the 10,000th iteration: We activate the color residual prediction modules by linearly reducing $\gamma$. In particular, $\gamma$ reaches 0.5 at the 20,000th iteration and remains constant thereafter.

In summary, the color residual prediction module is only utilized when the geometry represented by the Gaussians are sufficiently reliable, and our strategy of decaying $\gamma$ helps to avoid noisy gradient back-propagated to Gaussians during the initial training phase of the color residual prediction module.

---

3. Quantitative comparison of the reconstructed scene geometry

In Tab. 1 (supp. mat.), we provided the comparison of our method with others in reconstructing the scene geometry and novel-view synthesis. For all methods, given the rendered depth maps, TSDF fusion is applied to obtain reconstructed meshes, which are then evaluated by computing the F1 score against ground truth. The result reveals that our method achieves the second best performance in 3D scene reconstruction. Additionally, as requested by the reviewer, we provide the comparison for the rendered depth maps in the table below. Overall, our method produces the second best results in most scenes.

(We report the absolute relative error for the rendered depth maps in the table below)

While PGSR [1] achieves the lowest depth error, our method significantly outperforms PGSR in novel-view synthesis with 2.82 dB improvement in the PSNR metric (see Tab. 1, supp. mat). This shows IBGS not only obtains good geometric accuracy, but also illustrates a good balance between reconstructing the scene geometry and appearance.

Visualization of the rendered depths and normals and reconstructed meshes: In accordance with NeurIPS 2025 policies, we are unable to provide figures for the rendered depths, normals and images in the rebuttal. We thanks the reviewer for understanding and will provide the visualizations in the revised version.

---

4. Clarification on training time and memory/storage requirements

The table below shows the average training time (in minutes) for IBGS, 3DGS, and SuperGaussians across three datasets. All methods are trained on a single RTX 4090 GPU. Overall, IBGS requires more training time than 3DGS due to the additional computations involved in predicting color residuals. In contrast, compared to SuperGaussians which aim to learn per Gaussian spatially varying textures, our method requires less training time.

(We report the training time minutes, average across all scene in a dataset)

Regarding the storage requirement, we would like to clarify that IBGS requires to store all source images but will compute depth maps for neighboring source views during inference. In other words, it does not require storing depth maps for inference. In Tab. 3 and Tab. 6 (sup. mat.), we have reported (1) the total storage for storing the source images, source camera intrinsics/extrinsics, the color residual network's weights, the Gaussians and (2) rendering speed during inference. In the table below, we further show the detailed storage requirement for each dataset (in MB, average acrross all scenes within a dataset).

Additionally, we also provide a comparison with 3DGS and SuperGaussian in terms of peak GPU VRAM consumption during inference in the table below. Our approach requires pre-loading all source images into VRAM, resulting in higher peak memory usage compared to 3DGS and SuperGaussians. This design choice eliminates the overhead of on-the-fly GPU memory allocation for images, thereby accelerating rendering. However, we agree with the reviewer's observation that this strategy can become bottleneck especially in large and dense scenes. One solution to mitigate this issue is to load each image into VRAM only when it is selected as a neighboring view required for rendering. Nevertheless, this approach introduces a trade-off, as it can slow down the rendering speed of IBGS. We thank the reviewer for this insightful comment and will discuss this limitation of IBGS in the revised version.

(The VRAM memory is reported in GB)

---

5. Assumptions on Closed-loop Capture Trajectories:

IBGS does not require such assumption for the camera trajectories as it has demonstrated good performance on three datasets that do not exhibit closed-loop camera trajectories, including the Deep blending dataset (Tab. 1, main paper), Shiny dataset (Tab. 2, main paper) and DTU dataset (Tab. 2, supp. mat.). Regarding the out-of-frustum points, the predicted color residual for those points is 0 as there is no source views pass the visibility check (Eq. 15, main paper). In other words, we only use the Gaussian-rasterized color for out-of-frustum points.

---

6. Why exposure correction is only applied for TNT dataset

Following [1], we only apply the exposure correction to the TNT dataset since the camera auto-exposure setting is enabled while capturing images in this dataset, as discussed in [2]. In practice, the exposure correction module can be activated by default without concern for performance degradation, as it has minimal impact when there is no exposure variation across the training views. To validate this, we applied the exposure correction module to the MipNeRF-360 dataset and present the results below. It can be observed that the image quality remains largely unchanged when the exposure correction is applied.

---

7. Can adding more Gaussians result in more fine-grained details in the rendered images?

We agree with the reviewer that adding more Gaussians increases the capacity in modeling fine-grained details. However, it also leads to a heavier storage burden and still struggles to capture challenging view-dependent effects. In contrast, IBGS does not require a large number of Gaussians and is capable of modeling both fine-grained details and view-dependent color by leveraging information from the source images.

---

[1] Chen, Danpeng, et al. "Pgsr: Planar-based gaussian splatting for efficient and high-fidelity surface reconstruction." IEEE Transactions on Visualization and Computer Graphics (2024).

[2] Barron, Jonathan T., et al. "Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

We sincerely appreciate the reviewer for spending time reviewing our paper. The provided comments are very constructive and valuable to improve our paper. We address the reviewer's concerns as below.

---

1. Potential decrease in rendering speed

While IBGS introduces additional computations for predicting color residuals, resulting in slower rendering speed compared to 3DGS (as shown in Tab. 3 and Tab. 6, supplementary material), it consistently maintains rendering speeds above 30 FPS in most cases. This demonstrates that our method can still achieve real-time rendering, making it suitable for online applications. Furthermore, as demonstrated in Tab. 6 (supp. mat.), predicting the color residual at half resolution allows IBGS to nearly double its rendering speed while still achieving superior image quality compared to 3DGS and SuperGaussians.

---

2. Reliance on a large collection of source views

We would like to clarify that the "memory values" reported for our method in Tab. 1 and Tab. 2 (main paper) are the total storage required for both the Gaussians and the source images. The results indicate that our method uses at least 42% less storage compared to 3DGS, SuperGaussians, and TexturedGaussian. We further show the breakdown of the storage requirement of our method in the table below. The reported storage is in Megabytes (MB), averaged across all scenes within each dataset.