FreeSplat: Generalizable 3D Gaussian Splatting Towards Free View Synthesis of Indoor Scenes

To Reviewer ZuRj (#R4):

1. Computational Cost: The full training time is around 2 days for our 2-view and 3-view versions of our method and baselines, and 3 days for our free-view version. Although our training time is similar to pixelSplat [1] and MVSplat [2], we consume much fewer GPU hours due to our lower GPU requirements. As shown in our rebuttal pdf Table 1, the 3-view version of pixelSplat and MVSplat already consume 30~50 GB for training. It is due to their heavy patch-based 2D transformer and cross-view attention that results in quadratically increasing GPU requirements with increased image resolution and input sequence length. Therefore, their models are not suitable for high-resolution, long sequence inputs. In contrast, our proposed low-cost backbone can greatly reduce GPU consumption given a long sequence of high-resolution inputs, making whole scene reconstruction much more feasible. Furthermore, our proposed PTF module can effectively reduce redundant 3D Gaussians, improving the rendering speed from 39 to 72 FPS in whole scene reconstruction, which becomes even more important when reconstructing larger scenes to achieve real-time rendering. Overall, the efficiency of our model mainly includes: (1) lowering required GPU to make it feasible to train on long sequences of high-resolution inputs, and also reducing training cost (required GPU hours); (2) removing redundant 3D Gaussians to increase rendering speed and handle large 3D scenes.

2. Comparison with SurfelNeRF: As shown in our rebuttal pdf Table 3 and Figure 4, we conduct a comparison experiment with SurfelNeRF. We evaluate our FreeSplat-fv on the same novel views as theirs. Note that their input views are much denser than ours: e.g. they input 40 views covering a length of 120 images, with an approximate image interval of 3; while we still use relatively sparse inputs that have a fixed image interval of 20. We pick input views along their image sequences and evaluate at the same novel views. Note that the number of input views changes with the length of image sequence, while our FreeSplat-fv can perfectly fit to such varying numbers of input views. We outperform them largely in both rendering quality and efficiency. We analyze that such improvements are creditable to two main reasons:

   (1) Their surfel-based representation is less representative comparing to 3DGS;

   (2) Their model is not trained end-to-end with MVS encoder. SurfelNeRF uses external GT depth maps or MVS depth maps and do not jointly train an MVS-based encoder with the surfel-predictor, so their surfel-predictor does not have multi-view information, resulting in sub-optimal rendering quality.

3. Definition of FreeSplat-spec and FreeSplat-fv: You are correct that FreeSplat-spec is trained using fixed number of input views for a fair comparison with the baselines FreeSplat-fv uses Free-View Training to reconstruct larger regions for stricter supervision from broader view range, to enforce precise 3D Gaussian localization. We will add the above illustration in the final version to improve the clarity.

4. Comparison with 3DGS-based SLAM methods: Comparing to 3DGS-based SLAM methods, our proposed generalizable 3DGS-based method has the following two main strengths:

   (1) Efficiency: Although the existing 3DGS-based SLAM can reach 1~3 FPS for per-scene optimization, they still require 10~20 minutes to finish reconstructing the whole 3D scene (cf. MonoGS [3] paper Table 9, 10; Splatam [4] paper Table 6). In contrast, our method only requires 1~2 seconds to parallelly mapping all the input views to 3D Gaussians. Furthermore, our proposed PTF can largely improve the rendering FPS (cf. our rebuttal pdf Table 1), making the scene-level generalizable 3DGS more feasible. After fast feeding-forward, we can optionally conduct fast per-scene optimization using our predicted 3D Gaussians as the initalization to further improve the rendering quality.

   (2) Less constraints on Input Data: Majority of the existing SLAM methods require dense input RGB-D sequence to estimate camera trajectories on-the-fly. Although several works [3, 5] can handle monocular inputs, there is still a gap between their camera tracking accuracy and RGB-D methods (cf. MonoGS paper Table 1). Such on-the-fly methods may also suffer from the drift problem due to accumulated errors. In contrast, our method is designed to explore learning strong priors from training data and perform fast offline prediction instead of on-the-fly application, making it more compatible with COLMAP and recent feed-forward pose estimation methods like DUSt3R [6]. Furthermore, our method does not require sensor depths during training or inference, and only requires relatively sparse inputs, which can be more easily generalized to various domains.

[1] Charatan, David, et al. "pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Chen, Yuedong, et al. "Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images." arXiv preprint arXiv:2403.14627 (2024)."

[3] Matsuki, Hidenobu, et al. "Gaussian splatting slam." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Keetha, Nikhil, et al. "SplaTAM: Splat Track & Map 3D Gaussians for Dense RGB-D SLAM." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Teed, Zachary, and Jia Deng. "Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras." Advances in neural information processing systems 34 (2021): 16558-16569.

[6] Wang, Shuzhe, et al. "Dust3r: Geometric 3d vision made easy." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

In my previous rebuttal on the comparison with SurfelNeRF, we erroneously referred to rebuttal pdf Table 3 and Figure 4 when discussing the experimental results. We would like to clarify that the correct reference should be to rebuttal pdf Table 4 and Figure 5. We apologize for any confusion this may have caused and appreciate the opportunity to correct this error.

Thank you for your insightful feedback. To provide a more thorough comparison with 3DGS-based SLAM methods utilizing monocular inputs, we reproduced the monocular version of MonoGS (referred to as MonoGS-Mono) and conducted experiments on the scene0316_00 from the ScanNet dataset. We compared these results against our whole scene reconstruction approach. To ensure a fair comparison, we present their results both with and without ground truth (GT) camera poses. The results are summarized in the table below:

The results demonstrate that, on the real-world ScanNet dataset, MonoGS struggles to accurately track the camera trajectory and reconstruct a geometrically correct 3D scene with only color images as the input. The predicted camera trajectory suffers from significant deviation due to the drift problem and the lack of depth priors / sensor depth inputs. Interestingly, providing ground truth camera poses results in a notable performance drop, which we attribute to the difficulty of fitting reconstructed 3D Gaussians to the training frames without pose optimization. In contrast, our method effectively learns geometry priors from the training data and performs significantly faster feed-forward predictions. Our approach also significantly outperforms MonoGS-Mono in rendering quality, particularly in terms of PSNR and LPIPS metrics. These results demonstrate the advantages of our generalizable 3DGS-based method for whole scene reconstruction when using monocular color inputs, offering both effectiveness and efficiency.

We sincerely appreciate your comments, which prompted us to perform this comparison with 3DGS-based SLAM methods using monocular inputs. This experiment further solidifies our belief in the unique strengths of our approach as a foundational step toward feed-forward 3DGS-based whole scene reconstruction. We hope the above experiment and analysis can adequately address your concerns about the comparison with 3DGS-based SLAM methods.

To Reviewer Hd37 (#R3):

1. Evaluation Details: For view range, as mentioned in Line #431-432 in the appendix, the distance between nearby input views is fixed to 20 in ScanNet and 10 in Replica, thus the maximum gap increases linearly with the number of input views. Therefore, the long sequence reconstruction covers larger regions of the scene and supports whole scene reconstruction. During evaluation, we randomly select a sequence of images with specific length (2, 3, 10) and fixed interval between nearby views, e.g. we choose the 30, 50, 70-th images of a ScanNet scene as input views for the 3-view setting. When evaluating interpolation results, we randomly choose novel views between each interval of nearby views, and for extrapolation we choose novel views that are beyond the input image sequence as illustrated in Line #226-227 in our main paper. We will include the above illustrations in the final version to describe the evaluation details more clearly.

2. Decreased image quality when the number of input views increases: It is because of the increased difficulty when explicitly reconstructing longer sequences. Similarly as existing works, when evaluating interpolation results we select novel views that are within the view interval between nearby views. When using 2 or 3 input views, and the reconstructed 3D Gaussians only needs to give high-quality renderings from a relatively narrow view range, which means that they may still give good renderings even if the 3D Gaussians are erroneously located. However, when increasing the number of input views to 10, the reconstructed region needs to give reasonable renderings from a broader view range, where the distant novel views can serve as extrapolation views, thus evaluating the precision of Gaussian localisation. For example, given 10 input views of the 10, 30, 50, 70, 90, 110, 130, 150, 170, 190-th images of the scene, we render novel view from the pose of the 185-th image, then it may view at the same region as in the 10-th and 30-th images but from a significantly differently view direction. Such a case requires the reconstructed 3D Gaussians to be precisely localized, otherwise they may become floaters when viewing from extrapolated views. On the other hand, the depth map quality only evaluates the localization accuracy of 3D Gaussians, which becomes more precise when given more reference views and training on long sequences. Overall, training the model on long sequences is important for precise 3D Gaussian localization since the distant novel views can serve as extrapolated views to regularize the depth estimation, which is one of the key focuses of our paper.

3. Experiments on Re10k and ACID: To further evaluate our model's generalization ability across diverse domains, we train our model on RE10K using 2-View setting and 5-View setting respectively. The results are shown in our rebuttal pdf Table 2, 3 and Figure 3. Note that for the 5-View setting inference, we sample input views with random intervals between 25 and 45 due to the limited sequence lengths in RE10K and ACID. In the 2-View setting, we perform better than pixelSplat [1] and on par as MVSplat [2] on both datasets. In the 5-View setting, we outperform both baselines by a clear margin. We analyze the main causes of the above results as follows:

   In the 2-view comparison experiments with the baselines, the image interval between the given stereo images were set to be large. On average, the interval between image stereo is 66 in RE10K and 74 in ACID, which is much larger than our indoor datasets setting (20 for ScanNet and 10 for Replica). Such large interval can result in minimum view overlap between the image stereo (e.g. as shown in our rebuttal pdf Figure 4(b)), which means that our cost volume can be much sparser and multi-view information aggregation is weakened. In contrast, MVSplat uses a cross-view attention that aggregates multi-view features through a sliding window which does not leverage camera poses. pixelSplat uses a heavy 2D backbone that can potentially become stronger monocular depth estimator. In our 5-view setting, we outperform both baselines by clear margins. This is partially due to the smaller image interval and larger view overlap between nearby views. As a result, our cost volume can effectively aggregate multi-view information, and our PTF module can perform point-level fusion and remove those redundant 3D Gaussians.

   Therefore, our model is not specifically designed for highly sparse view inputs, but it is designed as a low-cost model that can easily take in much longer sequences of higher-resolution inputs, that is suitable for indoor scene reconstruction (we also offer a quantitative comparison on computation cost in our rebuttal Table 1 to emphasize our strengths). Comparing to RE10K and ACID, real-world indoor scene sequences usually contain more complicated camera rotations and translations, which results in the requirement of more dense observations to reconstruct the 3D scenes with high completeness and accurate geometry. Consequently, our model is targeting the fast indoor scene reconstruction with keyframe inputs, which contain long sequences of high-resolution images, while existing works struggle to extend to such setting as evaluated in our main paper.

   We really appreciate your question, which helped us dive deeper into broader experimental comparisons and explore the underlying reasons, reaching a better clarification of our research focus and contributions. We will add the corresponding results and illustrations in final version to improve the completeness and highlight the contributions of our paper.

[1] Charatan, David, et al. "pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction." CVPR. 2024.

[2] Chen, Yuedong, et al. "Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images." arXiv preprint arXiv:2403.14627 (2024)."

Thank you for your insightful question. To evaluate the effect of the number of input views when fixing the maximum view length, we conduct an experiment on ScanNet as follows: For each test scene, we first randomly pick the first frame (denoted as the x-th frame of the scene). Next, we set up three different view intervals (10, 20, 40) and fix the maximum view interval as 80, i.e. the chosen input views are [x, x+10, x+20, x+30, x+40, x+50, x+60, x+70, x+80] for view interval of 10, [x, x+20, x+40, x+60, x+80] for view interval of 20, and [x, x+40, x+80] for view interval of 40. For the target views, we select one target view between each interval: [x, x+10], [x+10, x+20], [x+20, x+30], ..., [x+70, x+80], and evaluate such target views for all the above settings to form a fair comparison. We evaluate the performance of our FreeSplat-fv as shown in the following Table:

where D is the interval between nearby input views. The results indicate that when fixing the maximum view length, more input views can lead to better rendering quality and more accurate geometry. Furthermore, when encoding denser input views, our PTF module becomes more important in removing increasingly redundant Gaussians. This experiment is a very good example evaluating that more accurate geometry leads to better rendering quality, and the increasingly important role of our PTF module when given denser inputs. Note that when setting D=10 with PTF (row #2), we give fewer Gaussians and +1.34dB PSNR comparing to D=20 without PTF (row #3), which clearly demonstrates the necessity of PTF in benefiting from denser inputs while removing those redundant Gaussians. On the other hand, the significantly decreased results when setting D=40 also evaluate that inputting sparser input views similarly as RE10K or ACID for indoor scenes leads to unsatisfactory results. We hope the above experiment and analysis can address your question.

To Reviewer emYN (#R2):

1. Experiments on Re10k and ACID: To further evaluate our model's generalization ability across diverse domains, we train our model on RE10K using 2-View setting and 5-View setting respectively. The results are shown in our rebuttal pdf Table 2, 3 and Figure 3. Note that for the 5-View setting inference, we sample input views with random intervals between 25 and 45 due to the limited sequence lengths in RE10K and ACID. In the 2-View setting, we perform better than pixelSplat [1] and on par as MVSplat [2] on both datasets. In the 5-View setting, we outperform both baselines by a clear margin. We analyze the main causes of the above results as follows:

   In the 2-view comparison experiments with the baselines, the image interval between the given stereo images were set to be large. On average, the interval between image stereo is 66 in RE10K and 74 in ACID, which is much larger than our indoor datasets setting (20 for ScanNet and 10 for Replica). Such large interval can result in minimum view overlap between the image stereo (e.g. as shown in our rebuttal pdf Figure 4(b)), which means that our cost volume can be much sparser and multi-view information aggregation is weakened. In contrast, MVSplat uses a cross-view attention that aggregates multi-view features through a sliding window which does not leverage camera poses. pixelSplat uses a heavy 2D backbone that can potentially become stronger monocular depth estimator. In our 5-view setting, we outperform both baselines by clear margins. This is partially due to the smaller image interval and larger view overlap between nearby views. As a result, our cost volume can effectively aggregate multi-view information, and our PTF module can perform point-level fusion and remove those redundant 3D Gaussians.

   Therefore, our model is not specifically designed for highly sparse view inputs, but it is designed as a low-cost model that can easily take in much longer sequences of higher-resolution inputs, that is suitable for indoor scene reconstruction (we also offer a quantitative comparison on computation cost in our rebuttal Table 1 to emphasize our strengths). Comparing to RE10K and ACID, real-world indoor scene sequences usually contain more complicated camera rotations and translations, which results in the requirement of more dense observations to reconstruct the 3D scenes with high completeness and accurate geometry. Consequently, our model is targeting the fast indoor scene reconstruction with keyframe inputs, which contain long sequences of high-resolution images, while existing works struggle to extend to such setting as evaluated in our main paper.

   We really appreciate your question, which also helped us delve into the comparison on a larger experimental scope and analyze the reasons behind the results. We hope that our analysis can help demonstrate our contributions more clearly. We will add the corresponding results and the above illustrations in our final version paper to improve the completeness of our paper and highlight the contributions of our paper comparing to existing works.

2. Failure cases and potential future works: As shown in our rebuttal pdf Figure 3, we visualize the whole scene reconstruction results highlighting (a) the errorenously estimated depth for the specular/texture-less regions, and (b) Difficulty of accurate depth estimation when the given input stereo has extremely large interval. Such errors are mainly due to the following aspects:

   (1) Lack of depth regularizations. The nature of our color-supervised depth estimation methods makes it difficult to accurately estimate depth for such regions. One potential solution is to leverage depth supervision / priors, e.g. regularize the depth estimation results using GT depth / sparse depth from COLMAP / monocular depth estimation methods. Future works can also explore the addition of geometric constraints on Gaussians localisation (eg. depth smoothness regularization), or multi-view consistency regularizations to enhance the 3D Gaussian localization.

   (2) Straightforward 3D Gaussian fusion method. Although we have proposed the PTF module to reduce redundant 3D Gaussians and improve the depth estimation performance, it is still not enough to reach satisfactory multi-view fusion results. Future works can learn from TSDF fusion methods [3] and enhance the unprojection of 3D Gaussians, e.g. given the initial 3D Gaussians from the first frame, search for its projections on all the remaining frames, and unproject them together.

   (3) Ineffectivity of MVS encoder when faced minimum view overlap between inputs: When the given inputs only have minimum view overlap, our MVS can hardly find correspondences on the reference view, which results in highly sparse cost volume and ineffectivity of multi-view feature aggregation. It would be beneficial to leverage a cross-view attention (e.g. as in MVSplat) which can work despite minimum view overlap. Although our transformer-free backbone may underperform in such extreme cases, it was specifically designed for relatively dense input of long sequences. In the main paper, we have evaluated that our low-cost backbone is essential for accurate indoor scene reconstruction.

   We will add the above illustrations in our final version to delve deeper into the analysis of failure cases and discuss potential future works that can be explored upon our method.

[1] Charatan, David, et al. "pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction." CVPR. 2024.

[2] Chen, Yuedong, et al. "Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images." arXiv preprint arXiv:2403.14627 (2024)."

[3] Choe, Jaesung, et al. "Volumefusion: Deep depth fusion for 3d scene reconstruction." CVPR. 2021.

To Reviewer gTxT (#R1):

1. Differences from existing MVS methods: Compared to traditional MVS methods [1,2], the main difference of our backbone lies in the unsupervised scheme of depth estimation supervised purely by color images, while reaching comparable depth estimation accuracy. Due to the unsupervised nature of our method, the pixel-aligned depth values are predicted through weighted summing along the candidate depth planes instead of predicting absolute depth values, in order to bound the depth values within a reasonable range. Compared to existing MVS-based NeRF methods [3, 4], our backbone design avoids expensive 3D CNN / ray transformer, and our explicit 3DGS reconstruction can greatly benefit from real-time renderings. Compared to MVS-based 3DGS methods [5, 6], our main contributions lie on designing a transformer-free 2D backbone to encode extended input sequences of high-resolution images, and adaptively formulate cost volumes within nearby views based on pose affinities. More importantly, we are the first to explore explicit long sequence reconstruction based on generalizable 3DGS. Both pixelSplat and MVSplat struggle to extend to long sequences of high-resolution inputs due to their patch-wise 2D Transformers / cross-view Transformers, which consume quadratically increasing GPU memory with image sequence length and resolution. On the other hand, our low-cost 2D backbone is designed to easily encode such indoor scene sequences. Therefore, although our 2D backbone bears high-level design similarity to existing MVS methods, the differences in our design are specifically targeting long sequence reconstruction with precise unsupervised depth estimation which we demonstrated to be significant for our goal.
2. Differences between FreeSplat-spec and FreeSplat-fv: FreeSplat-spec is trained with fixed number of input views (eg. 2, 3), and FreeSplat-fv is trained using our free-view training strategy using 2~8 input views, in order to extend our method to arbitrary length of inputs. We will include the above explanation in the final version.
3. Why providing 3-view version of baselines in Table 2: The main reason is the enormous GPU consumption of pixelSplat and MVSplat when inputting more views. As shown in our rebuttal pdf Table 1, pixelSplat and MVSplat already require 30~50 GB GPU when training with 3 input views and batch size of 1 due to their heavy patch-based 2D Transformers and cross-view attention. Their GPU requirements quadratically increase with respect to the inputs resolution and length, thus making training their 10-views version nearly infeasible. In contrast, our model can train on 8 input views while requiring a smaller GPU memory compared to 3 views version of pixelsplat and MVSplat, supporting training and inference on long sequences of high-resolution inputs. Therefore, in our main paper Table 2, we report pixelSplat and MVSplat's 10-views inference results using their 3-views trained models due to the infeasibility of training their models using more views. To form a fair comparison, we also compare our 3-view version on 10-view inference, where we consistently outperform the 3-views baselines. On the other hand, the further improvements brought by our FreeSplat-fv over our 3-view version are creditable to our low-cost backbone design.
4. Visualization of PTF for removing redundant 3D Gaussians: To illustrate our proposed PTF more intuitively, we draw an illustration figure in our rebuttal pdf Figure 1, and visualize the fusion process as shown in our rebuttal pdf Figure 2. Our proposed PTF module can greatly remove the redundant 3D Gaussians that lie very closely to the existing ones, such that we avoid redundant unprojection of 3D Guassians to the regions that are observed multiple times. PTF can also fuse the redundant 3D Gaussians latent features to aggregate multi-view observations at point-level, alleviating artifacts caused by lightning conditions, etc. Furthermore, as shown in our rebuttal pdf Table 1, applying PTF module can increase the rendering speed from 39 to 72 FPS during whole scene reconstruction, which becomes more essential when the 3D scene is larger with more input views. Since generalizable 3D Gaussians methods normally unproject pixel-wise 3D Gaussians for each input view that can easily result in redundancy, removing the redundant ones become more important to achieve real-time rendering.

[1] Yao, Yao, et al. "Mvsnet: Depth inference for unstructured multi-view stereo." Proceedings of the European conference on computer vision (ECCV). 2018.

[2] Sayed, Mohamed, et al. "Simplerecon: 3d reconstruction without 3d convolutions." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[3] Chen, Anpei, et al. "Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[4] Wang, Qianqian, et al. "Ibrnet: Learning multi-view image-based rendering." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[5] Charatan, David, et al. "pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[6] Chen, Yuedong, et al. "Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images." arXiv preprint arXiv:2403.14627 (2024)."

Dear reviewers,

As the reviewer-author discussion period is about to end by 8.13, please take a look at other reviewers' reviews and authors' rebuttal at your earliest convenience. It would be great if you could ask authors for more clarification or explanation if some of your concerns are not addressed by the rebuttal.

Thanks,

AC