Epipolar-Free 3D Gaussian Splatting for Generalizable Novel View Synthesis

Weaknesses 1. Overclaim Issue

We would like to clarify any confusion regarding our use of the term "epipolar-free." This term specifically indicates our method's avoidance of epipolar sampling and cost volume techniques, which are commonly used in generalizable novel view synthesis (GNVS) tasks. While approaches like UpSRT, LEAP, and PF-LRM also do not utilize epipolar geometry, our method targets multiview GNVS tasks with precise camera poses. Thus, our approach is epipolar-free within the GNVS context but not entirely devoid of geometric information. As mentioned in the related work section, "Solving 3D Tasks using Geometry-free Methods," our focus is distinct from those methods that are entirely geometry-free, such as LEAP and PF-LRM, which do not directly apply to the GNVS tasks addressed in this paper. We will reiterate our method's contributions and intuition. The novelty of our approach lies in the fact that most current multi-view GNVS methods heavily rely on epipolar line sampling or cost volume, which struggle to provide effective priors in non-overlapping and occluded areas. Therefore, we innovatively utilize a 3D cross-view pretraining model to obtain epipolar-free 3D priors and the cross-view Gaussians Alignment module to acquire accurate Gaussian attribute-matching features. The GNVS experiments demonstrate the advantages of our epipolar-free approach over existing methods dependent on epipolar line sampling or cost volume techniques.

Weaknesses 2. Related Work

We appreciate your suggestion to include more related works on GNVS using Gaussian splatting. To the best of our ability, we have discussed 6 recent works on GNVS using Gaussian splatting, except for some studies very close to the submission date, such as GS-LRM. We will add discussions on GS-LRM and other relevant works to provide a more comprehensive context for our contributions.

Weaknesses 3. Evaluation

We appreciate your valuable feedback on our evaluation section. We would like to emphasize that our method demonstrates clear advantages over both pixelSplat and MVSplat when the overlap is smaller. Specifically, our method achieves an improvement of 0.5dB PSNR compared to pixelSplat, along with faster rendering speeds. This improvement is comparable to the PSNR gain over pixelSplat reported in the MVSplat paper on the RE10K dataset. According to the evaluation criteria in the MVSplat paper, a PSNR increase of 0.5dB is considered a relatively significant improvement.

Thank you for your thoughtful response. Please find below our detailed comments on the novelty of our work and an analysis of the performance gain.

Novelty of the Work:

• Epipolar-Free Approach to Address Overlap Challenges. We have identified that mainstream multi-view GNVS methods, which rely on epipolar geometry, may encounter significant challenges in scenarios with large viewpoint overlaps. To address this, we explored the role of cross-view pretraining models that provide 3D prior knowledge for the GNVS task, and we introduced a cross-view mutual perception mechanism to partially mitigate the limitations of current methods.

• Iterative Cross-view Gaussians Alignment Module. Beyond cross-view mutual perceptive pretraining, we have specifically introduced the Iterative Cross-view Gaussians Alignment module. We observed that relying solely on the CroCo pretraining model is insufficient to effectively address challenges such as ambiguity in GNVS, primarily due to the absence of precise local feature matching. Unlike previous generalizable NVS methods utilizing Gaussian Splatting (e.g., MVSplat), our approach circumvents the need for time-consuming and less generalizable structured feature representations, bypasses complex depth sampling processes, and enables efficient per-pixel Gaussian attribute prediction. The ablation experiments presented in Table 3 and Figure 6 demonstrate the critical importance of the Iterative Cross-view Gaussians Alignment module.

Our approach not only effectively manages scenarios where epipolar geometry may fail but also eliminates the time-consuming process of sampling along epipolar lines. This makes our method particularly efficient and effective for GNVS tasks, especially in scenarios with extensive viewpoint overlaps.

Performance Gain Analysis:

With regard to Table 2 of the manuscript, the three subsets with overlaps below 0.7, 0.6, and 0.5 each represent challenging experimental conditions where epipolar geometry becomes unreliable, and it would be inaccurate to assert that one is inherently more challenging than the others. Moreover, the performance gain across these subsets does not follow a linear progression. The reasons are as follows:

• Distribution Gap Between Subsets and Training Set. These three low-overlap subsets differ from the training set and constitute a small fraction of the test set. The majority of training data involves input viewpoints that overlap well above 0.7. In the test scenes, the ratio of scenes with overlaps greater than 0.7 to those below 0.7, 0.6, and 0.5 is 45:8:5:1. Thus, from a dataset distribution perspective, these three subsets exhibit significant variability, and the overlap size does not directly correlate with difficulty. This distribution gap explains why the performance gain remains similar across these subsets, as the model’s performance is more heavily influenced by distributional differences than by the specific overlap percentage.

• Non-linear Error Growth with Decreasing Overlap. In 3D computer vision, local matching errors can propagate and affect the global output, meaning that overall error does not necessarily increase linearly as overlap decreases. This is because incorrectly matched features might be propagated as outliers into subsequent modules (especially in modules with global awareness, such as transformers), thereby amplifying their impact on overall performance. Once overlap falls below a certain threshold, traditional methods are significantly compromised. For instance, if the feature point matching error rate is too high, SFM algorithms may fail. Consequently, scenes with overlaps below 0.7–0.5 all represent scenarios where epipolar geometry is unreliable, and there is no strictly linear relationship for such "unreliableness". Thus, the similarity in performance gains for overlaps of 0.5 and 0.7 reflects the model’s consistent capability in managing these challenging scenarios where epipolar constraints are inadequate.

Finally, thank you again for your excellent suggestions, which have helped us to think more deeply about the essence of performance improvement in our method. We will incorporate this analysis of the performance gain into the revised manuscript.

Thanks for the detailed and helpful reply!

The motivation of using epipolar-free methods makes sense to me. While this field is super competitive as there are many recent works, releasing unnecessary inductive bias on geometric designs is promising to me, especially when having large data. The competition in this field makes me a bit tired, as I can feel most of the works are a kind of incremental. I appreciate the effort to further improving Dust3r training pipeline on the GNVS task using the proposed epipolar-free method and the iterative cross-view gaussian alignment method.

Regarding the performance analysis, maybe the reason for my previous question is the data distribution (45:8.5:1 as mentioned). The data amount with small overlap ratio, e.g. 0.5, can be too less, so the numbers are also noisy. I sincerely suggest the authors to find some other data that are more balanced to perform the analysis, e.g. DTU or DL3DV, where you can control the overlap ratio as the two datasets provide dense views. I won't request this experiment as the discussion period is coming to the end soon.

At the same time, we still want to let the authors know I expect the prior works can be correctly discussed, especially removing some controversial words like first "the first epipolar-free GNVS method". After the discussion, I don't want to reject this paper because of the overclaiming problem as I saw the effort behind this paper.

Thus, I would like to raise the score to a neutral borderline (with a score of 4.5, due to the uncertainty of how the over-claiming problem will be resolved) and I don't against accepting this paper. As there is no neutral borderline for the reviewer, I will just kept the current rating now, but I will let AC know if I keep the a neutral borderline until the end of discussion.

At the same time, if you have better wording for resolving the controversial claims, please let me know before the discussion period ends. If the revised version is satisfying to me, I will raise the score to a borderline accept.

Q1. The most influential module in Table 3

The results in Table 3 show that the absence of the epipolar-free cross-view mutual perception results in the most significant performance decline. This highlights the crucial role of the pretraining and network structure of Croco in providing 3D priors and cross-view features. Additionally, as illustrated in Fig. 6, the exclusion of this module leads to noticeable artifacts in both novel view images and depth maps, underscoring its importance. We will add this analysis to the ablation study section in the paper.

Q2. Supplementary analysis of the Iterative Cross-view Gaussians Alignment module

While the iterative nature of the Cross-view Gaussians Alignment module does slightly reduce the rendering speed during training or inference, it substantially enhances the reconstruction metrics. The supplementary experiments with iterations set to 1-3, as presented in the table below, demonstrate that iterating twice strikes a balance between effectiveness and efficiency, providing significant improvements without excessively increasing computational demands. For details, please see Table 2 and Figure 2 in the Rebuttal PDF.

Q3. Whether Cross-view Gaussians Alignment required during inference

Yes, cross-view Gaussians Alignment is required and plays a crucial role during inference, and it does not require much time (less than 0.061s, as reported in Tab 1 of the manuscript). The scene representation is obtained from input perspective images through Epipolar-free Cross-view Mutual Perception and Cross-view Gaussians Alignment, which matches the scene representation trained by the single-scene optimized 3DGS method, allowing for direct inference of new viewpoints. As shown in Fig. 2 of the manuscript, the pipeline of eFreeSplat remains the same for both the training and testing sets: it first uses Croco-pretrained ViT to provide feature maps with 3D priors, then employs the Cross-view Gaussians Alignment module to acquire pixel-wise 3D Gaussian primitives for the 3D scene representation, and finally, the inference is performed using the original 3DGS tile-based rendering method.

Weaknesses 1. Performances regarding cross-dataset generation

Thank you for your valuable suggestions on our work. We fully agree with your view on the importance of cross-dataset testing and, following your advice, have added cross-dataset test results on the DTU dataset. For specific experimental results, please see Table 1 and Figure 1 in the Rebuttal PDF.

These tests further validate eFreeSplat's generalization ability across different data distributions. The results demonstrate that even though training was conducted on the RE10K dataset, eFreeSplat performs excellently on the DTU dataset, which further confirms the robustness and generalization performance of our method.

Weaknesses 2. Supplementary analysis of the Iterative Cross-view Gaussians Alignment module

Thank you for your valuable suggestions. We have added ablation experiments for iteration counts of 1-3 and presented the corresponding depth maps based on your feedback. For specific experimental results, please see the Table 2 and Figure 2 in the Rebuttal PDF.

The results indicate that setting the iteration count to 2 achieves the best balance between reconstruction accuracy and rendering efficiency. When the iteration count is set to 3, there is no significant improvement in image reconstruction metrics, which may be due to training overfitting and the subsequent iterations losing the 3D perception provided by the original CroCo features.

Weaknesses 3. Fine-tuning of the CroCo model

Thank you for your valuable suggestions. We conducted relevant Experiments A and B regarding fine-tuning the CroCo model using the RE10K dataset. Experiment A involved fine-tuning the CroCo pretrained weights with the RE10K training set, while Experiment B involved training CroCo directly with the RE10K training set without loading the pretrained weights. Finally, we retrained eFreeSplat using the new pretrained weights. The CroCo pretraining for Experiments A and B was performed on 2 RTX 4090 GPUs, with total iterations of 4000 and 6000, respectively (due to time constraints), learning rates of 2e-5 and 2e-4, and a batch size of 12. The viewpoint overlap was set the same as in the main model training. For the quantitative and qualitative results of Experiments A and B, please see Table 3 and Figure 3 in Rebuttal PDF.

The results indicate that pretraining the backbone model on the RE10K training set effectively addresses the model's poor performance in low-overlap scenarios. However, in the RE10K test set, Experiment A's reconstruction metrics were slightly lower than those of the original model, which may be due to insufficient training iterations. We will further investigate the positive impact of fine-tuning the CroCo pretrained model on novel view synthesis and 3D reconstruction in future work.

Thanks so much again for the time and effort in our work. May I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Weaknesses 1. Missing experiments in a sparse-view setting and comparison with other sparse-view methods

We thank you for identifying the novelty of our idea. Nevertheless, maybe we did not illustrate the difference between GNVS and sparse-view NVS clear enough and make you miss the focus of our experiments. Please allow us to reiterate the concept of the generalizable novel view synthesis (GNVS) task: The GNVS task aims to render new viewpoint images by leveraging the generalization ability of cross-scene 3D representations. When encountering new scenes not present in the training set, this process requires no additional single-scene optimization, achieving rendering through a single network feed-forward pass. Therefore, sparse-view novel view synthesis and GNVS are distinct tasks. The methods and datasets we compare in our experiments are commonly used settings in the GNVS task. Moreover, eFreeSplat is theoretically straightforward to extend to multi-view inputs. Our choice of datasets and dual-view input settings is solely for fairer comparisons with current GNVS methods. For instance, recent works such as pixelSplat and MVSplat also use dual-view input in their experimental settings, with their datasets being RE10K and ACID. Through fair experiments, we have validated our method's generalization capability and its advantages in reconstructing challenging regions.

Weaknesses 2. Differences between Eq. (5) and cost volume construction method in MVSNet

Here, we provide a detailed explanation of the differences between Equation (5) and the coarse-to-fine cost volume construction method in MVSNet.

Equation (5) represents a commonly used warped transformation formula in multi-view geometry. Unlike MVSNet, which requires sampling along the canonical view ray N times to obtain N depth values $$ d_{i} $_{i=1}^{N}$ when constructing the cost volume, our method does not require multiple depth direction samples. Specifically, in each iteration, our method directly predicts the coarse depth value $d$ using a U-Net network (as detailed in Equation (4)), rather than sampling each depth plane to calculate the cost volume as MVSNet does.

The intuition behind this approach stems from the 3D prior knowledge provided by the CroCo pretrained model, which replaces the multiple depth sampling process required in the MVSNet cost volume. By leveraging CroCo's 3D priors, we can effectively obtain multi-view feature point matching relationships in each iteration, simplifying computation and improving efficiency.

Q1. Missing definition

Thank you very much for your thorough review and valuable suggestions. We sincerely apologize for the omission of the definition of $C$ in Equation (7). Your assumption is indeed correct: $C$ represents the feature dimension of the 3D Gaussian primitives. We will include a clear definition of $C$ in the revised manuscript.

Q2. More experiments on 512 x 512 resolution

Thank you for your valuable feedback. The baseline methods use the RE10K and ACID datasets, which have a resolution of 360 x 640, lower than 512 x 512. For a fair comparison, we did not conduct experiments at other resolutions that were different from the baselines and other popular methods.

According to our research, current 3DGS-based GNVS methods all use fixed, lower-resolution datasets for testing. Although high resolution and variable resolution are not the focus of this study, they could indeed make models more applicable in real-world scenarios, so we have included them in our future work plans. Thank you again for your suggestion.

Q3. The memory requirements

Thank you for your valuable suggestion. We have added ablation experiment results for iteration counts of 1-3, including comparisons of memory usage, rendering speed, and reconstruction metrics. For details, please see Table 2 and Figure 2 in the Rebuttal PDF.

The experiments show that while our method's memory usage varies with different iteration counts, the overall memory demand remains relatively low. Compared to DUSt3R, our method requires less memory and achieves better rendering efficiency and reconstruction accuracy with two iterations.