Structure Consistent Gaussian Splatting with Matching Prior for Few-shot Novel View Synthesis

We sincerely thank reviewer #42ua for recognizing our work and the valuable comments. Here, we will address the concerns point by point.

Q: It heavily relies on precomputed camera poses. When using COLMAP to obtain camera poses, SFM points are generated effortlessly as a byproduct.

• As the common setting of the few-shot NVS field, our target scene is the applications which have off-the-shuffle camera poses but only have sparse cameras, e.g., the poses between the sparse cameras in a driving car is known from the sensor or pre-computed (Not from COLMAP).
• In these sparse scenarios, the traditional COLMAP method is hard to extract SFM points, and the SFM points in existing public datasets (e.g., LLFF dataset) are generated from dense views. Therefore, for a fair comparison with NeRF methods, we don't use these SFM points for initialization.

Q: Comparisons with the direct triangulation.

As suggested, we further conducted comparisons with methods that directly initialize existing methods with the triangulation point clouds, in Tab. 3 and Fig. 2 of our uploaded PDF. Besides the 3DGS, we also test on the more advanced methods like ScaffoldGS [2] and OctreeGS [3].

• The results indicate that initializing with the triangulation points indeed improve the performance of existing methods. However, this improvement only comes from the better initialization and this strategy cannot mitigate the impact of the wrong matching priors. And the initialized Gaussian primitives face the same non-structure problem like the vanilla 3DGS.
• Our model can achieve more powerful performance because of our further designs on hybird representation and dual optimization, whose effectiveness has been demonstrated in Tab. 1 of our uploaded PDF.
• For a fair comparison, we train these models for more iterations. On a single RTX 3090 GPU, the optimization of the model based on 3DGS, ScaffoldGS and OctreeGS takes about 5m, 10m, and 13m, respectively. Their inference speed is about 220FPS, 160FPS and 135FPS, respectively. And as declared in the Efficiency section (from Line 292 to Line 296), our model runs at about 200FPS, converges in 1 minutes and still achieves the best performance.

Q: More explanation of our novelty.

• To our knowledge, we are the first method to attempt to integrate matching priors to the few-shot 3DGS and successfully mitigate the degradation in sparse scenarios.
• We propose a hybrid Gaussian representation consisting of ray-based Gaussian and the vanilla non-structure Gaussian, which restrics the Gaussian primitive move on the ray, making the Gaussian primitive more controllable and simplifing the model convergence space. This simplification is beneficial for few-shot NVS tasks, since the complex model tends to overfit, as demonstrated by DietNeRF [4] and FreeNeRF [5].
• We bind the ray-based Gaussian primitive to the matched ray and enable the direct optimization of the position of the Gaussian primitive besides the optimization of the rendering geometry. Combined with the ordinary photometric loss, we can properly alleviate the optimization ambiguity between the position and size of Gaussian primitives in the few-shot scenario.
• The ablation studies in Table 1 of our uploaded PDF demonstrate the effectiveness of each contribution.

Q: More experiments on Waymo dataset.

As suggested, we further conducted experiments on larger scenes. But since the full Waymo dataset is too large, we conducted comparisons with DNGaussian [6] on the sample scene provided by UC-NeRF [1], due to the time limit of rebuttal. Specifically, we took the middle 20 frames for experiments and trained the model with 5 input views at the resolution of 4x downsampling.

The quantitative comparisons are shown in the table below. Our method has a clear performance gain over the previous method DNGaussian. We show some visualization results in Fig. 3 of our uploaded PDF, from which we can see that our method can recover more details, including the texture of the road.

Q: More details of the matching prior.

We use the GIM model in our paper, and we use the default hyperparameters of GIM. In the setting of three training views, the matching model can extract about 5k pairs per image pairs and takes about 5 seconds per scene.

Q: The inclusion of geometric metrics, such as Chamfer Distance.

Thanks for this suggestion, and we have tried to conduct such evaluation on the DTU dataset. The results show that our $Thre_{10}$ metric (Percentage of pixels with error more than 10mm) of rendered depth is about 30% and NDGaussian is more than 99%. We speculate that there are some errors and mismatches between the camera pose and the ground-true depth. We will try to conduct the experiment on other suitable synthetic datasets.

And as the common practice of existing few-shot methods, we reported the qualitative comparisons of the rendering depth in Fig. 1 of our main paper, where our method performs much better in geometric accuracy, showing its superiority in geometry.

Reference

[1] UC-NeRF: Neural Radiance Field for Under-Calibrated Multi-view Cameras in Autonomous Driving, ICLR2024.

[2] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering, CVPR2024.

[3] Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians, arxiv2024.

[4] Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis, ICCV2021.

[5] FreeNeRF: Improving Few-Shot Neural Rendering With Free Frequency Regularization, CVPR2023.

[6] DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization, CVPR2024.

We sincerely thank the reviewer #42ua for recognizing our work and the valuable comments. As suggested, we conduct more ablation studies with different types of matching models and add more discussion of the failure cases.

Q: A discussion / ablation study on the pre-trained matching model (matching priors).

To study the effect of the matching priors to our model, in Tab. 2 of our uploaded PDF, besides the GIM prior (ICLR2024) [1] we used in our paper, we further conducted experiments with three more matching models, including SuperGlue (CVPR2020) [2], LoFTR (CVPR2021) [3] and DKM (CVPR2023) [4].

• The quality of the matching prior can indeed affect the performance of the model, but the results show that all these four kinds of matching priors can bring satisfactory improvement to the baseline. And this proved our designs are robust to different matching models.
• It's worth noting that our model has the cache and filter strategy, which can mitigate the negative effect of the wrong matching prior. Even in the worst case where there is no exact matching prior, our model can still achieve better performance than the baseline model, when using our hybrid representation, which has been proved in the results of Tab. 1 of our uploaded PDF ("+ Hybrid Rep." vs. "3DGS (Baseline)").

Q: More discussion of the failure cases.

We appreciate the great suggestion! We summarize some failure cases as follows:

• Our method is a reconstruction model, which requires the multi-view coverage. Therefore, it struggles to reconstruct the accurate scene structure when there is severe occlusion between views. And for this kind of scene, leveraging priors in generative models may be a feasible solution.
• The texture-less background is also a challenging problem for our model. This is an ill-posed problem, and we found that the implicit NeRF methods are much better in these regions because of their good smoothness. To solve this, we argue that more effective geometric supervision or a more generalizable model trained in large datasets is necessary.

Reference

[1] GIM: Learning Generalizable Image Matcher From Internet Videos, ICLR2024.

[2] SuperGlue: Learning feature matching with graph neural networks, CVPR2020.

[3] Loftr: Detector-free local feature matching with transformers, CVPR2021.

[4] Dkm: Dense kernelized feature matching for geometry estimation, CVPR2023.

We sincerely thank reviewer #k97m for recognizing our work and the valuable comments. We noticed that the main concerns are the performance of our designs and the robustness of different matching models and texture-poor scenes. Here, we will explain the specific questions individually:

Q: The performance and necessity of our designs. (Including the explanation of the second point of Weaknesses)

Our contribution is to exploit the matching prior to construct a structure consistent Gaussian splatting method. While leveraging matching prior to solve the challenge of few-shot 3DGS is one of our important contributions, our other main designs are in two folds:

• we propose a hybrid Gaussian representation consisting of ray-based Gaussian and the vanilla non-structure Gaussian (Hybrid Rep.), which makes the Gaussian primitive more controllable and simplifies the model convergence space through restricting Gaussian primitives to only move on the ray.
• we bind the ray-based Gaussian primitive to the matched ray and enable the direct optimization of the position of Gaussian primitive besides the optimization of the rendering geometry (Dual Optim.). Combined with the ordinary photometric loss, we can properly alleviate the optimization ambiguity between the position and size of Gaussian primitives in the few-shot scenario.

To prove these, as suggested, we conducted more detailed ablation studies and comparisons in Tab. 1, Tab. 3 and Fig. 2 of our uploaded one-page PDF.

• From Tab. 1, we can see that even without any matching priors, our "Hybrid Rep." can achieve more than 2dB improvement (from 16.46dB to 18.62dB), demonstrating the effectiveness of this design.
• To validate the effectiveness of our "Dual optim." design, we compare with the model only constraining the rendering geometry (Rend. Geo.) in Tab. 1. And the "Rend. Geo." is just the same as the setting of "w/ Matching prior" and "Straightforward combine matching priors", and we are sorry for not unifying and explaining it (Our explanation of the second point of Weaknesses). The results prove that our "Dual optim." can more completely combine matching priors to achieve the best performance (2.15dB improvement over the "Hybrid Rep." and 1.37dB improvement over "Rend. Geo.").
• As suggested, we further perform the comparison with methods initialize 3DGS, ScaffoldGS [1], and OctreeGS [2] models using point clouds directly triangulated from matching rays and camera poses, as shown in Tab. 3 and Fig. 2 of our uploaded PDF. Although they can achieve better performance than the vanilla 3DGS with better initialization, they struggle to mitigate the impact of the wrong matching samples and struggle to further optimize the position of Gaussian primitive like the vanilla 3DGS. On the contrary, our designs can efficiently optimize the position during the training, and our model has simpler convergence space with our hybrid representation, which is more suitable for few-shot NVS task and has been proved in DietNeRF [6] and FreeNeRF [7]. After all, since there may be surface points on this ray, there is no need to let the Gaussian primitives move throughout the entire 3D space.

Q: The robustness to different matching models.

As suggested, in Tab. 2 of our uploaded PDF, we conducted more experiments with different matching models, i.e., SuperGlue (CVPR2020) [3], LoFTR (CVPR2021) [4] and DKM (CVPR2023) [5].

• The results indicate that the matching prior from all matching models can bring satisfactory improvements to the baseline.
• The quality of matching prior does have an impact on the quality of the reconstruction, but our model has the cache and filter strategy which can mitigate the impact of wrong matching to some extent as demonstrated in Tab. 4 of our paper.

Q: The robustness to more texture-poor regions.

As suggested, in Tab. 4 and Fig. 1 of our uploaded PDF, we performed the comparisons with existing methods on the more texture-poor DTU dataset.

• The results show that our method can still achieve the best performance, especially compared to recent 3DGS method DNGaussian (CVPR2024) [8].
• The low texture is an ill-posed and common problem faced by all methods, so the scene that cannot get the ideal matching correspondence is still difficult for existing methods. Our model can still recover a more consistent structure with the insufficient matching prior, with our hybrid representation and dual optimization designs.

Reference

[1] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering, CVPR2024.

[2] Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians, arxiv2024.

[3] SuperGlue: Learning feature matching with graph neural networks, CVPR2020.

[4] Loftr: Detector-free local feature matching with transformers, CVPR2021.

[5] Dkm: Dense kernelized feature matching for geometry estimation, CVPR2023.

[6] Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis, ICCV2021.

[7] FreeNeRF: Improving Few-Shot Neural Rendering With Free Frequency Regularization, CVPR2023.

[8] DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization, CVPR2024.

Thanks for the authors' detailed reply. Most of my concerns are solved.

Here I have some additional problems:

1. Despite SCGaussian outperforms Scaffold-GS and Octree-GS, there seems no special design to delete wrong matching pixels compared to Scaffold-GS and Octree-GS besides "Cache & filter", as the ray-based Gaussians can not be pruned in my understanding. Meanwhile, the method can still work well without "Cache & filter" according to Table 4. So, how SCGaussian escape from the "struggle to mitigate the impact of the wrong matching samples"? If the flexibility of ray-based Gaussians along the ray is the reason, wish this feature to be added to the paper to help understand the design of ray-based Gaussian.

2. As illustrated in Figure 3, the goal of eq.(11, 13) is to optimize a pair of Gaussians converge to the same surface position with the same appearance. Then, is it really necessary to initialize two ray-based Gaussians for one surface point? Only one ray-based Gaussian seems to be sufficient for the framework. The only change may be on eq.(11), which can be modified as the loss between Gaussian position and ray intersection point, and all other parts can still work well. Although there is an ablation study in the rebuttal for Reviewer nvzF, would like to see more explanation about why it work.

3. After reading the experiment and discussion in the rebuttal, it's obviously easier for me to understand the method's effect than just reading the manuscript. Hope these valuable parts can be added to the paper if possible.

I'll temporarily keep my rating.

• I do think it's because the ray-based Gaussians can flexibly move when the matching prior is not so accurate, while will not go to some strange positions that may easily lead to overfitting, due to the ray and matching constraints. This can overcome some bad initialization points while keeping enough constraints for simplification. It's reasonable, but I can't directly tell this point from the words in the paper. To avoid confusion with previous structure-consistent GSs, maybe the authors can add some straight discussions like this in the revision.

• Well, I still think initializing two ray Gaussians is not so concise and elegant, personally. Nevertheless, it may not be a problem as no extra negative influence is introduced. The working principle is valuable to be further analyzed. The provided ablation study may be added to the paper.

Thanks for the response. I have no further questions. May consider raising the rating if there are no new problems found by other reviewers. Looking forward to the open access to the code and model.

We sincerely thank reviewer #nvzF for your time and valuable comments. We noticed that the main concerns are the differences with CorresNeRF and more explanation of the implementation details. Besides, we will address all concerns point by point.

Q1: The differences with CorresNeRF.

First of all, thanks for pointing out this important related work, and we will cite and add more analysis in our final version. Here, we give some analysis as follows:

One of our main designs is the dual optimization, instead of solely optimizing the rendering depth like CorresNeRF. We argue that optimizing the position of the Gaussian primitives is more important than optimizing the rendering geometry for learning consistent structures (Lines 51-56), because the rendering geometry is not consistent with the scene structure due to the interdependence of Gaussian attributes, a problem that NeRF methods do not face. The results in Tab. 1 of our uploaded PDF illustrated this (Improvement of 1.37dB of "Dual Optim." compared to "Rend. Geo.").

Q2: Which network is used for feature matching?

We adopt the GIM model for feature matching as declared in Line 147, and we further conducted with more different matching models in Tab. 2 of our uploaded PDF.

Q3: For the ray-based Gaussian primitives, do they have the shape? If not, can we call them Gaussian primitives?

Yes, the ray-based Gaussian primitives also have shape but with different representation of position，and thus we try to optimize both the position and shape of Gaussian primitives properly.

Q4: How do the authors select the value of N? Is it defined by the results of feature matching?

Yes, the value of N is determined by the results of feature matching.

Q5: Why does this work in line 177 define N pairs of ray-based Gaussian primitives rather than N ray-based Gaussian primitives?

We assign a pair of Gaussian primitives for each pair of matched rays, and this is more robust when the matching prior is not perfect. To demonstrate this, we conducted the comparison experiment shown in the table below.