Binocular-Guided 3D Gaussian Splatting with View Consistency for Sparse View Synthesis

We sincerely appreciate the reviewer gh5j for the invaluable feedback and time invested in evaluating our work. We respond to each question below.

Q1: The opacity penalty guides the remaining Gaussians closer to the scene surface.

Opacity penalty prune the Gaussians that far from the scene surface as much as possible, while preserving the Gaussians close to the scene surface. Our description may be ambiguous here, and we will revise it in the revision.

We perform ablation studies on the LLFF dataset with the value of $\lambda$ ranging from 0.96 to 1.0, and the results are shown in the Table G. We find that the best performance is achieved when the value of $\lambda$ is 0.995.

Q2: "the Gaussian close to surface has larger gradient and the Gaussian far from the surface has smaller gradient".

Our description may not be clear enough, the "gradient" refers to opacity. For instance, when initialized with a random point cloud, during the optimization process, Gaussians that are far from the scene surface will have their opacity reduced, thereby being pruned. Gaussians closer to the scene surface will see an increase in opacity, thus being preserved. However, not all Gaussians close to the surface have precise positions, hence, we employ opacity regularization to further eliminate Gaussians deviating from the surface, thereby enhancing the quality of novel view images.

Q3: Explanation and content of Opacity Penalty Strategy is not adequate, adding more analysis.

We provide more detailed ablation studies on the LLFF, DTU, and Blender datasets, as shown in Table B and Table C in the attached PDF, and the results prove that the Opacity Penalty Strategy is indeed very effective, although it is simple. We will correct some places that are not clear enough in the revision.

Q4: Advantages of sampling only in the horizontal direction.

Although some papers regard unseen views or adjacent training views as source views and warp them to the reference view, similar to our image-warping method based on binocular vision, these approaches do not consider the impact of the distance and angle between the source view and the reference view on the effectiveness of supervision. Figure A in the attached PDF shows the warped images and error maps when using different source views. It can be observed that when the source view is rotated relative to the reference view or is far away from the reference view, the warped image suffers from significant distortions due to depth errors and occlusions. This results in a large error compared with GT image, which hinders the convergence of the image-warping loss. In contrast, slight camera translations that cause minor changes in view without rotation and negligible impact from occlusions, allow the errors between the warped image and the GT image to primarily stem from depth errors, facilitating better optimization of depth.

Table A in the attached PDF shows the quantitative comparison using different source images on the LLFF and the DTU dataset. The binocular vision-based view consistency method achieves the best performance.

Q5: Using more diverse sampling space.

We perform ablation studies for horizontal sampling range $d_{max}$ on the LLFF and DTU datassets, as shown in Table H of the attached PDF, and the results show that the larger sampling range is not the better. There is almost no performance increase on both the LLFF and the DTU dataset when $d_{max}$ is greater than 0.4.

Meanwhile, the Table A shows that the camera views with rotation lead to performance degradation.

Q6: The impact of occlusion on performance.

Since we only move the camera slightly, there is no serious occlusion between the reference view and the source view, so we believe that the impact of occlusion can be ignored.

Q7: An inaccuracy about the equation of image warping.

Thank you for pointing out the inaccuracy in the equation, warping the right image to the left view requires the depth of left image instead of the depth of right image. We will correct the equation in our revision.

Q8: The impact of initialization on overall performance.

We conduct experiments on the LLFF and DTU datasets with several different initialization, including random initialization, sparse initialization, and using the LoFTR [1] matching network to construct initial point clouds. The results are shown in Table K of the attached PDF.

PDCNet+ can get more matching points than LoFTR, so using the initialized point cloud obtained by PDCNet+ result in better performance. For the LLFF dataset, the input views have large overlapping, SfM methods can generate point clouds with better quality than LoFTR. Therefore, using sparse point clouds generated by SfM as initialization yields better performance than LoFTR. However, for the DTU dataset, where there is small overlapping among input views, SfM-generated point clouds are too sparse. So, the performance as initialization is even worse than random initialization.

Q9: The results reported by FSGS differ significantly from those in the paper.

We cannot reproduce the results reported in the FSGS paper. The results listed in our paper are reproduced using their official code for a fair comparison. We will explain this in our revision. Note that even using the results reported in the FSGS paper, our results remain significantly better.

Q10: Some visualization results are not labeled with the corresponding methods.

Due to some unknown reasons, the method names labeled in the visualization results cannot be displayed in some PDF readers. Please use Chrome to visualize these figures. We will address this issue in revision.

Q11: Report the error bar or standard deviations.

We run our method 10 times for each scene in LLFF and DTU dataset. Error bars are shown in Figure D.

[1] LoFTR: Detector-free local feature matching with transformers. CVPR 2021.

Dear Reviewer gh5j,

Thanks for your feedback.

Q1: How about directly adjusting the threshold of the pruning operation.

It is clear that the opacity penalty strategy is different from setting a fixed pruning threshold. The opacity penalty strategy is applied to all Gaussians and acts as a global constraint. In contrast, directly adjusting the pruning threshold only affects Gaussians with opacity lower than the threshold. For Gaussians with opacity higher than the threshold but not on the scene surface, a fixed pruning threshold is ineffective.

We are conducting experiments on the LLFF dataset with different pruning threshold, the results will be provided later.

Q2: The range control of sampling space can be achieved by any existing sampling methods and the camera can be translated in all direction.

Yes, the existing methods might be able to change the position and angle of the unseen view to achieve the same effect as ours, but they didn't do it. To the best of our knowledge, our method is the first to attempt using viewpoint translation based on Gaussian Splatting to optimize the problem of sparse view synthesis, and it has been proven that it is more effective than other sampling methods.

Indeed, the camera can be translated in any direction, but that still falls under translation without rotation, which we believe is essentially no different from horizontal movement.

Q3: The method is relatively sensitive to the matching model and have a great reliance of the matching model.

Dense initialization is an indispensable part of our method, as we mentioned in Section 3.4 of our paper, our method needs a dense initialization to achieve better geometry initialization, thereby improving the quality of 3DGS. Additionally, experiments have shown that the matching model we selected is effective across different datasets.

Although our method performs poor when using random initialization, it still outperforms the baseline 3DGS, especially on the DTU dataset, which proves that the other two components of our method are effective.

We sincerely appreciate the reviewer UVce for the invaluable feedback and time invested in evaluating our work. We respond to each question below.

Q1: Comparison to generalizable methods.

Thank you for listing some of the latest generalizable novel view synthesis methods. We run the state-of-the-art MuRF method on the LLFF dataset for comparison with our method under the same input conditions. For the DTU dataset, we directly use the evaluation results from the original paper. As shown in Table D and Table E in the attached PDF.

Our method outperforms the state-of-the-art MuRF method on the LLFF dataset, and also shows better performance when using 6 views and 9 views as input on the DTU dataset. MuRF performs better on the DTU dataset when using 3 views as input. However, generalizable methods require large scale datasets and time-consuming training to learn a prior.

Figure C in the attached PDF shows the visual comparison between our method and MURF on LLFF dataset.

Q2: Computational performance comparison.

Thank you for your advice. We provide a comparison on computational performance of our method and prior works. All the results are tested under one single RTX3090 GPU. The times are listed in Table F in the attached PDF.

Q3: Ablation studies for hyperparameters $\lambda$ and $d_{max}$.

Ablation study for $\lambda$. We perform ablation studies on the LLFF dataset with the value of $\lambda$ ranging from 0.96 to 1.0, and the results are shown in Table G in the attached PDF.

We find that the best performance is achieved when the value of $\lambda$ is 0.995. We set the $\lambda$ to 0.995 for all datasets.

Ablation study for $d_{max}$. We perform ablation studies for $d_{max}$ on the LLFF and DTU datassets. The value $d_{max}$ gradually increases from 0.1 to 0.8, the results are shown in Table H in the attached PDF.

We find that there is almost no performance increase on both the LLFF and the DTU dataset when $d_{max}$ is greater than 0.4, so we set the $d_{max}$ to 0.4 for all datasets.

The values of camera shift are sampled uniformly.

Q4: Compare depth maps with other methods and pseudo-GT depth.

Figure 3 in our paper shows visual comparison of rendered novel images and depth maps between our method and others including DNGaussian. We apologize that due to unknown reasons, the method labels below the images cannot be displayed in some PDF readers. Please use Chrome to open our pdf to see the visual comparisons. We will address this issue in the revision.

Additionally, the Figure B in the attached PDF presents comparisons of depth maps from several methods, including pseudo-GT depth. Pseudo-GT depth is obtained by training using all available views in baseline 3DGS.

Q5: Experimental results from multiple run times.

We run our methods 10 times for each scene in LLFF and DTU dataset. Error bars are shown in Figure D of the attached PDF.

Q6: Apply the disparity consistency loss to the existing methods.

We conduct experiments to train DNGaussian on LLFF dataset using disparity consistency loss. The results are shown in Table I of the attached PDF.

Although we could not reproduce the performance reported in the original paper, we find that the performance is improved by comparing the quantification results before and after using disparity consistency loss. It indicates that disparity consistency loss is effective in other method.

Q7: Apply depth priors as supervision.

We experiment with L1 depth loss and DNGaussian Depth Regularization on the LLFF and DTU datasets, employing dense initialization and opacity penalization by default. Monocular depth is obtained by pre-trained DPT, same as DNGaussian. We calibrate monocular depth using sparse point clouds from SfM when using L1 depth loss. The results are shown in Table J of the attached PDF.

The L1 depth loss and DNGaussian Depth Regularization both result in performance degradation. We attribute this mainly to errors in monocular depth, even after calibration, leading to decreased performance.

Q8: Put the zoom-in areas side-by-side.

We really appreciate your suggestion, and we will make appropriate adjustments to the layout of zoom-in areas in the revision.

Dear Reviewer UVce,

Thanks for your feedback.

Q1: Whether the choice of $\lambda$ is empirically easy.

We conduct ablation studies for $\lambda$ on the DTU and Blender datasets, the results are as follows:

Table G2: Ablation studies for $\lambda$ on DTU dataset with 3 input views.

Table G3: Ablation studies for $\lambda$ on Blender dataset with 8 input views.

We can see that the effect of $\lambda$ on performance is generally consistent across different datasets.

We sincerely appreciate the reviewer sng2 for the invaluable feedback and time invested in evaluating our work. We respond to each question below.

Q1: Opacity regularization does not seem enough to consider it a contribution.

Although opacity regularization is a very simple strategy, it significantly contributes to improving the quality of sparse view synthesis. Similar to FreeNerf [1], its primary contribution lies in the use of frequency regularization, which despite its simplicity, greatly aids in performance enhancement. We provide more comprehensive ablation studies on the LLFF, DTU, and Blender datasets in Table B and Table C in the attached PDF. It can be observed that there is a significant performance improvement when opacity regularization is applied. We believe the simple yet effective regularization term will greatly benefit other researches on sparse view synthesis.

Q2: Prior free claim.

Indeed, we did not describe 'Prior Free' clearly. What we intended to convey is that, different from methods such as the FSGS [2] and DNGaussian [3] that use priors as supervision, we do not require any prior as supervision, since we can mine supervision using the self-supervised method. We will correct this claim in the revision.

Q3: 1.5dB higher than baseline 3DGS without any component used in Table 4.

Our method employs opacity penalty strategy, which conflicts with the opacity reset operation in baseline 3DGS, hence we do not use opacity reset operation. The first row in Table 4 shows the performance without using opacity reset operation. We inadvertently omitted an explanation of it in the ablation study, which we will add in the revision.

Q4: Missing ablation studies.

We provide more comprehensive ablation studies on LLFF, DTU, and Blender datasets, as shown in the Table B and Table C in the attached PDF.

Q5: The main performance gains come from the dense keypoint initialization.

Although dense initialization brings significant performance gains, relying solely on dense initialization cannot achieve or approach optimal performance. View consistency loss and opacity penalty strategy also contribute significantly to performance improvement, as evidenced on ablation studies in Table B and Table C in the attached PDF..

Q6: Inconsistent performance improvements in Table 4.

The three components we propose to improve the quality of novel view synthesis are essentially to make the Gaussians more accurately distributed on the scene surface, and the three components are complementary to each other. When only dense points are used as the initialization, the quality of Gaussians has a significant improvement space over the baseline 3DGS. Continuing with view consistency loss and opacity penalty further advances the already enhanced Gaussians, reducing the improvement potential. Therefore, there is less performance improvement when the three components are used together.

Additionally, this is also verified by other works based on 3DGS, such as the ablation study in DNGaussian [3], which shows that using only Depth Regularization leads to a performance improvement of 2.5dB, whereas adding depth Normalization results in only a 1dB performance improvement.

[1] Freenerf: Improving few-shot neural rendering with free frequency regularization. CVPR2023.

[2] FSGS: Real-time few-shot view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00451 (2023).

[3] DNGaussian: Optimizing sparse-view 3d gaussian radiance fields with global-local depth normalization. CVPR2024.

We sincerely appreciate the reviewer q3N3 for the invaluable feedback and time invested in evaluating our work. We respond to each question below.

Q1: Whether depth-warping loss can be viewed as a contribution

Although some papers regard unseen views or adjacent training views as source views and warp them to the reference view, similar to our image-warping method based on binocular vision, these approaches do not consider the impact of the distance and angle between the source view and the reference view on the effectiveness of supervision. Figure A in the attached PDF shows the warped images and error maps when using different source views. It can be observed that when the source view is rotated relative to the reference view or is far away from the reference view, the warped image suffers from significant distortions due to depth errors and occlusions. This results in a large error compared with GT image, which hinders the convergence of the image-warping loss. In contrast, slight camera translations that cause minor changes in view without rotation and negligible impact from occlusions, allow the errors between the warped image and the GT image to primarily stem from depth errors, facilitating better optimization of depth. This is what we would like to report. We will highlight this in our revision.

Table A in the attached PDF shows the quantitative comparison using different source images on the LLFF and the DTU dataset. The binocular vision-based view consistency method achieves the best performance.

Q2: Prior free claim

Indeed, we did not describe 'Prior Free' clearly. What we intended to convey is that, different from methods such as the FSGS and DNGaussian that use priors as supervision, we do not require any prior as supervision, since we can mine supervision using the self-supervised method. We will correct this claim in the revision.

Q3: Ablation study on DTU and Blender datasets

Thank you for your suggestion, we provide ablation studies on DTU and Blender datasets with 3 input views. The results are shown in Table B and Table C in the attached PDF. We can see that our modules are effect on other datasets. We will update our ablation studies in our revision.