Flow Distillation Sampling: Regularizing 3D Gaussians with Pre-trained Matching Priors

[Q1]:How to determine the value of the hyperparameter $\sigma$?

[A1]:Thank you for your question. we have derived the physical meaning of this hyperparameter $\sigma$, which represents the average radius of the 2D flow between the current input view and the unobserved sampling view. Therefore, this parameter directly corresponds to a measurable property and can be determined by the flow length that the optical flow model predicts the most accurately, which is typically related to the training data of the optical flow model. This parameter can be shared across different types of datasets, including DTU datasets (a object centered dataset with dense sampling) and Mushroom datasets (an indoor datasets with sparse sampling.) We have updated the results of FDS in our paper.

Derivation:

As noted in [1], the rotation flow in image warping is independent of depth. Therefore, we can set the rotation part to identity matrix. so that the transformation between the input view $i$ and its sampled view $s$ is pure translation, we can change the Equ.(4) in our paper to:

where $K$ is the intrinsic matrix of the camera, after solving the above equation, we get:

We set $t_3 = 0$ in our camera sampling scheme and assume camera intrinsic parameters: $f_x\approx f_y=f$. The radiance flow $F^{i\rightarrow s}(u_1, v_1) = \begin{bmatrix} u_2 - u_1\\ v_2 - v_1\\ \end{bmatrix}^T$from the training view $i$ to its sampled view $s$ is shown below:

We aim to keep the value of $||F^{i\rightarrow s}(u_1, v_1)||_2$ constant for the pixel $x = (u_1, v_1)$ during each camera sampling. By setting $||F^{i\rightarrow s}(u_1, v_1)||_2 = \sigma$, we get:

Thus, the radius of translation in our camera sampling is defined as $\epsilon_t = \sigma \frac{D_i(u_1, v_1)}{f}$ which helps maintain stable flow. The parameter $\sigma$ can be tuned as a hyperparameter. Given that pixel depths vary within an image, we use the mean depth $\bar{D_i}$ of the image and set the radius of our translation $\epsilon_t = \sigma \frac{\bar{D_i}}{f}$. $||F^{i\rightarrow s}(u_1, v_1)||_2 = \sigma$, demonstrating that $\sigma$ represents the average radius of the 2D flow between the current input view and its unobserved sampling view.

[Q2]: Is FDS robust to different radius ?

[A2]: As noted in A1, the parameter radius represents the average radius of the 2D flow between the current input view and its unobserved sampling view. So our FDS is robust to different radius in a proper range included in the training dataset of optical flow model. We added an experiment to validate our statements on the Mushroom dataset. The results are shown below:

It can be observed that both increasing and decreasing the radius allow FDS to achieve a consistent level of improvement.

[1]. Bian, Jia-Wang, et al. "Auto-rectify network for unsupervised indoor depth estimation." IEEE transactions on pattern analysis and machine intelligence 44.12 (2021): 9802-9813.

Thank you for your detailed rebuttal. I recommend acceptance of the paper.

[Q1]:This paper lacks evaluation on widely used geometry reconstruction and novel view synthesis benchmarks such as DTU, Tanks and Temples, and MipNeRF 360.

[A1]: Thank you for your advice. Since DTU is an object-level dense observation dataset, our FDS is primarily designed to mitigate the issue of insufficient sampling in observation regions. Therefore, we did not prioritize testing on the DTU dataset initially. We have added the results of our FDS on the DTU dataset, which are shown below. While FDS performs better under sparse observations, it still achieves notable improvements on the DTU dataset with dense observations.

[Q2]:While the metrics in Table 3 appear strong, it’s unclear why the loss significantly outperforms multi-view depth supervision, which does not suffer from inaccurate prior flow. More explanation and analysis are needed to clarify this point.

[A2]: Thank you for your question. First, the Mushroom dataset has limited overlap between input views, making multi-view depth estimation unreliable. In contrast, our prior flow is extracted from a controlled overlap between the input view and a sampled unobserved view, which is more reliable than offline multi-view depth from uncertain input views.

Second, to reduce inaccuracies caused by relatively blurry sampled views, FDS adopts a random sampling strategy. For any single input viewpoint, FDS effectively generates a sufficient number of sampled viewpoints during training. This acts as a form of model ensemble, helping to average out errors. To validate this claim, we conducted experiments using fixed sampling viewpoints on the Mushroom dataset instead of random sampling. The results showed a significant decline in performance, highlighting the importance of this approach.

[Q3]:Given that RAFT is computed at every time step, how does the training time for 2DGS + FDS compare to 2DGS?

[A3]: We have included our training time analysis in Table 1 of the revised paper. On the Mushroom dataset, adding the FDS loss led to a half-hour increase in training time. Despite this increase, the overall training time remained comparable to other baselines.

We are very delighted to receive your response and suggestion! Thank you for raising the rating and for your support.

[Q4]: It is recommended that the authors include a detailed report on the training and inference times of the model in the experimental section, along with comparative metrics against other existing methods.

[A4]: Thank you for your advice. We have included our training time analysis in Table 1 of the revised paper. On the Mushroom dataset, adding the FDS loss led to a half-hour increase in training time. Despite this increase, the overall training time remained comparable to other baselines. Since FDS is only applied during the training process, it does not affect the inference time.

[Q5]: The review of existing prior art aimed at improving 3DGS performance should be more comprehensive and clearer. There are sentences with grammatical errors that require careful review and correction. In Algorithm 1, there are notation errors that need careful checking and correction.

[A5]: Thank you for your advice. We have carefully reviewed and revised our paper in the related work and methods part. In our updated paper, we used colors to highlight the changes. Specifically, we adopted a more concise and clear discussion approach in the related work section and added a discussion on the application of priors in optical flow models. In the method section (including Algorithm 1), we used more rigorous notation and corrected some errors.

[Q6]: The comparison of depth reconstruction experiments needs to be supplemented with results from other methods, it is recommended that they include more baseline methods for a more robust evaluation.

[A6]: Thank you for your advice. We have compared the depth reconstruction results in Table 1 and Table 2 of our paper, where a lower "Abs Rel" metric indicates better reconstruction quality. Compared with GOF, PGSR, and 2DGS, which are SOTA 3DGS-based 3D reconstruction methods, our FDS achieves the best depth reconstruction performance. We choose 2DGS as the baseline method for our FDS because it performed the best on the Mushroom dataset compared with GOF and PGSR.

[Q7]: In the dataset section, the paper mentions that the authors have evaluated their method on the Replica dataset, but the experimental results are not presented.

[A7]: The experimental results of the Replica dataset are shown in our supplementary materials due to page limitations. We also updated our Replica results using a more advanced optical flow model.

[Q1]: The reliability of the prior flow cannot be assured under certain sparse viewpoint configurations.

[A1]: Our FDS introduces two strategies to improve the reliability of prior flow. The first is controlled overlap between the input view and its sampled unobserved view, which ensures controlled overlap and average flow. According to the derivation in our updated paper, the hyperparameter $\sigma$ in FDS represents the average radius of the 2D flow between the current input view and sampled view. This ensures that the optical flow model can generate accurate optical flow under suitable overlap. Secondly, the random sampling introduced by FDS also provides an advantage. FDS effectively generates a sufficient number of sampled viewpoints during training which can help to average out errors. We tested our FDS by fixing the sampling viewpoints instead of using random sampling on the Mushroom dataset. The results are shown below:

We can see that the random sampling further helps 2DGS improve geometric accuracy.

[Q2]: The method's reliance on the performance of a pretrained optical flow model restricts its generalization.

[A2]: Although our model is limited by the accuracy of the optical flow model, the upper bound of our FDS will continue to improve with the emergence of more annotated data and larger models. To validate our idea, we replae RAFT with the more advanced SEA-RAFT model in our updated paper, the accuracy of geometric reconstruction has been further enhanced.

[Q3]: It is important to point out that these datasets are primarily limited to indoor scenes.

[A3]: To test our FDS on more diverse datasets, we have added results of our FDS on DTU datasets shows below. Although our FDS is primarily designed to mitigate the issue of insufficient sampling in observation region, we still achieve improvements on the DTU dataset with dense observation.

We hope our response has addressed your questions. As the discussion phase is coming to a close, we are looking forward to your feedback and would like to know if you have any remaining concerns we can address. We are grateful if you find our revisions satisfactory and consider raising your score for our paper.

Thank you once again for the time and effort you have dedicated to reviewing our paper.

Best regards

Flow Distillation Sampling Authors

We would like to sincerely thank you for your valuable feedback and the time you've dedicated to reviewing our paper. As the extended discussion phase is now nearing its end, we would greatly appreciate your feedback on our revisions. Please let us know if there are any remaining issues we can address.

Thank you once again for your support and effort!

Best regards

Flow Distillation Sampling Authors

[Q1]: Baseline methods usually use more diverse datasets.

[A1]: Thank you for your advice. We did not prioritize testing on the DTU dataset initially since DTU is an object-level dense observation dataset, and our FDS is primarily designed to mitigate the issue of insufficient sampling in observation regions. To test our FDS on more diverse datasets, We have now included results of our FDS on the DTU dataset, as shown below. While FDS performs better under sparse observations, it still achieves notable improvements on the DTU dataset with dense observations.

[Q2]: Do not provide evidence or explanations for depth distortion loss and it is unclear how this influenced the quantitative comparison to 2DGS.

[A2]: We found that the depth distortion loss tends to negatively impact performance on indoor scenes. To validate this observation, we evaluated the performance of “2DGS (with depth distortion loss)” and “2DGS(with depth distortion loss)+ FDS”. The results on the Mushroom dataset are shown below.

For the DTU results in A1, the depth distortion loss is used by default in both 2DGS and 2DGS+FDS. Our FDS also improves reconstruction results.

[Q3]: Comparisons to neural field based methods such as Geo-NeUS or NeuralAngelo would significantly strengthen the claim of state-of-the-art performance.

[A3]: Thank you for your advice. We are testing NeRF-based methods, such as NeuralAngelo, on the Mushroom dataset. We are running the relevant program and will update the results in about two days.

[Q4]: The overall quality and the presentation should be improved.

[A4]: Thank you for your advice. We have carefully revised our updated paper to improve the presentation quality and address identified errors in method and conclusion section.

[Q3]: Comparisons to neural field based methods such as Geo-NeUS or NeuralAngelo would significantly strengthen the claim of state-of-the-art performance.

[A3] (Supplement): We evaluated the geometric reconstruction performance of the NeuralAngelo method on the Mushroom dataset. We used the original method's data preprocessing script, set the scene type to "indoor," and processed the ground truth camera poses from Mushroom along with the initial point cloud from other experiments in this paper to obtain the bounding sphere of interest. The experimental parameters were configured exactly as in the authors' paper for the TnT dataset, with a batch size of 16 and 500k iterations.

Since the NeuralAngelo method was not designed to reconstruct scene geometry from sparse viewpoints, its performance on Mushroom was suboptimal. Specifically, in the “sauna” scene, using the default preprocessing script caused the NeRF model to diverge. Following the recommendations of the NeuralAngelo authors, we tried adjusting the radius of the bounding sphere of interest. While the model no longer diverged, it tended to overfit to the input viewpoints during training.

To ensure a fair comparison, we exclude the 'sauna' scene, where reconstruction failed by NeuralAngelo, and compare the average results of the remaining four scenes, as shown in the table below: