Boost 3D Reconstruction using Diffusion-based Intrinsic Estimation

Thank you very much for reviewing our submission. We have addressed all concerns expressed in your reviews.

• Comment 1: The reviewer finds the ablation study in Tab. 4 insufficient, as it lacks a detailed analysis of each DM-Calib component’s contribution to accuracy.

  Answer: We would like to clarify that Table 4 is not the ablation study. Table 4 highlights the advantages of our proposed DM-calib in downstream applications like sparse-view 3D reconstruction. The ablation studies are actually presented in Tab. 6 and Tab. 7, where Tab. 6 analyze each DM-Calib component’s contribution to accuracy.

   

• Comment 2: The reviewer notes that DM-Calib shows minimal improvement over the baseline in Table 3 (comparison of affine-invariant depth estimation), with gains falling within the standard deviation range.

  Answer: Our primary contribution lies in the calibration of pinhole cameras, achieving state-of-the-art performance as shown in Table 1. We leverage the camera image for metric depth estimation (Table 2) and perform scale-shift alignment to derive affine-invariant depth, as most monocular depth estimation methods focus on this setting (Table 3). Notably, recovering metric depth is significantly more challenging than affine-invariant depth, yet our approach delivers competitive and comparable performance without sophisticated designs.

   

• Comment 3: The reviewer points out that baseline comparisons lack an ablation study of DM-Calib (e.g., without Camera Image or grayscale encoding) to clarify each component’s impact.

  Answer: We conducted the mentioned ablation study in Tab. 6. We also shown this table below, demonstrating the effectiveness of the proposed camera image.

  Multi-Res		Camera Image		Mean Error ($^{\circ}$ )$\downarrow$		Median Error ($^{\circ}$) $\downarrow$
  $\times$		$\times$		24.36		25.74
  $\checkmark$		$\times$		9.10		7.00
  $\times$		$\checkmark$		6.72		6.33
  $\checkmark$		$\checkmark$		2.54		2.01

Thank you for your insightful summary and for acknowledging our contribution. We provide detailed responses to all concerns in the following sections of this document.

• Comment 1: The reviewer notes that the proposed method closely resembles DiffCalib, which also fine-tunes Stable Diffusion 2.1 to estimate both the incident map for intrinsics and the depth map.

  Answer: To the best of our knowledge, DiffCalib was posted on arXiv several months ago and may not have undergone peer review. Furthermore, we believe that this work and ours were developed concurrently, and our method differs significantly from DiffCalib in two key aspects:

  • Our proposed camera image provides a superior formulation for intrinsic estimation compared to the incident map introduced by DiffCalib, as demonstrated in Fig. 2 and Tab. 1. Below, we present the average errors for focal length ($e_f$) and principal point ($e_b$) for both methods (lower is better).

    		$e_f$		$e_b$
    DiffCalib		0.122		0.030
    Ours		0.078		0.017

  • We estimate metric depth through a one-step deterministic process, unlike DiffCalib's diffusion-based affine-invariant depth. Notably, the estimated metric depth and intrinsics enable point cloud reconstruction, which is not directly achievable with affine-invariant depth [1].

   

  [1] Yin, Wei, et al. "Learning to recover 3d scene shape from a single image." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

   

• Comment 2: If the VAE is fine-tuned or simply rescaled to match the distribution before the diffusion model training, will the gray image still be necessary?

  Answer: We froze the VAE encoder and fine-tuned the decoder while training it concurrently with the U-Net for metric depth estimation, as shown in Fig. 5. The grayscale image remains necessary, as it is used to form the three-channel camera image. The VAE encoder is pre-trained on three-channel images and is only compatible with this format, which is why the grayscale image is reserved as part of the input.

   

• Comment 3: The diffusion model is stochastic. The standard deviation in the experiments should also be reported.

  Answer: Thanks for point this out, we report our standard deviation as a complementary of our Tab. 1 below:

  	Waymo		RGBD		ScanNet		MVS		Scenes11		Average
  $e_f$	$0.115\pm0.008$		$0.041\pm0.002$		$0.089\pm0.002$		$0.087\pm0.006$		$0.061\pm0.006$		$0.078\pm0.006$
  $e_b$	$0.036\pm0.001$		$0.010\pm0.000$		$0.024\pm0.000$		$0.008\pm0.000$		$0.010\pm0.001$		$0.017\pm0.001$

  Specifically, to reduce the stochasticity of the process, we aggregate five predicted camera images by taking their mean. This significantly minimizes the randomness of the diffusion model, as evidenced by the small standard deviation in our results.

  Without the aggregation, the standard deviation is sometimes not negligible, as presented below:

  	Waymo		RGBD		ScanNet		MVS		Scenes11		Average
  $e_f$	$0.115\pm0.035$		$0.041\pm0.010$		$0.089\pm0.024$		$0.087\pm0.008$		$0.061\pm0.009$		$0.078\pm0.017$
  $e_b$	$0.036\pm0.012$		$0.010\pm0.001$		$0.024\pm0.001$		$0.008\pm0.001$		$0.010\pm0.001$		$0.017\pm0.001$

  We will include this discussion in our revised version.

   

• Comment 4: Explaining the ray vector $\vec{r}$ at its first appearance would benefit novice readers.

  Answer:Thanks for your advice. The ray vector $\vec{r}$ is the normalized camera ray originating from the camera center and passing through the pixel in the camera coordinate system.

Thank you very much for reviewing our submission. Your constructive suggestions and detailed comments have been invaluable in improving the quality of our work.

• Comment 1: The reviewer appreciates the Camera Image representation but notes the lack of distortion handling, suggesting it's limited to the Pinhole model, and asks for clarification in the abstract about only recovering pinhole intrinsics.

  Answer: Thank you for your comments and suggestions. We will clarify in our article that our focus is on recovering the Pinhole model from a single image without considering distortion parameters. Additionally, single-image undistortion is a well-studied problem with existing tools available for implementation [1, 2], which can be used as a preprocessing step for the input image of our method.

     

  [1] Fan, Jinlong, et al. "Wide-angle image rectification: A survey." International Journal of Computer Vision 130.3 (2022): 747-776.

  [2] cui,xingxing (2024). Single-Image-Undistort (https://github.com/cuixing158/Single-Image-Undistort), GitHub. Retrieved November 17, 2024.

   

• Comment 2: The reviewer questions whether the metric-depth estimation method, which uses a single-step pass through the UNet of Stable Diffusion, qualifies as a diffusion-based approach.

  Answer: Thank you for pointing that out. The one-step deterministic forward process proposed in our work is not a true diffusion-based method, as it does not involve a denoising process. Instead, we directly predict the target metric depth in a regression manner rather than noise in a diffusion scheme.

   

• Comment 3: The reviewer seeks clarification on whether the metric-depth estimation model uses a ''GT" camera image or a one from our diffusion model DM-calib.

  Answer: We did not use the ground truth camera image but instead relied on intrinsic parameters predicted by our diffusion model for all of our downstream 3D vision tasks, including Monocular Metric Depth Estimation, Sparse-View 3D Reconstruction & Pose Estimation, Monocular 3D Metrology (discussed in the main article), as well as mesh and point-cloud reconstruction (presented in the appendix).

   

• Comment 4: Comparison with the recent methods like ''GeoCalib" [1].

  Answer: Thank you for pointing that out. GeoCalib was released within a month of our submission, and we did not notice this approach in our evaluation. We have now compared our method with GeoCalib [1] as follows:

  Method		Waymo		RGBD		ScanNet		MVS		Scenes11		Average
  GeoCalib		0.285		0.203		0.137		0.104		0.344		0.215
  Ours		0.115		0.041		0.089		0.087		0.061		0.078

  As we can see that our method outperforms GeoCalib [1] across all datasets. We will include these table in our revised version.

     

  [1] Veicht, Alexander, et al. "GeoCalib: Learning Single-image Calibration with Geometric Optimization." arXiv preprint arXiv:2409.06704 (2024).

   

• Comment 5: The reviewer appreciates the applications of Camera Intrinsic but feels that describing too many. They suggest focusing on calibration in the main text and moving some applications to the appendix for better clarity.

  Answer: Thank you for the suggestion. We will provide more detailed discussions on Camera Intrinsic and relocate some descriptions of downstream applications to the Appendix in our revised version.

Thank you very much for reviewing our submission. Your constructive suggestions and comments have been invaluable in improving the quality of our work. We have addressed all concerns expressed in your reviews.

• Comment 1: The reviewer finds it strange that the author's monocular method achieves better intrinsic estimation than the multi-view method, Dust3r.

  Answer: Thank you for your comment. Our method complements sparse-view reconstruction methods like Dust3r by providing intrinsic information, rather than serving as a direct comparison. Dust3r delivers less accurate intrinsic estimation because it focuses on sparse-view reconstruction by generating point clouds for image pairs and performing global alignment to jointly optimize intrinsic calibrations and poses. This process is less robust and often converges to a local minimum. In contrast, our method is specifically designed to recover camera intrinsics. The results demonstrate that Dust3r achieves more accurate reconstruction when equipped with our estimated intrinsics.

   

• Comment 2: The reviewer concern that most training datasets have a principal point at the image center and often $f_x = f_y$. They wonder whether a simpler representation [$f_x$, $f_y$, gray] might perform well or not, and request examples to demonstrate that $c_x$ and $c_y$ are not being ignored by the model.

  Answer: We have a significant amount of data where the principal point does not lie at the image center in certain datasets, and our model effectively learns the position of the principal points rather than ignoring them. To validate this, we conduct an ablation study comparing the error when assuming the principal point lies at the image center ($e_b$) with the error of our estimated principal point ($\hat{e}_b$). The results are presented below (lower is better):

  	NuScenes	KITTI	CityScapes	NYUv2
  $e_b$	0.051	0.021	0.055	0.050
  $\hat{e}_b$	0.007	0.014	0.011	0.009

   

  As we demonstrate the importance of accurately estimating the principal point, a formulation such as $[f_x, f_y, \text{gray}]$ is not reasonable.

  Furthermore, not all datasets have $f_x = f_y$ (e.g., CityScapes dataset with $f_x = 2268.36$ and $f_y=2225.54$). And our method is inherently capable of solving for both $f_x$ and $f_y$ and we take this into account to ensure more robust estimation and support future broader applications and datasets such as Diode [1].

     

  [1] Vasiljevic, Igor, et al. "Diode: A dense indoor and outdoor depth dataset." arXiv preprint arXiv:1908.00463 (2019).