Omnidirectional 3D Scene Reconstruction from Single Image

We would like to extend our sincere thanks to the reviewer for the valuable and insightful comments, which are instrumental in improving our manuscript. Please find our point-by-point responses to each of the reviewers' comments in the following.

Overall pipeline and Homography

• Overall pipeline. We agree with the reviewer that the proposed pipeline has the potential to be simplified in future work. One possible way is generating 3D Gaussian splats straight from a single image input by a 3D fundamental model, bypassing the 2D (multi-view images) intermediate steps. This would not only simplify the overall pipeline but also significantly accelerate the inference speed.
• Homography transformation. Indeed, Homography transformation has a strong planar geometry assumption. However, it achieves image-level viewpoint transformation with only 8 parameters, and therefore can be used as an effective yet time-efficient way for the coarse correctness or alignment of camera views, especially when drifting viewpoint in a small angle. Beyond Homography, we also employ flow maps for pixel-level warping and learn residuals for the correctness of each pixel value, which play the role for finer correctness of generated views. As such, we achieve a hierarchical coarse-to-fine process of pose-view optimization. Its effectiveness has been validated in the experiments.

Details about MVD

Thanks for the questions regarding MVD. We will describe the MVD more detailly in the revised manuscript.

• Training MVD for four orbits. Instead of controlling MVD by camera poses, we have trained four MVD models for the four target orbits (left, right, up and down), respectively. Specifically, before training the MVD models, we first reconstruct the 3DGS of each sample in the training set, and then render it along the four target orbits to obtain the training samples of sequential images in the four orbits, respectively. Then, the MVD model for each orbit is trained by the samples of the corresponding orbit.
• Initial poses. Although the MVDs show good performance for multi-view generation along the target orbits, it may inevitably occur slight errors between the generated orbits and the ideal orbits. Therefore, we employ MASt3R to calculate the initial poses of the generated views, instead of the poses predicted by the ideal orbits, to reduce the initial error of pose estimation before the PVO process.
• Frozen MVD. After training the MVD models, they are frozen during the following processes, i.e., MASt3R, PVO and Gaussian optimization.

Design of the refinement network and ablation studies

Thanks a lot for the insights on the refinement network.

• Lightweight CNN vs. direct optimization of parameters. In the proposed PVO process, we use a lightweight CNN to overfit the geometric parameters instead of directly optimize the parameters, since we observed instability (loss may increase) during the iterative optimization when directly optimize the geometric parameters themselves (denoted as w/o lightweight CNN). This is probably because that the lightweight CNN increases the nonlinearity of the network and are more compatible with the Adam optimizer, and therefore it is easier to adjust the learning rate during training. In the following, we show the ablation results in Table 1 in this rebuttal. It can be seen from Table 1 that directly optimizing the parameters (w/o lightweight CNN) leads to worse performance than the proposed lightweight CNN due to the worse convergence of PVO. Note that, in the case of instability, PVO has to be early stopped when the loss starts to increase, to avoid the crash of the whole pipeline.

  
  Table 1: Ablation study on the lightweight CNN vs. direct optimization of parameters (w/o lightweight CNN)

  Methods	Tanks and Temples	Mip-NeRF 360	DL3DV
  	PSNR $\uparrow\ \ \$ SSIM $\uparrow\ \ \$ LPIPS $\downarrow\ \ \$	PSNR $\uparrow\ \ \$ SSIM $\uparrow\ \ \$ LPIPS $\downarrow\ \ \$	PSNR $\uparrow\ \ \$ SSIM $\uparrow\ \ \$ LPIPS $\downarrow$
  Omni3D (w/o lightweight CNN)	15.92 $\ \ \ \ \$ 0.5227 $\ \ \ \$ 0.5175	15.21 $\ \ \ \ \$ 0.2605 $\ \ \ \$ 0.7262	16.57 $\ \ \ \ \$ 0.6244 $\ \ \ \$ 0.3890
  Omni3D (w/ lightweight CNN)	16.30 $\ \ \ \ \$ 0.5308 $\ \ \ \$ 0.5166	15.89 $\ \ \ \ \$ 0.2859 $\ \ \ \$ 0.6369	17.08 $\ \ \ \ \$ 0.6649 $\ \ \ \$ 0.3348

• Replace by a shared network. If using a pre-trained (shared) network to refine the generated views according to estimated poses, it should learn to extract features from initially estimated poses and the corresponding initial re-projected images, in addition to the input image pair. The current lightweight CNN takes the inputs only the pair of reference and generated views without pose-related conditions, and therefore this network (or even its upscaled version) cannot play the role of a pre-trained network for the refinement of generated views. To employ a shared network, it is necessary to re-design the network structure. During the rebuttal period, it is hard to re-design and propose the pre-trained shared network, which are largely different from the proposed method. This may be an interesting topic that is valuable to be explored in future research.

Evaluation in other camera distributions

Thanks a lot for the valuable comments on the experimental setup. In this rebuttal, we have conducted additional experiments on the evaluation of rendering quality at forward-facing camera views, following the experimental setup of LiftImage3D. The results are shown in Table 2, where we can see that the proposed Omni3D approach also improves the reconstruction quality for the forward-facing camera views.

Table 2: Evaluation of rendered views at forward-facing camera distributions

Task setting

Thanks a lot for the insightful comment on the task setting. Indeed, sparse-view based solutions have been widely studied in the community and have achieved promising results on 360-degree 3D reconstruction. Lifting a single image to the 3D representation (e.g., 3DGS) of the entire omnidirectional space is much more challenging, and meanwhile has the potential to be practical for the scenarios of AI-powered immersive media, such as the 3D experience of online meeting with mobile or laptop camera, and immersive 3D experience of the scene in a photograph. In recent years, there are a number of works in this direction, however, the existing works, e.g., LiftImage3D, ViewCrafter, etc., only effectively reconstruct 3DGS in limited angles, encountering significant hurdles to reconstruct omnidirectional 3D scenes. In this work, we pioneered in omnidirectional 3D reconstruction from single image, thanks to the proposed pipeline and the PVO method in our Omni3D approach.

Comparison of time consumption

Thank the reviewer very much for the valuable comments on the comparison of time consumption.

• Fairness for the comparison on 8 x A100 GPUs. In the original manuscript, the proposed Omni3D and all compared methods are tested on identical hardware, i.e., 8 x A100 GPUs. Moreover, we have also applied parallelism computation to all compared methods as significant as possible, such as paralleled MVDs for various orbits, to fully leverage the capabilities of the hardware. Therefore, it ensures fairness in the comparison of time consumption. We will declare this issue more clearly in the revised manuscript.

• Time consumption on 1 x A100 GPUs. Indeed, as suggested by the reviewer, it is necessary to report the computational time on a single GPU as well. We show the results in Table 3 in this rebuttal as below. It can be seen from Table 3 that when the parallelism capability is limited to a single A100 GPU, the proposed Omni3D approach consumes additional 46.2% of computational time compared with LiftImage3D, and still faster than ZeroNVS (CVPR'24). However, our Omni3D approach is able to reconstruct the entire omnidirectional 3D space, instead of only the forward-facing views, and also achieves better reconstruction quality. In the revised manuscript, we will include the time consumption results on a single GPU in the paper.

  
  Table 3: Time consumption on a single A100 GPU

  	MVD	Pose calc.	3DGS	Total
  ZeroNVS [33]	-	-	-	133.7 min
  ViewCrafter [51]	4.3 min	-	12.8 min	17.1 min
  LiftImage3D [3]	12.0 min	1.5 min	67.4 min	80.9 min
  Our Omni3D	21.6 min	83.9 min	12.8 min	118.3 min

We would like to thank the reviewer very much for the positive comments, and thanks a lot for the valuable questions and for pointing out the typo. We response to the reviewer's questions point-by-point as follows:

Generalizability on various kinds of scenes

In our experiments, we evaluated the proposed method on three commonly used datasets including various kinds of natural scenes, which are not limited to object-centric scenes. The generalizability of scene generation depends on training dataset of the Multi-View Diffusion (MVD). As described in Section 3.2, the MVD in our method is trained on DL3DV-10K, which is the largest scale dataset of 3D scenes as far as we know at the time point of submitting this manuscript. Employing larger-scale datasets, such as SceneSplat-49K (published in June 2025), may further enhance the generalizability of our approach. Besides, please note that the proposed Pose-View Optimization (PVO) method, which is the core contribution of this manuscript, is independent from the specific MVD, and can be generalized to various kinds of MVDs (as validated in Section 4.4).

Time for optimizing the lightweight network

Table 3 in our manuscript shows the analysis on the time consumption of each component of our Omni3D approach. The lightweight network is optimized in an online training manner during the proposed PVO scheme in our approach, which averagely takes 10.5 min on a test sample. This time consumption is less than the time used for 3DGS optimization, and therefore, does not significantly increase the overall time consumption.

Evaluation protocol

Thanks a lot for the suggestion regarding the evaluation protocol. During the proposed PVO process, the poses of generated views only slight drift in most cases, probably because of the relatively high capacity of the Multi-View Diffusion (MVD) model. In theory, our evaluation protocol, that incorporate the groundtruth views into the coordinate of the generated and optimized views to align the coordinates, should be equivalent to registering the optimized poses of generated views to the coordinate of groundtruth poses. However, our evaluation protocol has a benefit that it can be applicable to the evaluation with unposed groundtruth views.

Dependency on MASt3R

We agree to the reviewer that the proposed pipeline highly depends on a reliable pose estimation method. We utilize MASt3R in this manuscript, which performs effectively on all test datasets. Moreover, some advanced methods, e.g., MV-MASt3R, VGGT, etc, were proposed recently and have shown better performance than MASt3R, and therefore, they may potentially provide more reliable pose estimation to our proposed Omni3D approach.

Typos

Thanks a lot for pointing out the typo. We will undertake a thorough review of the entire manuscript to correct this and any other typographical errors before final submission.

Thanks to the reviewer for the positive comments on the motivation of the proposed PVO method and the performance of the videos in the supplementary material. We would like to also thank the reviewer very much for the valuable questions, which are incredibly helpful in improving the quality of our manuscript. We provide point-by-point responses to the reviewer’s comments as follows:

Novelty and contributions vs. LiftImage3D

LiftImage3D proposes a distortion-aware 3DGS method to reduce the distortion of 3DGS during the Gaussian optimization process with the generated multi-view images which suffer from content and geometric inconsistencies. Although the distortion-aware 3DGS mitigates the inconsistencies among multi-view images to a certain degree, it may be an indirect way with incremental effectiveness and low time efficiency. On the contrary, in this paper, we proposed a novel Pose-View Optimization (PVO) method to directly and jointly correct both the estimated poses and the generated contents by an iterative self-supervised learning. This way, the geometric inconsistencies and the error of pose estimations can be directly and effectively refined before the optimization of 3DGS. The experiments validated that our approach is a more effective way with better quality performance and also achieves higher time efficiency (when parallelism applies) than LiftImage3D. More importantly, thanks to the propose PVO method, we are able to achieve the challenging task of omnidirectional 3D reconstruction, instead of only reconstructing 3DGS in narrow angles in LiftImage3D.

Experimental setup

Thanks a lot for the valuable comments on the experimental setup. The reason for the lower reported numbers in our manuscript than those reported in the original papers is that we evaluate the quality of rendered images in the entire omnidirectional space, instead of only the surrounding views of the input (please see line 11 of Sec. 4.1 in the LiftImage3D paper). We will release the codes publicly with our evaluation protocol, which ensures the reproducibility of our approach, and furthermore may contribute to benchmarking the future works in this direction.

We agree with the reviewer that it is beneficial to conduct additional experiments with the settings aligned with the original paper of compared methods. Since the SOTA method LiftImage3D does not provide an evaluation protocol in their open-sourced codes, we are not able to exactly know their detailed experimental setup, such as which groundtruth views are selected in each sample. Hence, we tried our best to set up the settings where LiftImage3D performs similarly to the original numbers reported in their paper, and then evaluate our proposed Omni3D approach. The results are shown in Table 1 in this rebuttal below. It can be seen from Table 1 that our Omni3D approach still achieves superior performance under this setting. We will add the results of this experiment to the revised manuscript.

• Table 1: Evaluation on the surrounding views of input (align with LiftImage3D)

  Methods	Tanks and Temples	Mip-NeRF 360	DL3DV
  	PSNR $\uparrow\ \ \ \$ SSIM $\uparrow\ \ \$ LPIPS $\downarrow\ \ \$	PSNR $\uparrow\ \ \ \$ SSIM $\uparrow\ \ \$ LPIPS $\downarrow\ \ \$	PSNR $\uparrow\ \ \ \$ SSIM $\uparrow\ \ \$ LPIPS $\downarrow$
  ZeroNVS [33]	13.98 $\ \ \ \ \$ 0.4856 $\ \ \ \$ 0.6225	14.25 $\ \ \ \ \$ 0.2669 $\ \ \ \$ 0.8657	13.73 $\ \ \ \ \$ 0.5380 $\ \ \ \$ 0.6476
  ViewCrafter [51]	15.79 $\ \ \ \ \$ 0.5054 $\ \ \ \$ 0.5625	15.99 $\ \ \ \ \$ 0.2717 $\ \ \ \$ 0.7012	17.85 $\ \ \ \ \$ 0.6459 $\ \ \ \$ 0.3843
  LiftImage3D [3]	16.14 $\ \ \ \ \$ 0.5232 $\ \ \ \$ 0.5346	16.09 $\ \ \ \ \$ 0.2776 $\ \ \ \$ 0.5553	20.77 $\ \ \ \ \$ 0.6834 $\ \ \ \$ 0.4557
  Our Omni3D	18.32 $\ \ \ \ \$ 0.5919 $\ \ \ \$ 0.4689	16.96 $\ \ \ \ \$ 0.2880 $\ \ \ \$ 0.4807	22.02 $\ \ \ \ \$ 0.7649 $\ \ \ \$ 0.3011

Computational costs and potential optimizations

We agree with the reviewer that although we employed significant parallelism in computation, the inference time is still substantial. This is mainly because the diffusion-based generation model (i.e., MVD) and Gaussian optimization are time-consuming. Other image-to-3D methods, e.g., ViewCrafter and LiftImage3D, also suffer from the same problem. As Table 3 in our manuscript shows, the proposed PVO method consumes only ~30% of the total computational time and the proposed Omni3D achieves faster speed than LiftImage3D. Based on above analysis, a potential way to effectively reduce time consumption is training a (diffusion-based) model to generate 3D Gaussian splats straight from a single image input, bypassing the 2D (multi-view images) intermediate steps. This way, it saves the computational time for generating hundreds of multi-view images and also eliminates the optimization process for 3DGS. As such, the computational complexity is expected to be significantly reduced. This can be seen as interesting and valuable future work.

Error accumulation in iterative refinement

We thank the reviewer very much for pointing out the possible risk of error accumulation. We analyze this issue from two aspects as follows:

• Error in the iterative PVO of each pair. In the iterative PVO process of each pair, the generated views and estimated poses and intrinsics are jointly optimized by minimizing the reprojection error until convergence. This process should not suffer from error accumulation, since the error (loss) continuously reduces until nearly unchanged throughout the optimization. In Section C of the Supplementary Material, we have conducted the ablation study regarding the quality performance and the iterations of pose updates in PVO. We also show the results as follows. It can be seen from Table 2 in this rebuttal that the reconstruction quality continuously increasing along with the process of iterative PVO (the 0 to 4 iterations of pose updates). This confirms the continuous reduction of error without accumulation. Besides, the performance only negligibly increases at the 4-th iteration, hence it verifies the reasonability for setting the iterations of pose updates to 3 in our PVO method.

  
  Table 2: Ablation on iterations of pose updates in PVO, in addition to the initial pose estimation

  Iterations	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
  0 (w/o PVO)	15.56	0.5198	0.5346
  1	15.62	0.5207	0.5325
  2	15.91	0.5254	0.5296
  3	16.30	0.5308	0.5166
  4	16.33	0.5311	0.5162

• Error in the progressive paring scheme. Recall that in Section 3.3.1, we proposed a progressive pairing scheme with the window size of $N$, i.e., in each generation orbit, the first $N$ generated views undergo PVO with the reference of $\boldsymbol{x}_0$, and then the next $N$ views takes the optimized $N$-th view $\boldsymbol{\hat{x}}_N$ as reference. This progressive scheme continues until all generated views within the orbit have been processed and refined.

  This scheme is a trade-off between the risk of error accumulation and the robustness of PVO. That is, the large size of $N$ (consistently use the initial view as reference for a large range of other views) would lead to progressively larger viewpoint disparities and therefore challenge the robustness of PVO, and on the contrary, using immediately preceding optimized view (very small $N$) may introduce error accumulation. We have conducted the ablation study on this trade-off balance in Section C of the Supplementary Material, and we show the results in Table 3 in this rebuttal. It can be seen from Table 3 that utilizing views with large disparities as references (e.g., $N=24$ or $48$) can significantly impair the robustness of pose estimation and diminish the efficacy of PVO. Conversely, employing each immediately preceding optimized view, $\boldsymbol{\hat{x}}_{i-1}$ as the reference for the current view $\boldsymbol{x}_i$ ($N=1$), may inadvertently introduce error accumulation and propagation along the generation path of the orbits. Therefore, setting $N=12$ in our approach results in the best performance, indicating a reasonable balance of the trade-off issue.

  
  Table 3: Ablation on progressive pairing

  	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
  w/o PVO	15.56	0.5198	0.5346
  $N=1$	16.24	0.5305	0.5170
  $N=12$	16.30	0.5308	0.5166
  $N=24$	16.19	0.5281	0.5179
  $N=48$	15.98	0.5206	0.5254

We would like to sincerely thank the reviewer for the valuable and insightful comments, which help us a lot for improving the quality of our paper. We will follow the reviewer's suggestion to provide more rendering results of the back side of the input view in the experiments section. We also thank the reviewer for the willingness to upgrade the rating.

Thanks to the authors for further clarification. It is recommended that the authors provide more rendering results of the “back side” of reference views in the manuscript to support the claim of omnidirectional spatial rendering. Anyway, I will raise my rating to "Borderline accept".

We would like to thank the reviewer very much for the constructive suggestions and insights into the qualitative results and the ablations on progressive pairing and hyperparameters.

Qualitative results

• User study. According to the suggestion of the reviewer, we conducted a user study with 10 non-expert users, who are requested to rate the reconstructed 3D scenes with scores from 0 (poorest quality) to 10 (perfect quality). In the user study, we render the images with omnidirectional trajectories from the 3DGS generated by our Omni3D and the compared methods and send them to the users in video format, to reduce the hardware requirements for the users’ personal computers. The average ratings are shown in Table 1. It can be seen from Table 1 that our Omni3D approach has obvious superior perceptual quality performance, which aligns with the numerical results reported in the paper. We will add this experiment to the revised manuscript.

  
  Table 1: Results of user study on our Omni3D and compared methods

  Methods	Tanks and Temples$\ \ \ \ \$	Mip-NeRF 360$\ \ \ \ \$	DL3DV
  ZeroNVS [33]	1.0	1.3	0.8
  ViewCrafter [51]	4.3	4.7	7.4
  LiftImage3D [3]	5.1	4.5	5.8
  Our Omni3D	7.6	7.9	8.2

• Visual examples in videos. Besides, we have uploaded three visual examples in video format in the Supplementary Material. It can be obviously seen from the videos that the proposed Omni3D approach achieves 3D scene reconstruction in the omnidirectional space with higher visual quality and better geometry consistency than the compared approaches.

• Figure 3 in the paper. In the revised manuscript, we will follow the reviewer’s suggestion to highlight the regions with obvious differences with zoomed in views to increase readability.

Ablations on the progressive pairing strategy and hyperparameters

We have provided additional ablation studies on the progressive pairing strategy and hyperparameters in Section C in the Supplementary Material. We also show the results in the following.

• Ablation study on progressive pairing. Recall that in Section 3.3.1, we proposed a progressive pairing scheme with the window size of $N$, i.e., in each generation orbit, the first $N$ generated views undergo PVO with the reference of $\boldsymbol{x}_0$, and then the next $N$ views takes the optimized $N$-th view $\boldsymbol{\hat{x}}_N$ as reference. This progressive scheme continues until all generated views within the orbit have been processed and refined.

  Table 2 shows the ablation study on the selection of $N$ on the Tanks and Temples dataset [13]. It can be seen from Table 2 that $N=12$ results in the best performance. This outcome is likely attributable to the fact that utilizing views with large disparities as references (e.g., $N=24$ or $48$) can significantly impair the robustness of pose estimation and diminish the efficacy of PVO. Conversely, employing each immediately preceding optimized view, $\boldsymbol{\hat{x}}_{i-1}$ as the reference for the current view $\boldsymbol{x}_i$ ($N=1$), may inadvertently introduce error accumulation and propagation along the generation path of the orbits. Besides, setting $N=1$ also considerably limits parallelism.

  
  Table 2: Ablation on progressive pairing

  	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
  w/o PVO	15.56	0.5198	0.5346
  $N=1$	16.24	0.5305	0.5170
  $N=12$	16.30	0.5308	0.5166
  $N=24$	16.19	0.5281	0.5179
  $N=48$	15.98	0.5206	0.5254

• Ablation study on iterations of pose updates in PVO. Recall that in Section 3.3.2, we proposed a pairwise iterative PVO method. In the proposed PVO, given the initially estimated poses and intrinsics, we refine the generated view by minimizing the reprojection error until convergence, and then the poses and intrinsics are updated based on the refined views. This cycle is iteratively conducted until estimated poses converge. Table 3 illustrates the performance on the Tanks and Temples dataset [13] with different iterations of pose updates, in addition to the initial pose estimation. It can be seen that the performance converges at 3 iterations, i.e., optimize the views and then update poses for three time. When the iteration number comes to 4, the performance negligibly increases. This verifies the reasonability for setting the iterations to 3 in our approach.

  
  Table 3: Ablation on iterations of pose updates in PVO, in addition to the initial pose estimation

  Iterations	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
  0 (w/o PVO)	15.56	0.5198	0.5346
  1	15.62	0.5207	0.5325
  2	15.91	0.5254	0.5296
  3	16.30	0.5308	0.5166
  4	16.33	0.5311	0.5162