From an Image to a Scene: Learning to Imagine the World from a Million 360° Videos

We thank the reviewer for their thorough review. We are happy that they find our dataset crucial for modern data-driven methods and find our correspondence search pipeline to be innovative. We have addressed comments and questions below and are happy to engage in further discussion. Note that we could not address all points within the character limit and will address the remaining questions during the discussion phase in the official comments.

The camera trajectory (x, y, z) is controlled by the camera operator, thereby limiting the flexibility to select diverse viewpoints from different positions.

We found that the increased scale and diversity of camera views in the 360-1M dataset compensates for this limitation as shown qualitatively and quantitatively in our results. The 1 million in-the-wild videos provides camera movements with extremely diverse trajectories, for example people sky-diving, climbing mountains, and walking many blocks through cities. Though each video has a fixed (x,y,z), the diversity of viewpoints across the data gives our model flexibility in generating from different viewpoints compared to models trained on previous datasets such as ObjaVerse and Co3D.

The quality of the data is unknown and can be unstable across different data points. For instance, the author acknowledges that it is infeasible to guarantee the uniqueness of video content.

While it’s true that we do not crawl every frame, we contend that the quality of the model is a strong indicator of the data quality. In our case we show that the scale and the quality of the 360-1M leads to improved model performance compared to previous datasets.

Although the work claims that 1 fps is sufficient in lines 126-128, this claim lacks experimental support.

We initially ran small-scale proxy experiments at various FPS before scaling to the full dataset. Performing such experiments at 1 million scale was computationally infeasible. We found that below 1 FPS there was no performance loss. We’ve also included the computational cost difference. Below we’ve included the results of our experiments on DTU and have added this to the appendix. For this experiment we kept the window size for correspondence searching fixed and used 50k videos from 360-1M.

Larger dataset may need more training time and computational resource.

We agree that considering computational cost is important. The task of finding corresponding pairs in video and estimating their relative pose was the main computational cost for us. By providing the image pairs and relative pose, other researchers can benefit from large-scale multi-view datasets without the computational burden of parsing a million videos.

One of the most challenging aspects is the impact of motion over time. This aspect lacks analysis. For instance, the importance of the hyperparameter in Eq. 3 is unknown.

We’ve included an ablation below over the hyperparameter lambda in Eq. 3 and have added discussion about choosing lambda to the appendix. We found that if the lambda value was too small, then the model would mask all objects in the scene and the reconstruction quality was poor. If the lambda value was too large, then smearing would occur near dynamic objects as the model was forced to predict the motion of the object.

I wonder whether the motion masking has a negative impact on synthesizing dynamic objects when they are static.

Qualitatively we found that motion masking helped in reconstructing dynamic objects even when they are static such as parked cars. We hypothesize that due to the scale of the dataset, objects which can be both dynamic and static are seen multiple times in both settings.

As mentioned, there are some large-scale real-world datasets, such as CO3D, albeit with limited diversity. A related work, OmniObject3D, should be mentioned.

Thanks, we have added OmniObject3D to the related works. We’ve changed line 5-6 to be more concrete. Co3D is impressive and relatively large-scale, but orders of magnitude smaller than our proposed dataset (20 thousand vs 1 million videos).

In lines 8-12, the phrase "...diverse views" may be too broad.

Often datasets such as DTU, MVImageNet and Co3D have images taken from positions close in space around an object (1-5 meters apart). By diverse views we meant varying differences in camera pose (up to 50 meters apart and 360 degree rotation) and from various locations, not only around objects or landmarks. We will make it more precise what we mean by diverse viewpoints in the abstract and throughout the paper.

I hope the author can discuss the difference between a multiview dataset that can capture images from different locations at the same timestamp and one that can only change the camera orientation.

One limitation of a multi-view video dataset is that camera poses at different points in the (x,y,z) trajectory will have different timestamps. Learning novel view synthesis from dynamic scenes is still an open problem which temporal masking addresses to an extent, but there is still significant progress to be made. Current novel view synthesis (ZeroNVS, MegaScene) works seem capable of training on datasets where time and spatial location change simultaneously. Datasets which capture images from different locations at the same timestamp are ideal, but we believe this would be extremely difficult to manually collect at scale, and harnessing existing video data is an appealing alternative.

I just realized that my discussion is not public to the authors. Anyway, I copy and paste it here...

I have an open question regarding data maintenance. The dataset is large-scale because it includes existing data accumulated over time from YouTube, in addition to the data collected through your pipeline. My concern is whether the data growth rate might slow down in the future. Perhaps the authors could consider plotting the total amount of data over time or by year, so that readers can understand the data growth rate. This could also help the authors estimate the expected time needed to collect a new large amount of data and update the dataset to a second version.

We thank the reviewer for their thorough review. We are glad they appreciated the diversity and scale of our proposed dataset and the new capabilities of our resulting model. We provide responses to their comments and questions below and are happy to engage in further discussion.

A few things that would have been useful to show are 1) The generated images along the trajectories in fig. 4 in the form of both images and videos. Only the final 3d reconstruction by dust3r and not the intermediately generated images by ODIN are shown, 2) Images of the renderings of different methods in table 1 and 2, and 3) Generated images and the corresponding 3d reconstructions on Google Scanned Objects, so qualitative comparison corresponding to table 3 for all methods.

Great suggestion, we’ve included generated videos in the one page pdf for better visualization and have added 3D construction examples and further image examples to the paper.

There are a few missing references and as a result too strong claims in the paper ZeroNVS also constructs a 3d scene by training a NeRF with Score distillation sampling. Due to this I think e.g. the claim on line 48 that ODIN is the first to reasonably reconstruct a 3d scene from a single image is too strong.

Thanks for the suggestion. We’ve removed this sentence from the introduction and softened the language overall. Additionally we’ve added the missing references you’ve pointed us to.

I wonder how important the use of dust3r is for the 3d reconstruction. Is the confidence measure by dust3r used, and if so, how? I would imagine that it would be useful to get a consistent 3d reconstruction even if the views generated by ODIN are not consistent. By consistent I mean both that the views might not adhere to a specific camera model well enough for traditional sfm method to work well, or that the different views contain contradicting contents. Is that why there are some holes in the reconstructions in fig. 4 (e.g. the middle part in the leftmost 3d model of the cathedral), because it looks from the positions of the cameras that they should not be empty? What would happen if colmap was used on the generated images, would it still result in a reasonable 3d model?

For the 3D reconstruction portion colmap performs qualitatively the same. You’re correct that the hole in fig. 4 is due to low confidence from the Dust3R model in that region. In general, as long as enough images are generated, experimentally, we found that colmap can perform similarly. To summarize, Dust3r paired with the graphical search is necessary for finding corresponding frames while constructing the dataset, but at inference the model is agnostic to the method for 3D reconstruction.

Did you notice any failure cases related to just conditioning on the previously generated image for the 3d generation? The camera motions in fig. 4 are fairly simple with little rotations and objects do not disappear and reappear, but if there would be more rotations I would imagine that it needs to be more carefully handled, e.g. as in the paper “WonderJourney: Going from Anywhere to Everywhere” (CVPR 2024).

Yes, as the reviewer mentioned the most common failure case is if an object comes in and out of occlusion it can be inconsistently generated by the model. With long-range camera trajectories more parts of the scene will come in and out of view which can lead to inconsistent generations, a typical failure mode of conditional NVS models. We found that the SDS anchoring introduced by ZeroNVS was effective in improving this consistency.

For 3d reconstruction, many methods (e.g. zeroNVS) use a NeRF model and score distillation sampling instead of direct 3d reconstruction from multiple generated images like it is done in this paper. Were any experiments like that performed?

In general the dataset and model are agnostic to the downstream 3D reconstruction method. We originally chose direct 3D reconstruction so that the geometry of the scene could be inferred in real time and we found it better for large-scale scenes and inferring geometry compared to distilling into a NeRF model.

We thank the reviewer for their thorough review. We are glad they found our algorithm for transforming 360° video into multi-view data to be especially significant and see its potential for scaling 3D datasets. We provide responses to their comments and questions below and would be happy to engage in further discussion.

The paper relies heavily on the 360-degree videos collected from YouTube, which may contain varying video qualities and resolutions. The described preprocessing steps does not fully account for this diversity in video quality, which may affect the quality of the extracted multi-view data negatively.

We experimented with filtering techniques such as CLIP filtering, where we fine-tuned a CLIP model to classify low quality frames from high-quality frames. For the fine-tuning data we hand-labeled 10k example frames. We found that the confidence scoring used to find image-pairs was sufficient for removing videos that were low in quality such as blank frames, corrupted or blurry frames, and static videos. Intuitively, frames that are low in quality are unlikely to be paired since relative pose and depth are difficult to estimate. For resolution we filtered out videos below a resolution of 2K. We also provide the available resolutions, view count, etc. for each video in the meta-data so users can filter videos based on various criteria.

It is mentioned in line 220 that optimizing Equation 2 directly results in a degenerate solution and Equation 3 is introduced to mitigate this problem. However, there is no ablation to demonstrate this. It may help make the paper more comprehensive.

Thanks for the suggestion. We’ve included an ablation below over the hyperparameter lambda in Eq. 3 and have added discussion about choosing lambda to the appendix. We found that if the lambda value was too small, then the model would mask all objects in the scene and the reconstruction quality was poor. If the lambda value was too large, then smearing would occur near dynamic objects as the model was forced to predict the motion of the object.

In Tables 2 and 3, it is shown that ODIN outperforms other approaches on the LPIPS and PSNR metrics but not the SSIM metric consistently. However, there is no discussion of possible reasons for this discrepancy.

On line 247 we briefly note that previous works [1,2,3] have shown that SSIM and PSNR are not accurately correlated with better novel view synthesis. For example, in [3] figure 7 they show that a frame with uniformly grey pixels can outperform far more reasonable generations. In general, PSNR is sensitive to the low level pixel statistics which is tangential to the content of the image. We will better clarify this point in section 6.1.

1. Zero-Shot 360-Degree View Synthesis from a Single Image
2. Generative novel view synthesis with 3d-aware diffusion models
3. Nerdi: Single-view nerf synthesis with language-guided diffusion as general image priors

Thank you very much for your comprehensive efforts in addressing my concerns. In particular, I find your response on how and why the long-range image pairs are used particularly helpful. I also appreciate your efforts on the additional ablation experiments, given the time and computational constraints on your end as well as empirically quantifying the effects of using 1 FPS in other responses. After reading all of the reviewers' feedback as well as the corresponding responses from the authors, I will retain my original rating.

We thank the reviewer for their thorough review. We are glad they find our model demonstrates important capabilities for large-scale scene understanding, the idea of using 360° video is interesting, and our dataset will be useful for future researchers. We provide responses to their comments and questions below and are happy to discuss further.

Besides its major contribution as a large-scale and open-world training dataset compared to ZeroNVS, the model primarily aligns with Zero-1-to3, which somewhat limits its technical contributions.

We contend that the technical contribution of this work goes significantly beyond a new dataset. We've enumerated our contributions below.

• We present a new technique capable of finding image-pairs from in-the-wild 360° videos. Corresponding image pairs are crucial for training novel view synthesis (NVS) models and current approaches such as colmap and hloc are not capable of extracting such pairs from large-scale video.
• We introduce temporal masking, a method which enables training on dynamic scenes which has been a limitation of novel view synthesis works.
• We provide a new dataset consisting of over 1 million 360° videos, multi-view image pairs from the videos with their relative camera pose, and meta data for each video.
• Our model demonstrates novel capabilities in generating full, real-world scenes along long-range trajectories consisting of both rotation and translation compared to Zero1-to-3 which is limited to synthetic objects and camera rotations about the object.

Could you provide a reason why Zero-1-to-3 is not compared on the DTU benchmark, which is suitable for Zero-1-to-3 given its focus on single objects placed on tabletops?

We originally omitted Zero-1-to-3 since it is very similar to ZeroNVS and strictly worse in performance. We have added it back to Table 1 for completeness.

I wonder how Zero-1-to-3 is applied in Fig. 3, as its camera pose involves both elevation and azimuth angles, which may differ from the video’s camera poses. Could the authors clarify how they calculate the camera pose?

It’s true that Zero1-to-3 is limited to specific relative camera poses. We fit the azimuth, elevation angles, and radius of rotation to minimize L2 distance between the Zero1-to-3 camera pose and the target pose. Due to this limitation of Zero-1-to-3, we primarily focused our comparisons on ZeroNVS as it is designed for unconstrained camera movement, real-world scenes, and has a similar architecture.

This paper presents quantitative results only for experiments on DTU and MipNeRF360 (Tables 1 and 2). Including visual comparisons could help understanding.

We’ve included videos of our generation for better visual understanding in our one page pdf. Additionally we will add qualitative comparison for MipNeRF360 and DTU to the main paper.

Why does Table 4 in the supplementary materials show a higher Chamfer Distance compared to ZeroNVS?

Thanks for catching the type in the appendix. That should be a comparison with Zero-1-to-3. We’ve corrected the error.

I wonder if there are potential methods to expedite this process, such as initializing from a pretrained Zero-1-to-3 model?

We tried initializing from Zero-1-to-3 and ZeroNVS. We found that it led to slightly faster convergence, about ~10% fewer training iterations. The performance at convergence was the same when starting from both pretrained models.

Could the authors elaborate on how long-range image pairs contribute to the model?

The long-range image pairs in the training data are crucial for generating long-range trajectories and modeling large-scale scenes with our model such as in figure 1l. Models are limited to types of camera movements in their training distribution. We observe this with zero1-to-3 being limited to only rotating the camera around objects.

It’s challenging to distinguish correspondence in Fig. 9 from the appendix, as the images appear not to overlap. While such pairs may assist the model in extrapolating unseen regions, could they potentially hinder learning the viewpoint changes of observable content?

The trees and the steps in the background are shared by both images and Fig. 9 shows that Dust3r is capable of accurately finding the relative camera poses. Empirically we did not find that extrapolating to unseen regions decreased performance of the model for observable content.

The visualization in Fig. 3 could be enhanced by including relative changes in viewing angles, as it’s currently difficult for humans to find correspondence between the output and input view images.

Thanks for the suggestion, we will add this to figure 3 to improve the visualization.