CAT3D: Create Anything in 3D with Multi-View Diffusion Models

Thank you for your time and careful review of our work. Below we address your questions and weaknesses mentioned:

Most techniques have been proposed.

We agree with you that CAT3D leverages existing techniques for individual components of the system. The innovation of CAT3D lies in the effective integration and scalability of those components: a multi-view diffusion model, efficient sampling strategies, and a robust 3D reconstruction pipeline, resulting in a powerful and practical system for 3D content creation given any number of input views. CAT3D decouples the generative prior from the 3D extraction process, which not only contributes to its efficiency and simplicity, but also allows for potential improvements in either component without affecting the other in future research.

Results from the 3D reconstruction / multi-view diffusion.

All results in the main text are renderings of optimized mip-NeRF models, but the Appendix also includes quantitative results of images produced by the multi-view diffusion model (Table 3), and the anonymous website in the supplementary contains qualitative samples from the multi-view diffusion model. We will clarify this in the revised draft, and are happy to include more results from the diffusion model in the main text if the reviewer deems this to be useful.

Run time on one GPU.

Thank you very much for pointing out the differences in timings. We have added a note to the Table indicating that our timings take place on 16xA100 GPUs, and are working to evaluate the timing for a single GPU. One advantage of our approach is that it can benefit from parallelism when generating novel views, while other methods built around distillation and feedforward prediction do not.

Support arbitrary conditions and targets.

Interestingly, we find our model can work on test settings different from what the model is trained for to some extent. For example, we find that doubling the target views or tripling the conditional views still works (see Figures 2 and 3 in the rebuttal PDF). We still anticipate training or fine-tuning the model on the actual test settings would lead to better results.

Full-model generalization.

This is a good question and a valid concern. We tried to verify the generalization ability of our model empirically by, e.g., testing our model on a wide variety of captured or generated images that are far out-of-distribution from what the model is trained on (see gallery.html in the supplementary). In Table 3, we observe that the model initialized from the pretrained text-to-image model performs better than the model initialized from scratch quantitatively (suggesting that the pre-trained priors are still present or useful in some capacity). To further maintain the generalization ability, a possible future direction is to jointly train the model on our datasets mixed with the text-to-image data that the model is pretrained on.

Images with different aspect ratios

While the multi-view latent diffusion model we trained only supports 512x512 images, we found the model still performed well when padding non-square images to square. However, this method often reduces resolution, so we also run our model on a square-cropped version of the inputs, and then compose the square-cropped outputs with the edges from the padded outputs to create a different aspect ratio image. We will add these details to the Appendix.

shift of the noise scheduler.

As we describe in the text (lines 143-145), we add $\log(N)$ to the log signal-to-noise ratio, where $N$ is the number of target images. More concretely, assuming for the base text-to-image model each time step $t$ is mapped to a log signal-to-noise ratio $\log \lambda_t$; then for our model, $t$ is mapped to $\log \lambda_t - \log(N)$. In settings with a mixed number of target views (e.g. 5 and 7 target views), we pick $N$ to be the smallest number of target views (i.e., 5). The logic is similar to [67] where the noise schedule is shifted for higher resolution generation (see Eqn (4) in [67]).

Drop the text embedding.

It is removed completely for model simplicity. While architecture changes like this can impact the stability of fine-tuning, we found that a learning rate warmup is sufficient to mitigate potential instabilities.

Mask and SD inpainting.

The mask we used is to specify conditioning vs. target images. It is constant per image (i.e., not spatially varying), whereas the mask in SD inpainting is used for indicating unknown pixels within an image, which may not align with our task.

3DGS.

As far as we know, all radiance field models including 3DGS and NeRFs are data hungry and 3DGS doesn’t necessarily require fewer captured (or generated) images. It’s worth noting that generating novel views is not the main computational bottleneck in the whole system. For example, in the single-image-to-3D setting, it takes 5 seconds to generate 80 views while it takes 55 seconds to run 3D scene extraction.

Thanks for the reply. Please find our response below:

It is interesting to see that CAT3D can be generalized to various condition and target views. Is this capability attributable to that no positional encoding is used in CAT3D? To my knowledge, video models with positional encoding fail to be extended to inference with arbitrary lengths.

While CAT3D does not use an embedding of time (e.g. positional encoding of the time index for each frame), it does use an embedding of camera pose (via the raymap). Unlike time embeddings where during training you only see a small but finite number of time embeddings, we see a much larger continuous set of pose embeddings which may aid in generalization. In video and language models, one still can get generalization depending on how you structure and interpolate the time embeddings (see e.g. [1, 2]).

[1] Chen, Shouyuan, et al. "Extending context window of large language models via positional interpolation."

[2] Kazemnejad, Amirhossein, et al. "The impact of positional encoding on length generalization in transformers."

For the shift of the noise scheduler, I think the conclusion of using log(N) to shift the scheduler (N is the number of target images) is not convincing. Because when CAT3D is trained with 5 and 7 target views, it is just evaluated with N=5 in the scheduler. This only confirms that using large N helps multi-view training. The specific relationship between the number of views and the scheduler has not been thoroughly assessed in this paper, leaving the log(N) shifting here somewhat ambiguous. Furthermore, it's important to note that multi-view images should not simply be equated with high-resolution images, given that the overlap among multi-view images can vary greatly and is stochastic.

It is common practice in training video diffusion models to shift the noise schedule based on the number of target frames, to compensate for the amount of redundant information which may exist across pixels. This is similar to what is done when increasing a models spatial resolution. In reality, the amount of redundant information in a video is a function of the amount of camera and scene motion, i.e., how many pixels are similar or the same across frames, but the number of frames serves as a reasonable approximation. In practice, our model is adapted from a single image diffusion model, and therefore modifying the noise schedule from that base model is necessary, since supervising and predicting multiple frames strictly has more redundant information. And indeed, in practice, we found shifting noise (while training and sampling) improves the quality of results. It is true that we don’t dynamically adjust the schedule based on the number of target frames, we just use the same shift of log(5) for both 5 and 7 target frames, but we found that the difference between that shift in practice is small (e.g. the average LPIPS on the in-domain diffusion samples is 0.235 for shifting log(5) vs. 0.240 for shifting log(7)). The precise formula for log(N), while not explicitly defined in prior work, was something we derived approximately from the numerical noise schedule information provided in the video model literature [3,4]. We will add this detail to the paper. Future work on multi-view models may want to instead condition the model on logSNR and use different shifts when training and sampling with different numbers of frames.

[3] Blattmann, Andreas, et al. "Align your latents: High-resolution video synthesis with latent diffusion models."

[4] Blattmann, Andreas, et al. "Stable video diffusion: Scaling latent video diffusion models to large datasets."

I think that providing multi-view generation from diffusion before Nerf learning is very important to confirm the capacity limit of stablediffusion. Even if the model itself cannot be publicly released, sharing samples—such as 80 views generated from Stable Diffusion before NeRF optimization (in the gallery.html)—would greatly benefit the community. This would enable researchers to reproduce performance based on these multi-view images in their own NeRF optimizations.

Great point, we will include more samples alongside our NeRF results in the project page (and will also need to combine those videos with a serialized form of the camera pose trajectories).

Thank you for your time and feedback on our work.

larger diffusion models can boost the performance

We certainly expect that larger models will lead to improved performance and can generate more consistent novel views. One relevant piece of evidence: we experimented with different model variants (Table 3) and found that increasing the amount of computation (e.g., by adding 3D attention operations at every resolution layer of the UNet) can improve performance (albeit at the expense of efficiency in training and sampling). We chose one of the smaller models as our showcase result with the intention of striking a balance of efficiency and performance.

Thank you for the careful reading and kind words. Below we address your questions and weaknesses mentioned:

performance with increasing sparsity and decreasing pose accuracy

In terms of performance while varying the number of input views with accurate camera poses, Table 1 includes qualitative results for 3, 6, and 9 view sparse reconstruction. The CAT3D model was not trained with missing or noisy poses, but some of our training data (CO3D) has imperfect pose which leads to some small amounts of robustness. To verify this, we conducted an experiment by perturbing the camera rotations of the input conditioning 3 views by a certain number of degrees, and measuring the error between the generated and ground truth target images. See Figure 1 in the rebuttal PDF for average PSNR across varying amounts of rotation perturbation on the MipNerf-360 dataset. Our model can handle small rotation perturbation. Fine-tuning the model with more perturbations on camera pose should lead to improved performance in this setting, which would be an interesting direction for future work.

manually designed camera trajectories

We agree with the reviewer that jointly learning a trajectory model or inferring trajectories given scene content is an exciting direction for future work. Empirically, we found a handful of simple heuristic trajectories worked well for our experiments, but ideally the trajectories should cover the scene without being placed inside of objects or walls as mentioned in line 167-169.

overstates its accomplishments

We are happy to revise language to better reflect limitations. The “state-of-the-art performance across nearly all settings” is referring to Table 1, where our method indeed exhibits stronger performance than all prior work. Were there any other passages of text that you would like us to alter? We discuss several limitations in the discussion section, and can expand for the camera ready (especially discussing the challenges we mentioned above regarding trajectory selection).

reproducibility

In the Appendix, we aimed to provide all details necessary to reproduce CAT3D on top of an open-source latent diffusion model. If there are any additional details that the reviewer thinks are missing, we are happy to include them in an updated draft.

constant camera intrinsics, other limitations

It’s worth noting that our model does not rely on constant camera intrinsics during training. This is an artifact of our current training dataset where the camera intrinsics are approximately fixed for each scene. If we were to capture and include additional training data with camera intrinsics varying within a scene, we expect our model would be able to perform camera intrinsic manipulation. As far as limitations in expressiveness of the base model, one notable example is that our model performs poorly on human faces, as the base model was not trained on much human data. A showcase of the limitation of producing a small number of output views can be seen in the Supplementary website, where the generated spin video is clearly not perfectly 3D consistent. The need for manual camera trajectories is shown in Fig. 8, by the fact that it was necessary to create different types of trajectories based on characteristics of different datasets. We will further emphasize these points in the paper text.

Thank you for your time and careful review of our work. Below we address your questions and weaknesses mentioned:

cumbersome to generate a large number of viewpoints

We agree that jointly generating all target frames from the multi-view diffusion model would enable more consistent samples. However, simultaneously generating a large number of frames with video diffusion-like architectures is still an active area of research, with many SOTA text-to-video models using autoregressive generation in time to produce a larger number of frames (while losing consistency due to limited context length). By splitting the generation into anchors and then independent blocks of generation at different camera locations, we can efficiently generate a large number of frames in parallel. Building efficient architectures that can support the joint generation of a larger number of frames is an exciting direction for future research.

trajectory influences quality

Our multi-view model can generalize to arbitrary camera trajectories, and we show results for other input capture types, e.g., forward-facing scenes like LLFF which contain several images pointed towards the same scene target. It is true that the model performance varies for different camera trajectories, and this is likely due to some of the biases inherited from our relatively limited training dataset. Mixing in training data with more diverse camera distributions could be an interesting future direction.

only use anchor views for reconstruction

Our model was only trained to produce at most 8 views total (conditioning + target), and the 3D reconstruction methods we use (i.e., Zip-NeRF) are seldom able to reconstruct plausible geometry from 8 views (as a reference, see results with Zip-NeRF and 9 real views which are strictly better than 8 generated views in Table 1). Including additional frames through AR generation is critical to yield the high-quality reconstruction and NVS results we present in the paper (see also Fig. 6 that compares 80 vs. 720 frames).

densify views through video interpolation

Using a video interpolation model to generate frames is an interesting idea. However, it is not clear how to use such a model to incorporate more than 2 frames, and how to leverage the resulting frames in 3D reconstruction without explicit camera control. It’s also not clear whether this approach would be more efficient or would yield better quality results: if using an architecture similar to ours, the compute cost would be similar, and one would also have to estimate camera poses for the resulting frames (which adds an additional computational expense that our pose-conditioned multi-view diffusion models do not incur).