Consistent123: Improve Consistency for One Image to 3D Object Synthesis

Consistent123 may produce smooth texture in synthesized views for some cases. Have the authors identified potential strategies or modifications to address this shortcoming?

It is an important but challenging task for NVS methods to predict novel views with finer details. To further solve the smooth texture problem, the following techniques may help:

• Lower the PCFG guidance scale and increase inference steps. For all cases in the paper, we use the fixed schedule for progressive classifier-free guidance without per-case tuning to perform fair comparisons. For some specific cases as mentioned above, the starting and ending points of progressive classifier-free guidance can be tuned (lower for better texture) to alleviate the smoothness problem. Moreover, using larger DDIM sampling steps (e.g., 200) also helps to improve the synthesized texture.
• Improve the quality of training data. As mentioned in other concurrent works (e.g., Instant3D), there is a lot of low-quality training data in Objaverse for novel view synthesis, which may result in the smoothness of the synthesized texture. A careful dataset pruning strategy may further alleviate this problem.

What is the performance with a random number of views as well during training?

It is interesting to use a random number of views during training. We implement it by selecting a random number of views dynamically for each batch, which ranges from 8 views to 16 views (since we only render 18 views for each object). We evaluate the consistency score on GSO as shown in the following table. It shows that training at a fixed number of views is already effective, and training at a random view may degrade the performance. This is mainly because as the number of training views increases, the difference among views is smaller. It becomes easier for cross-view attention to predict the novel views, thus harming the robustness of the learned model and may lead to the overfitting issue.

Can Consistent123 be trained from scratch?

For a fair comparison, we train Consistent123 from scratch on the same renderings as Zero123. We show the consistency score evaluated on GSO in the following table:

In our experiments, we found that it is challenging to simultaneously maintain the synthesized view quality and consistency. Due to the limited scale of the existing 3D dataset, the model is easy to be trapped in overfitting when directly trained from scratch. To reduce the overfitting, it is reasonable to enable the model with novel view synthesis ability by training like Zero123, then improve the view consistency as Consistent123. Moreover, due to the slow training process (over 7 days for zero123), there is no further time to explore a more effective optimization strategy. A better training strategy (e.g., Efficient 3DiM) may make it possible to train Consistent123 from scratch.

It makes the paper stronger if you can show better 3D reconstruction results (e.g. use Magic123).

Better 3D reconstruction results using Magic123 are shown in the video supplementary (magic123.mp4) and Figure 17.

Minor suggestions mentioned in the review.

We thank you for all the kind suggestions and will consider them in the revised paper.

The attention mechanisms in Consistent123 are borrowed from the video diffusion model and Cao et al.

Although the high-level idea of attention mechanisms is inspired by video generation, they still remain unexplored in 3D generation and can not be directly applied for novel view synthesis. We make several key modifications to adapt these components for better view consistency.

• Shared self-attention is an inference-only strategy in Cao et al., but we adapt it for both training and inference in Consistent123 to further enhance the alignment between novel views and input views.

• To attend to the input view latent, we fix the first pose of the view sequence to be the same as the input image, thus avoiding additional inverse operations and making the training process more efficient.

• We apply shared self-attention in all the diffusion timesteps, which only part of timesteps are used in Cao et al. We also conduct an ablation study in Table 3(b) to decide the position to apply for shared self-attention.

• Finally, other important components of Consistent123 also significantly contribute to the view consistency, including the flexible number and pose at inference (while the frame length in video generation is typically fixed) and progressive classifier-free guidance.

How will the shared self-attention act if there exists a large pose difference between the input view and the synthesized view?

Intuitively, with a larger pose difference between the input view and synthesized view, the shared self-attention would become less activated. For these poses, the non-shared self-attention (i.e., U-Net Encoder part) and cross-view attention layers can serve as complementary and the performance can be maintained.

For a better understanding of how shared self-attention works, we conducted an ablation study. For the poses whose azimuth angle with input view is larger than 90°, we set the shared self-attention to vanilla self-attention by force. We evaluate the consistency score on GSO in the following table. The table shows that the constrained version of shared self-attention achieves similar performance as the vanilla self-attention. This indicates that the performance will degrade when disabling shared self-attention when a large pose difference exists, which makes the model harder to learn when attending keys and values from different views (half of views attend to input view, half attend to themselves).

The authors only picked up 100 objects for Table 1, which seems far from enough.

We also agree that more evaluation objects help to reduce the randomness of quantitative results. However, the evaluation of the consistency score is quite time-consuming (constructing a Nerf per object), preventing us from using a large number of evaluation objects. Moreover, our evaluation settings is mainly borrow from Zero123, which only use 20 objects in GSO (as well as most concurrent works).

To demonstrate more convincing results, we extend the number of evaluation objects to 1k in the following table, which shows that the proposed model consistently outperforms baselines by a large margin.

Can Consistent123 be compared with recent work magic123, zero123 xl, sync-dreamer?

• Magic123 is an SDS-based method for image-to-3D based on Zero123, which can be regarded as a downstream application of Consistent123. Therefore, we show the experiment results of Magic123 based on Consistent123 rather than directly comparing with it. Please refer to the video supplementary (magic123.mp4) and Figure 17 for more details.

• Both Zero123 XL and Syncdreamer are used for novel view synthesis, so we directly compare them in synthesized multi-views. Note that Syncdreamer is constrained at fixed poses, thus we only compare with it qualitatively. Please refer to supplementary Figure 12 for a qualitative comparison with these baselines.

Besides the qualitative comparison in Figure 17 and Figure 12, we also evaluate the consistency score on GSO dataset for the quantitative comparison.

Why the reconstructed results are no better than the published methods in threestudio?

The reconstruction results of dreamfusion are highly dependent on hyper-parameter tuning. For a fair comparison, all the dreamfusion experiments shown in the paper follow the same hyper-parameter settings without further per-case tuning (and note that all these experiments are also conducted under the implementation of threestudio). As for some failure textures, we could use stronger optimization-based reconstruction methods (e.g., Magic123) to improve the quality. To further demonstrate the effectiveness of Consistent123, we show better 3D reconstruction results using Magic123 in the video supplementary (magic123.mp4) and Figure 17.

Are there any thoughts on further improving the consistency so the quality gap between diffused output and render-view can be minimized?

• First, introducing geometry-aware conditioning may help. For instance, spatial volume or epipolar attention can be used to further enhance the consistency. However, such architecture may reduce the quality and flexibility of synthesized views (e.g., the number and pose of view must be fixed at inference).
• Second, a better data filtering strategy may help. There are lot of low-quality data in our training dataset Objaverse as indicated by other concurrent methods (e.g., Instant3D). Training on the full dataset of Objaverse may result in oversmoothed textures and increase the difficulty of being consistent.

How it generalizes towards more sophisticated images?

More sophisticated cases are shown in supplementary Figure 13, indicating that Consistent123 can be generalized to sophisticated images while preserving good consistency.

Please also provide some failure cases for a better understanding of the limitations.

Failure cases are provided in supplementary Figure 14. The main problem falls on the latent diffusion decoder, which struggles to reconstruct some complicated pattern of input images (e.g., text, thin objects). A direct solution is to use a stronger latent decoder like the Dalle3 decoder (OpenAI). Using a cascaded diffusion model may be another solution.

V_c should be after softmax in Eq. 5

Thanks for your careful reading and pointing out the typo. We will fix it in the revised paper.

Why is the name "shared self-attention"? It seems a cross-attention from the novel views to the input view.

The shared self-attention is similar to the cross-attention between the novel view and input view, but they have several key differences:

• It reuses the pre-trained weight of vanilla self-attention layers in Zero123.
• It is non-trainable but only changes the calculation of self-attention.

Therefore, we name it “shared self-attention” to distinguish these differences.

Only qualitative results were presented for image-to-3D tasks.

Following the settings of Zero123, we calculate the Chamfer Distance and Volume IoU of image-to-3D reconstruction based on SDS loss. Note that these metrics only evaluate the geometry quality, ignoring the texture improvement of Consistent123 (which is more significant compared with geometry improvement). Please refer to the updated video supplementary (dreamfusion.mp4) for more qualitative comparisons.

How many views were used in the NeuS experiment?

3D reconstruction methods like Nerf and Neus typically require a large number of views. Therefore, we use 64 views for all Neus experiments.

No texture on the spray bottle in Figure 7?

In the spray bottle case, we deliberately remove the texture from the object to show the synthesized geometry of Neus, which more concentrates on geometry reconstruction. We also add the textured version of this case into the supplementary Figure 15.

Is the Super Mario a failure case since the object has flattened?

Super Mario is not a failure case. As a Lego toy, it is “flat” by design (feel free to google for more Lego Mario images). This case also shows that the object geometry can be constructed more reasonably with improved consistency.