TOSS: High-quality Text-guided Novel View Synthesis from a Single Image

Thank you very much for the insightful feedbacks. We’ve provided more results according to your comments in the new revision.

1. Over-claim multiview-consistency: We agree that we do not introduce explicit modules to encourage multiview-consistency like in SyncDreamer. However, we claim that text prompts improve the multiview-consistency with semantic constraints. We support the conclusion with both qualitative and quantitative comparisons. For qualitative comparison, we refer to the minion in Figure 16 as an example. As we provide a front view to be the input condition image, Zero123 generates severe multi-view inconsistent results (minion with/without backpack), even under similar camera poses. However, our method significantly improves multi-view consistency with semantic constraint from text prompt "minion without backpack". For quantitative comparison, we evaluate 3D consistency scores in Table 2, which shows that our method indeed improves the multiview-consistency. In comparison to SyncDreamer, our improvement on multi-view consistency is not limited by azimuth or elevation angles. We also agree that it's extremely challenging to achieve 100% consistency with single-view novel view synthesis as eventually, this problem is severely under-constrained. We will revise the manuscript to avoid over-calim on being multivew-consistent but rather with improvements on multivew-consistency. We'll also cite more works on explicit multiview-consistency improvement for readers' reference.
2. Emphasize the controllability: Thanks for pointing this out. We now provide more results on controllability in Figure 15.
3. Replace SD model for better performance: We replace the initilazation weights of UNet and finetune it with higher resolution images. The other parameters such VAE encoder and decoder remain frozen.

Thank you very much for the insightful questions and suggestions. We’ve provided more results according to your comments in the new revision.

1. Time/Memory cost: As suggested, we analyze time/memory cost per module. (1) Token-level attention maintains the same time complexity but requires slightly more memory consumption for the VAE embeddings of the condition image. (2) Text prompt attention raises the attention token number from CLIP embedding's 1 to 77. In experiments, we evaluate time/memory cost on Nvidia A100 (80G) GPU with batchsize 128. During mixed-precision training, our method costs 2.13s and 52G per batch, while Zero123 costs 1.61s and 47G. It's worth noting that our method is sufficiently stable, allowing us to employ half-precision training without any performance degradation, while Zero123 does not exhibit the same stability. Therefore time/memory cost of out method could further decrease to 1.21s and 45G per batch with half-precision training, which is more efficient than Zero123.
2. Effectiveness of expert denoisers: For completeness, we follow the reviewer's suggestion and supplement the results of expert denoisers under the 160M setting in Table 1, showing quantitatively better results of expert denoises across different training budgets.
3. More comparative experiments: Thank you for the suggestion, we provide the results of DietNerf and Image Variation on GSO datasets for more comparative experiments in Table 1.
4. Section 3.2.1: Thanks a lot for pointing this out. We have removed the title of section 3.2.1 in the revision.
5. "inference w/o text" and "w/ expert denoisers" experiments for 160M setting: We've now included results under the 160M setting in Table 1.

Thank you very much for the insightful questions and suggestions. We’ve provided more results according to your comments in the new revision.

1. Plausibility results: In Figure 14 we've provided more qualitative results emphasizing plausibility improvements via text prompts.

2. Camera pose details for fair comparison: For all the NVS results, we randomly sample the camera pose similar to how Zero123 renders the Objaverse objects for training. That means camera poses are not strictly limited in certain relative azimuth or elevation angles, which is quite different from Sycdreamer[1]. Furthermore, we use the same sets of camera poses for all methods for fair comparison. [1]Liu, Yuan, et al. "SyncDreamer: Generating Multiview-consistent Images from a Single-view Image." arXiv preprint arXiv:2309.03453 (2023).

3. More 3D reconstruction baselines: We've further included the comparison to Magic123 in Table 3. Unfortunately, the code of consistent123 is not open-sourced yet, we plan to add its result and comparisons to more methods upon availability. As Magic123 utilizes Stable Diffusion and Zero123 separately without joint training on image and prompts, the quality of its results is highly reliant on the quality of text prompts. Therefore we use the same Blip2-generated captions across all methods for fair comparison.

4. Ablation on cross attention process: We supplement results of three ablation experimets on cross attention process below: (1) reverse the order of token-level attention and text prompt attention; (2) inject the camera pose information to text prompt attention instead of to token-level attention; (3) replace the token-level attention with gated attention and train it as an adapter with other parameters freezed. We can conclude from results of (1)(2) that different order of attention modules or different camera pose injecting choices lead to decrease in performance. Besides, experiment (3) reveals that finetuning significantly enhances model performance compared to adapter training.

   Method	PSNR	SSIM
   (1) Reverse the order of attention	16.63	0.8328
   (2) Inject the camera pose with text prompts	16.84	0.8275
   (3) Gated attention	13.51	0.7105
   TOSS	16.95	0.8369

5. More detailed elaborations on expert denoisers: We discover that during sampling, novel view pose is dedicated at large timesteps and details are further refined at small timesteps. The observation is similar to the observation in diffusion-based 2D image generation that structure forms during early sampling steps and later sampling steps are responsible for image details. Therefore, we draw inspiration from ensemble of expert denoisers [1] to train two expert denoisers that are specialized for denoising at different timestep intervals.
   [1] Balaji, Yogesh, et al. “ediffi: Text-to-image diffusion models with an ensemble of expert denoisers”.

Thank you very much for the insightful questions. We’ve provided more results according to your comments in the new revision.

1. Contribution of text prompt: As requested, we've provided additional visual results showing contributions of the introduced modules in Figure 13. We can see that even though "Token-level attention" indeed improves the model the most in terms of metrics, the "Text prompt" still significantly enhances the quality of the generated novel views visually, including more plausible and intricate details (for example the lights on the top of the police car). We would like to point out that the contribution of text prompts exceeds what the metrics demonstrate. The reviewer's intriguing question precisely aligns with the observation in 3DiM[1]: The standard metrics (PSNR/SSIM) are not sufficient to effectively evaluate geometry-free models for view synthesis. Since NVS from a single image is an ill-posed problem (e.g. novel view generation of the backview given the front view), sometimes a plausible generation which is inconsistent with the ground truth may result in a decrease in PSNR/SSIM.
   [1]Watson, Daniel, et al. Novel View Synthesis with Diffusion Models. Oct. 2022.
2. Effect on Human Portrait: Thank you very much for the intriguing question. We provide additional 3D reconstruction results on Taylor Swift, young male and more in Figure 20. We acknowledge that there remains a quality gap between our method and the human-centric approaches, but our results are of higher quality than zero123. We agree with the reviewer that the quality concern in human portraits comes from the pretrain datasets (Objaverse), which lack sufficient human data. Therefore we believe the issue can be alleviated by finetuning on human-centric datasets which remains for future work.