3D-free meets 3D priors: Novel View Synthesis from a Single Image with Pretrained Diffusion Guidance

Thank you sincerely for your constructive and detailed feedback. We fully understand the points you have raised and have provided responses below.

(1) The writing is poor and many sentences are extremely confusing without any support. I will list several examples below and there are more similar problems in the submitted paper. / In Line 265. (...) / In Line 146. (...) / In Line 159. (...)

Thank you for pointing that out. We have identified the typo you mentioned and corrected it accordingly.

(2) In section 3.1, why does Fig.3 illustrate these findings? I believe the findings here are that CLIP struggles with 3D comprehension, but I do not see any examples in Fig.3 supporting this viewpoint. Moreover, if CLIP itself can control camera pose change, there is no need for training on 3D-aware data pairs. I do not see the necessity of including this discussion.

Section 3.1 describes the process of generating images solely from a text prompt, excluding the guidance image required by the base HawkI model while providing angle-related information. As you can see, the resulting images are inconsistent, and the camera control angles are not properly followed.

One example is elevation. Specifically, as shown in Figure 3, when the elevation was set to -20, the model still generated a view from an aerial perspective and failed to reflect the elevation value of -20. This demonstrates that the CLIP included in the View Synthesis method, HawkI, is still not perfect.

Hence, our hypothesis is that the guidance image plays a critical role. This naturally leads to the question of why the guidance image is important.

This experiment highlights the importance of guidance images and plays a significant role in shaping the main idea of this paper, which involves providing guidance images using a 3D-prior model and applying regularization loss.

(2) The performance improvement is weak. From the visualized comparison in Fig.5, I don’t see much difference between Zero123++ and Ours. Is the camera view control more precise or is the image more realistic? From the waterfall example, I think the result of Zero123++ is more realistic.

Please refer to A2 of the global rebuttal at the top!

(3) The novelty is weak and the method is not well explained. I think a training-free method is a good motivation. But the author still requires heavily on a pretrained NVS model.

Please refer to A2 of the global rebuttal at the top! There have been improvements in the results/methodology.

(4) Related work is not well discussed. To cope with the complexity of real scene instead of object-level, many works have been proposed and the author should include them. I will list several below. / [1] ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image, CVPR 2024 / [2] MegaScenes: Scene-Level View Synthesis at Scale, ECCV 2024 / [3] Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis, ECCV 2024

Thank you for your insightful feedback. We have added the three papers you mentioned to the Related Works and Introduction sections, along with additional related works.

(5) Minor: In Tab.1 and Tab.2, since the author reports the LPIPS metric, I believe the commonly used PSNR and SSIM metrics can also be reported.

Thank you for the valuable feedback. We have revised the evaluation metrics and included PSNR and SSIM as part of the updates. We followed the evaluation approach of the baseline model, HawkI. However, metrics like View-CLIP, CLIPD, and View-CLIPD, while relevant to our task, were deemed somewhat mismatched (and potentially unfair), so we decided to exclude them. Instead, we added metrics commonly used in other view synthesis tasks, such as PSNR and SSIM, as well as DINO and SSCD, to enhance the reliability of our evaluation. The LPIPS metric used by Zero123++ has already been included in the previous paper we submitted.

(6) The reviewer suggests that the author revise the writing, conduct more thorough experiments, and propose a clearer explanation of motivation and method.

Please refer to A2 of the global rebuttal at the top!

There have been significant improvements in the results and the paper, and based on the positive evaluations from several reviewers regarding the novelty and potential of our idea, we would like to kindly and respectfully ask if you could consider raising the review rating for our paper.

Thank you sincerely for taking the time to provide such detailed reviews. If you have any additional questions, please feel free to reach out at any time!

Dear Reviewer,

We would like to sincerely thank you once again for the time and effort you have devoted to reviewing our paper. With the discussion period ending in two days, we would greatly appreciate it if you could review our rebuttal at your convenience, should your schedule permit!

If you have any questions, require additional explanations, or see any areas that need improvement, please feel free to let us know, and we will address them immediately!

We are truly grateful for the valuable feedback you have provided on our research. Additionally, considering the improved methodology and results in the current version, we would be sincerely thankful if you could consider raising your review rating!!!

Hope you have a vibrant start to the week! Thank you very much!!!

With warm regards, The Authors

You can see which metrics were commonly used in previous studies. For instance, Magic123 [1] uses CLIP and LPIPS for evaluation, Zero-1-to-3 [2] employs LPIPS, ZeroNVS [3] also incorporates LPIPS, and Zero123++ [4] exclusively uses LPIPS. Therefore, I strongly believe that LPIPS should be used when comparing model performance.

According to the LPIPS metric, the performance gap between our model and the model with the lowest performance is 0.1712 (0.7452 - 0.5740, in the HawkI-Syn $(-20^\circ, 210^\circ)$ settings), which is approximately 5.187 times greater than the 0.033 gap for Zero123++ (refer to Table 1, 0.210 (Zero-1-to-3) - 0.177 (Zero123++ (Ours))). We have incorporated this feedback and made some revisions to the introduction. We kindly ask you to review the changes.

Hence, we respectfully request that you reconsider the evaluation using LPIPS, as it is the most widely used evaluation metric.

[1] Qian, Guocheng, et al. "Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors." arXiv preprint arXiv:2306.17843 (2023).

[2] Liu, Ruoshi, et al. "Zero-1-to-3: Zero-shot one image to 3d object." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

[3] Sargent, Kyle, et al. "Zeronvs: Zero-shot 360-degree view synthesis from a single real image." arXiv preprint arXiv:2310.17994 (2023).

[4] Shi, Ruoxi, et al. "Zero123++: a single image to consistent multi-view diffusion base model." arXiv preprint arXiv:2310.15110 (2023).

2. Oversaturation issues

Oversaturation issues can occur in Stable Diffusion. Thank you for pointing this out. The results in this paper were generated using Stable Diffusion 2.1, which employs classifier-free guidance. A paper uploaded to arXiv (https://arxiv.org/abs/2410.02416) in October 2024, titled 'Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models' [5], proposed a new methodology to address this issue. Once the code is officially released, we will consider applying this methodology as well.

[5] Sadat, Seyedmorteza, Otmar Hilliges, and Romann M. Weber. "Eliminating Oversaturation and Artifacts of High Guidance Scales in Diffusion Models." arXiv preprint arXiv:2410.02416 (2024).

4. Application limitations

We are well aware of the concern that Zero123++ can only generate images at six fixed camera poses. Since our guidance image input is provided in the form of images, it is true that applying an excellent 3D-prior model other than Zero123++ could be even more beneficial. We want to clarify that the 3D prior used in our proposed idea is not limited to Zero123++.

We had some practical reasons to use Zero123++ to demonstrate our approach, is clearly stated in Global Rebuttal A3. (This is a question that you did not mention in your initial review. However, other reviewers raised this point, so we included it in the Global Rebuttal.)

For example, the comparison model ZeroNVS requires 7,500 seconds to generate a camera-controlled image, and the Stable Zero123 model takes 1,278 seconds to generate an image. As more general and faster models come up in the future, our approach can be combined with them. We mentioned in this detail in the conclusion part.

5. Equation and Writing

The issue you mentioned with the '?' marks was caused by duplicate references, and we have resolved this error.

Additionally, to avoid confusion between plain text and mathematical notation, **we have revised all equations to use mathematical notation. **

We have corrected all the errors you pointed out and have uploaded a new version accordingly.

Furthermore, after acceptance, the final version will undergo proofreading by a professional writer.

Closing Remarks

Your primary concerns seem to focus on the results and evaluation, and we sincerely appreciate you bringing these points to our attention. Once this paper is accepted as a camera-ready paper, we will consider further improvements to these aspects.

We believe that perspectives on the results may differ among individuals. However, we would like to note that reviewers BRTu, uyxj, and hAuB have previously provided positive feedback on the results.

Additionally, following the rebuttal, a reviewer who had initially given a score of 6 (uyxj) increased it to 8, and reviewer hAuB also stated that the rating for this paper should be raised from 6 to 7. In light of these points, we respectfully request that you reconsider your review rating.

We deeply appreciate the time and effort you have devoted to reviewing our work. While we respect your opinion and acknowledge that everyone may have different perspectives, we kindly ask for your consideration, as the majority of the reviewers have encouraged this work.

With warm regards, The Authors

Dear Reviewer yRXV,

Since multiple reviewers have mentioned this issue, we would like to post an additional rebuttal regarding image consistency.

As shown in the results attached at the anonymous URL below, we demonstrate that consistency can be improved by utilizing the Runway Gen-3 Alpha Turbo model as a 3D-prior model instead of Zero123++.

URL: https://github.com/2024nvs/2024nvs

Please kindly note that the core idea of this work is not the improvement of consistency itself but rather the ability to achieve camera control by combining 3D priors with a 3D-free architecture, which is the novel insight this work has introduced.

We hope that minor artifacts due to our initial choices in implementation using Zero123++ should be alleviated, after we demonstrate the new results on using Runway Gen-3 Alpha Turbo as 3D prior.

We kindly request you to reconsider your rating, given our new implementation and results using Runway Gen-3 Alpha Turbo.

With warm regards, The Authors

Thank you sincerely for your constructive and detailed feedback. We fully understand the points you have raised and have provided responses below.

(1) The method is quite convoluted with 4 different test-time optimization steps. No time comparisons are provided. It would be important to mention typical in inference times.

Please refer to A1 of the global rebuttal at the top!

(2) Looking into the intermediate and final results, as well as the method, the main task of aligning the camera appears to be the result of the Zero123++ method, while the test-time optimizations generally improves the image quality. This is also reflected in the particular choice of angles, as only those angles are trained for in Zero123++. However, for small angles (30,30), often the test-time optimization worsens the results, e.g., figure 6, angles = (30,30) for cats. Same for figure 5, angles = (30,30)- the house changes completely after optimization. Due to this reason, it may be a bit dubious to stipulate object + background novel view synthesis as contribution. After the evaluations, both the contributions (1) & (2) could be attributed - at least in part - to Zero123++.

Please refer to A2 of the global rebuttal at the top!

(3) The architecture diagram in Figure 2 is not complete in the info provided. Missing: which losses are used? These should be made clear together with the equations (1,2) since this is the crux of the proposed method.

(7) The English could benefit from some slight polishing at times

(8) It would be good if the authors clearly define what they mean by azimuth and elevation angles. The typical definition of the latter would be relative to the horizontal plane. Is that the case here as well, or are angles relative to the guidance view ? In any case, the angles specified in the image captions do not seem to follow one consistent set of definitions... Elevation angles are taking up values larger than 90 degrees, for instance. Related work that could be mentioned: Depth self-supervision for single image novel view synthesis, 2023 Giovanni Minelli et al.

We have made the requested revisions and improved the clarity of the four steps used in our paper.

Additionally, we have updated the method figure to clearly indicate which losses are used, making it easier to understand.

Moreover, please note that improvements have also been made to the writing of the paper.

Furthermore, as you suggested, we have clarified the definitions of azimuth and elevation. We use the azimuth and elevation as defined by Zero123++.

(4) L215--L216- "Our model eliminates the need for training data, 3D data or multi-view data". This is somewhat misleading. The camera alignment largely appears to be performed by the guidance method Zero123++, which is trained on Objaverse and more importantly on the same task. The method implicitly uses Objaverse via Zero123++, a competing method used in the proposed method. It is not clear how Zero123++ performs test-time optimization without the 3D data.

That is an excellent point.

To clarify, what we mean by '3D-free' is that our method does not require any "additional 3D" data for training or generating results beyond the data provided to the pretrained model.

Many of the methods mentioned in our Introduction and Related Works sections rely on additional 3D data to achieve good performance.

In contrast, our method does not require such additional 3D data.

(5) The paper glosses over some of the limitations in the results, like the house not being shown in a very consistent fashion when viewpoint is changed. Probably there is a difference in performance between renowned monuments with many photographs that can be scraped from the Internet, compared to rather unique objects / situations.

Please refer to A2 of the global rebuttal at the top!

(6) Given the complete approach, the novelty of the work mainly consists of the combination of methods, whereas the actual novelty drags in the cost of extensive inference optimization.

Please refer to A1 of the global rebuttal at the top!

(9) The creation of videos with continuously changing viewpoints would be telling when it comes to consistency and angle correctness…

Thank you sincerely for your excellent suggestion. Consistency has been significantly improved through the regularization loss, as demonstrated in the paper. If we have the opportunity to submit a camera-ready version, we will consider ways to further emphasize consistency in the paper.

Given the improved results/methodology presented in this revision, and your initial assessment of Acceptance (6), we would be deeply grateful if you could raise your review ratings that better reflect your encouraging evaluation!

Thank you sincerely for taking the time to provide such detailed reviews. If you have any additional questions, please feel free to reach out at any time!

Dear Reviewer,

We would like to sincerely thank you once again for the time and effort you have devoted to reviewing our paper. With the discussion period ending in two days, we would greatly appreciate it if you could review our rebuttal at your convenience, should your schedule permit!

If you have any questions, require additional explanations, or see any areas that need improvement, please feel free to let us know, and we will address them immediately!

We are truly grateful for the valuable feedback you have provided on our research. Additionally, considering the improved methodology and results in the current version, we would be sincerely thankful if you could consider raising your review rating!!!

Hope you have a vibrant start to the week! Thank you very much!!!

With warm regards, The Authors

Thank you sincerely for your constructive and detailed feedback. We fully understand the points you have raised and have provided responses below.

(1) The proposed method relies on a pre-trained multi-view diffusion model, Zero123++, and thus shares its limitations: restricted view generation (Zero123++ generates six fixed-view images around the object), modification of background and object, handling multiple objects, and pose misalignment.

Please refer to A3 of the global rebuttal at the top!

(2) The evaluation can be seem unfair. For apple to apple comparison, the paper could compare with running the proposed method on Zero123++. Also, measure the CLIP score seem unfair as the proposed method optimizes the CLIP text embeddings.

Thank you for mentioning the metrics.

We followed the evaluation approach of the baseline model, HawkI. However, metrics like View-CLIP, CLIPD, and View-CLIPD, while relevant to our task, were deemed somewhat mismatched (and potentially unfair), so we decided to exclude them.

Instead, we added metrics commonly used in other view synthesis tasks, such as PSNR and SSIM, as well as DINO and SSCD, to enhance the reliability of our evaluation. The LPIPS metric used by Zero123++ has already been included in the previous paper we submitted.

(3) The proposed method introduces additional computational costs: generating a multi-view guidance images from a pre-trained diffusion model and the optimization of text embeddings to reconstruct $I_{input}$ and $I_{view}$

Please refer to A1 of the global rebuttal at the top!

(4) L.86, L.90: Some terms are used in contradicting manner. 3D-free methods like Zero123++ and 3D-based models like Zero123++. / L.95: we (1) we / Eqn. 1: The optimization task is provided without details. $L, f, x_{t}$ is not introduced before.

(5) The method could be presented with more detail. Specifically, how do Eq. 1 and Eq. 2 work? Does the diffusion model output clean predicted samples? If not, how is the loss computed between the noisy latent $x_{t}$ and the clean image $I_{input}$?

We have reviewed and revised the writing as you suggested, and added more detailed information about the variables $L, f, x_{t}$ improving the overall quality of the paper.

In addition, as another reviewer pointed out that the method was somewhat difficult to understand, we have explained all the loss functions used in the four-step process and improved the method figure by adding numbered labels for better clarity.

There have been significant improvements in the results and the paper, and based on the positive evaluations from several reviewers regarding the novelty and potential of our idea, we would like to kindly and respectfully ask if you could consider raising the review rating for our paper.

Thank you sincerely for taking the time to provide such detailed reviews. If you have any additional questions, please feel free to reach out at any time!

Dear Reviewer,

We would like to sincerely thank you once again for the time and effort you have devoted to reviewing our paper. With the discussion period ending in two days, we would greatly appreciate it if you could review our rebuttal at your convenience, should your schedule permit!

If you have any questions, require additional explanations, or see any areas that need improvement, please feel free to let us know, and we will address them immediately!

We are truly grateful for the valuable feedback you have provided on our research. Additionally, considering the improved methodology and results in the current version, we would be sincerely thankful if you could consider raising your review rating!!!

Hope you have a vibrant start to the week! Thank you very much!!!

With warm regards, The Authors

I appreciate the authors for clarifications on evaluation metrics and optimization tasks.

Still, the applicability of the proposed method remains unclear to me due to the dependency of Zero123++ which I list as follows.

1. Zero123++ can only generate images at six fixed camera poses which means the proposed method can also be limited in generating these 6 poses. This is also reflected in the result section, where the evaluation is performed only using a discrete set of views. This limitation may be critical as in most NVS tasks, users may want to freely move around the object/scene.

2. While the outputs from Zero123++ generate quite consistent images, it does not guarantee the preservation of the source view image. This is also reflected in Figs. 5-6 where the generated outputs of the proposed method alter the content of the source view image.

3. Zero123++ may produce frames with incorrect pose information. While the model is capable of generating plausible target view images, it does not incorporate explicit physical constraints, which can result in inaccuracies when generating images with precise pose information.

While I do not claim that the use of Zero123++ is flawed, I believe the paper should explain how these issues are addressed in their proposed method and how adopting the multi-view diffusion model can outperform inpainting-based methods or pose-conditioned video generative models? This leads to additional concern as the authors does not present comparisons to other types of methods.

Additionally, the optimization tasks seem like to introduce a lot of burdens as it accumulates the gradient of all timesteps (Eqs. 1-3). It raises questions on how the gradients are backpropagated.

Once again, I appreciate the authors for providing clarifications. It would be great if the authors can clarify the concerns raised above.

Dear Reviewer 9Fq5.

Thank you sincerely for providing insightful comments and making such valuable points.

1. Zero123++ can only generate images at six fixed camera poses which means the proposed method can also be limited in generating these 6 poses. This is also reflected in the result section, where the evaluation is performed only using a discrete set of views. This limitation may be critical as in most NVS tasks, users may want to freely move around the object/scene.

We are well aware of the concern that Zero123++ can only generate images at six fixed camera poses.

Since our guidance image input is provided in the form of images, it is true that applying an excellent 3D-prior model other than Zero123++ could be even more beneficial. We want to clarify that the 3D prior used in our proposed idea is not limited to Zero123++.

We had some practical reasons to use Zero123++ to demonstrate our approach, is clearly stated in Global Rebuttal A3.

For example, the comparison model ZeroNVS requires 7,500 seconds to generate a camera-controlled image, and the Stable Zero123 model takes 1,278 seconds to generate an image.

As more general and faster models come up in the future, our approach can be combined with them. We mentioned in this detail in the conclusion part.

2. While the outputs from Zero123++ generate quite consistent images, it does not guarantee the preservation of the source view image. This is also reflected in Figs. 5-6 where the generated outputs of the proposed method alter the content of the source view image.

3. Zero123++ may produce frames with incorrect pose information. While the model is capable of generating plausible target view images, it does not incorporate explicit physical constraints, which can result in inaccuracies when generating images with precise pose information.

We are well aware of the concern that Zero123++ may fail to preserve the content of the source view image or provide incorrect pose information, and this has been explicitly stated in Section 5.1.

To address this issue, we applied a regularization loss to resolve it. Accordingly, as indirectly mentioned in Section 4.1.5, we clarified that the 3D-prior model performs weak or pseudo guidance. This explanation is identical to the response provided to Reviewer BRTu. Please refer to the details and please check Section 5.3, Figure 7, and Table 2.

In particular, the examples highlighted in the ablation study shown in Figure 7 include scenes of a Hawaiian beach and a waterfall. In both cases, the Zero123++ model failed to provide accurate viewpoints and guidance to the model. However, the application of the regularization loss significantly improved the consistency of the textures in the Hawaiian beach scene and enhanced the consistency of the rock textures generated in the waterfall scene.

We kindly ask you to refer to Section 5.3 for more detailed information on this aspect.

Part of sentences from Section 5.3: "For the Hawaiian beach scene, the output generated with the regularization term exhibits a more consistent style compared to the output produced without it. Despite the inaccuracies in the Zero123++ guidance images, the regularization term facilitates more reliable camera control than the results generated without it. Similarly, in the waterfall scene, the regularization term enhances the consistency of the generated rock textures surrounding the waterfall. Without the regularization term, these textures are inconsistently represented; however, with it, the style is maintained more faithfully. Once again, Zero123++ does not provide accurate guidance in this case, underscoring the significant contribution of the regularization loss to improved control and visual coherence in the generated images."

We also conducted an ablation study to compare the results with and without the application of the regularization loss. By analyzing seven image evaluation metrics presented in the existing evaluation — LPIPS, CLIP, DINO, SSCD, CLIP-I, PSNR, and SSIM — we demonstrated superior performance, thereby providing quantitative evaluations. We have attached the results in a table below for your reference, and we would appreciate it if you could take a look.

4. While I do not claim that the use of Zero123++ is flawed, I believe the paper should explain how these issues are addressed in their proposed method and how adopting the multi-view diffusion model can outperform inpainting-based methods or pose-conditioned video generative models? This leads to additional concern as the authors does not present comparisons to other types of methods.

Thank you for the insightful comment. However, inpainting-based methods or pose-conditioned video generative models differ significantly from our method, making direct comparisons challenging.

First, regarding pose-conditioned video generative models, a representative example would be Pix2Pix3D [1] (https://arxiv.org/abs/2302.08509). Our method operates in an environment where an input image and text are given to perform camera control, whereas Pix2Pix3D uses a 2D label map as input, which inherently defines a different setting.

Moreover, our approach is not limited to simple environments such as faces or objects but tackles a more complex task by performing camera control on synthetic data with backgrounds or real-world data with backgrounds.

As for inpainting-based methods, we can take the latest study, MVInpainter [2] (https://arxiv.org/pdf/2408.08000), as an example. Since this study uses 3D data, it is not 3D-free, and therefore, it cannot serve as a direct comparison to our method.

[1] Deng, Kangle, et al. "3d-aware conditional image synthesis." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Cao, Chenjie, et al. "Mvinpainter: Learning multi-view consistent inpainting to bridge 2d and 3d editing." arXiv preprint arXiv:2408.08000 (2024).

5. Additionally, the optimization tasks seem like to introduce a lot of burdens as it accumulates the gradient of all timesteps (Eqs. 1-3). It raises questions on how the gradients are backpropagated.

We will explain this part based on our implemented code.

Our implementation uses configurable gradient_accumulation_steps, which allows for flexible control over how gradients are accumulated. This ensures that the computational overhead can be managed by balancing memory and performance considerations.

Additionally, the use of accelerator.backward() optimizes gradient computation and synchronization across devices in distributed setups.

Gradients are only computed for parameters explicitly involved in the optimization task (e.g., text_embeddings), while other parts of the model are frozen using requires_grad_(False) to minimize unnecessary computations.

To reduce computational costs, we employ techniques such as gradient checkpointing, selective accumulation, and efficient memory utilization through frameworks like accelerator. These techniques ensure that the optimization process remains computationally tractable without compromising model performance.

Closing remarks

Your primary concerns seem to focus on the applicability of Zero123++, and we sincerely appreciate you bringing these points to our attention. Once this paper is accepted as a camera-ready paper, we will consider further improvements to these aspects.

We believe that perspectives on the results may differ among individuals. However, we would like to note that reviewers BRTu, uyxj, and hAuB have previously provided positive feedback on the results.

Additionally, following the rebuttal, a reviewer who had initially given a score of 6 (uyxj) increased it to 8, and reviewer hAuB also stated that the rating for this paper should be raised from 6 to 7. In light of these points, we respectfully request that you reconsider your review rating.

We deeply appreciate the time and effort you have devoted to reviewing our work. While we respect your opinion and acknowledge that everyone may have different perspectives, we kindly ask for your consideration, as the majority of the reviewers have encouraged this work.

With warm regards, The Authors

Dear Reviewer 9Fq5,

We greatly appreciate your time and effort in reviewing our work and providing detailed feedback. Your insights have been invaluable in helping us refine our paper. As the discussion period is coming to a close in a few days, we kindly request you review our second-round rebuttal at your earliest convenience.

Thank you once again for your thoughtful consideration.

With warm regards, The Authors

Dear Reviewer 9Fq5,

Since multiple reviewers have mentioned this issue, we would like to post an additional rebuttal regarding image consistency.

As shown in the results attached at the anonymous URL below, we demonstrate that consistency can be improved by utilizing the Runway Gen-3 Alpha Turbo model as a 3D-prior model instead of Zero123++.

URL: https://github.com/2024nvs/2024nvs

Please kindly note that the core idea of this work is not the improvement of consistency itself but rather the ability to achieve camera control by combining 3D priors with a 3D-free architecture, which is the novel insight this work has introduced.

We hope that minor artifacts due to our initial choices in implementation using Zero123++ should be alleviated, after we demonstrate the new results on using Runway Gen-3 Alpha Turbo as 3D prior.

We kindly request you to reconsider your rating, given our new implementation and results using Runway Gen-3 Alpha Turbo.

With warm regards, The Authors

(Q6) It's interesting to see that images generated from Zero123++, which is trained on Objaverse, can generate reasonable images' guidance with scene image input. Combining this with previous question, what's the problem of using some open-sourced scene-level NVS models for pseudo guidance? For example, I'd assume ZeroNVS[1] can provide images with any viewpoint input condition. [1] Sargent, Kyle, et al. "Zeronvs: Zero-shot 360-degree view synthesis from a single real image." CVPR 2024.

Please refer to Global Rebuttal Response A1 for reference.

First, the reason for applying Zero123++ over other models is its faster image generation speed and minimal resource usage. For example, ZeroNVS, which you mentioned, allows for a full 360-degree azimuth transformation but cannot handle elevation transformations, which does not align with our goals.

Additionally, when we tested ZeroNVS using an animal image from our HawkI-Real dataset, the results were less natural compared to Zero123++. Furthermore, generating an image took 7,500 seconds (2 hours and 5 minutes) with significant GPU consumption, making it even less suitable for our application.

Therefore, we chose to apply the Zero123++ model as it best guarantees the concept of our idea.

Given the improved results/methodology presented in this revision, and your initial assessment of Acceptance (6), we would be deeply grateful if you could raise your review ratings that better reflect your encouraging evaluation!

Thank you sincerely for taking the time to provide such detailed reviews. If you have any additional questions, please feel free to reach out at any time!

Thank you sincerely for your constructive and detailed feedback. Please find our responses below.

(Q1) The connection between the section3 and section4 is very confusing. First, the detailed experiment setting in section3 is not clearly explained (see questions), then it suddenly shift from section3 with CLIP embeddings to section4 on leveraging a diffusion model. I suggest authors revise the high-level motivations and present them more clearly in the paper.

(Q3) What is the detailed procedure of experiments related to figure 3. and figure 4.? What is the actual model that generates those images? CLIP itself isn't a generative model. Does the author mean using Stable Diffusion with different text embeddings? And how is the image guidance been used in (I assume) this standard SD model?

Above all, we deeply appreciate your constructive feedback on our work.

To elaborate further on Section 3, the experiments were conducted based on a 3D-free stable diffusion-based view synthesis method called HawkI, which utilizes a guidance image. Section 3.1 describes the process of generating images solely from a text prompt, excluding the guidance image required by the base HawkI model, while providing angle-related information.

As you can see, the resulting images are inconsistent, and the camera control angles are not properly followed. Hence, our hypothesis is that the guidance image plays a critical role. This naturally leads to the question of why the guidance image is important.

To explore this, we conducted experiments where $(\alpha_{elev}, \alpha_{azi}) = (30^\circ, 30^\circ)$ was provided, but guidance images were supplied at various angles. The results confirmed that this model heavily relies on the guidance image.

Consequently, our experiments naturally led to the question, "What role does the guidance image play?" This inquiry served as the foundation for deriving our method, which provides 3D prior information through the guidance image. We hope this conveys the logical progression of our work. We have revised the paper to emphasize the points you mentioned.

We have also clarified the experimental section. As mentioned above, we used HawkI, but we believe there is a possibility of misunderstanding from the reader's perspective that only CLIP itself is being used. Additionally, to eliminate the possibility of misunderstanding that the guidance image is directly incorporated into the Stable Diffusion model (We use a custom pipeline. Stable Diffusion does not follow the approach of providing guidance images), we have made the HawkI reference more explicit. The preprocessed guidance image is passed through a Variational Autoencoder (VAE) to encode it into a latent distribution. This step transforms the image into a compressed latent space representation. So it can provide critical contextual or conditional information to guide the process.

(Q2) The figure 2 for pipeline is actually making it harder to understand. I'd suggest at least add some orderings in it can better illustrate the whole pipeline.

We have updated the method figure based on your suggestions and added step numbers to it. Additionally, we have reorganized the Method section into step-by-step stages to make it easier to understand and improved the paper accordingly.

(Q4) Although inference-time optimization with efficient methods like LoRA and embedding finetuning is not that slow, I'd suggest authors also provide the memory consumption and time used in this process, addition to optimization iterations.

Please refer to A1 of the global rebuttal at the top!

(Q5) Are the specific evaluation viewpoints come from Zero123++ settings? So the method is actually limited to certain viewpoints?

Please refer to A3 of the global rebuttal at the top!

Dear Reviewer uyxj,

First, I would like to sincerely thank you for maintaining a positive review rating. Additionally, thank you for highlighting the importance of our initial paper's idea and the impressive results as its strengths.

Regarding the mentioned Figure 2, As you mentioned, we have revised the numbering to align with the numbering in the method section. We will further consider ways to make it more impactful for readers and improve it accordingly. We will also do our utmost to address the points you raised, even after acceptance.

I am reaching out to kindly ask if you might reconsider raising the review rating, if you mind!

There have been further developments in the idea you highlighted as a strength, the results have been further refined, and we have thoroughly addressed and revised all the points you mentioned, with most of the highlighted areas improved.

Given such improvement, positive feedback, and your final 'Accept' evaluation, I would respectfully suggest considering a higher score that better reflects your encouraging assessment!!!

Thank you very much!!!

With warm regards, The Authors

Dear Reviewer,

We would like to sincerely thank you once again for the time and effort you have devoted to reviewing our paper. With the discussion period ending in two days, we would greatly appreciate it if you could review our rebuttal at your convenience, should your schedule permit!

If you have any questions, require additional explanations, or see any areas that need improvement, please feel free to let us know, and we will address them immediately!

We are truly grateful for the valuable feedback you have provided on our research. Additionally, considering the improved methodology and results in the current version, we would be sincerely thankful if you could consider raising your review rating!!!

Hope you have a vibrant start to the week! Thank you very much!!!

With warm regards, The Authors

Thank you sincerely for your constructive and detailed feedback. Please find our responses below.

(Q1) While the authors acknowledge the limitation of inference-time optimization in terms of real-time applicability and scalability, a direct runtime comparison with baseline models would provide additional clarity.

Please refer to A1 of the global rebuttal at the top!

(Q2) This method is incapable of generating arbitrary camera viewpoints. this might be due to the choice of the 3D prior, Zero123++, which can provide guidance images at six fixed viewpoints. Clarifying this limitation or demonstrating the capability of generating arbitrary viewpoints is much needed.

Please refer to A3 of the global rebuttal at the top!

(Q3) The novelty of the proposed framework is somewhat limited. The approach builds heavily on the HawkI, with modifications such as replacing guidance images generated via Inverse Perspective Mapping with those from a 3D prior method (Zero123++). This modification, while practical, could be seen as a straightforward combination.

Please refer to A2 of the global rebuttal at the top!

(Q4) The generated images across different viewpoints appear inconsistent. Does the proposed method include any mechanisms to encourage consistency across these viewpoints?

Please refer to A2 of the global rebuttal at the top!

Considering the improved results/methodology presented in this revision, and considering your positive evaluation regarding approach/method/result/analysis, we would be deeply grateful if you could raise your review ratings based on these enhancements.

Thank you sincerely for taking the time to provide such detailed reviews. If you have any additional questions, please feel free to reach out at any time!

Dear Reviewer,

We would like to sincerely thank you once again for the time and effort you have devoted to reviewing our paper. With the discussion period ending in two days, we would greatly appreciate it if you could review our rebuttal at your convenience, should your schedule permit!

If you have any questions, require additional explanations, or see any areas that need improvement, please feel free to let us know, and we will address them immediately!

We are truly grateful for the valuable feedback you have provided on our research. Additionally, considering the improved methodology and results in the current version, we would be sincerely thankful if you could consider raising your review rating!!!

Hope you have a vibrant start to the week! Thank you very much!!!

With warm regards, The Authors

Dear Reviewer BRTu,

First of all, we would like to thank you for pointing out this important aspect. While the rebuttal itself does not include experimental results, we would like to highlight that all the results have been updated in the paper. Specifically, we have conducted quantitative and qualitative evaluations on the inclusion and exclusion of the regularization loss, as emphasized in Section 5.3, Figure 7, and Table 2.

In particular, the examples highlighted in the ablation study shown in Figure 7 include scenes of a Hawaiian beach and a waterfall. In both cases, the Zero123++ model failed to provide accurate viewpoints and guidance to the model. However, the application of the regularization loss significantly improved the consistency of the textures in the Hawaiian beach scene and enhanced the consistency of the rock textures generated in the waterfall scene.

We kindly ask you to refer to Section 5.3 for more detailed information on this aspect.

Part of sentences from Section 5.3: "For the Hawaiian beach scene, the output generated with the regularization term exhibits a more consistent style compared to the output produced without it. Despite the inaccuracies in the Zero123++ guidance images, the regularization term facilitates more reliable camera control than the results generated without it. Similarly, in the waterfall scene, the regularization term enhances the consistency of the generated rock textures surrounding the waterfall. Without the regularization term, these textures are inconsistently represented; however, with it, the style is maintained more faithfully. Once again, Zero123++ does not provide accurate guidance in this case, underscoring the significant contribution of the regularization loss to improved control and visual coherence in the generated images."

We also conducted an ablation study to compare the results with and without the application of the regularization loss. By analyzing seven image evaluation metrics presented in the existing evaluation — LPIPS, CLIP, DINO, SSCD, CLIP-I, PSNR, and SSIM — we demonstrated superior performance, thereby providing quantitative evaluations. We have attached the results in a table below for your reference, and we would appreciate it if you could take a look.

| Dataset, Angle, Method | LPIPS ↓ | CLIP ↑ | DINO ↑ | SSCD ↑ | CLIP-I ↑ | PSNR ↑ | SSIM ↑ | |----------------------------------------------|------------|-----------|-----------|-----------|------------|-----------|-----------| | HawkI-Syn (30°, 30°) w/ regularization | 0.5661 | 29.9563 | 0.4314 | 0.3638 | 0.8317 | 11.0664 | 0.3162 | | HawkI-Syn (30°, 30°) w/o regularization | 0.5867 | 28.5417 | 0.4122 | 0.3640 | 0.8243 | 10.8272 | 0.2954 | | HawkI-Real (30°, 30°) w/ regularization | 0.6201 | 29.8850 | 0.3346 | 0.2588 | 0.8152 | 9.4009 | 0.2184 | | HawkI-Real (30°, 30°) w/o regularization | 0.6257 | 29.0798 | 0.3357 | 0.2401 | 0.8231 | 9.1957 | 0.2014 | | HawkI-Syn (30°, 270°) w/ regularization | 0.5744 | 29.1800 | 0.4148 | 0.3684 | 0.8327 | 11.0661 | 0.3047 | | HawkI-Syn (30°, 270°) w/o regularization | 0.5952 | 28.9866 | 0.4098 | 0.3350 | 0.8248 | 10.8656 | 0.2850 | | HawkI-Real (30°, 270°) w/ regularization | 0.5868 | 30.5489 | 0.4126 | 0.3424 | 0.8708 | 10.6177 | 0.2687 | | HawkI-Real (30°, 270°) w/o regularization | 0.6114 | 29.9184 | 0.4003 | 0.3075 | 0.8541 | 10.2958 | 0.2615 | | HawkI-Syn (-20°, 210°) w/ regularization | 0.5740 | 29.1144 | 0.4277 | 0.3529 | 0.8280 | 10.9697 | 0.2837 | | HawkI-Syn (-20°, 210°) w/o regularization | 0.5860 | 28.6385 | 0.3969 | 0.3559 | 0.8171 | 10.8401 | 0.2792 | | HawkI-Real (-20°, 210°) w/ regularization| 0.6185 | 30.6729 | 0.3610 | 0.2880 | 0.8448 | 10.3130 | 0.2223 | | HawkI-Real (-20°, 210°) w/o regularization | 0.6338 | 29.1693 | 0.3817 | 0.2605 | 0.8263 | 10.0794 | 0.2117 | | HawkI-Syn (-20°, 330°) w/ regularization | 0.5624 | 29.2144 | 0.4487 | 0.3892 | 0.8559 | 11.2175 | 0.3048 | | HawkI-Syn (-20°, 330°) w/o regularization | 0.5714 | 28.4089 | 0.4476 | 0.3870 | 0.8492 | 11.0409 | 0.2947 | | HawkI-Real (-20°, 330°) w/ regularization| 0.5925 | 29.5090 | 0.3899 | 0.3127 | 0.8689 | 10.6183 | 0.2971 | | HawkI-Real (-20°, 330°) w/o regularization | 0.5894 | 28.8531 | 0.3704 | 0.2954 | 0.8506 | 10.5213 | 0.2828 |

Table 2: Quantitative Results of Ablation Study. Evaluation of seven metrics demonstrates the superior results of the regularized method over the non-regularized one.

The quantitative and qualitative evaluations you mentioned demonstrate that applying the regularization loss term is an effective solution for maintaining consistency.

Additionally, as highlighted in the initial Strengths section that you wrote, we kindly ask you to consider the fact that this model is fundamentally a 3D-free method and operates under the constraint of no additional training.

Most novel view synthesis (that achieves the 'crucial criterion' you mentioned) papers rely on training with 3D datasets, whereas our approach does not.

It is within these constraints that we introduced the regularization loss term to address viewpoint consistency. We would greatly appreciate it if you could reconsider your evaluation, taking into account the strong results achieved without training using 3D datasets.

With warm regards, The Authors

Dear Reviewer BRTu,

We greatly appreciate your time and effort in reviewing our work and providing detailed feedback. Your insights have been invaluable in helping us refine our paper. As the discussion period is coming to a close in a few days, we kindly request you review our third-round rebuttal at your earliest convenience.

Thank you once again for your thoughtful consideration.

With warm regards, The Authors

(A2) Results and Idea Improvement Through Regularization Loss (Reviewers BRTu, 9Fq5, hAuB, yRXV)

Given that the CLIP model lacks an understanding of camera control information, it is essential to leverage 3D prior information from pretrained model to address this limitation.

By incorporating this prior knowledge, our goal is to enhance the CLIP model's understanding of viewpoints (e.g., elevation and azimuth) to generate the desired camera control outputs.

To address issues such as maintaining consistency across angles in the same image and improving image generation performance in complex scenes, as pointed out by the reviewers, we introduced a regularization term between the text embedding containing elevation and azimuth information ($e_{target}$) and the optimized text embedding ($e_{view}$) in addition to the pretrained guidance model.

This regularization term not only enriches viewpoint knowledge in 3D space but also addresses the limitations of the 3D-prior guidance model, which struggles with complex scenes.

By applying this regularization loss, we hypothesize that we can improve camera control results by constructing a model that uses the guidance image as supplementary information rather than relying solely on it. To improve viewpoint consistency, we added this additional regularization loss term to the reconstruction loss. The term, calculated as

$L_{\text{reg}} = \| e_{\text{view}} - e_{\text{target}} \|^2$

provides a constraint between the embedding $e_{\text{view}}$ and the intended angular information $e_{\text{target}}$, reinforcing the target viewpoint in the generated outputs.

Consequently, images predicted using the 3D-based NVS method, Zero123++, serve as weak or pseudo guidance. The optimization strategy conditions the embedding space with knowledge related to the input image and its view variants, leveraging the guidance image's prior knowledge to facilitate view transformation and guide viewpoint adjustments.

Through this approach, we have addressed the concerns raised about image consistency and camera control performance in complex scenes. The results and detailed implementation details can be found in the revised paper.

(A3) Limitations of Zero123++ (Reviewers BRTu, uyxj, 9Fq5)

Thank you for your important comments regarding the limitations of Zero123++.

While Zero123++ has the drawback of being unable to generate images from specific angles, it was chosen due to its speed and minimal impact on computation time. For instance, although ZeroNVS (Thanks to reviewer uyxj) can perform azimuth transformations (i.e., 360-degree rotation), it cannot handle elevation transformations, which are essential for our task. In addition, ZeroNVS requires 7,500 seconds to generate an image, making it unsuitable for general use. Moreover, Stable Zero123 has a memory consumption of 39.3GB and takes 1,278 seconds to generate an image.

Therefore, applying Zero123++ is the best choice to emphasize the concept of our framework, given its speed and practical utility. Zero123++ also consumes approximately 10.18GB of memory, making it feasible to run even on consumer-grade GPUs, thus enhancing its applicability. This was briefly mentioned in the previous version's qualitative analysis, and in this revision, we have explicitly stated in the conclusion that addressing these limitations will be a focus of future research.

Closing Remarks

We sincerely thank reviewers BRTu, uyxj, and hAuB for positively evaluating the potential and results of our research.

Considering the improved results/methodology presented in this revision, we would be deeply grateful if you could reassess your review ratings based on these enhancements.

Once again, we sincerely thank you for your constructive feedback and for recognizing the value of our ideas.

We will provide individual responses to any additional comments in separate replies.