Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priors

Thank you for your time and helpful feedback. We respond below to your questions:

Q: Can authors provide additional insights onto the effectiveness of CLIP-based termination and its potential drawbacks?

Thank you for the suggestion to conduct more experiment to fully assess the effectiveness of CLIP-guided termination. We follow the setting of your mentioned simple baseline, and the fixed training iteration for 3D phase is set to 3000. The results on RealFusion15 and C10 are shown in the following table.

We agree that monitoring the change of CLIP-rate can be stuck at local minima, and this is the main drawback of CLIP-based termination. To mitigate the problem of local minima, we introduce a sliding window that takes into account the CLIP-rate of change over a certain time interval. As for concern about "the change of CLIP-rate may exhibit large variations across object categories", that's why we propose the case-aware detection mechanism. We conduct additional experiments on the threshold $\delta$ and sliding window size $L$. The results:

Q: Can authors provide additional insights onto the importance of annealing strategy?

We conduct the experiment in which we activate 2D prior in a binary way and the prolong the 3D phase. The results are shown in teh table below.

Q: Can authors provide additional details on the C10 dataset and justify the significance of evaluation results?

C10 consists of 100 Internet images collected from 10 categories. C10 images were not obtained by rendering 3D objects. The main purpose of our evaluation on C10 is to evaluate the ability of the method to generalize over different categories of objects.

Q: Can authors clarify the confusing parts of the method as well as experiments?

the input image is described but it is unclear at which stage it is applied.

Specifically, in the code implementation, in one of the four iterations (either Stage 1 or Stage 2), a reconstruction of the reference viewpoint is performed, by reducing the gap between the rendering of the input image viewpoint and the input image.

The normal maps are shown in Figure 3

We sincerely apologize for our mistake. Indeed, we do not utilize the normal map, and in the revision, we have replaced the normal map with the mask image.

if the results are evaluated on the reference image (input) or novel views.

For quantitative evaluation, we adopt three metrics, namely CLIP-similarity, PSNR and LPIPS. Due to the absence of GT 3D model, the PSNR and LPIPS results are only evaluated on the reference image (input). We got CLIP-similarity by calculating the average CLIP distance between the rendered images of novel views and the reference view.

dataset the ablation of Table 2 is performed on.

We randomly selected 5 objects for each category in the RealFusion15 and C10 image set as our data for the ablation experiment of Table 2. We have made it clear in the revision.

Misc: Page 2, it claims that “with 3D structure priors, … (prior works) avoid multi-face issues, but struggle to obtain consistent reconstruction”. In my opinion, these are the same thing, which can be broadly categorized as “content drifting”.

Thanks for the correction. The sentence you mentioned is indeed poorly expressed. We have modified it in realvision to read: "With 3D structure prior, Liu et al. (2023) and Qian et al. (2023) can stably recover the 3D structure of an object, but struggle to obtain highly consistent reconstruction "

We concur with your perspective that solely evaluating the differences between predicted values and ground truth in reference views for image-to-3D tasks does not adequately measure the quality of reconstruction. Consequently, following the approach of Zero-1-to-3, we have chosen Chamfer Distance and volumetric IoU as metrics which directly response to the gap between two meshes. We calculate these metrics on the Google Scanned Object (GSO) 3D dataset to compare the differences between the generated meshes and the ground truth meshes in the dataset. This comparison reflects the overall reconstruction quality of our method relative to previous image-to-3D approaches.

It's noteworthy that in the cases randomly selected from the GSO dataset, our method outperforms both Zero-1-to-3 and Magic123 in terms of Chamfer Distance and volumetric IoU. The detailed experimental results are as follows:

Instead of calculating the gap in image quality evaluation metrics between multi-view images, we directly choose to measure the gap between the generated values and the true values at the mesh level, which allows for a more intuitive and comprehensive evaluation of the results of our approach against those of other image-to-3D work. We hope this result helps you to understand the result quality of our method.

Q1: Regarding the display of results for the multi-head issue you mentioned, please refer to Section A.1 in the Appendix. There, we have detailed the multi-view sampling results of the Realfusion15 and C10 dataset tests, demonstrating how our algorithm effectively avoids the multi-head problem by integrating 3D and 2D priors.

As mentioned in our paper's introduction, we observed the effects and final outcomes of the optimization process using pure 3D and 2D priors. We believe that the need for 3D and 2D priors varies at different timesteps during representation optimization. Hence, in the early optimization stages, where structural guidance is more crucial, we use pure 3D priors to ensure the 3D assets acquire correct geometric information (effectively resolving the multi-head issue). Following this, we employ a pretrained image-text model to assess if the structure is sufficiently optimized. Once stability in the assessment indicators is achieved, we gradually increase the 2D priors that offer high-resolution detail textures, thereby providing the most needed information at each optimization step.

Q2: In Equation 3, we used weight proportions for different components referencing the default ratios in the stable-dreamfusion framework (λ_rgb=1000, λ_mask=500, λ_depth=10). We also supplemented our paper with experimental results showing different orders of magnitude in parameter swapping and scenarios without using depth priors for reconstruction (λ_rgb=1000, λ_mask=500, λ_depth=0; λ_rgb=10, λ_mask=1000, λ_depth=500; λ_rgb=500, λ_mask=10, λ_depth=1000).

We used the same depth estimator as the stable-dreamfusion framework. EPFL-VILAB Omnidata GitHub Repository This tool is also integrated into stable_dreamfusion/preprocess_image.py.

Q3: We appreciate your attention to the "dynamic prior" used in our second stage. The "adaptive" process you mentioned is not reflected in Equation 8. The design around Equation 8 primarily aims to gradually increase the weight of 2D priors, which provide rich texture information, and decrease the weight of 3D priors, which offer structural information, once the 3D assets have acquired geometric data. The adaptive process occurs during the first stage of optimization using pure 3D priors, where the CLIP model assesses the optimization boundaries through images sampled from 8 different perspectives. Due to varying degrees of geometric and texture complexity in different cases, the first stage, which primarily optimizes geometry, requires different timesteps for different cases to acquire sufficient structural guidance. Thus, the Optimization Boundary Judgement (OBJ) is an adaptive process to detect whether 3D assets have gained adequate geometric information. The visualization of the weight changes during the optimization is presented in Fig 11 in the Appendix. Our work, similar to Magic123, optimizes representations by merging 2D and 3D priors. However, a core difference from Magic123 is that we do not use a fixed ratio of 2D and 3D throughout ($\mathcal{L}_{g}=\lambda _{2D/3D} \mathcal{L} _{2D}+ 40 \mathcal{L} _{3D}$). We divide the optimization into two stages: (1) In the first stage, we use pure 3D priors (see Equation 4) to quickly enable NeRF to acquire high-quality geometric information. (2) We use optimization boundary judgement to assess if the assets have received sufficient structural information from the 3D prior. (3) When the OBJ determines that the asset is sufficiently structured, we follow Equation 8 to gradually reduce the weight of the 3D prior in the optimization target, while increasing the weight of the 2D prior. This ensures that the 2D priors, which provide richer texture information, gradually dominate the optimization targets while maintaining geometric accuracy.

Q4: For most cases, each of the two stages takes about 15 minutes, totaling approximately half an hour.

Q5: The experimental results for the depth comparison can be found in Q2.

Q6: We greatly appreciate your valuable suggestions regarding the experiments. Regarding the Objaverse dataset you mentioned, it is used for training the Zero123 model (the 3D prior we use), making it unsuitable for testing purposes. Therefore, we have supplemented our appendix with additional experimental results on the NeRF4 datasets (see Figure 10 in the Appendix). We could not obtain reasonable reconstruction on NeRF4, but these are also the failure cases for all the methods compared.

Thank you for your detailed and constructive feedback. We have carefully considered each of your comments and have made efforts to improve our paper accordingly.

Q1: In response to your concern about not introducing previous work in Sec 3. Methodology to demonstrate the development of our ideas, we believe that a thorough review of the field and the positioning of our problem to be solved have already been addressed in the introduction and related work sections. a. Regarding the use of depth prior, our algorithm's implementation is indeed based on both the Magic123 and Stable-dreamfusion frameworks. The depth estimator we used for our experiments is primarily based on the one integrated in Stable-dreamfusion, available at [GitHub Link]. We will make modifications to Sec 3.1 of the original text to reflect this. b. Our algorithm, like Magic123, is implemented based on the Stable-Dreamfusion framework. As you mentioned, the 3D prior loss proposed by Zero123 and the 2D prior loss proposed by Stable Diffusion are both integrated into this framework. c. Considering your concerns about similarities with Magic123, we would like to re-emphasize our differences: (1) In the first stage, we use pure 3D priors (see Equation 4) to optimize NeRF for rapid acquisition of high-quality geometric information. (2) We employ optimization boundary judgement to assess whether the optimized assets have received sufficient structural information from the 3D prior. (3) When OBJ determines that the asset is adequately structured, we gradually reduce the weight of the 3D prior in the optimization target and increase the weight of the 2D prior according to Equation 8. Besides both using 3D and 2D priors, our method of applying these priors throughout the optimization process distinctly differs from Magic123.

Q2: You are correct that our initial explanation of the R and T parameters could lead to ambiguity. The R parameter represents the camera's orientation in the form of a rotation matrix, while the T parameter represents the camera's translational movement in the coordinate system. Therefore, we have revised our original statement from “R and T mean the positional coordinate parameters of the camera” to “R and T represent the rotation and translation parameters of the camera.”

Q3: Magic123's use of 3D priors does help mitigate the Janus problem associated with purely 2D-based methods to some extent, but it still fails to adequately cover all cases, particularly those with complex geometric information. This phenomenon, to a certain degree, motivates our paper. One reason why dependence on 3D priors still results in the Janus problem is the lack of sufficient 3D prior guidance during the geometric initialization phase. Our method ensures sufficient geometric information is acquired through Optimization Boundary Judgement before gradually integrating 2D prior information.

Q4: We have decided to adopt your suggestion regarding additional citations and comparative experiments. As shown in Fig. 10, we could not obtain reasonable reconstruction results for the images in NeRF4, but this is also a failure case for all the methods compared.

Q5: As shown in Figure 10 in the Appendix, we could not obtain reasonable reconstruction results for the images on NeRF4, but these are also the failure cases for all the methods compared.

Q6: We agree that the elevation of the input image will affect the reconstruction quality. Like DreamGaussian, we will adopt the elevation estimation module to get the elevation information, which is later added into the textual description.

We have also addressed the grammatical errors you pointed out in the revision.

Thank you for your time and helpful feedback. We respond below to your questions:

Weaknesses 1

We agree that, by using a dynamic mechanism and a proper $\sigma$, the two stages can be merged into a single one. Specifically, the combined one stage reconstructs the objects using equation $\mathcal{L} _{DP}=e^{-\frac t{\sigma T}} \mathcal{L} _{3D} + \left(1-e^{-\frac t{\sigma T}}\right) \mathcal{L} _{2D}$, where the value of $\sigma$ is 1.5. The results on RealFusion15 and C10 are shown in the following table.

Weaknesses 2

Sorry we didn't specify the source of the text description. Except for Zero123, all other methods of comparison use a description of the reference image, which is obtained using the image caption model. We do the same, but we apologize for not using the icon of the imgae caption model in the pipeline(Figure 3) to illustrate the source of the textual description. We have fixed it in the revision.

Weaknesses 3

Thank you for the interest in our case-aware CLIP-based detection mechanism. We employ CLIP-similarity as it can serve as a metric for evaluating global structure. This is attributed to the fact that the distance between textual descriptions and the rendering of arbitrary viewpoints can be computed in the CLIP latent space. If opting for PSNR, we will calculate the PSNR value between the rendering of the current step and the last step from the same viewpoint. And the change rate of this PSNR value only accounts for the convergence of a certain perspective optimization, but fails to judge if an object's structure is sufficiently initialized. If you could kindly suggest metrics that enable the assessment of overall consistency across multi-view images, we would be delighted to explore these possibilities and present our findings.

Weaknesses 4

Thanks for the correction. We add the citation in our revision. About the equations 4 and 7, they are gradients, which are the same as described in DreamFusion. Our description of them as losses creates confusion, and we make correction in revision.