Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion

Dear Reviewer Ef4X,

Thanks for your careful review and constructive comments. We appreciate you recognizing our paper as an impactful work with significant industrial value. We note that you still have several concerns about the performance of Phidias in extreme cases and the novelty of the proposed solution. We would like to provide more details regarding your questions and hope your concerns can be addressed.

W1: Limitation of inappropriate 3D reference in extreme cases.

We have already discussed this limitation in Appendix (B). Note that when the 3D reference significantly conflicts with the concept image, our performance still equals the existing SOTA image-to-3D generation method, as shown in Fig. 7 (b). Please refer to the 2nd point of our Common Response above for more details.

W2: Combining ControlNet could be criticized for lacking novelty.

As you said, "Simplicity often leads to elegant and effective solutions". While the combination of ControlNet for reference-augmented multi-view generation seems straightforward, it is not that easy to adapt ControlNet to our task due to the Misalignment Dilemma, where the discrepancy between the concept image and the 3D reference can lead to conflicts in the generation process (L78-80). Our novelty comes from introducing the novel task of RAG-based 3D generation, our specific architecture designs for tackling misalignment dilemmas, and extending our framework for unified 3D generation from multi-modality inputs with various useful applications.

Q1: How can we improve the method when there are significant differences?

This depends on the user’s intention.

• Currently, for cases that emphasize image more than the 3D reference (the default setting of Phidias), such as image-to-3D generation, Phidias will follow the concept image and ignore the 3D reference if there are significant differences between them, as shown in Fig. 7 (b). Please refer to the 2nd point of our Common Response above for more details.

• Currently, for cases where users want to decide which side to follow most, Phidias allows users to adjust the reference strength manually using a “control knob”, as shown in Fig. 12 and L483.

• In the future, if we use the native 3D diffusion model for the RAG-based 3D generation, we may be able to leverage the information of all kinds of 3D references even if they are irrelevant to the image during inference, by utilizing their spatial structure/distribution (sparse or dense) and number of faces as conditions.

Q2: Increase the rating

Again, thank you for recognizing the value of our paper. We hope our responses above address your concerns about our paper, and it would be appreciated if there is an increased rating. We would also like to hear more from you if you have further questions or comments.

Bests,

Authors

We sincerely thank the authors for their detailed response. For using complex images to edit the texture of 3D objects, the following articles could be helpful:

[1] StyleTex: Style Image-Guided Texture Generation for 3D Models
[2] Texture: Text-guided texturing of 3D shapes
[3] TextureDreamer: Image-guided Texture Synthesis through Geometry-aware Diffusion
[4] Ipdreamer: Appearance-controllable 3D object generation with image prompts
[5] Paint3D: Paint Anything 3D with Lighting-Less Texture Diffusion Models

There is no need to include additional experiments. Instead, a simple analysis of the similarities and differences between your paper and these methods would suffice. Additionally, consider citing these articles. A concise response would be appreciated, and I will raise my rating accordingly.

Dear Reviewer eNa9,

Thanks for your constructive comments. We hope our responses below address your concerns about our paper. We kindly invite you to watch the attached demo video in the supplementary if you haven’t watched it yet. We would like to hear more from you if you have further questions or comments.

W1, W2: Limited analysis of retrieval mechanism. Performance may degrade given irrelevant references. Analyze scenarios where the input concept conflicts with the reference.

• Regarding the retrieval mechanism, we have detailed our retrieval data and method in Appendix (A.1) and analyzed the limitation of the current retrieval mechanism based on Uni3D in Fig. 15 and Appendix (B). We want to clarify that the specific retrieval mechanism is not our contribution and is replaceable. The main contribution of Phidias is the proposed reference-augmented multi-view diffusion model for enhancing the generative quality, generalization ability, and controllability using an additional 3D reference that can be achieved in many ways, such as (1) retrieved from a 3D dataset, (2) provided by users (Fig. 9), and (3) created manually (Fig. 10).

• Regarding the performance given irrelevant reference and analysis on scenarios where the input concept conflicts with the reference, please refer to the 2nd point of Common Response above for more details. In short, Phidias is designed to adapt to 3D references with different similarity levels thanks to the proposed meta-ControlNet and dynamic reference routing. If the retrieved references are irrelevant, phidias’s performance would not degrade but equal existing SOTA image-to-3D generation methods without 3D reference, as shown in Fig. 7 (b). However, retrieved reference models with misaligned poses or structures can lead to undesired outcomes, as the two failure cases shown in Fig. 15. We believe advanced retrieval mechanisms and a larger 3D database would alleviate this problem, which we leave as further works (L944-945).

Q1: Why does the point cloud for retrieval only contain 10K points?

We follow the training setting of Uni3D to sample 10K points from the mesh surface with colors to leverage the pre-trained prior in the Uni3D encoder fully.

Q2: Why are the results of Fig. 14 (currently Fig. 15) different from Fig. 4?

Fig. 15 shows the actual CCMs of reference we used as a condition of the meta-ControlNet, while Fig. 4 shows the meshes of 3D reference under poses that better present their 3D structures. The actual poses we used to render CCMs of 3D references in Fig. 4 match the input image.

Q3: Inference speed and resource

As suggested, we have detailed the inference speed in Appendix (A.3, L858) and the 3rd point of Common Response. In short, Phidias can generate a final 3D model from the input image in less than 10 seconds, including 9 seconds for multi-view generation, 0.03 seconds for image-based 3D retrieval, 0.4 seconds for rendering the CCM maps, and 0.02 seconds for sparse-view 3D reconstruction. The inference time is tested using a single NVIDIA A100 GPU. The precomputed embeddings of 3D models in the dataset are stored as a tensor of size Nx1024, where N is the size of the dataset. The retrieval latency will not significantly increase even if we scale up the dataset thanks to the super-fast matrix multiplication using GPU.

Thank you again for your efforts in reading and reviewing our paper. We hope our explanation has alleviated your concerns. Please feel free to reply if you have more questions regarding our paper.

Bests,

Authors

Dear Reviewer mgWn,

Thanks for your constructive comments. We hope our responses below address your concerns about our paper. We kindly invite you to watch the attached demo video in the supplementary if you haven’t watched it yet. We would like to hear more from you if you have further questions or comments.

W1: Dependence on Retrieval Quality

As a reference-augmented generation pipeline for image-to-3D generation, the upper bound of Phidias depends on the quality of retrieved 3D reference. However, as stated in the 2nd point of the Common Response (Please refer to the common response above for more details), Phidias's lower bound equals SOTA image-to-3D generation without 3D reference, even if the 3D reference is significantly different from the concept image. The reason is that, during inference, meta-ControlNet will produce adaptive control signals based on the similarity between the 3D reference and concept image (L229-232), and dynamic reference routing will try to alleviate local conflicts (L406-410), which allows Phidias to depend less on retrieval quality for stable inference (Fig. 7 (a) and (b)).

W2: Database Limitations

Note that our promising results are based on a small 3D database of 40K objects. This can be significantly enhanced if the dataset is scaled to the whole Objaverse, ObjaverseXL, and other 3D databases. Similar to RAG in a large language model, which requires an additional large-scale database (all information throughout the network) for achieving promising results, we believe that the advantages of incorporating an additional database for RAG-based 3D content generation outweigh its potential disadvantages, given its potential industrial value and scalable ability.

Q1: Missing citation regarding CCM

Thank you for pointing this out. We have included the reference [a] you suggested in L183. In our first manuscript, we use the term “canonical coordinate map (CCM)” following CRM [Wang et al., 2024a].

Thank you again for your efforts in reading and reviewing our paper. We hope our explanation has alleviated your concerns. Please feel free to reply if you have more questions regarding our paper.

Bests,

Authors

Dear Reviewer tCL1,

Thank you for your constructive suggestions. We hope our responses below address your concerns about our paper. We kindly invite you to watch the attached demo video in the supplementary if you haven’t watched it yet. We would like to hear more from you if you have further questions or comments.

W1: Inference speed

As suggested, we have detailed the inference speed in Appendix (A.3, L858) and the 3rd point of Common Response. In short, retrieving the 3D model and rendering the CCMs do not introduce significant latency during inference. Phidias can generate a final 3D model from the input image in less than 10 seconds, including 9 seconds for multi-view generation, 0.03 seconds for image-based 3D retrieval, 0.4 seconds for rendering the CCM maps, and 0.02 seconds for sparse-view 3D reconstruction. The inference time is tested using a single NVIDIA A100 GPU. The retrieval process is fast as we only need to compare the query vector of an input image (dim=1x1024) with precomputed embeddings of 3D models in the dataset (dim=Nx1024, where N is the size of the dataset) using cosine similarity, as stated in L783-785. The retrieval latency will not significantly increase even if we scale up the dataset thanks to the super-fast matrix multiplication using GPU.

W2: The effectiveness of dynamic routing in Tab. 3 seems limited

Dynamic reference routing focuses on dealing with local conflicts (L235-239), which may not bring significant performance enhancement when computing quantitative metrics but is very important for preserving local details of input concept images during inference. As shown in Fig. 6 (b), the wire in the astronaut will be ignored without dynamic reference routing because the 3D reference does not have this wire.

Q1: Is the meta-ControlNet trained on images of different resolutions?

Yes, meta-ControlNet is trained on images of different resolutions depending on the sampled timestep (noise levels) during each training step. We have detailed meta-ControlNet training with dynamic reference routing in Appendix (A.2, L804-808).

Thank you again for your efforts in reading and reviewing our paper. We hope our explanation has alleviated your concerns. Please feel free to reply if you have more questions regarding our paper.

Bests,

Authors

Hi Reviewer tCL1,

Thank you for your response! In our experiment, 0.02s is an approximate time for generating 3D Gaussians from sparse-view images. The time reported in the LGM paper includes additional time for saving the Gaussians into a .ply file and rendering a visualization video.

For a fair comparison, we report the step-wise time for Gaussian generation, saving, and rendering given the mushroom teapot in Fig. 7 (b) of our revised paper, using the same device (a single A100 GPU).

The tested time of LGM is 0.019s (Gaussian generation), 0.095s (saving .ply), and 0.529s (render a video with 180 frames).

The tested time of Phidias is 0.023s (Gaussian generation), 0.143s (saving .ply), and 0.756s (render a video with 180 frames).

As can be seen, after increasing the number of input images and the resolution, the time for sparse-view 3D reconstruction has increased but is still fast.

Thank you again for your response! We hope our additional experiments and discussions can address your concerns regarding the inference speed of Phidias. Please feel free to let us know if you have any other questions.

Best regards,

Authors

Dear Reviewer phbC,

Thanks for your careful review and constructive comments. We appreciate you recognize our paper as a great work that provides additional controllability for current image-to-3D generation models. We note that you still have several concerns about the performance of Phidias, the majority of which come from misunderstandings regarding the methods and experimental settings. We would like to provide more details regarding your questions and hope your concerns can be addressed.

W1: Model behavior

You said, “As A.4 (current A.5 after we updated the pdf) stated, the augmented ones are still very similar to the ground truth.” Yes, but the self-augmented 3D objects are only part of our curriculum training strategy (L261-265). As stated in Sec. 3.4, during training, we start from these self-augmented 3D objects to avoid the ‘idleness’ of reference, i.e., pushing our model to learn to use the reference frist because they are similar to the ground truth. Then, we will increase augmentation strength and incorporate retrieved references (L879-880) to simulate significant misalignment during inference. Thus, Phidias does see references with different similarity levels during training. Once trained, Phidias performs well with a variety of references, even those retrieved ones that are not very similar (L265).

W2: Retrieval robustness As suggested, we have added a more comprehensive analysis of the bad influence of the unreasonable retried reference with explicit conflicts with the confect image, as shown in Fig. 7 (b). Please refer to the second point of our Common Response for details. Here, we want to emphasize that the lower bound of our performance equals an image–to-3D generation method without reference since our model is designed to ignore the bad influence of unreasonable references. That is also why we achieve promising results with a small retrieval database. If we scale up the size of the database, our model will benefit more from the retrieved reference as the similarity levels increase.

W3,W4: Restricted/unfair experiment setting

• Regarding the input view, Phidias can improve the results significantly given front-view input image, but it also works well with random-view inputs, as shown in Fig. 14 (camel) and Fig. 18 (robot, dinosaur, pig, horse, chest, car, etc.). As stated above, the lower bound of Phidias equals to SOTA image-to-3D generation methods without reference. If not restricted to only-front-view input (either a random view or 2-4 views), other methods will not be better than Phidias, i.e., if their results are reasonable, the results of Phidias will also be reasonable or even better. The core of Phidias’s design is to use an additional 3D reference to improve the results’ quality and controllability adaptively, which is unique from all previous methods.

• Regarding your concern about the quantitative results in Tab. 1, there should be a misunderstanding. (a) the retrieved 3D objects, in most cases, are not very similar to the concept image in the front view when we compute the metrics; (b) the differences between the retrieved reference and GT will result in a marginal reconstruction loss, even the results of Phidias are plausible and better than the baselines. Thus, we do not aim to surprise you with better results for the row “Ours (GT)” and we also know that the row “Ours (Retrieved Ref.)” is still marginal. What we want to emphasize in Tab. 1 is that the actual performance of Phidias should be between “Ours (Retrieved Ref.)” and “Ours (GT Ref.)”. We have included these discussions in the revised paper (L358-361). We have also conducted additional user studies to evaluate our method based on human preference (Tab. 2 and L362-366).

W5: Model usage When there are conflicts between retrieved 3D objects and concept images, Phidias is trained to adaptively adjust the guidance strength of the 3D reference according to their similarity levels (L229-232). If there are significant conflicts, Phidias will follow the concept image, as shown in Fig.7 (b), because the default setting of our task is image-to-3D generation, and we need to follow the image first. However, Phidias also allows users to adjust the guidance strength manually, just like the “control knob” as you said. We have included this functionality as a new application in Fig. 12 and L483-485.

W6: Novelty We hope our explanation above and the 1st point of Common Response have alleviated your concerns regarding our novelty. The novelty of our paper is to bring the concept of RAG into 3D generation. Meta-ControlNet and reference-based inference are solutions for incorporating conflicted 3D reference into the image-to-3D generation pipeline, i.e., the Misalignment Dilemma in L78-79. Regarding why we use diffusion models and the following reconstruction model, please refer to our Q2 and Q3 responses below.