IPDreamer: Appearance-Controllable 3D Object Generation with Complex Image Prompts

Thank you for your questions and suggestions. Below are the responses to the points you raised.

Q1: The written of the paper is not so clear, some details are lack:

Q1.1: The description on how to adopt GPT-4v to generate localization prompts is lack in the paper.

A1.1: Thank you for your feedback. In the revised version of the paper, we have added details on how GPT-4v is used to generate localization prompts in Appendix A.2.5. Specifically, the prompt provided to GPT-4v is as follows:

"You will act as an image analysis agent. Based on the input complex image condition < image >, you need to analyze which parts of the image can be used to guide 3D object synthesis. The multi-view renderings of the initialized 3D model will be provided to you in the form of a video < video >. Based on the input image condition, you are required to generate textual prompts that describe the segmented partial images, which will be used to guide the segmentation of partial image features. Each segmented partial image must also have a corresponding localization prompt, mapping these partial image features onto the 3D object. Please respond in the following format:

Partial image textual prompts: < text1 >, < text2 >, ... <br> Corresponding localization textual prompts: < y1 >, < y2 >, ...

Note that the numbering of the partial image textual prompts must correspond one-to-one with the localization textual prompts."

Q1.2: In Figure 1 (b), the author gives comparison between VSD and IPSDS on text-based generation. But is the proposed method IPSDS need an image prompt? How to compare IPSDS with VSD on text-based generation? Moreover, for the cases in Fig (a), could the author provide the images parts extracted from the reference image of the castles. It’s hard to understand how could we break such things into parts.

A1.2: Thank you for your valuable questions. First, to clarify, Recent advancements in text-to-image generation have allowed single-image-to-3D methods to leverage these techniques for text-based 3D generation. Existing single-image-to-3D research has already showcased results in text-based 3D synthesis. However, it is crucial to note that the images generated by text-to-image models often lack distinct, well-defined foregrounds. This limitation implies that current single-image-to-3D methods struggle to consistently achieve reliable text-to-3D generation. In contrast, IPDreamer leverages complex image prompts to guide the 3D synthesis process, allowing it to produce high-quality, stable results for text-to-3D tasks.

Regarding Fig. 1(a), it is important to clarify that Mask-guided Compositional Alignment combined with IPSDS is necessary only when there are substantial appearance and semantic differences between the initialized coarse 3D object and the reference image, or when multiple complex images are used to guide the optimization of a single 3D object. For the case in Fig. 1(a), high-quality 3D object generation can be achieved using only equations (2) to (7), without the need to extract partial images from the reference complex images. This further demonstrates the effectiveness of our IPSDS.

Q1.3: For eq.9 and eq.10,the author highlights that “they localize the features of the multiple images onto 3D object” in many places such as Line 321-322, 349-350, which makes me very confused. I think the author is adopting eq.9 and eq.10 to fuse information from different image parts to do SDS loss. Therefore, this description is inaccurate and leads to misunderstanding.

A1.3: Your understanding is correct. Equations (9) and (10) modify equation (8) when there is a significant difference between the initialized coarse 3D object and the reference image, or when there are multiple complex images served as conditions. As you rightly pointed out, equations (9) and (10) are used to adjust the cross-attention mechanism during the SDS loss calculation process, which enables the localization of features from multiple images onto the 3D object. Based on your feedback, we have updated the description in the revised paper for greater clarity. If any points remain unclear, please feel free to ask.

Q1.4: Some annotation in the equations are missing, like Z in eq.9.

A1.4: Thank you for pointing this out. We appreciate your careful review. The variable Z in Equation (9) is defined earlier in the text (see line 283) as the query features. Additionally, we have carefully reviewed the paper and added missing annotations to ensure that all variables are clearly described in the revised paper.

Dear Reviewer 4vLr,

We sincerely thank you for your constructive feedback and insightful comments. The core contribution of this paper is the ability to use single or multiple complex images to guide 3D object synthesis, enabling high-quality texture editing and generation without a clear main subject. Specifically, IPSDS and Mask-guided Compositional Alignment are the key contributions of our work, and the use of MLLM analysis serves to further refine the method.

Regarding your concern that "the input image cannot be divided into reasonable semantic regions," we have not encountered any issues in our experiments where IPDreamer was unable to effectively guide the 3D results using complex images. Besides, the ability to guide 3D object synthesis with multiple complex images represents a significant improvement over the generalization ability of current single-image-to-3D methods. While we have not encountered unsatisfactory results in texture editing so far, we acknowledge that your consideration is valid. In response, we have added the following statement in the Future Work section (Appendix A.4) of the revised paper: "Although our IPDreamer has successfully guided high-quality 3D object synthesis in most cases using complex images, there may still be cases of failure. Future work will focus on identifying and addressing such failure cases, while enhancing the generalization ability of IPDreamer and improving the quality of the generated 3D objects."

Once again, thank you for your valuable suggestions, which have greatly helped us refine the paper.

Sincerely,
The Authors

Thank you for your thoughtful review and support. We address each of your questions in detail below.

Q1: The paper leverages GPT-4v as MLLM inputs. How accurate should the MLLM be, in case people don't have access to this advanced MLLM algorithm? Would the output become much worse?

A1: The incorporation of MLLM serves to automate the analysis of reference complex images and 3D objects, specifically to:

1. Identify regions requiring segmentation in reference images;
2. Generate appropriate localization prompts for positioning these segments on the 3D object.

As detailed in Appendix A.3.3 of the revised paper, we present examples of partial images, corresponding localization textual prompts, and optimized 3D objects generated through the addition of Mask-guided Compositional Alignment during the IPSDS optimization process. These results demonstrate that analyzing input conditions is not particularly difficult, and partial images and localization prompts can be manually obtained.

To demonstrate the robustness of our approach without relying solely on GPT-4v, the revised paper includes partial images obtained using the open-source Qwen-VL model combined with SAM. The partial images generated by Qwen-VL and SAM were comparable to those produced by GPT-4v and SAM, showing that even without access to GPT-4v, reasonable analysis results can still be achieved using alternative MLLMs such as Qwen-VL.

By the way, it is worth noting that the generation of high-quality 3D objects guided by multiple complex images relies more on the effectiveness of the Mask-guided Compositional Alignment strategy.

Q2: It's very nice to conduct user studies for genAI works in general. Could authors provide more demographics information in the appendix section? (age, gender, background, etc)

A2: Thank you for your thoughtful question. While we must maintain participant privacy by withholding specific personal details such as age and gender, we can share the professional composition of our study participants to demonstrate the diversity of our sample:

Our 80 volunteers comprised:

• 30 university students
• 20 internet company professionals
• 30 participants from non-computer science backgrounds

This balanced distribution of participants, including both those with and without technical expertise, helps establish the reliability and generalizability of our user study results. We have included these demographic details in Appendix A.2.4 of the revised manuscript.

Q3: I don't fully understand Fig 1b, especially the right images -- what is the contents in the input and what is the actual real-world application of this particular input/output pair?

A3: Thank you for your question. First, I would like to clarify that Fig. 1(b) displays 2D rendered results of 3D objects. Additionally, in the revised version of the paper, we have included the results of single-image-to-3D generation in Fig. 1(b). The purpose of this figure is to demonstrate that existing text-to-3D and single-image-to-3D methods fail to generate reasonable results for cases where the subject is unclear, such as "Leaves flying in the wind" or "Ripples on the water." In contrast, our method can leverage complex images to guide 3D object synthesis, enabling the generation of rational, high-quality 3D objects even for these challenging examples.

In industrial scenarios, 3D modeling is often applied to create special effects for complex scenes, such as "Leaves flying in the wind." These effects typically do not have a distinct physical entity, making it difficult for text-to-3D and single-image-to-3D methods to generate accurate results. Our method can handle such cases, demonstrating its practical value in producing realistic and high-quality scene effects.

Dear Reviewer sW6c,

We sincerely thank you for your suggestions. Your feedback has been instrumental in helping us improve our paper. In the revised paper, we have made modifications based on the recommendations provided by all reviewers. Additionally, we plan to submit a second version of the revised paper before the revised paper submission deadline. In the new version, we will include more text-to-3D generation results based on the reviewers' new suggestions. If you have any further suggestions, please feel free to share them with us—we greatly appreciate your valuable input.

Once again, thank you for your patience in reviewing our work and for providing constructive feedback.

Sincerely,
The Authors

Thank you for your patience and for taking the time to review our paper. Below, we respond to the points you raised.

Q1: The paper primarily focuses on controlling the generation of 3D objects from complex input images. As noted in line 537, "IPDreamer addresses the limitations of existing text-to-3D and single-image-to-3D methods." However, the paper does not include comparisons with relevant single-image-to-3D methods, such as Zero123++ and SV3D. Could the authors clarify why these comparisons were omitted?

A1: Thank you for your question. We would like to clarify that both LRM and LGM, which were already included in our comparisons, are single-image-to-3D generation methods. To avoid any confusion for future readers, we have now explicitly stated in the revised paper that LRM and LGM are single-image-to-3D methods. Additionally, in response to your suggestion, we add comparisons with Zero123++ and SV3D. The experimental results show that our method still achieves the highest FID and CLIP scores.

Q2: In Figure 7, the qualitative comparison presents different samples for each method. Conventionally, all methods are evaluated on the same samples to ensure consistency in comparisons. Could the authors provide insight into this choice?

A2: The reason we presented the qualitative comparison in this manner was to showcase a broader range of examples within the limited space of the paper. This allowed us to better demonstrate the versatility and superior performance of our method across diverse scenarios. Additionally, in the revised paper, we have updated Fig. 7 to show comparisons of our method with all other methods using the same sample. Based on the experimental results, we would like to point out that single-image-to-3D methods rely on clear foreground subjects as input, however such images with distinct subjects are relatively difficult to obtain. For example, objects like "the shining sun," which emit rays of light, are challenging for both single-image-to-3D and text-to-3D methods to generate effectively. In contrast, our IPDreamer can generate them much more accurately.

Q3: The proposed method incorporates several additional components beyond the standard SDS pipeline, including ChatGPT, SAM, ControlNet, and IPAdapter. Could the authors provide details on the runtime overhead introduced by each component, as well as the overall runtime?

A3: The runtime for running ChatGPT, SAM, and ControlNet is around 3-4 minutes, while the optimization time for the 3D object is approximately 1 hour and 20 minutes. Therefore, the overall optimization time is comparable to other methods that generate high-quality 3D results, such as Fantasia3D and MVDream. While our method isn't the fastest, it is on par with the speed of Fantasia3D and faster than ProlificDreamer, which also generates high-quality results. The relative computational speed ratio of our method, Fantasia3D, and ProlificDreamer is approximately 1:1:1.5. Although real-time performance remains a challenge for current high-quality 3D generation methods, our approach, leveraging complex image prompts, enables the production of high-quality 3D assets, making it a practical solution for detailed and controllable 3D generation.

Q4: The method illustration in Figure 2 appears challenging to interpret. It does not effectively aid in understanding the proposed pipeline, and I found it difficult to correlate it with the text. A more intuitive figure might improve readability and clarity.

A4: Thank you for your suggestion. The original version of Fig. 2 was designed to highlight the core contributions of our paper. In response to your feedback, we have updated Fig. 2 in the revised paper to present a clearer and more intuitive depiction of the full IPDreamer pipeline. In the new framework, we illustrate the complete generation process. If you have any further questions about the pipeline, please feel free to raise them.

Q5: I do not understand Figure 1b. What is being generated, a 3D shape or an image? both the leaves and the water ripples look like images!

A5: Thank you for your question. The results shown in Fig. 1(b) are the rendered results of 3D objects. In the original version of our paper, the background of rendered results in Fig. 1(b) was not set to white. We have updated the Fig. 1(b) in the revised paper. We also appreciate your recognition of the quality of the generated 3D results in this challenging example.

Q6: What is the difference between equations (11-13) and (14-17)? Are both used during optimization?

A6: Thank you for your question. When there are multiple input images, the optimization process consists of two key stages: First, equations (11-13) implement Mask-guided Compositional Alignment to localize features from multiple images onto specific regions of the 3D object. Subsequently, equations (14-17) perform global optimization based on IPSDS to enhance the overall quality and coherence of the generated 3D result.

To demonstrate the necessity of both components, we have included comprehensive ablation studies in Appendix A.3.1, comparing results generated under three conditions:

1. Using only Mask-guided Compositional Alignment
2. Using only global optimization
3. Using our full guidance approach

The results clearly show that without Mask-guided Compositional Alignment, features from input images fail to properly localize on the 3D object. Conversely, without global optimization, the generated 3D objects exhibit notable artifacts. These comparisons validate the essential role of both components in achieving high-quality 3D synthesis.

Q7: What is the impact of employing the super-resolution model, ControlNet tiling, on the final generated quality?

A7: Thank you for your question. The super-resolution model is used in our paper to enhance the partial image prompts, especially when the resolution of these images is too low. In Appendix A.3.2 of the revised paper, we compare the 3D results generated with and without partial image enhancement. As shown in the visual comparison, using low-resolution partial images as prompts can lead to 3D results that contain unnatural noise. In contrast, the enhanced partial images result in significantly better 3D outputs, demonstrating the effectiveness of employing super-resolution to improve the quality of the final generated results.

Thank you for taking the time to respond to my questions and comments! I still have several concerns about the paper.

1- As noted by PRnJ, the generated 3D objects do not align with the input image (Figure 1, 5 ). This is a fundamental requirement for image-to-3D approaches, which is not fulfilled by your method. For artists, it is crucial that the generated 3D assets align well with their input images.

2- I understand that your approach is inspired by IPAdapter in Text-to-Image diffusion models, but I am still wondering how it could be useful in the 3D generation domain! In text-to-3D, the input is text, and the model is free to generate any style, while for image-to-3d, it needs to abide by the input image. Your approach seems to be in the middle between the two cases, making it difficult to position and compare against existing approaches for 3D generation.

3- I still have concerns about using different samples for different methods in Figure 7. To be able to position your approach amongst existing approaches, the same samples should be used for all comparisons. "The shinning sun" example that you showed at the bottom of the figure is not sufficient to judge.

Minor: a- The caption of table 1 and 2 say "text-to-3D", but both Zero123 and SV3D are "image-to-3d".

Dear Reviewer zssc,

Thank you for your feedback. First, we would like to clarify that Figure 1 is not simply a style transfer; it adjusts the appearance based on the shape of the initial 3D object. Please take another careful look at the texture editing results presented in the paper (Fig. 1, Fig. 5, Fig. 6, Fig. 9, Fig. 10, Fig. 11, Fig. 12, etc.). Each result is not just a style transfer but aligns the features of the input complex image with the shape and semantics of the initial 3D object, generating high-quality and rational results while preserving the shape and semantics of the initial object. Moreover, for geometric details, as shown in Fig. 8, our IPSDS-geo also fine-tunes the geometric details, which is definitely not just a style transfer.

At the same time, the Large Reconstruction Models that you mentioned, which offer very fast generation times, are typically trained on datasets like Objaverse, which consist of clean, single-object 3D models. Generating complex scenes, such as "leaves flying in the wind," is still a challenging task for these methods. Therefore, existing fast-generation methods struggle with the synthesis of complex, high-quality 3D objects.

We sincerely appreciate your feedback and fully respect your judgment, but please allow us to express that overlooking the key contributions of the paper would also be unreasonable. Thank you for helping us refine the paper, and we wish you all the best.

Sincerely,
The Authors

Thank you for your thoughtful review and recognition of our research contributions. We now provide clarifications for the points you raised:

Q1: Fig.1 is not clear. It's not able to showcase that existing methods struggle with complex images.

A1: Thank you for your suggestion. We have updated Fig. 1(b) to include the generated results from the single-image-to-3D methods, along with the corresponding input image. It is now clearer that, for the input image with unclear subject, the single-image-to-3D methods cannot generate rational 3D results. To provide context for our observation about existing single-image-to-3D methods' limitations with complex images: these approaches are typically trained on datasets like Objaverse, which are characterized by clean, single-object 3D models. Consequently, they are inherently optimized for processing images with well-defined, isolated foreground objects. This fundamental constraint explains their limited performance when handling complex images characterized by rich visual content and sophisticated compositional elements, as detailed in our manuscript.

Q2: The results showcased are not quite aligned with the input image.

A2: Thank you for your question. IPDreamer's core innovation lies in balancing two objectives: preserving the structural integrity of the initial 3D object while incorporating visual elements from complex input images. While the provided coarse 3D objects may be semantically similar to the input complex images, they differ in structural morphology. Our method aligns the appearance of the generated 3D objects with the input images, while maintaining the structure and semantics of the coarse 3D objects as much as possible. This is why our results are not "quite" perfectly aligned with the input image.

Our method makes two key contributions:

1. Generating meaningful 3D objects from complex images with ambiguous boundaries;
2. Performing high-quality texture editing on coarse 3D objects using complex image references.

This design is particularly valuable in industrial applications, where workflows often begin with a predefined basic 3D model provided as a structural starting point. Artists or designers then refine this model by adding intricate details, textures, and stylistic elements based on reference images. Maintaining the basic 3D structure while enhancing visual details is often more practical than achieving perfect alignment with reference images. IPDreamer simplifies this process by enabling efficient editing of coarse 3D objects, reducing the need for extensive manual refinement and allowing for faster deployment in industrial pipelines.

We appreciate your feedback and have included a discussion about potential improvements in alignment accuracy in our Future Work section (Appendix A.4).

Q3: The masks in Fig.4 are not quite aligned with the corresponding parts.

A3: Thanks for your valuable question. In the original paper, the masks shown in Fig. 4 were chosen to clearly demonstrate which parts the textual prompts focus on, instead of using strict binary (0-1) masks. In the actual optimization process, we utilize strict binary (0-1) masks for the Mask-guided Compositional Alignment strategy. We have updated Fig. 4 in the revised paper to reflect these binary masks for better clarity and accuracy.

Q4: It's hard to see the effectiveness of mask-guided compositional alignment.

A4: Thank you for raising this important concern. The effectiveness of mask-guided compositional alignment is most evident in challenging scenarios, specifically:

1. When there are significant disparities in both appearance and semantics between the reference image prompts and the initialized 3D objects
2. When multiple complex images are used as input conditions for 3D synthesis

As demonstrated in Fig. 3, Fig. 6(a), and Fig. 11, without mask-guided compositional alignment, the generated 3D objects are unreasonable. In contrast, with our proposed alignment approach, the results show marked improvement in quality and coherence. These comparisons directly illustrate the crucial role of mask-guided compositional alignment in achieving high-quality 3D synthesis. Please feel free to ask if you need any further clarification.

Dear Reviewers,

We extend our heartfelt gratitude to all reviewers and the Area Chair for their thorough evaluation and constructive feedback on our submission. In our response, we have diligently addressed all the reviewers' questions and incorporated their suggestions to adjust the main paper. Additionally, we have included some extra analyses in the appendix.

We sincerely hope that our revisions address your concerns and meet your expectations. Your insights have greatly improved our work, and we welcome further suggestions to refine it. We look forward to engaging in meaningful discussions and are eager to address any remaining concerns.

Thank you once again for your time and thoughtful comments.

Best regards,
The Authors

Dear Reviewers,

We would like to once again express our sincere gratitude to the AC and all the reviewers for their valuable suggestions, which have helped us significantly improve the paper. We are confident that our work will positively contribute to the 3D community. Therefore, after the final decision, we plan to share the OpenReview link of this paper with as many relevant researchers as possible. Our Q&A record will help those unfamiliar with this paper better understand its contributions and potentially inspire new ideas.

Therefore, we invite the reviewers to feel free to ask any further questions about details that might still be unclear, and we will be happy to provide answers before the end of the discussion.

Once again, we sincerely thank the AC and all the reviewers for their patience and support. We respect and support everyone's judgment.

Sincerely,
The Authors