Integrating View Conditions for Image Synthesis

Thank you for your insightful comments. We will appropriately revise some details and add additional visualization results, which is presented as Appendix A.4 Additional Visualization Results of the revised PDF now, and we encourage you to review these to address any remaining concerns you might have.

Q: Can the authors provide more outdoor examples, especially with realistic lighting effects?

A: We appreciate the suggestion. In the revised manuscript, we include more outdoor examples to showcase the model's capability to handle diverse lighting conditions, which is presented as Appendix A.4 Additional Visualization Results of the revised PDF now. From Appendix 4, we can conclude that based on stable diffusion's ability to understand the entire image, our framework has a certain ability to render images with realistic lighting effect, but it still cannot achieve the same effect as physics-based models.

Q: Certain details, like text on the hat in Fig. 1, are modified. Could the authors provide improvements for stricter shape fidelity?

A: We acknowledge this concern. This is indeed an issue with Stable Diffusion 1.5. Although there have been partial improvements in SDXL, it is still hard to achieve complete fidelity preservation, even if we provide a very intricate text prompt. In the revision, we'll provide a discussion on this limitation that SDXL performs better but it lacks corresponding community now.

Q: Showcase more qualitative results, emphasizing the model's controllability.

A: Agreed. We will include additional qualitative results in various scenarios, showcasing the model's controllability and its response to varying textual descriptions for object generation within different contexts. This part is presented as Appendix A.4 Additional Visualization Results of the revised PDF now.

Q: How does the model obtain view and text conditions during testing?

A: On the one hand, LLM Planner will combine the view conditions of the input image, reference image, and those from the input text, calculating the resultant view conditions to be output. These calculated conditions are then fed into the pose synthesis module. On the other hand, LLM Planner will combine Segmentor and BLIP to obtain detailed object description.

There is no need to expand relative camera view labels into sentences, because our pose estimation module accept camera view parameters directly. What LLM needs to do is to pass some numbers to a specific python function.

Q: How does the model address conflicting conditions?

A: The model's strategy in handling conflicting conditions relies heavily on the configuration of parameters like $text~threshold$ and $box ~threshold$ within the Segmentor component.

When provided text descriptors (like sofa color or shape) diverge from the original reference, the segmentor may struggle to extract relevant information. Consequently, in such scenarios, the resulting output may be the same as the original image. However, the effectiveness of the model in these situations largely relies on the robustness of the segmentor. If the segmentor is robust enough, it can work normally.

We show a successful case and a failed case, and this part is presented as Appendix A.4 Additional Visualization Results of the revised PDF now.

Q: Are there examples involving more intricate subjects?

A: We appreciate the suggestion. Defining the default pose information of intricate subjects like Pikachu, a Picasso painting, or a badminton-playing prince poses a challenge. In such cases, we need collect a series of similar images that given the same view to achieve synthesizing the reference object with the same pose as the source image. And view conditions might be invalid due to the absence of a default perspective. Therefore, our framework currently focuses on specific categories, such as indoor furniture, where default perspectives are well-defined.

However, we can also give specific relative view conditions towards the reference images by text prompt, without concerning the view conditions from the source image. And in this case, our model works well. So we also add some examples involving more intricate subjects. This part is presented as Appendix A.4 Additional Visualization Results of the revised PDF now.

Thank you for your constructive comments. We will appropriately revise the evaluation table and add additional visualization results, which is presented as Appendix A.4 Additional Visualization Results of the revised PDF now, and we encourage you to review these to address any remaining concerns you might have.

Q: Why integrate view conditions for image synthesis. What is technically unique and novel in it?

A: While drawing upon established methodologies, our unique innovation rests in the integration of view conditions for object synthesis. This novel approach presents a distinct and pragmatic solution to a prevalent challenge in image synthesis. The strength of our framework lies in its adaptability across diverse domains while requiring minimal training resources (e.g., 1 GPU for 6 hours), setting it apart from resource-intensive alternatives (e.g., a concurrent method that demands 8 GPUs for 144 hours). We underscore the novelty within the synthesis framework itself, rather than focusing solely on individual components.

Our system's innovation stems from the fusion of established components, specifically designed for viewpoint-guided image editing. The uniqueness lies in our methodological approach, a novel combination and application of these components, specifically tailored to tackle the complexities of viewpoint-based image synthesis. Similar to the best paper "Planning-oriented Autonomous Driving" showcased at CVPR 2023, our method extends and refines existing methodologies to effectively address the practical challenges in this domain.

Q: Incomplete evaluation.

A: We appreciate your feedback regarding the "Incomplete evaluation". In response, our revised version will feature additional experiments involving outdoor scenes, aiming to illustrate the versatility of our method. Besides, it's essential to note that our method primarily focuses on modifying specific objects under defined view conditions, irrespective of the background. Hence, the distinction between indoor and outdoor scenes may not significantly impact our core findings.

Furthermore, while we have outlined an evaluation of controllability in Sec. 4.3.3, we acknowledge the need for a more comprehensive assessment by human evaluation. To address this, we conduct a human evaluation where evaluators will judge the alignment of synthesized object poses with input conditions. The resulting scores from this evaluation will be incorporated into the revised manuscript to provide a more detailed and comprehensive evaluation of controllability.

Q: Details about personalization module.

A: In Fig. 2, the personalized pre-trained diffusion model denotes a model fine-tuned to accommodate unique object characteristics, such as Dreambooth, LoRA, and their successive versions. The personalization module represents some trainable parameters, and it is optimized by images acquired throughout the pose synthesis process.

The specific method utilized within this personalization module is not explicitly detailed due to the rapidly development of this domain. The selection of method for personalization should be guided by practical applications, aiming to find an optimal balance between efficiency and performance. For example, if your want to preserve more details of the reference object, Dreambooth is a good choice. But if you need faster training speed, LoRA might be a better choice.

Q: Can the system work well if there are more than one object to be replaced in the source image?

A: Yes, this is equivalent to going through the process multiple times, processing only one object each time. And our system can sucessfully handle it, seen as Appendix A.4 Additional Visualization Results of the revised PDF. This will work well when the Segmentor module works well.

Q: What dataset is used for the comparisons in Sec. 4.2?

A: We used the dataset comprised of product images spanning various categories, gathered by our team from publicly available repositories, as described in Sec. 4.1 of our manuscript. It's important to note that we strictly maintain a clear distinction between the training set and the test set, ensuring that the data used in comparisons have not been part of the training process.

Thank you for your careful comments. We will appropriately revise the pose estimation performance table and add additional visualization results, which is presented as Appendix A.4 Additional Visualization Results of the revised PDF now, and we encourage you to review these to address any remaining concerns you might have.

Q: Why integrate view conditions for image synthesis. What is technically unique and novel in it?

A: While drawing upon established methodologies, our unique innovation rests in the integration of view conditions for object synthesis. This novel approach presents a distinct and pragmatic solution to a prevalent challenge in image synthesis. The strength of our framework lies in its adaptability across diverse domains while requiring minimal training resources (e.g., 1 GPU for 6 hours), setting it apart from resource-intensive alternatives (e.g., a concurrent method that demands 8 GPUs for 144 hours). We underscore the novelty within the synthesis framework itself, rather than focusing solely on individual components.

Q: Similar orientations in figures 1 and 3.

A: The similar orientations in the reference images depicted in figures 1 and 3 can be attributed to their adherence to the default view predetermined within our pose estimation module. Acknowledging the necessity for a more varied representation, we include a broader array of examples showcasing with objects in diverse orientations, which is presented as Appendix 4 in the final section of the revised PDF now. Specifically, we estimate both the relative view information ($v1$) of the original object and the relative view information ($v2$) of the reference object. With the relative view information ($v3$) provided by user prompt, by computing $|v2-v1+v3|$, we derive the relative view information required to transform the reference object effectively into the view of the original object.

Regarding scale information, if the original objects and reference objects are of the same type, there are no issues. However, when dealing with different types of objects, our approach involves employing a series of traditional transformations to adjust the reference object to the mask area. Consequently, there might be some challenges in open domain data. We leave it for future work, by providing a solution where users can manually scale objects through the graphical user interface (GUI).

Q: Details about table 3.

A: We appreciate your feedback and have updated Table 3 as follows with more experiments with more suitable hyperparameters to improve accuracy.

Q: Details about section "Effects of View Conditions".

A: We apologize for any confusion arising from our previous explanation regarding the "Effects of View Conditions". Allow us to clarify further. Our initial intent was to highlight the robustness of our predictions within a reasonable variance. Nevertheless, it is crucial to claim the importance of precise pose estimation and accurate view conditions in achieving realistic object synthesis.

The slight column indicates a level of robustness in our predictions because in these tests, they can combine the semantic information of the background and the pose information of the reference object to comprehensively determine the generated pose information. But in many cases, such as empty outdoor scenes, there is no clear background semantic information to guide the pose of generated objects. Besides, when comparing these slight variations to the perfect column, it becomes evident how crucial precise pose estimation and accurate view conditions are in achieving a more accurate outcome in object synthesis. Or in other words, robustness determines the lower bound of the generated effect, and effectiveness determines the upper bound of the generated effect.

Thank you for your insightful comments. We will appropriately revise the title and add additional visualization results, which is presented as Appendix A.4 Additional Visualization Results of the revised PDF now.

Q: Why integrate view conditions for image synthesis?

A: The integration of view conditions in image synthesis addresses one limitation in current image editing methods, which often lack precise control over view-related information. It's essential to highlight that, similar to text-guided and image-guided image editing techniques, our approach specifically applies view conditions to the object of interest rather than the entire image. This utilization of view conditions significantly enhances controllability. The title, "View-conditioned Image Synthesis," aligns well with this approach, and we elaborate its significance in the Abstract.

Inspired by diverse text-guided and image-guided regional image editing methodologies, we formulated this title. Your constructive feedback is immensely appreciated, and we intend to incorporate your suggestions. Here are some refined title options:

"Integrating Object View Conditions for Image Synthesis"

"Integrating View Conditions for Regional Image Synthesis"

"Integrating View Conditions for Image Editing"

Q: Provide more details about the pose estimation module.

A: In general, without the view estimation of reference object, our pipeline needs the presentation of the new object in a canonical view. However, we have devised strategies to achieve this without relying explicitly on a canonical view. For instance, we estimate both the relative view information ($v1$) of the original object and the relative view information ($v2$) of the reference object. With the relative view information (v3) provided by user prompt, by computing $|v2-v1+v3|$, we derive the relative view information required to transform the reference object effectively into the view of the original object.

The images generated in the presented results align with both the 6D pose of input images and accompanying text descriptions. While we have showcased only a selection of examples without text descriptions, our revised version will include more diverse instances, such as "Replace A with B, and turn left 90 degrees." Generally, when the new pose harmonizes with the background, the resulting effect tends to be favorable.

In scenarios where the newly added object and the existing object do not belong to the same category, it still works well. Nevertheless, in practical applications, replacing objects across different categories is rare. Our system is tailored to address realistic scenarios, ensuring, for example, that a shoe would not inadvertently substitute for a hat. Despite this, our revised version will include more scenarios where where the newly added object and the existing object do not belong to the same category.

We highly value your feedback and aim to address these concerns comprehensively in the revised manuscript by incorporating additional examples and explanations. We have included some visualization results in Appendix 4 of the revised PDF, and we encourage you to review these to address any remaining concerns you might have.