CustomNet: Zero-shot Object Customization with Variable-Viewpoints in Text-to-Image Diffusion Models

W1. Limitation of 256$\times$256 Resolution.

• Limitation of Resolution

  • The constraint of the resolution of $256\times256$ can be addressed in two ways. For the first one, we can use an improved Zero1-to-3, e.g., The Zero1-to-3 ++ [8], which has extended the resolution to $512\times512$. The second one is to train our CustomNet on large-resolution data from the current zero-1-to-3 checkpoint.

  • We update more real-world object customization cases in the update paper, shown in Figures 12, 13, 14.

W2. Non-rigid transformations and style changes.

• It can be solved by training our model on a larger web-scale dataset with our dataset construction pipeline. The Non-rigid transformations and style changes can be controlled by text prompts, which has been proven in the SOTA Text-to-Image Diffusion model. Current Text-to-image diffusion models present promising results by training with large web-scale (text, image) pair datasets. With our dataset construction pipeline, we can convert the web-scale text-image pair datasets into (text, image, and novel-view image) pair datasets, then train our model to achieve non-rigid transformations and style changes.

Q1. Difference between the proposed CustomNet and existing methods. How does CustomNet achieve better performance?

• Difference from other works.

  • We should kindly remind you that we novelty incorporate the 3D novel view synthesis capability into the task of object customization, obtaining more flexible customization results with various backgrounds. Note that previous object customization methods (optimization-based Dreambooth[4], Textual Inversion[5], and encoder-based Paint-by-example[2], Anydoor[3]) do not consider the 3D properties of the inserted object but learn a compressed token embedding to represent the object visual information. Therefore, they struggle to balance identity preservation and output diversity. For example, Paint-by-example[2] loses identity, while Anydoor [3] produces copy-paste results, lacking diversity. other optimization-based methods Dreambooth[4], and Textual Inversion[5] suffer long test-time optimization times and are prone to over-fitting.

  • In 3D novel-view synthesis fields, following Zero-1-to-3, most works focus on how to improve it to generate better geometry and quality, e.g. Magic123 [6], One-2-3-45 [7]. They still have the limitation in Zero-1-to-3 that can not be applied in real-world customization applications directly. However, CustomNet provides such a possibility to bridge the 3D novel-view synthesis and object customization with flexible multiple controls.

• CustomNet achieves better identity preservation and varied output generation in the following three aspects:

  • Model architecture. Other methods mainly extract object identity visual information into a compressed embedding vector (whose dimension is normally 1 $\times$ 768), Which makes it hard to preserve the identities of different objects. CustomNet preserves objects' identity by contacting objects corresponding latent with the UNet input latent, and sending them into UNet together. the object latent is the output of the VAE encoder, which can reconstruct the object input image by the VAE decoder. Therefore, the object latent contains the most information as the total model (VAE + Unet) can do.

  • Explicit control with 3D prior. Other methods change the object generation by text control. The text prompt may be ambiguous in semantics, and the words that are not used be describe the object could also affect the object's identity. We design a separate viewpoints control branch that learns to warp the object into different view synthesis. 3D viewpoint warping is a rigid process that has less uncertainty than text description. Therefore, with our explicit control, we can preserve object identity better when generating various results.

  • Dataset construction. Other methods mainly use the text-image pair dataset for customization, while we further generate a multi-view for the object image which provides more information for the model to learn the object visual concepts.

Dear Reviewer h7FN:

Thanks again for all of your constructive comments and suggestions, which have helped us improve the quality and clarity of this paper

We sincerely hope that our added experiments and analyses could address your concerns.

Since the deadline for discussion is approaching, please feel free to let us know if there are any additional clarifications or experiments that we can offer. Your suggestions are highly appreciated.

Best wishes,

Authors

Thanks to the authors for the rebuttal. In light of all the clarifications, additional figures, and open-sourcing the synthetic dataset+pipeline, I am raising my rating.

Q7. Visualization samples of the dataset.

• We have updated more dataset visualization samples in the paper, please see the updated Figure 9.

Q8. Artifacts in the limitation section.

• First, this is a hard case in which other models usually fail to preserve their identity, see the updated Figure 11. Some minor changes would indeed occur when customizing objects with highly detailed textures. Compared to previous methods, our identity preservation is highly competitive. We will mention it in the limitation, and continuously improve the identity in future work.

• The white square in the background causes the artifacts. When testing another method, we place a white square to mask the original object. When testing our CustomNet, we mistakenly keep the white square and the model interprets the white pixel as the background. Our CustomNet will generate a good image without artifacts when provided with a pure background image. Please refer to the revised Figure 4 for further clarification.

• The model allows for modifications to be made to the location bounding box and viewpoints as specified by the user. We have set an overhead view of the dog in the figure, resulting in it appearing squished. Such an artifact can be addressed by modifying the location bounding box and viewpoints.

[1] Yukai Shi, et al. "TOSS: High-quality Text-guided Novel View Synthesis from a Single Image." ArXiv preprint arXiv:2310.10644.

[2] Binxin Yang, et al. "Paint by Example: Exemplar-based Image Editing with Diffusion Models." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[3] Xi Chen, et al. "AnyDoor: Zero-shot Object-level Image Customization." ArXiv preprint arXiv:2307.09481.

[4] Nataniel Ruiz, et al. "DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[5] Rinon Gal, et al. "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion." In International Conference on Learning Representations, 2023.

[6] Guocheng Qian, et al. "Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors." ArXiv preprint arXiv:2306.17843.

[7] Minghua Liu, et al. "One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization." ArXiv preprint arXiv:2306.16928.

Q1. Camera translation and object location.

• We should clarify that the definition and design of RT is consistent with the setting in Zero-1-to-3. More specifically, Zero-1-to-3 renders 3D objects into images with a predefined camera pose that always points at the center of the object. This means the viewpoint is actually controlled by polar angle $\theta$, azimuth angle $\phi$, and radius $r$ (distance away from the center), not the full rotation and translation matrices, resulting all objects to be central of the rendered image. Thus, the Zero-1-to-3 can only synthesize objects at the image center, lacking the capacity to place the object at the arbitrary spatial location in the image.

• Note that zero-1-to-3 cannot perform translation and can only generate objects localized centrally. We tackle this problem by specifying the bounding box at the desired location to the latent. This design is actually consistent with your comments.

Q2. Translation in Zero-1-to-3.

• Note that the zero-1-to-3 model cannot synthesize objects with arbitrary spatial translations, and can only control the size of the objects. Please refer to Q1 for the specified reasons. We address this limitation by placing objects in a specified location bounding box.

Q3. Generation and composition branches.

• You are right. We provide two ways to control the background: textual description and given background image, referred to as generation and composition branches, respectively. For the generation branch, We added a new text branch to enable generating the background complying with the given text description with the Dual-Attention module.

• As for the composition branch, when both given text and background image, the model generation will be dominated by the background images, as it provided more information than text when training.

Q4. Delicate or intricate.

• We will modify our phrase more precisely. Our goal that customizing an object with identity preservation and diverse control is straightforward. However, to achieve this goal, we have several careful designs from dataset construction (both synthetic and real datasets) to model architecture (DualAttn for text and viewpoints, and location control in initial latent concat).

Q5. Definition of RT.

• The RT is controled by the polar angle $\theta$, azimuth angle $\phi$, and radius $r$ (distance away from the center). Following your advice, We added the detailed statements in the paper in Appendix A.2, which is the same as the following sentence. "The definition and design of RT is consistent with the setting in Zero-1-to-3. In Zero-1-to-3, they render 3D objects into images with a predefined camera pose that always points at the center of the object. This means the viewpoint is actually controlled by polar angle $\theta$, azimuth angle $\phi$, and radius $r$ (distance away from the center), resulting in all objects being central to the rendered image. Therefore, the rotation matrix R can only control polar angle $\theta$, azimuth angle $\phi$, and the translation vector T can only control the radius $r$ between the object and camera. Thus the Zero-1-to-3 can only synthesize objects at the image center, lacking the capacity to place the object at the arbitrary location in the image. We tackle this problem by specifying the bounding box at the desired location to the latent."

Q6. Definition of d.

• Thank you for pointing out the mistakes. We have updated the definition on it in the Sec. 3.2.

W1. Novelty and object placement.

About Novelty

We would like to kindly remind you that CustomNet is designed for the task of object customization that can place the object into desired scenes harmoniously with the explicit viewpoint, location, and background controls. We are not a simple application of zero-1-to-3 in the object customization field, but we introduce it to image customization for the first time, by addressing the inherent limitations of zero-1-to-3. Compared to previous image customization methods, CustomNet is the first to investigate the novel view synthesis in the customization field. It can preserve the object's identity while controlling the viewpoint and increasing the output diversity.

• Improvements of zero-1-to-3.

  • Our module designs, together with the training dataset construction, are important improvements for the final outcome of object customization. They cannot be viewed as a simple addition to zero-1-to-3. They are specialized designs for object customization. Without those designs for object customization in the zero-1-to-3 model, the synthesized image lacks the background, and the object is centrally localized, which is far from our goal of object customization. The comparison to the Zero-1-to-3 limitation in our updated Figure 10 shows that our designs are necessary and effective. Besides, the concurrent paper TOSS [1] also mentions in their paper that the extra text input could improve the generation quality of novel views over zero-1-to-3. In our CustomNet, incorporating real-world data with text could also benefit the generation results, which is insightful to improve the novel-view synthesis model.

  • Specifically, as described in the main paper, we demonstrate that Zero-1-to-3 has the following limitations when applied to object customization, and we make specific designs to address those limitations and adapt them to the task of object customization.

    • I. Zero-1-to-3 cannot be controlled by text for background. v.s. We design a dual-attention mechanism to simultaneously control viewpoint and text.
    • II. Zero-1-to-3 cannot generate/composite backgrounds with a given background image. v.s. We designed a unified pipeline to combine it.
    • III. Zero-1-to-3 can only process images located in the image's center. v.s. We design the concatenation module to place the object in arbitrary locations flexibly.
    • IV. Zero-1-to-3 usually generates unreal effects. v.s. We design a real dataset construction pipeline to extend its capability on real data.

• Difference from other works.

  • We should kindly remind you that we novelty incorporate the 3D novel view synthesis capability into the task of object customization, obtaining more flexible customization results with various backgrounds. Note that previous object customization methods (optimization-based Dreambooth[4], Textual Inversion[5], and encoder-based Paint-by-example[2], Anydoor[3]) do not consider the 3D properties of the inserted object but learn a compressed token embedding to represent the object visual information. Therefore, they struggle to balance identity preservation and output diversity. For example, Paint-by-example[2] loses identity, while Anydoor [3] produces copy-paste results, lacking diversity. other optimization-based methods Dreambooth[4], and Textual Inversion[5] suffer long test-time optimization times and are prone to over-fitting.

  • In 3D novel-view synthesis fields, following Zero-1-to-3, most works focus on how to improve it to generate better geometry and quality, e.g. Magic123 [6], One-2-3-45 [7]. They still have the limitation in Zero-1-to-3 that can not be applied in real-world customization applications directly. However, CustomNet provides such a possibility to bridge the 3D novel-view synthesis and object customization with flexible multiple controls.

• Object placement. The original Zero-1-to-3 model struggles to synthesize novel views of an object at an arbitrary location, in their camera settings, they cannot control translation but only azimuth angle, polar angle, and size. Therefore, we introduce a localized bounding box as guidance. It is a simple but effective design for solving this problem. When applying the same location control strategy to zero-1-to-3 directly, we can see that significant artifacts would occur, while our CustomNet has reasonable results in the updated Figure 10.

W2. Dataset construction pipeline.

• Thank you for your advice. Our dataset can help CustomNet obtain more harmonious results since the real images serve as the model target. We will release the dataset, together with the full pipeline for constructing the dataset to contribute to the development of future research.

W1. Novelty and Resolution Constraint.

About Novelty

We would like to kindly remind that CustomNet is designed for the task of object customization that can place the object into desired scenes harmoniously with the explicit viewpoint, location, and background controls. We are not a simple application of zero-1-to-3 in the object customization field, but we introduce it to image customization for the first time, by addressing the inherent limitations of zero-1-to-3. Compared to previous image customization methods, CustomNet is the first to investigate the novel view synthesis in the customization field. It can preserve the object's identity while controlling the viewpoint and increasing the output diversity.

• Improvements of zero-1-to-3.

  • Our module designs, together with the training dataset construction, are important improvements for the final outcome of object customization. They cannot be viewed as a simple addition to zero-1-to-3. They are specialized designs for object customization. Without those designs for object customization in the zero-1-to-3 model, the synthesized image lacks the background, and the object is centrally localized, which is far from our goal of object customization. The comparison to the Zero-1-to-3 limitation in our updated Figure 10 shows that our designs are necessary and effective. Besides, the concurrent paper TOSS [1] also mentions in their paper that the extra text input could improve the generation quality of novel views over zero-1-to-3. In our CustomNet, incorporating real-world data with text could also benefit the generation results, which is insightful to improve the novel-view synthesis model.

  • Specifically, as described in the main paper, we demonstrate that Zero-1-to-3 has the following limitations when applied in object customization, and we make specific designs to address those limitations and adapt them to the task of object customization.

    • I. Zero-1-to-3 cannot be controlled by text for background. v.s. We design a dual-attention mechanism to simultaneously control viewpoint and text.
    • II. Zero-1-to-3 cannot generate/composite backgrounds with a given background image. v.s. We designed a unified pipeline to combine it.
    • III. Zero-1-to-3 can only process images located in the image's center. v.s. We design the concatenation module to place the object in arbitrary locations flexibly.
    • IV. Zero-1-to-3 usually generates unreal effects. v.s. We design a real dataset construction pipeline to extend its capability on real data.

• Difference from other works.

  • We should kindly remind that we novelty incorporate the 3D novel view synthesis capability into the task of object customization, obtaining more flexible customization results with various backgrounds. Note that previous object customization methods (optimization-based Dreambooth[4], Textual Inversion[5], and encoder-based Paint-by-example[2], Anydoor[3]) do not consider the 3D properties of the inserted object but learn a compressed token embedding to represent the object visual information. Therefore, they struggle to balance identity preservation and output diversity. For example, Paint-by-example[2] loses identity, while Anydoor [3] produces copy-paste results, lacking diversity. Other optimization-based methods Dreambooth[4], and Textual Inversion[5] suffer long test-time optimization times and are prone to over-fitting.

  • In 3D novel-view synthesis fields, following Zero-1-to-3, most works focus on how to improve it to generate better geometry and quality, e.g. Magic123 [6], One-2-3-45 [7]. They still have the limitation in Zero-1-to-3 that can not be applied in real-world customization applications directly. However, CustomNet provides such a possibility to bridge the 3D novel-view synthesis and object customization with flexible multiple controls.

Limitation of Resolution

• The constraint of the resolution of $256\times256$ can be addressed by in two ways. For the first one, we can use an improved Zero1-to-3, e.g., The Zero1-to-3 ++ [8], which has extended the resolution to $512\times512$. The second one is to train our CustomNet on large-resolution data from the current zero-1-to-3 checkpoint.

• We update more real-world object customization cases in the update paper, shown in Figure 12, 13, 14.

W2. Details of the Dataset construction pipeline.

We provide more details about our dataset construction pipeline.

• How to specifically determine the foreground object: Given an image, We use BLIP2 (Github repo: https://github.com/salesforce/LAVIS/blob/main/examples/blip2_instructed_generation.ipynb) to extract the foreground object with the following instruction: ''' {"image": image, "prompt": "Question: What foreground objects are in the image? find them and separate them using commas. Answer:"} ''' Then we feed the queried object and its corresponding text to SAM, SAM can receive text as input and output the segmentation mask of the corresponding object.

• When the selected foreground object falls outside the Zero-1-to-3 training domain, we may get unsatisfying results. We have tried to alleviate this issue in our CustomNet

  1. We jointly train with both synthetic and real datasets, which will complement each other and narrow the gap between the synthetic and real datasets.
  2. During training, though the input object image may be unsatisfying, we adopt the original natural image with the object as the target. It is helpful for model training as the target image provides information about how to place an object into a result image harmoniously and naturally, which helps in real-world scenarios. On the other hand, the synthetic dataset endows the model with the warping ability among multiple views. Combined with the two datasets, CustomNet demonstrates natural and high-fidelity customization.
  3. In our CustomeNet, except for the image input, CustomNet also receives text as input. The text helps to improve the viewpoints control ability that Zero-1-to-3 is hard to manage, as shown in the updated Figure 10.

W3. Broad Discussion with other methods.

• To achieve customization, We can use a one-stage model like the optimization-based method Dreambooth[4] or Encoder-based Paint-by-example [2]. The optimization-based method suffers long test-time optimization times and is prone to over-fitting, while the encoder-based method may lose identity. We can also use two-stage methods that first get the foreground object image, then paste it to a new background directly or with diffusion inpainting. The OVTrack [9] uses stable diffusion background inpainting while preserving the appearance of foreground objects for a data hallucination strategy tailored to appearance modeling in multi-object tracking. Our Customnet can generate different views of the foreground object, and use text to generate background for it.

• We have conducted quantitative and qualitative comparisons to the state-of-the-art encoder-based and optimization-based image customization methods in Sec. 4 of the main paper. We update further discussion with the OVTrack[9] and other models, such as Blended latent diffusion[10], etc. in the Discussion in Sec. 3 in the updated paper.

[1] Yukai Shi, et al. "TOSS: High-quality Text-guided Novel View Synthesis from a Single Image." ArXiv preprint arXiv:2310.10644.

[2] Binxin Yang, et al. "Paint by Example: Exemplar-based Image Editing with Diffusion Models." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[3] Xi Chen, et al. "AnyDoor: Zero-shot Object-level Image Customization." ArXiv preprint arXiv:2307.09481.

[4] Nataniel Ruiz, et al. "DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

[5] Rinon Gal, et al. "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion." In International Conference on Learning Representations, 2023.

[6] Guocheng Qian, et al. "Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors." ArXiv preprint arXiv:2306.17843.

[7] Minghua Liu, et al. "One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization." ArXiv preprint arXiv:2306.16928.

[8] Ruoxi Shi, et al. "Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model." ArXiv preprint arXiv:2310.15110.

[9] Li, Siyuan, et al. "OVTrack: Open-Vocabulary Multiple Object Tracking." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[10] Omri Avrahami, et al. "Blended Latent Diffusion." In SIGGRAPH, 2023.

Dear Reviewer 96bJ:

Thanks again for all of your constructive comments and suggestions, which have helped us improve the quality and clarity of this paper

We sincerely hope that our added experiments and analyses could address your concerns.

Since the deadline for discussion is approaching, please feel free to let us know if there are any additional clarifications or experiments that we can offer. Your suggestions are highly appreciated.

Best wishes,

Authors