Cross-domain Adaptation for Few-shot 3D Shape Generation

Thanks for your precious time for reviews. Here we provide corresponding responses and results to cover your concerns.

1.$**Notations**$: We have added the definition of the four functions to the revised manuscript near Eq.1-2, and 9-10. Thanks for your advice.

2.$**Textures**$: Since we only use silhouettes as training samples, it is difficult for our approach to get perfect textures. It is designed to maintain similar texture distributions of source samples, leading to some unrealistic textures in some cases. We also provide the method of learning both geometry and texture information from few-shot data in Appendix C. As the first work to tackle few-shot 3D shape adaptation, we hope to achieve better texture adaptation in future work. The directly fine-tuned models are fine-tuned on textured shapes. They overfit and replicate training samples compeletely, leading to more "realistic" textures than ours. Besides, there are mature methods to generate textures given geometry structures. Recent 3D generation works like MeshDiffusion and DiffusionSDF only synthesize the geometry structures and generate textures with additional methods or models. It is also convenient to fix some problems in textures with modern software for finer results. Moreover, as discussed in Appendix B, GET3D also produces some low-quality features, which may influence the visualized results in this paper.

3.$**Additional Experiments**$: We have conducted experiments on Tables and added the results to the revised manuscript in Appendix L.

4.$**Quantitative Ablations**$: We have moved the quantitative ablations of our approach to the main paper in the revised manuscript following your advice. Thanks!

5.$**Batch Size**$: Yes, N corresponds to the batch size. The N latent codes are sampled randomly in every iteration. We set N=4 following most prior few-shot adaptation works. We also tried a larger N of 16 and got very similar results to N=4. Adapted models are guided to learn from amounts of batches (about 3K-5K in our experiments) of N randomly sampled latent codes to model its distributions of pairwise similarities. In every iteration, we employ N=4 samples to keep the relative distances between them while guiding these samples to get close to the target samples. The key idea is to keep the generation diversity during domain adaptation. We empirically find that N=4 works well for the adaptation setups in our paper.

Please let us know if our responses solve your concerns. If you still have any unclear parts about our work, please let us know as well. Thanks for your review.

Thank you for the detailed response. Most of my concerns have been addressed.

One concern I still have is with the added results on large domain gaps. While I expect the results to look worse than adapting between smaller domain gaps, the results shown seem to have significantly less diversity in terms of geometry. In particular, many of the generated samples are highly similar, while only being similar to 1 or 2 of the few-shot training examples. This makes me question whether maintaining pairwise relative distances makes sense when adapting across larger domain gaps, where the level of diversity in these domains may vary significantly. For example, say you have a source domain with little diversity (where objects may be similar at the feature level) and are adapting to a target domain with greater diversity. If you are only learning the target geometry through adversarial learning, maintaining pairwise relative distances seems like it would encourage mode collapse. This sort of seems like what is happening in the results in Figure 14, where the source Table samples are considerably less diverse than the target modern chairs.

Thanks for your response. Here we add further explanation about our approach on large domain gaps.

1. We randomly displayed some samples in the original Fig. 14. Firstly, we add additional examples to Fig. 14 to the revised manuscript to show that our approach achieves similar diversity as the training samples of modern chairs. Furthermore, we add quantitative results of adaptation from different source datasets in Table 7 in the revised manuscript. It can be seen that the adaptation from tables achieves relatively lower diversity than the adaptation from chairs. However, it still achieves apparently greater diversity than directly fine-tuned models on Chairs $\rightarrow$ 10-shot Modern and Lawn Chairs.
2. Our approach is designed to adapt source models pre-trained on large source domains to target domains using limited data. It is mainly designed for related source/target domains. We add experiments to show that it is compatible with relatively larger domain gaps like Tables $\rightarrow$ Chairs. Compared with samples adapted from the model pre-trained on Chairs, we get lower quality with some incomplete detailed structures and some blurred textures. For some target shapes very different from source samples, it is harder for the proposed approach to produce similar results (similar to the mode collapse mentioned in your response), resulting in lower diversity. Despite this, it achieves reasonable and diverse target shapes.
3. Our approach proposes to keep relative distances to avoid adapted models from overfitting to training samples and replicating them directly, which is not wanted by generative tasks. Adapted models are encouraged to learn the common features of target domains by our approach instead of replicating them directly. For domain adaptation methods, the generation diversity is limited by source models inevitably. However, our approach could still serve as an effective approach to avoid overfitting and maintain diversity when the diversity of source models is relatively limited, as shown by the better quantitative results than directly fine-tuned models on related source/target domains in Table 7.
4. As the first work to tackle the few-shot 3D shape adaptation task, we have provided a practicable solution for related source/target domains. It is also compatible with source/target domains of relatively larger gaps. We will work on better methods to achieve better results in domain adaptation with large domain gaps like Tables $\rightarrow$ Chairs or even larger gaps like Cars $\rightarrow$ Planes in future work.

We hope the explanation can solve your concern. Looking forward to your response.

Thanks for your positive feedback and precious time in reviewing our paper.

Dear reviewer DPcy,

We thank you for your valuable reviews and positive feedback. We have revised the manuscript and provided further response to cover your concern of our approach on large domain gaps. Today is the last day for discussion between authors and reviewers. We are looking forward to your reply.

Thanks for your review!

Thanks for your precious time for reviews. Here we provide corresponding responses and results to cover your concerns.

1. $**Domain Gaps and Datasets**$: The source model is trained on Cars in ShapeNetCore v1 dataset. There are only 31 truck samples in all 3533 car samples, making it hard to generate truck samples with the source model. Besides, the GET3D model is unconditional and cannot be controlled. We cannot say that the source model cannot produce trucks at all. However, we aim to guide target models to generate target samples only. As shown in the visualized samples, the adapted models synthesize trucks while the source model gets other cars with the same noise inputs. We also add CLIP results of the source samples and adapted samples in Appendix K to prove this illustration quantitatively. We have provided some detailed explanations in Appendix G. Actually, it doesn't influence our approach's evaluation even if the few-shot target shapes are directly picked from the source datasets. The source models still fail to produce diverse target samples, which is the target of the adapted models obtained through our approach. Instead, the source models generate lots of samples out of target domains. Prior 2D GAN-based few-shot image generation works directly use samples in FFHQ to adapt the source model pre-trained on FFHQ to a smaller target domain like babies or people wearing sunglasses.

2. $**More Accurate Description**$: Thanks for your advice, we have changed the "few-shot 3D shape generation" related to "first" to "few-shot 3D shape adaptation" in the revised manuscript to ensure its accuracy.

3. $**Learning Geometry Only**$: Our approach is mainly designed for geometry learning, which is mainly considered in 3D shape generation. Our approach is designed to further democratize 3D shape generation. With our approach, only few-shot silhouettes without textures are needed to realize 3D shape generation. We also provide the method of learning both geometry and texture information from few-shot data in Appendix C. As the first work to tackle few-shot 3D shape adaptation, we hope to achieve better texture adaptation in future work. Besides, there are mature methods to generate textures given geometry structures. Recent 3D generation works like MeshDiffusion and DiffusionSDF only synthesize the geometry structures and generate textures with additional methods or models. It is also convenient to fix some problems in textures with modern software for finer results. Moreover, as discussed in Appendix B, GET3D also produces some low-quality features, which may influence the visualized results in this paper.

4. $**Novelty Discussion**$: We have discussed the novelty of our approach in Appendix F. The proposed losses share similar formats with CDC Ojha et al. (2021) and prior contrastive learning methods but still have significant differences. Firstly, CDC only uses features in generators for the computation of losses. Our approach uses both feature-level and shape-level (image-level) information and achieves higher-quality and more stable results compared with using feature-level losses only, as shown in the ablation results. Secondly, our approach is designed for 3D shape generators and applies to geometry and texture information separately. Thirdly, we propose the concept of shared masks to alleviate the influence of geometry for texture adaptation, which helps our approach better preserve the texture information of source samples during domain adaptation. Besides, the novelty of this work also comes from the research topic. This is the first work to study the few-shot 3D shape adaptation task. In addition, we also introduce several metrics for few-shot 3D shape generation evaluation.

5. $**Explanation of Feature Maps**$: The texture generator in GET3D produces tri-plane feature maps. Any point in the generated shape can be projected into the tri-plane feature maps to synthesize its textures. Given a view to synthesize masks, we can extract the corresponding features in the tri-plane feature maps, which are used to synthesize RGB images (including textures) in correspondence to masks with the same camera parameters (angles) in GET3D. That's the reason why we can apply masks to texture features in Eq. 10. GET3D uses RGB images and corresponding masks for 2 different discriminators respectively.

Please let us know if our responses solve your concerns. If you still have any unclear parts about our work, please let us know as well. Thanks for your review.

Thanks for your positive response. Here we add further explanation and experiments for your concerns.

1.$**Key Idea**$: The key idea of our work is to preserve the diversity of source models while adapting to target domains to get diverse results using limited data. With a larger dataset, the source models would have greater diversity, resulting in greater diversity in the few-shot adaptation model produced with our approach. For example, if a source model produces 100 different car examples, our approach transfers it to 100 truck examples. If a source model produces 1000 car examples, our approach transfers it to 1000 truck examples. Our approach is designed to adapt pre-trained source models to a related target domain for diverse results with few-shot data (e.g., 10-shot silhouettes). If we have a large dataset of target samples (e.g., 1000 truck samples), we can directly train a new generative model on this dataset to produce diverse truck samples. However, this work focuses on few-shot tasks, and it would be an efficient method to obtain diverse target 3D shapes using limited data. Users only need to provide 10-shot silhouettes for domain adaptation instead of collecting large amounts of 3D shapes as data.

2.$**DreamBooth**$: DreamBooth shares different targets with this work. DreamBooth is subject-driven and aims to synthesize novel scenes of the same subject shown in few-shot training samples, for example, novel views or novel contexts. The input includes the pre-trained Stable Diffusion Model and a reference set of the target subject. The output is a fine-tuned Stable Diffusion Model, which could synthesize novel scenes of the target subject following corresponding text prompts. The key features of target subjects are preserved. For example, given a reference set of a dog, DreamBooth synthesizes images of the same dog.

Our work follows prior few-shot image generation methods and is domain-driven. It is designed to generate high-quality and diverse samples of target domains. We aim to extract the common features of limited data and maintain generation diversity by adapting source samples to target domains. The subjects in adapted samples are a lot more diverse than few-shot training data. The input includes the source model pre-trained on a large source domain and few-shot samples of target domains. The output is a fine-tuned model which could generate target samples. For example, we use silhouettes of 10-shot SUVs as training data and adapt the source model of cars to the target domain of SUVs. The adapted model is not trained to synthesize SUVs in training data. Instead, it is trained to generate diverse SUVs (cars sharing common features with few-shot training data).

Besides, 2D DreamBooth is not qualified for domain-driven tasks. For example, it cannot learn the domain knowledge (e.g., styles) from few-shot data. $**We have added visualized samples of DreamBooth on domain-driven tasks and this discussion to Appendix F in the revised paper.**$ It's hard for DreamBooth to learn domain knowledge and link it to an identifier like [V]. Therefore, applying DreamBooth to 3DGen may fail to accomplish this task.

3.$**LOGAN and UNIST**$ are based on VAEs trained on enough data from two domains (e.g., table and chairs). Then translators are trained to transfer samples from one domain to the other based on the latent space provided by VAEs. They tackle a different task from this work and aim to build a translation between two domains. Our approach aims to produce diverse results given few-shot data, the domain adaptation is our method instead of our target. Besides, LOGAN and UNIST are not qualified for few-shot data since both VAEs and translators need enough data for training to avoid overfitting. $**This discussion has been added to the related work part**$. Thanks for your advice!

4.$**Our approach on large domain gaps:**$ Based on your review, we further add experiments of our approach on Tables $\rightarrow$ Modern Chairs and Lawn Chairs (10-shot silhouettes as training data). The visualized results have been added to Appendix E. $**It can be seen that our approach is also qualified for domain gaps like Tables$\rightarrow$Chairs used in LOGAN and UNIST.**$ We get diverse chair samples using the model pre-trained on Tables and fine-tune it on 10-shot silhouettes of chairs. In addition, we add quantitative results of the adaptation setups of larger domain gaps in Appendix E as well.

5.$**We have highlighted the modifications in blue in the revised manuscript.**$

Looking forward to your respond. Thanks for your review again!

Dear reviewer KzHF,

We thank you for your valuable reviews and positive feedback. We have revised the manuscript and provided further response to cover your concerns. Today is the last day for discussion between authors and reviewers. We are looking forward to your reply.

Thanks for your review!

Thanks for your precious time for reviews. Here we provide corresponding responses and results to cover your concerns.

1. $**Related domains**$: As the first few-shot 3D shape adaptation work, our approach is designed for related source/target domains. As illustrated in Sec. 5, we also view it as a limitation and hope to achieve domain adaptation for large domain gaps in future work. We have added experimets on relatively large domain gaps like Tables $\rightarrow$ Chairs in Appendix E and show that our approach is qualified for such domain gaps to get diverse target samples using 10-shot silhouettes as training data.

2. $**Application to Future Models**$: This work designs loss functions for 3D shape GANs without modifying the network structures to realize few-shot generation. We only need to extract features and 2D images (masks) for loss computation, which makes it possible to apply our approach to other 3D shape GANs in the future. The rendered 2D images and the features can be extracted from any 3D generative models. We only implement our approach with GET3D since it is the only 3D shape GAN that synthesizes textured 3D shapes directly from random noise inputs yet.

3. $**Equations 4 and 5**$: Eq. 4 & 5 are designed to keep the relative distances between two masks at image-level. Eq. 1 & 2 are designed to keep the relative distances between the features in the geometry synthesis networks (mask is one of the representations of geometry shapes). The geometry features correspond to the whole 3D geometry structure synthesis, making it impossible to identify which parts of features correspond to the common regions of two masks.

Please let us know if our responses solve your concerns. If you still have any unclear parts about our work, please let us know as well. Thanks for your review.

Dear reviewer xeiS,

We thank you for your valuable reviews. We have revised the manuscript and provided detailed response to cover your concerns. Today is the last day for discussion between authors and reviewers. We are looking forward to your reply.

Thanks for your review!

Thanks for your precious time for reviews. However, we find that you have some misunderstanding of our work. Here we provide corresponding responses and results to cover your concerns.

1. $**Task:**$ Our work is the first few-shot 3D shape adaptation work. It is a 3D shape generative task using limited data to synthesize textured 3D shapes instead of synthesizing images of 3D shapes or synthesizing images for domain adaptation tasks. We need to point out that the "cross-domain adaptation" is not the motivation. Instead, it is the method to achieve our motivation of few-shot 3D shape generation. We propose to adapt generative models pre-trained on source domains to target domains using limited data, pursuing high-quality and diverse synthesized textured 3D shapes. The cross-domain adaptation method in our work is different from the traditional domain adaptation task for object recognition, which is usually evaluated with 2D image datasets containing many categories of objects like Office Data or DomainNet.

2. $**Extensive Experiments:**$ We agree with you that the experiments for evaluating the effectiveness of the synthesized 3D shapes are necessary. CD, FID and Intra-LPIPS results are provided in our paper to evaluate generation quality and diversity. Here we add more. Instead of evaluating our data with the traditional domain adaptation models trained on Office data (about 4k images) or DomainNet (about 600k images), we add experiments to evaluate the synthesized data with the CLIP model trained on 0.4 billion text-image pairs. We randomly sample 1024 3D shapes and get 24 views of each as the synthesized datasets. We compute the cosine similarity between the embeddings encoded from these images by the CLIP image encoder and the embeddings encoded by the CLIP text encoder with text (e.g., “a photo of a truck”). The results of training datasets and source samples are provided as reference.

Dear reviewer 4te8,

We thank you for your valuable reviews. We have revised the manuscript and provided detailed response to cover your concerns. Today is the last day for discussion between authors and reviewers. We are looking forward to your reply.

Thanks for your review!

Dear all reviewers,

There are about two days left for discussion. We have provided the rebuttal for your reviews. Could you please look at our responses and let us know what concerns are still not solved or if you still have any unclear parts about our work? We are looking forward to your responses.

Thanks for your review!

Dear all reviewers,

There are only about 4 hours left for discussion. Could you please look at our responses and let us know what concerns are still not solved or if you still have any unclear parts about our work? We are looking forward to your responses sincerely.

Thanks for your review!