Few-shot Hybrid Domain Adaptation of Image Generator

Thank you for your insightful comments about our work. We have added additional results with highlighting in the Appendix. Here we provide a point-by-point response to the issues raised by you.

Why is hybrid domain adaptation a meaningful task

Current generative domain adaptation approaches typically employ the discriminator to discern whether generated images belong to the target domain. When we require to generate images with integrated attributes, they need to collect images from the hybrid target domain (e.g., a $*smiling*$ $*baby*$ with the style of $*sketch*$ in Fig. 1). However, these images tend to be more difficult to collect compared with single domain in real-world scenarios. In contrast, HDA only requires collecting data for several independent target domains rather than for each possible combination as mentioned by Reviewer 2zhA. Under such circumstances, HDA offers greater flexibility and versatility to adapt the generator to more composite and expansive domain.

Results using different numbers of shots

We perform experiments on lower shots (5-shot and 1-shot) as suggested using various datasets. As shown in Fig. 11 of Appendix, the results of 5-shot are close to those of 10-shot, which integrates the attributes and maintains the consistency with source domain. Although multiple attributes have been learned, the results of 1-shot exhibit relatively lower cross-domain consistency compared with 10-shot and 5-shot. This is because when there is only one reference image per domain, the subspace in our directional subspace loss degenerates into a single point (see Fig. 2). Then distinct generated images corresponding to different noise tend to converge towards the same image in CLIP's embedding space, which comprises cross-domain consistency as depicted in Section 3.3.

Results of conditional generation

As shown in Fig. 12 of Appendix, we conduct the experiments on conditional generation. Specifically, we collect 10-shot images with $*red hair*$ and $*sunglasses*$. Then we use masks to separate these attributes and adapt the generator with masked images for $*red hair*$ and $*sunglasses*$ respectively. We can observe that the generated images possess the corresponding attribute for both single DA and hybrid DA. Simultaneously, these images also maintain consistency with source domain.

Effect of different pre-trained image encoders

As shown in Fig. 13 of Appendix, we conduct experiments on pre-trained Swin and Dinov2 to explore the impact of different image encoders on generated images. Their results all achieve HDA, indicating that our method is agnostic to different pre-trained image encoders. Although they exhibit slight stylistic differences, these are due to their different approaches to extract features into separated subspaces, as depicted in Fig. 8 of Appendix. To converge to the exact realization of the target domain, our method employs the ensemble technique that exploits both Swin and Dinov2. As shown in the figure, the results closely resembles the attributes of the target domain while maintaining the best consistency with source domain.

Compare with DALL-E and Imagen

DALL-E and Imagen are both text-to-image models known for their ability to generate images based on natural language descriptions. They exhibit the advantage of autonomously producing realistic images with specific attributes while maintaining considerable diversity. However, in comparison to our generator, they exhibit two primary drawbacks.

(1) They necessitate intricate and laborious adjustments of prompts to generate images with specific attributes. Additionally, certain attributes are challenging to accurately describe using text, such as artistic paintings. Conversely, our generator can preserve the domain-specific attributes of reference images without the need for intricate prompt engineering.

(2) It is hard to control the characters within images via prompts for them. In contrast, our adapted model maintains strong consistency with the source generator, preserving the same characters from the same noise.

Besides, ours differs from the current trend in customized text-to-image models like DreamBooth. DreamBooth aims to retain the individuals from the training images, requiring similar individuals across the training set. Differently, our goal is to acquire the attributes of target domain like $*baby*$ and preserve consistency with the source generator, ensuring that images generated from the same noise exhibit similar individuals.

Thank you for your insightful comments about our work. We have added additional results with highlighting in the Appendix. Here we provide a point-by-point response to the issues raised by you.

Why we claim to introduce Hybrid generative DA

Prior techniques like style-mixing [1] can generate images from hybrid domain via interpolating latent codes. DoRM is just one of those methods. However, they lack a detailed investigation of HDA, which is merely mentioned as an additional feature. For example, they fail to provide systematic definition and proper evaluation metrics to distinguish good from bad.

With regarding to the method in achieving HDA, they encounter two primary issues:

(1) DoRM is not an end-to-end pipeline to address HDA. They primarily focus on single domain adaptation, necessitating the separate training and interpolating multiple models to accomplish HDA. This may not be the right way to approach this task.

(2) DoRM necessitates intricate tuning of hyperparameters that interpolate between the source and target domains. As demonstrated in the DoRM paper, they assign a weight of 0.005 to the $*baby*$ domain during single domain adaptation. Varying these hyperparameters significantly affects the outcomes. This complexity and sensitivity in parameter tuning make HDA's tuning more intricate and demanding.

How sensitive is our method to the domain coefficient $\alpha$

In our method for hybrid domain adaptation, this parameter controls the composition ratio of the attribute from each domain. As depicted in A.4 of Appendix, we use $\alpha_i = 0.5$ for most experiments without the need for complex and intricate adjustments. To further explore the sensitivity, we conduct the study for simple traversal of $\alpha_i$. As shown in the Fig. 14 of Appendix, the attributes of generated images transit smoothly between domains. Our method produces the similar attribute blending effect when $\alpha_i \in$ {0.4, 0.5, 0.6} .

Comparison to methods for diffusion model personalization

Current trend in customized text-to-image models like DreamBooth aim to mimic the appearance of subjects in a given reference set. Similar to ours, DreamBooth fine-tunes a pre-trained generator for the personalization. However, there are two key differences between personalization and HDA.

(1) DreamBooth aims to retain the individuals from the training images, requiring similar individuals across the training set. Differently, our goal is to acquire the attributes of target domain like $*baby*$ and preserve consistency with the source generator, ensuring that images generated from the same noise exhibit similar individuals.

(2) Dreambooth utilizes text-to-image generator which necessitates intricate and laborious adjustments of prompts to synthesize images with specific attributes. Additionally, certain attributes are challenging to accurately describe using text, such as artistic paintings. Conversely, our adapted model adeptly preserves the domain-specific attributes of reference images without the need for intricate prompt engineering.

Conceptual comparison to Domain Expansion

Domain Expansion aims to expand the pre-trained generator to have the capacity to generate images from multiple domains. To the end, it proposes to repurpose dormant directions in latent space for the new domains. While our approach involves training images from multiple target domains, our objective is to create a unseen composition of given target domains that integrates attributes from them. For example, given training images from $*baby*$ and $*sketch*$, Domain Expansion aims to expand the generator to have the capacity to generate images from $*baby*$ or $*sketch*$. Differently, we aim to adapt the generator to unseen hybrid domain $*baby with the style of sketch*$.

Details of the training time comparisons

In Table 2, our comparison about training time is fair. Specifically, we measure all the time on a single NVIDIA TITAN GPU. Due to the absence of open-source code for DoRM, we implement it following their description in the paper, based on the official implementation of StyleGAN2-ADA. The Seq method is also developed in a similar manner. As for StyleNADA, we utilize their open-source code for training. For an apples-to-apples comparison, we set the same batch size as 4 and the same resolution as $256 \times 256$.

[1] Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110–8119, 2020.

Thank you for your insightful comments about our work. We have added additional results with highlighting in the Appendix. Here we provide a point-by-point response to the issues raised by you.

The results of more domains

Consistent with prior work for few-shot generative domain adaptation (like CDC[1], RSSA[2], and DoRM[3]), our experimental data encompasses both global style ($*sketch*$ and $*baby*$) and local attributes ($*smile*$ and $*sunglasses*$). The combination of these domains demonstrates the effectiveness of our method since they encompass all types of generative domain adaptation.

To provide more comprehensive evidence of validity, we conduct additional experiments on $*Raphael*$ and $*Caricature*$ for both single and hybrid domain adaptation. As shown in Fig. 15 of Appendix, the results integrate the characteristics from multiple target domains and maintain robust consistency with source domain, which further demonstrates the effectiveness of our method.

The generalizability to other domains

To verify the generalizability of our method to other domains, we conduct experiments on church domain following prior work. We adapt the pre-trained generator from $*LSUN Church*$ to $*Van Gogh's house paintings*$, $*haunted houses*$, and the combination of them. As shown in Fig. 16 of Appendix, the results acquire the corresponding style and showcase the preservation of good cross-domain consistency. This aligns with the results observed in the face domain.

The advantages and disadvantages of few-shot domain adaptation techniques with discrimination

Few-shot domain adaptation techniques with discrimination (like DoRM, DualStyleGAN and 3DAvatarGAN) excel in generating images closely resembling the style of the target domain as its discriminator tends to memorize training images. However, they suffer from the notorious issue of model collapse. Especially in few-shot scenarios, they easily overfits to the target images, compromising the diversity of generated images. Additionally, they are challenging to extend into an end-to-end hybrid domain adaptation approach. Their approach to hybrid domain often involves separately training multiple models and interpolating style codes, necessitating multiple model size and training time.

Why is hybrid domain adaptation necessary in these cases

Image editing indeed enables local adjustments resembling the original domain, like $*smile*$ and $*baby*$. However, in this scenario, our method holds several advantages.

Firstly, our objective is to generate images with attributes of target domain while maintaining considerable diversity, which can be applicable in scenarios like data collection. Image editing, on the other hand, requires an original image as input, rendering it impractical for such applications.

Moreover, hybrid domain adaptation has the capability to generate images from hybrid domain containing multiple target attributes like $*baby with the style of sketch*$ or $*smiling person with the style of sketch*$. Image editing, however, lacks the ability to perform such global stylistic modifications, limiting its broader applications.

The editability before and after the domain adaptation

We conduct the editing on the images generated by original and adapted model for both single and hybrid domain adaptation. As shown in Fig. 17 of Appendix, the results indicate the adapted generator maintains similar editability like pose to the original generator. This verifies that the our method effectively preserves original generator's attributes.

Results in 3D GAN setting

We conducted experiments using the popular 3D-aware image generation method, EG3D[4]. Specifically, we replace the discriminator as we did in Fig. 2 for both single and hybrid domain adaptation. As shown in Fig. 18 of Appendix, we adapt the pre-trained generator from FFHQ to $*sunglasses*$ and the hybrid of $*sunglasses*$ and $*smile*$ with 10-shot training images per domain. We can observe that the results effectively integrate the attributes and preserve the characters and poses of source domain.

Comparisons with $*Mind the GAP*$ and $*StyleCLIP*$

While Mind the GAP has a similar loss with direction loss, one term of our proposed directional subspace loss, their motivations differ. Mind the GAP is proposed for one-shot domain adaptation, thus representing the entire target domain using the embedding of a single image. In contrast, for the few-shot setting, we propose representing the entire domain using the subspace formed by the embeddings of multiple images. Furthermore, we utilize pre-trained encoders to obtain separated subspaces corresponding to different domains, enabling us to accomplish hybrid domain adaptation.

The direction loss in StyleCLIP is computed based on the cosine similarity between the generated images and textual prompts within the CLIP embedding space. It does not depend on the source image or domain. In practice, this loss leads to adversarial solutions and sees no benefit from maintaining diversity as depicted in Style-NADA[5]. A mode-collapsed generator producing only one image may be the best minimizer for the distance to a given textual prompt. Differently, our direction loss aims to preserve more characteristics from source domain and maintains its diversity.

[1] Few-shot image generation via cross-domain correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10743–10752, 2021.

[2] Few shot generative model adaption via relaxed spatial structural alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11204–11213, 2022.

[3] Domain re-modulation for few-shot generative domain adaptation. arXiv preprint arXiv:2302.02550, 2023.

[4] Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16123–16133, 2022.

[5] Stylegan-nada: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108.00946, 2021.