MoLE: Enhancing Human-centric Text-to-image Diffusion via Mixture of Low-rank Experts

We thank the reviewer for the positive comment, which is truly invigorating and encouraging! Below is our pointwise response, hoping to address your concerns.

Q1: The text description process is inadequate, particularly regarding the manual filtering of data generated by the LLaVA model.

A1: Sorry for the confusion. We take an image and a prompt "Describe this image in one sentence with details" as input for LLaVA to generate the caption of the image. Afterward, we will streamline the long LLaVA caption manually. Specifically, if the caption is very long, we will streamline it with a new shorter caption by ourselves while aligning with the content of the image. We also remove unrelated and uninformative text patterns including "creating ...", "demonstrating .....", etc. For example, if the caption contains, e.g., "creating a peaceful atmosphere", we will remove it to make the model focus on informative words more. We will add a detailed description of the process in our paper.

Q2: How did the authors address the issue of hallucinated descriptions. Why did the authors not consider using some human/clothing attribute prediction methods and then use the generated attributes as prompts for the MLLM to make the results more reliable?

A2: To alleviate this issue, we use CLIP to filter image-text pairs with lower scores. By doing so, we can effectively filter the hallucinated descriptions whose content does not appear in the image.

For the second question, we also appreciate the reviewer's suggestion of using human/clothing attribute models. In our preliminary experiments, we qualitatively compared several CLIP-filtered LLava-generated captions with those generated by attribute models. Both approaches produced captions rich in detail and similar in their expression of global semantics. However, while the attribute model captions were more detailed due to the nature of these models, they occasionally appeared somewhat awkward and less natural compared to the normal descriptions of human habits. (For example, "A full-body shot, an Asian adult female, outdoor, black straight above chest hair, a black silk shirt" vs. "an Asian adult female with black straight hair falling just above her chest, wearing a black silk shirt."). Another potential issue is that the attribute model may ignore the behavior of a person in an image, e.g., running and reading. Therefore, we opted not to use them in our paper, also considering that the former method was simpler to implement. Regardless, we appreciate the reviewer's valuable suggestion and will consider it to combine with our current method for future work.

Q3: Could the authors show some full-body comparison results?

A3: Yes, we show several full-body comparison results based on MoLE (SD v1.5) and MoLE (SDXL) in Fig.4 of the attached PDF file. One can see that even in the case of full-body images where the face and hand areas are smaller, MoLE still can refine these parts. We thank the reviewer's suggestion for this and will add this result to our paper.

Q4: To better evaluate its generalization capability, have the authors considered applying the MoE architecture to transformer-based diffusion models, such as PixArt?

A4: We thank the reviewer's suggestion. To verify the generalization of our method, we attempt to build our MoLE based on PixArt-XL-2-512x512. To compare the performance, we randomly sample 3k prompts from COCO Human Prompts and calculate HPS for MoLE (PixArt) and PixArt. The evaluation process is repeated three times. Our method achieves $21.79 \pm 0.03$ HPS(%) and outperforms PixArt ($21.33 \pm 0.08$ HPS). These results demonstrate the generalization of our method. We are willing to cite and add the discussion to our paper.

We thank the reviewer for the encouraging comment! Below is our pointwise response, hoping to address your concerns.

Q1: An analysis of the ratios of different races in one prompt (A Beautiful Woman)? Provide a discussion that if the ratios of different races are the same, will the generations of different races be improved?

A1: We follow the reviewer's suggestion and use the prompt "A Beautiful Woman" to show the ratios of different races in MoLE. Specifically, we generate 10K images with this prompt. With the help of DeepFace (https://github.com/serengil/deepface), we find that approximately 51.08% individuals identify as white, 5.29% as Asian, 10.18% as Black, 4.31% as Indian, 24.66% as Latino Hispanic, and 4.48% as Middle Eastern.

To verify if the generation of different races can be improved by using a race-balanced dataset, based on our dataset we use DeepFace to reconstruct a new dataset with the same ratio of different races. The newly created dataset comprises 30K images. We use it to train a MoLE model and generate 10K images using the same prompt "A Beautiful Woman". We find that approximately 45.56% individuals identify as white, 7.17% as Asian, 8.32% as Black, 13.41% as Indian, 13.44% as Latino Hispanic, and 12.10% as Middle Eastern. This result has a relatively higher balance of races compared to the previous result, demonstrating that MoLE is beneficial to alleviating race biases with the reconstructed dataset. We are willing to cite and add this discussion to our paper.

Q2: With the introduction of many human images featuring limbs in this dataset, assess whether this approach can effectively mitigate limb deformations.

A2: To assess whether MoLE can effectively mitigate limb deformations, we perform a user study by sampling 20 image pairs from SD v1.5 and MoLE, and inviting 50 participants to evaluate which model produces better human limbs with less deformations. Among the participants, we find that 62% of participants select MoLE, which indicates that MoLE also can effectively mitigate limb deformations.

Q3: The generation of two global scalars may be tricky (Lines 180-182), current experiments show less evidence about the effects of the global scalars.

A3: We feel sorry for the confusion. Actually, we have shown the efficacy of global scalars in our ablation study of Mixture Assignment in Tab 3 where only employing global assignment can also improve the performance compared to SD v1.5 (as well as the model in Stage 1). Moreover, in (a) and (b) of Fig 8, we can see that global assignment is content-aware. For example, when generating a close-up image, e.g., a face image, the global assignment consciously produces large global scalars for the face expert and small global scalars for the hand expert. From this perspective, the global assignment is meaningful.

Q4: The values of S in Figure 1 are more arbitrary to decide the qualities of a generation, so how to decide these values in MoLE?

A4: In MoLE, we use a mechanism called Soft Mixture Assignment to determine these values. This mixture assignment contains two parts: global assignment and local assignment. Specifically, the global assignment takes as input the entire feature map to produce adaptive global scalers, which are allocated to each expert. Additionally, we introduce the local assignment, which takes as input each token to produce local scalers that similarly determine how much weight for each token sent to the experts.

Q5: Since the dataset is considered to be part of the innovation, will the dataset be released in the future?

A5: Yes, we are very willing to release this dataset to advance the development of our community. Moreover, we promise to select a suitable license to ensure compliance with legal regulations and emphasize its exclusive use for academic purposes only.

Q6: The negative board impacts are insufficient in the Conclusion. The authors should analyze whether their approach will introduce more biases on race and other impacts, such as fake faces.

A6: We thank the reviewer's advice. Through our analysis in A1 above, though MoLE may not introduce more biases on race, it also inherits the biases in the training data like pervious methods. As for other impacts such as fake faces, since our method primarily focuses on human-centric image generation, it also inevitably generates fake faces like other SD models, which requires users to leverage these generated images carefully and legally. These issues also warrant further research and consideration. We maintain transparency in our methods with open-source code and dataset composition, allowing for continuous improvement based on community feedback. We will highlight these discussions in the Broader Impact part.

We thank the reviewer for the positive comment! We hope our response presented below can address your concerns.

Q1: Provide and analyze the distribution of the local weight sent to each expert by the sigmoid function.

A1: We thank the reviewer's advice. We provide the distribution of the local weight sent to each expert in Fig.3 of the attached PDF file. To obtain this, we generate 10 samples for close-up images and normal human images, respectively, and collect local weights for each expert. In Fig.3, one can see that for close-up images, e.g., face, the corresponding expert receives more weights with higher values. We think this effectively demonstrates the efficacy of the soft assignment mechanism in MoLE, which adaptively activates the relevant expert to contribute more to the generation of close-up images. When generating normal human images involving face and hand, the two experts contribute equally, and generally, the face expert receives relatively more weights with higher values as the area of the face is typically larger than that of the hand. We will add this result and discussion in our paper.

Q2: Some minor writing errors in the paper

A2: Thanks for pointing out the typos. We will check our manuscript carefully and fix them.

We thank the reviewer for the positive comment! Below is our pointwise response, hoping to address your concerns.

Q1: Comparison with existing datasets like CosmicMan is essential to contextualize its value.

A1: We thank the reviewer's advice. Below, we give a comparison of the differences between CosmicMan and our newly collected dataset, which primarily lie in four aspects:

• From the aspect of image diversity, due to different motivations, CosmicMan only contains human-in-the-scene images while our dataset also involves two close-up datasets for face and hand, respectively. Moreover, to the best of our knowledge, the high-quality close-up hand dataset is absent in prior related studies.

• From the aspect of image content distribution, there is a relatively severe gender imbalance in CosmicMan where female makes up a large proportion around 75% (Please see Fig 3 of Appendix in its paper) while our dataset is relatively balanced (58% vs 42%).

• From the aspect of image size, though CosmicMan and our dataset are both of high quality, our collected images (basically over 1024 × 2048) are relatively larger than CosmicMan whose average size is 1488 × 1255.

• From the aspect of data sources, our dataset is legally collected from various websites including unsplash.com, gratisography.com, morguefile.com, and pexels.com, etc, while CosmicMan is sourced from LAION-5B (See https://huggingface.co/datasets/cosmicman/CosmicManHQ-1.0 ). What sets our dataset apart is not just its wide collection, but also the freshness of the data. As a trade-off, the quantity of our dataset (1M) is relatively smaller than that of CosmicMan (5M).

We will add this comparison to our paper to highlight the value of our dataset.

Q2: Comparison with state-of-the-art methods like HanDiffuser and HyperHuman.

A2: We thank the reviewer's advice. Regrettably, we find that the code for HanDiffuser (https://supreethn.github.io/research/handiffuser/index.html) and HyperHuman (https://github.com/snap-research/HyperHuman) has not been made available. As a result, we are unable to directly compare these methods with our work. We attempt to reimplement HyperHuman (it is relatively simpler) based on our understanding of the paper. However, due to our constraints in time and computational resources, we were unable to complete the reimplementation. Hence, we resort to a user study. Specifically, we invite 50 participants to compare the visualization presented in the two methods' papers with our generated images, respectively. In the user study, we prepare 10 MoLE-HyperHuman pairs and ask participants to select the best one from each pair according to their preference in terms of hand quality. Some compared images are presented in Fig.1 and Fig.2 of the attached PDF file to show the differences between our generated images and theirs. The results show that 58% of participants think our generated images are better than that of HyperHuman. Similarly, for HanDiffuser, we also prepare 10 MoLE-HanDiffuser pairs and ask participants to select the best one. We find that 48% of participants vote for MoLE, slightly inferior to HanDiffuser (52%). All these results demonstrate that our method is effective and competitive with the state-of-the-art methods. More importantly, our method is user-friendly because both HanDiffuser and HyperHuman rely on additional conditions to enhance human and hand generation: HyperHuman takes text and skeleton as input; HanDiffuser needs text, a SMPL-H model, camera parameters, and hand skeleton. In contrast, MoLE only relies on text without the need for any additional conditions, offering greater flexibility and ease of use while maintaining competitive performance.

Q3: Ablation study on experts to see the results of training only one expert compared to training two experts.

A3: Following the reviewer's advice, we train only one expert and compare the performance with that of two experts. We find that one expert achieves $20.19 \pm 0.03$ HPS(%), inferior to that of two experts ($20.27 \pm 0.07$ HPS), which demonstrates the necessity of using one expert for face and hand, respectively.

We hope that our responses above can address your concerns, and turn your assessment to the positive side. If you have any questions, please let us know during the rebuttal window. We appreciate your suggestions and comments!