Clothing-disentangled 3D character generation from a single image

Technical Contribution

1. We respectfully disagree with the opinion that the "main novelty lies in fine-tuning the diffusion model." As demonstrated in our experiments (Table 1 and Figure 3 in the original manuscript), directly fine-tuning the diffusion model (i.e., the baseline method) struggles to correctly disentangle individual part images or combine them into a coherent clothed human image. These capabilities are critical for applications such as virtual try-on.
2. This paper introduces a novel task: generating disentangled multi-view images for each part of a character from a single input image. This task is fundamentally different from holistic clothed human generation. To address this challenge, we propose a novel two-stage disentanglement method that first disentangles each component in the 2D image space and then generates multi-view images for each part.
3. Specifically, we introduce a multi-part attention mechanism for 2D component disentanglement and a combination attention mechanism for multi-view component integration, enabling high-quality part disentanglement and seamless clothing combination.
4. Furthermore, we construct the first large-scale clothing-disentangled character dataset to support future research in this area.
5. Taken together, we believe the contributions of this paper are substantial and address a significant gap in the field.

3D Gaussian Splats

1. As discussed earlier, the primary focus of this paper is on generating disentangled multi-view images. For the multi-view reconstruction, we adopt an existing method, LGM, which supports high-quality rendering generation. While LGM outputs 3D Gaussian splats, other methods such as [a] and [b], which produce mesh-based representations, can also be employed within our framework if desired.

Body Shape Diversity

1. Intuitively, our method does not rely on predefined body shape priors but instead learns the combination in a data-driven manner. As a result, our approach is capable of handling significant differences in body shapes. In the revised manuscript, we will include additional virtual try-on results showcasing the model's ability to manage large variations in body shapes.

Question 1

1. Yes, our model generates one part of the entire image at a time, conditioned on the specific part type provided.

Question 2

1. Without the combination attention module, the multi-view generation model is limited to generating multi-view images for each individual part separately. To address this, we integrate the combination attention module into the model, enabling it to combine these individual parts into a unified clothed human. For each batch, the original condition input includes only the images of individual parts. However, our goal is to generate multi-view images not only for each part but also for the combined clothed human. To achieve this, we introduce a special combination condition image as a flag, which signals the model to generate the multi-view images of the combined clothed human. As demonstrated in our experiments, this design significantly improves the quality of the combined results, producing more cohesive and realistic clothed human models.

Question 3

1. We will include the citation in the revised manuscript.

Applications of Existing Methods

1. Use of Stable Diffusion Architecture

   We do not believe that adopting a similar architecture to Stable Diffusion [1] diminishes the novelty of our work. Many influential papers [a][b][c] have also adopted similar architectures, demonstrating that the framework is a robust foundation for diverse applications. Our contributions lie in addressing the novel task of clothing disentangled generation.

2. Attention Module

   We respectfully disagree with the assertion that our proposed attention module is similar to that of Animate-Anyone [2]. In Animate-Anyone, the attention module is designed to model temporal interactions. In contrast, our attention mechanism serves two distinct purposes: (1) enhancing information exchange between different body parts in Stage 1 and (2) combining these parts into a unified clothed human in Stage 2. These approaches address entirely different problems. Moreover, our method is the first to generate disentangled multi-view images for each part from a single image, which we believe establishes its novelty.

3. Multi-View Reconstruction Model

   Regarding the multi-view reconstruction model, we directly utilize an existing method, LGM [3], as an off-the-shelf solution. We do not claim this aspect as part of our contributions, as the focus of our work lies elsewhere.

Lack of References

1. Thank you for the reviewer’s valuable feedback. While we provide detailed explanations in Section 2, we acknowledge the need for more specific references in the mentioned lines. We will include the corresponding references in the revised manuscript to improve clarity and readability.

Comparative Analysis

1. As discussed in the paper, the task of clothing-disentangled generation from a single image is a novel problem, and most existing methods are designed solely for generating a holistic clothed human. To the best of our knowledge, multi-view diffusion models hold the potential to be extended to this new task. Therefore, we adapted the state-of-the-art multi-view diffusion method as our baseline, enabling it to effectively handle multi-part decoupling and generation.
2. We have provided a detailed explanation of the baseline method in Lines 350–356. Our proposed method and this baseline are fundamentally different approaches, targeting the same task from distinct perspectives.

Various Poses and Perspectives

1. This paper represents the first attempt to address the novel task of clothing-disentangled generation from a single image. It is impractical for a single paper to tackle all the challenges in this new domain. The main contributions of this work are:

   a) proposing a new task focused on disentangled generation,

   b) developing the first method capable of achieving high-quality disentangled results, and

   c) constructing a large-scale dataset to support this research.

   While handling various poses and perspectives is not the primary focus of this paper, we believe our work lays a solid foundation for future research to build upon.

2. Additionally, existing studies [d][e] have shown that recovering T-pose or A-pose representations from images with diverse poses and perspectives is relatively straightforward. These findings indicate that such challenges can be addressed in future work or by integrating existing techniques.

Question 1

1. Yes, similar to previous works [a][b][c], our model leverages the pretrained weights of the Stable Diffusion model.

Question 2

1. The output 3D model in our method is represented by 3D Gaussians, which are fundamentally different from traditional mesh representation. Metrics such as Chamfer Distance or Point-to-Surface are not suitable for evaluating this type of representation.

Question 3

1. The multi-view reconstruction module is not the primary focus of this paper. We directly adopt the LGM model, which utilizes four views for reconstruction. Further exploration of the optimal number of views could be an interesting direction for future work.

[a] Zero-1-to-3: Zero-shot one image to 3d object

[b] Mvdream: Multi-view diffusion for 3d generation

[c] Wonder3d: Single image to 3d using cross-domain diffusion

[d] Charactergen: Efficient 3d character generation from single images with multi-view pose canonicalization

[e] Adding Conditional Control to Text-to-Image Diffusion Models

Combination Attention and Combination Condition Image

1. Without the combination attention module, the multi-view generation model is limited to generating multi-view images for each individual part separately. To address this, we integrate the combination attention module into the model, enabling it to combine these individual parts into a unified clothed human. For each batch, the original condition input includes only the images of individual parts. However, our goal is to generate multi-view images not only for each part but also for the combined clothed human. To achieve this, we introduce a special combination condition image as a flag, which signals the model to generate the multi-view images of the combined clothed human. As demonstrated in our experiments, this design significantly improves the quality of the combined results, producing more cohesive and realistic clothed human models.

More Evaluation

1. Based on our observations, the test images include characters and clothing types that were never seen in the training data. Additionally, the test set features many clothing styles that differ significantly from those in the training data. We appreciate the reviewer’s valuable suggestion. In the revised manuscript, we will include additional results on out-of-domain images.

Question 1

1. The baseline of Wonder3D uses the full character image as the input. It is important to note that the decomposed part images are generated as outputs of our method in Stage I.

Question 2

1. We do not adjust the opacity values of the Gaussians during the optimization process. Once the spatial positions of each part are accurately recovered, the rendering of the composite Gaussians will appear consistent with the rendering of the individual parts.

Differences Between Our Method and ICON [A], HiLo [B], and D-IF [C]

1. Focus on Clothing-Disentangled Generation

   It is important to clarify that the primary focus of our work is on clothing-disentangled generation, rather than treating clothing and the human body as a unified entity, as in [A], [B], and [C]. Our approach addresses this new task by generating not only the complete 3D clothed human model but also the individual 3D models of each clothing item from a single input image. This capability facilitates efficient clothing editing and provides a significant advancement in flexibility and functionality.

2. Feed-Forward Generation Without Optimization

   As demonstrated in Figures 2 and 5 (single GS), our method is fully capable of producing 3D models of both clothed humans and their disentangled components in a feed-forward manner, without requiring an optimization process. Optimization, while optional, is introduced in our method as a refinement step to align the individual 3D parts in a unified space, thereby further enhancing rendering quality. This hybrid strategy ensures our approach maintains both efficiency and precision.

Baseline Methods

1. Clothing-Disentangled Generation as a New Task

   As mentioned earlier, the task of clothing-disentangled generation from a single image is a new problem, and most existing methods are designed to generate only a holistic clothed human. Based on occupancy networks, some works, such as ICON [A], HiLo [B], and D-IF [C], directly predict the holistic clothed human mesh. However, due to the non-watertight nature of VRoidHub 3D models and the complexity of multi-layer occupancy representation, it remains unclear how these methods could be adapted to the task of clothing-disentangled generation.

   In contrast, multi-view diffusion models provide a more natural extension to this setting. To this end, we extended the state-of-the-art multi-view diffusion method to serve as our baseline, enabling it to handle multi-part decoupling and generation effectively.

Obtaining the Rotation Matrix Rc​

Yes, the optional optimization process is utilized to determine the mutual positional relationships between different parts (Rc, Tc, sc). As illustrated in Figure 5 (right), while the feed-forward strategy is already capable of generating a plausible 3D clothed human model (Single GS), the optimization step further enhances rendering quality (Combined GS).

Table 1 Results

Regarding the comparison methods, please refer to the discussion in Baseline Methods. As for the evaluation metrics, we follow the conventions established by previous multi-view diffusion studies [Wonder3d]. Thank you for the reviewer’s valuable suggestion. We will incorporate additional metrics in the revised manuscript.

Image Resolution

Consistent with previous multi-view diffusion methods, we adopt a resolution of 256×256 for generation, primarily due to the limitations of GPU computational resources. When more computational resources become available, higher-resolution inputs can be explored to further enhance the results.

1. In line with previous methods [A][B][C] that generate multi-view images from a single image, we adopt PSNR, SSIM, and LPIPS as evaluation metrics to assess image quality. We believe these metrics are sufficient to evaluate the results of our method and the baseline. Regarding the Clip score, it is typically used to evaluate the alignment between images and text. Since our paper focuses solely on image-based input rather than text-based input, we do not find the Clip score to be an appropriate metric in this context.

2. It is important to emphasize that the main focus of our work is on clothing-disentangled generation, rather than comparing with unified reconstruction methods. As the reviewer has noted, methods such as ICON, D-IF, and HiLO are unable to perform clothing-disentangled generation. Therefore, we do not believe it is necessary to compare our method with these approaches, as it does not detract from the contributions or experimental validation of our paper.

Overall, we believe that the experiments and evaluations presented in the paper are sufficiently comprehensive to demonstrate the effectiveness of our proposed method for the task of clothing-disentangled generation.

[A] Wonder3D: Single Image to 3D using Cross-Domain Diffusion

[B] SyncDreamer: Generating Multiview-consistent Images from a Single-view Image

[C] Zero-1-to-3: Zero-shot One Image to 3D Object