HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priors

We deeply appreciate your recognition of the insight behind our method and its model designs. Below are our clarifications for your concerns.

Q1: How the ‘video’ diffusion model can be used for the ‘novel-view’ synthesizer? Why the authors chose this generative model for the novel-view synthesizer? Which novel view is modeled with the video diffusion model given no camera pose?

A1:

1. SV3D [91] is a cutting-edge video diffusion model for object novel-view synthesis, and it's finetuned from SVD [93], an image-conditioned video generative model. SV3D takes an image and the relative camera orbit elevation and azimuth angles as inputs to generate several corresponding novel-view images, which are continuous in 3D space and can be regarded as a video with the camera moving.
2. Thanks to the seamless continuity of images in the generated video, SV3D presents state-of-the-art 3D consistency in the object novel-view synthesis task, so we adopt it as the foundational model for our novel-view synthesizer.
3. In our work, we finetuned SV3D on human datasets with relative camera elevation and azimuth angles to generate 4 orthogonal views, which we found to be sufficient for reconstruction (refer to SIFU [101]).
4. We sincerely appreciate the reviewer's keen observation and commit to addressing omission of target camera pose input in Equation 1 in the revised manuscript.

Q2: For the hierarchical loss, how can the authors access the target human image of certain body parts? Do the authors already have GT body part segmentation? While there could be hair and loose clothes in the image, how do the authors deal with it?

A2:

1. Definition. We employ the widely-used definition of 24 human body parts, such as "rightHand", "leftHand", "head" etc [a]. The GT body part segmentation results are obtained from GT SMPL meshes fitted from scans with predefined semantic vertices and faces. Specifically, we calculate the visible face triangles given the mesh and camera parameters. Each visible triangular face is then assigned the corresponding segmentation label, repeated across all available scans.

2. Explanation.

• Robustness to hair and clothing? First, The hierarchical loss is specifically focused on the head and hands, excluding hair and clothing which can be further supervised using reconstruction loss $L_{Rec}$. Second, instead of using segmentation to reconstruct and then concatenate different components, we design our loss function based on part segmentation. This approach is tolerant of inaccuracies in segmentation.
• Comparison. We also compare our model supervised with a Human Parsing model while still showcasing the current Hierarchical Loss's superiority, as shown in the following table. Besides, compared to 2D image-based human parsing methods, our approach can be utilized online without pre-computation. Additionally, it provides labels with superior 3D consistency, and the supervision signals it offers help to avoid issues associated with training instability.

Q3: Lack of comparison to SOTA methods Please compare with more SOTA methods, such as ICON and ECON.

A3: We appreciate the reviewer's suggestion and provide the following detailed explanation:

1. Focus on Appearance Results: While ICON (CVPR-2022) and ECON (CVPR-2023) primarily focus on geometry, our work emphasizes appearance results, evaluated through image rendering metrics. These metrics are crucial for assessing final reconstruction fidelity, which geometry-focused methods alone do not capture. Therefore, our comparisons target methods that prioritize both appearance and geometry.
2. Comparison to Recent SOTA Methods: We have compared our results with the latest SOTA methods: GTA (NeurIPS 2023), SIFU (CVPR-2024), and TeCH (3DV 2024). These methods have been shown to outperform ICON and ECON in their respective papers. Notably, TeCH is a follow-up work by the same authors of ICON and ECON.

As suggested, we will highlight the rationale for selecting the latest SOTA methods for comparison and cite ECON in the main paper during the revision. Additionally, we will consider including a comparison of our appearance reconstruction results with ICON and ECON's geometry reconstruction results in the supplementary materials.

Q4: Lack of in-the-wild results? As the proposed work aims to generalizable 3D human reconstruction, there should be more diverse generalized results.

A4: We thank the reviewer for the insightful suggestion. In response, we provide more in-the-wild results with diverse human poses, identities, and camera poses in Figure 2 and Figure 3 of the PDF attachment in our "global response" in the rebuttal.

Reference：

[a]. https://meshcapade.wiki/assets/SMPL_body_segmentation/smpl/smpl_vert_segmentation.json

We are pleased to hear that our responses have addressed your major concerns. We truly appreciate your constructive comments throughout the review process, which have greatly helped in improving our work.

Dear Reviewer eC4L,

Thank you for your insightful comments and suggestions regarding our manuscript. We have thoroughly reviewed each point and have made substantial revisions to address the concerns highlighted.

• Clarified the Novel-view Synthesizer (Sec 3.3) and Hierarchical Loss (Sec 3.5)
• Compared with HumanSGD on Adobe Stock images.
• Added qualitative results on in-the-wild images, which reinforce our claims and address your concerns about diverse input images.

If you feel that the rebuttal addresses any of your concerns, we kindly request that you consider updating your score in light of these substantial improvements. If you have any further questions, please do not hesitate to contact us.

We genuinely value the opportunity to enhance our work based on your recommendations.

Best Regards,

Authors of #3255

Thank you very much for your valuable comments! Below are our clarifications for your concerns.

Q1: Why are the input views in low quality (e.g., Fig. 4)? Is this a sign of poor model capacity or inappropriate objective functions?

A1:

1. (a) HumanSplat reconstructs 3DGS from a single image in a generalizable manner, a process that is inherently ill-posed. To enhance its capability to handle diverse scenarios, the quality of the input views is sacrificed for overall balance. For additional details, please refer to Fig 3 of the attached PDF. (b) Similar artifact patterns in the input views have been reported in Fig. 4 of GTA [30] and Fig. 5 of SIFU [101].

2. To further confirm the model's capacity, we adjusted the weights of the reconstruction loss for the input views ($\times$4), illustrating that the current model with reweighting loss has sufficient capacity to render input views with higher fidelity, as demonstrated in Fig. 1 of the attached PDF.

3. Ablation on Reweighting Loss Function on 2K2K Dataset [104]. While qualitative improvements are noticeable by reweighting loss, there are slight declines in quantitative metrics, showing that reweighting loss is not "a bag of freebies". We originally chose to balance the quality of the novel view and the input view, and as your suggestion, we will include this analysis in the revision.

Q2: How are human body parts defined? What if there are prediction errors in the 'ground truth' semantic labels? Is the proposed method robust to such errors?

A2:

1. Difinition. We employ the widely-used definition of 24 human body parts, such as "rightHand", "leftHand", "head" etc. The GT body part segmentation results are obtained from SMPL meshes with predefined semantic vertices [a]. Specifically, we calculate the visible face triangles given the mesh and camera parameters. Each visible triangular face is then assigned the corresponding semantic label. our approach is efficient and can be utilized online without pre-computation. Additionally, it provides labels with superior 3D consistency, and avoid issues associated with training instability.

2. Robustness? There are two primary sources of "Hierarchical Loss Error": SMPL misalignment and the influence of hair and clothing. (1) Robustness to SMPL accuracy? During training, we leverage ground truth SMPL parameters to ensure training stability. (2) Robustness to hair and clothing? To validate this, we compare our model supervised with a Human Parsing model, and the corresponding metrics are as follows:

The ablation study demonstrates the superiority of our method, primarily because the hierarchical loss is specifically focused on the head and hands, excluding hair and clothing. In contrast, the 2D Human Parsing model does not provide supervision that is consistent in 3D.

Q3: Is the proposed method very sensitive to geometric priors? How does it handle challenging poses, and what are the advantages over methods that do not use geometric priors?

A3:

1. SMPL provides geometric priors that alleviate broken body parts and artifacts (please refer to Fig. 3 of the attached PDF). Prior-based methods reduce the demand for large datasets, enhance generalizability, and prevent overfitting. For instance, HumanLRM requires 9.4K scans compared to our method which requires only 3.5K human scans.

2. We have implemented two key strategies to handle misalignment from imperfect SMPL: (1) Using human priors as keys and values in the Latent Reconstruction Transformer, and (2) Setting a window size $k_{win}$ for fault tolerance. Our generalization ability for challenging poses is shown in Fig. 3 of the attached PDF. In the following table, we simulate erroneous poses with random SMPL initialization and demonstrate that our window strategy helps reduce the influence of erroneous human priors.

3. For extreme cases, we acknowledge these as limitations. We will add discussion to the limitation part and consider integrating a better SMPL parameter optimization process.

Q4: Why there are no qualitative comparisons with SIFU?

A4:

1. Up to now, the official repository of SIFU [101] only generates coarse geometry and texture. This is inconsistent with their paper because texture refinement significantly impacts the qualitative outcomes. This is unfair to report the qualitative results (Fig.1 of attached PDF).

2. if they update their code during the revision, we promise to complete qualitative results.

Q5: Change Figure 1 (b) to a different style.

A5: Thanks for your suggestion, and we will make it more distinct.

We promise to incorporate the addressed issues into the final version.

Reference：

[a]. https://meshcapade.wiki/assets/SMPL_body_segmentation/smpl/smpl_vert_segmentation.json

Thank you for your thoughtful review and for taking the time to reconsider our submission after evaluating our rebuttal.

The concern you raised regarding "why HumanSplat cannot replicate the input view in high quality?" is indeed thought-provoking. Upon revisiting our methodology and related works, we recognize that HumanSplat represents 3D content using a set of colored Gaussians ${{\mathcal{G}_i}}$, inherently lacking a direct "shortcut" from the input view to the 3D representation.

Conversely, HumanSGD [22] is equipped with a "Multi-view Fusion" module (please refer to Figure 5 of HumanSGD). This module optimizes the UV texture map by minimizing the LPIPS and L1 loss between the input and rendered views. However, the downside is a time-consuming optimization process that may lead to random and unstable outcomes, such as the multi-face Janus issue in novel views.

For the unsatisfactory input view's results, we will add a discussion to the limitation. Notably, compared to generalizable methods such as GTA (NeurIPS 2023), SIFU (CVPR 2024), and LGM (ECCV 2024), HumanSplat exhibits fewer blurry and unnatural details in the input view. Compared to these optimization-based methods (e.g., HumanSGD and TeCH), $\textcolor{blue}{**we offer faster reconstruction times and enhanced robustness.**}$

We deeply value your expertise and time. If you have any further questions, please do not hesitate to contact us.

Thank you for your insightful and valuable feedback. It is truly inspiring to know that you appreciate the intuitive and clear design of our model, which offers competitive reconstruction times and can render novel views at real-time speeds.

Q1: Missing ablation on human prior? The paper should include ablation experiments where there is no human structure prior, as well as experiments where the human structure prior is estimated instead of GT. These experiments can provide insight on the benefit of the specific human structure prior fusion design.

A1: Thank you for your suggestions. Below are our clarifications for your concerns.

1. During training, we employ SMPL parameters fitted from scans (Ground Truth) to ensure multi-view semantic consistency when computing the hierarchical loss, which helps maintain training stability.

2. During testing, we utilize the off-the-shelf PIXIE [89] to predict SMPL parameters. We report both the estimated and Ground Truth SMPL parameters and compare the model that does not incorporate a human structure prior, as shown in Table 2 (b) and (c) of this paper. We further demonstrate the results of ablation experiments as follows:

These experiments provide insight into the benefit of incorporating a human structure prior, as significant performance degradation is observed without it. Furthermore, thanks to our model's fusion design ($k_{win}$), HumanSplat exhibits robustness in testing against challenging poses with imperfect estimation (additional in-the-wild results can be found in Figure 3 of the supplementary material PDF), with only minor declines in metrics using predicted SMPL parameters.

Q2: How was the reconstruction time computed? PIFu is very lightweight and should not take 30 seconds per human.

A2: The reconstruction time in Table 1 was calculated from a single image (512$\times$512) to the corresponding 3D representation. Although PiFu is a lightweight network, its inference involves densely sampling the 3D grid and evaluating each point to determine its inclusion in the surface, followed by marching cubes to extract the surface mesh. This runtime and reconstruction resolution are dependent on the number of sampling points. The official implementation of this process takes approximately 30 seconds on two NVIDIA GV100 GPUs, as detailed in the follow-up work [a] by the same authors of PiFu.

Reference:

[a]. Li, Ruilong, et al. "Monocular real-time volumetric performance capture." Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII 16. Springer International Publishing, 2020.

Dear Reviewer zEoa,

We appreciate your insightful comments and suggestions on our manuscript. We have thoroughly reviewed each point to address the highlighted concerns:

• Detailed the ablation experiments both without human structure priors and with estimated human structure priors (PIXIE).
• Explained the PIFu's time-consuming procedure.

If you find the rebuttal addresses your concerns or have any further questions, please feel free to contact us.

We truly value the chance to refine our work based on your invaluable feedback.

Best Regards,

Authors of #3255

We thank the reviewer for the insightful and thorough feedback. It is inspiring to hear that you find the paper well-written, organized, and easy to follow, that the method in the context of human avatar creation is novel, and that our experiments are comprehensive. Below are our clarifications for your concerns.

Q1: Measuring the 3D consistency of novel views generated by fine-tuned SV3D and providing any quantitative insights into 3D consistency?

A1: Your insight regarding the novel-view synthesizer's 3D consistency metrics is enlightening. As suggested, we utilize the cutting-edge local correspondence matching algorithm, MASt3R [a], to measure the 3D consistency of the novel-view synthesizer. Specifically, we performed one-to-one image matching between input views and their generated novel views, using the average number of matching correspondences as a metric.

Quantitatively compared to the original SV3D [93], our synthesizer demonstrates a significant enhancement in 3D consistency for human novel-view generation after fine-tuning using human datasets.

We will incorporate it into the ablation study that fine-tuning on human datasets improves the multi-view consistency metrics.

Q2: The generalizability of the novel-view synthesizer (including in-the-wild scenarios)? Is it limited to full-body images?

A2: The novel-view synthesizer, inheriting the generalization and multi-view consistency of SV3D [93] and fine-tuned on human datasets, exhibits robust multi-view generation capabilities and has enhanced generalization abilities within the human domain for different regions and in-the-wild scenarios, including 'head only,' 'upper body,' and 'full body.' Our SV3D model outputs intermediate feature maps, which are not visually indicative of quality. However, the human GS generation based on in-the-wild images demonstrates the generalization ability of our synthesizer, as shown in Figure 2 and Figure 3 of the PDF attachment.

Q3: The qualitative comparison between HumanSGD and HumanSplat is missing. How does it compare with HumanSGD in "in-the-wild" images from the Adobe Stock Website?

A3: In response to your insightful request and suggestion, we provided a qualitative comparison against HumanSGD in-the-wild image from the Adobe Stock Website in Figure 2 of the PDF attachment. Notably, our results demonstrate satisfactory outcomes without per-instance optimization.

Q4: Minor issues.

A4: We will rename it to "projection-aware" and modify L206-207 to "Therefore, to enhance the fidelity of reconstructed human models, we optimize our objective functions to preferentially focus on the facial regions." in our revision, eliminating non-technical parts and ensuring consistency throughout the manuscript.

Reference：

[a]. Leroy, Vincent, Yohann Cabon, and Jérôme Revaud. "Grounding Image Matching in 3D with MASt3R." arXiv preprint arXiv:2406.09756 (2024).

Dear Reviewer 2EjT,

We appreciate your insightful comments and suggestions on our manuscript. We have thoroughly reviewed each point to address the highlighted concerns:

• Measured the 3D consistency metric of novel-views synthesizer based on your recommendation.
• Conducted comparisons with HumanSGD on Adobe Stock images.
• Addressed and clarified minor issues highlighted in your feedback.

If you find the rebuttal addresses your concerns or have any further questions, please feel free to contact us.

We truly value the chance to refine our work based on your invaluable feedback.

Best Regards,

Authors of #3255