Expressive Gaussian Human Avatars from Monocular RGB Video

Q1: How can the effectiveness of this confidence be ensured?

A1: The predictor could leverage the information contained in rendered RGB and depth to adaptively learn the confidence in an end-to-end data-driven manner. The experimental results have validated the effectiveness of CL and we also present some visualization in Figure 3 of the rebuttal pdf. Another benefit of CL is to reduce the number of Gaussians from 21,038 to 19,993.

Q2: GaussianAvatar also demonstrates photorealistic 3D human body generation, and the authors need to include more discussions with it.

A2: GaussianAvatar is one of the pioneering animatable 3D Gaussian models which is learned from a monocular RGB video. Its representation is further enhanced via two key components for final photorealistic quality. Dynamic properties are designed to support pose-dependent appearance modeling, while joint optimization of motion and appearance helps tackle inaccurate motion estimation. We will add this discussion in related work.

Q3: Suggestion on revising some writing issues, such as a too-brief related work section, inconsistent formula definitions (e.g., in Eq. 11), and some missing punctuation in formulas.

A3: Thanks for pointing out these issues. We will discuss more works in the related work section, e.g. GaussianAvatar, fix the inconsistent formula definitions, and add the missing commas in Eq. 7, 8, 9, 10 and 13. We will incorporate these modifications in the revised manuscript.

Q4: Specific requirements for videos in terms of the method.

A4: Since our method is reconstruction-based, our main requirement is that the areas of interest should be captured in the video. For example, for the full-body X_Human dataset, the monocular RGB video includes observations around the person of interest. On the other hand, for the upper-body UPB dataset, since the area of interest for sign language is only on the upper body, we only need to capture the corresponding appearance of the person. We note that we keep the input RGB frames of all comparison methods consistent for fair comparison.

Q5: How does the method perform when the magnitude of the action being driven is out-of-distribution?

A5: It is hard to quantify the magnitude of out-of-distribution actions in a principal way (although if the reviewer would like to indicate a specific metric/evaluation, we would be happy to explore it). With that being said, on the UPB benchmark for instance, our evaluation is conducted on novel poses that are not used during training, so our evaluation indicates the performance on the novel pose setting. Moreover, qualitatively, we demonstrate the results of our avatar driven by an in-the-wild SMPL-X sequence with unseen poses and unseen identities. By inspecting the results, the expressive details are quite faithful. However, if the motion used at test time is completely different, it is likely we will observe artifacts in the reconstruction.

Q6: Inference speed of the method.

A6: Our method could render the 1080p image with the inference speed of 361.02 fps on one RTX A5000.

Q1: One of the contributions of using and optimizing SMPL-X for expressive avatar is not novel. This has been done in previous works GVA.

A1: GVA is a concurrent work and has not been accepted. Compared with GVA, our fitting design is quite different, since ours explicitly focuses on the fine-grained areas (especially on hand) to meet the requirement of expressiveness. Considering GVA is not open-sourced, we are not able to compare with it. However, we have visualized that our fitting method outperforms the current SOTA SMPL-X estimation method in both Figure 4 (main paper) and Figure 1 (rebuttal pdf).

Q2: More comparison with recent methods such as SplattingAvatar.

A2: Thanks for this suggestion. The results of SplattingAvatar are shown in A1 of General Response. We will add these experiment results in Table 1.

Q3: Clarification on inconsistent ablation study across the datasets.

A3: Table 2 demonstrates the ablation results on the XHumans dataset. For XHumans, we directly utilize its provided accurate SMPLX annotation. That is to say, we do not need the SMPL-X fitting involved. For Table3, we further update the ablation results.

Q4: PSNR, SSIM, and LPIPS should be calculated from four novel views to ensure the robustness of the method across different perspectives. Additionally, animation and novel pose metrics can be included to display differences between methods.

A4: We calculate PSNR, SSIM and LPIPS from different viewpoints. To our best knowledge, we cannot find any animation and novel pose metrics reported in relevant works (GART, GauHuman, SplattingAvatar, etc.)

Q5: Clarification on CL role in the framework.

A5: We observed that CL helped with perceptually improving the results, which is also reflected in the consistent performance improvement on the LPIPS metrics. Moreover, another benefit of CL is that it led to a more compact representation (e.g., reducing the number of Gaussians from 21,038 to 19,993). We further present some qualitative results (w/o and w confidence-aware loss) in Figure 3 of the rebuttal pdf.

Q6: Project page for visualization including a 360-degree rotating video, animation, and zooming.

A6: Due to the NeurIPS policy ("all the texts you post (rebuttal, discussion and PDF) should not contain any links to external pages"), we are unable to provide the external link to our built project page. In fact, we have included 360-degree rotating video (Supp video) and animation (Supp video) and zooming results (both Supp video and figures in main paper). We promise to release the project page afterward.

Q7: Is e and λt in Equation 14 learned or fixed?

A7: Both e and λt are fixed.

Q8: Discussion on failure cases and outlining potential future work.

A8: Thanks for pointing out this issue. For the fine-grained expressive areas, the failure cases happen when hand interaction exists, which is mainly caused by the false driven SMPLX signal (e.g. the driven SMPLX has implausible interpenetration between hands). From the holistic view, some 'floaters' may exist sometimes, where general 3DGS modeling also exists.

We outline the potential future works as follows,

1. Modeling capability on non-rigid elements, such as loose cloth (dress). Modeling cloth on the avatar has been a challenging topic, even separately studied in several works. A potential solution could be parameterizing the cloth deformation or adding more prior on cloth type, so as to provide more driving signals.

2. Generalizable human avatar from monocular RGB video. Current methods in this topic mostly need per-subject optimization, which needs to be re-trained to any new given subjects. It is worth exploring if we could get an avatar from a monocular RGB video of any given subject with a single feed-forward pass.

3. Robustness. It is worth exploring if we could build the human avatar well with more limited source inputs, e.g. a few images with even mutual occlusion, a single image, etc.

Q9: Information on the optimization time compared to previous methods and the resource requirements.

A9: For the optimization time, we generally divide it into two stages. Take a real-world 1080p video as an example, the first stage (data preprocessing) needs 10.2 minutes, while the second stage (avatar modeling) needs 7.5 minutes on RTX A5000. Compared with the avatar modeling stage of EVA, 3DGS+SMPLX needs 7.5min, GART+SMPLX needs 12.9 min, GauHuman+SMPLX needs 4.5min, and SplattingAvatar needs 45.2 min. The resource requirement could be further lowered down to consumer GPU cards like 3090, but at the expense of speed.

Thank you for the detailed responses. I also recommend including the fixed values of e and λt​ in the paper, along with a justification for how these values were determined. Given the overall quality of this submission, I will recommend the acceptance of this paper. Please include the additional experimental results in your revision.

Q1: Suggestion on simplifying some aspects or providing more intuitive explanations.

A1: Thanks for this suggestion. We commit to releasing our code upon acceptance. Our work makes a step towards building an expressive avatar from the real-world video. In short, our method aims to answer two key questions below,

1. How to generate a well-aligned SMPLX mesh to provide accurate correspondence for expressive avatar modeling? We propose a fitting-based alignment method explicitly focusing on fine-grained details (e.g. hand).

2. How to enhance the expressiveness during avatar modeling? We propose context-aware adaptive density control, in coordination with the feedback mechanism.

Q2: More discussion on the generalizability to other types of monocular videos or environments.

A2: Currently, our work has included various types of monocular RGB videos from controlled capturing setting to those collected from the Internet. The involved human poses contain large body motion (e.g., playing basketball, weight lifting, do kicks, dancing), along with fine-grained motions (e.g. finger counting, using tools, sign language). Among them, considering the complexity and practicability of sign language, we separately involve it to evaluate expressiveness and collect a new benchmark (UPB) from the Internet. We acknowledge that there are more types of videos and environments that could be further explored (such as when the people are occluded by other objects, changing light conditions) and will add this discussion as promising future work.

Q3: Suggestion on including more diverse baselines or variants to demonstrate the robustness of the proposed approach.

A3: Thanks for your suggestion! We have used the Splatting Avatar method from CVPR 24 and evaluated it on the datasets we used. The results are shown in A1 of General Response. We will add these experiment results in Table 1.

Q4: How does EVA perform with videos that have significant occlusions or low lighting conditions?

A4: EVA can deal well with self-occlusions, which usually exist in each frame, especially on the hand areas. This can be attributed to our proposed SMPL-X fitting method which could provide reliable correspondence to map the pixels to the canonical space. Both qualitative and quantitative results demonstrate the effectiveness of our method. Besides, we expect that EVA could also handle low lighting conditions (as long as the lighting is not changing significantly), since our SMPL-X fitting method should be robust to low lighting conditions and with good SMPL-X estimates, we expect the Gaussian Splatting stage to perform well.

Q5: Are there any specific preprocessing steps required for the input monocular RGB videos?

A5: Since our method is reconstruction-based, our main requirement is that the areas of interest should be captured in the video. For example, for the full-body X_Human dataset, the monocular RGB video includes observations around the person of interest. On the other hand, for the upper-body UPB dataset, since the area of interest for sign language is only on the upper body, we only need to capture the corresponding appearance of the person. We note that we keep the input RGB frames of all comparison methods consistent for fair comparison.

Q6: Can the proposed method be extended to capture other expressive elements, such as cloth or hair movements?

A6: We believe our work could be extended to capture other expressive elements. One possible solution is to embed the driving signal with more modeling capabilities considering the other expressive elements that we need to incorporate. In this way, our method could benefit from explicit signals to perform modeling. However, this is beyond the scope of our current work.

Q7: The performance heavily relies on the quality of the datasets used. Real-world applications might present challenges not covered by the current benchmarks.

A7: The high-quality dataset is needed to ensure sufficient resolution on fine-grained expressive areas for training and evaluation. With the popularization of low-cost consumer cameras, high-quality data (e.g. 1080p) becomes much easier to capture, even via personal phones. With that being said, our proposed SMPLX fitting simplifies and robustifies the whole avatar generation pipeline, such that it works with sufficient robustness in Internet videos (e.g. see the UPB videos that come from YouTube).

Q8: The method requires significant computational power, which might limit its applicability in real-time scenarios or on devices with limited resources.

A8: We only need one GPU for both training and testing. Once our model is trained, it is efficient to run, which could render the 1080p image at 361.02 fps on one RTX A5000.

Q1: Clarification on reprojection losses.

A1: We want to clarify that the reprojection losses (keypoint, mask, perceptual loss) are not claimed as our contributions. Instead, we include them in our paper to describe the necessary components of our framework and make sure our paper is self-contained.

Q2: Clarification on confidence-aware losses.

A2: We observed that the Confidence-aware loss helped with perceptually improving the results, which is also reflected to the consistent performance improvement on the LPIPS metrics. Moreover, another benefit of using the Confidence-aware loss is that it led to a more compact representation (e.g., reducing the number of Gaussians from 21,038 to 19,993). We also present some qualitative results (w/o and w confidence-aware loss) in Figure 3 of the rebuttal pdf.

Q3: Ablation on hyperparameter.

A3: We observed that one of the most crucial hyperparameters was the value $\lambda_{t}$ in the context-aware adaptive density control. We perform an ablation on this hyperparameter.

Q3: Typo on 'video'.

A3: Thanks for pointing out this issue. We will revise the usage of 'video' in the manuscript. We clarify that only one monocular RGB video of one subject is used to train the corresponding avatar.

Q4: What's 'E' in Eq (16)? It looks like the so-called feedback module. But what's the design? Is it a learning-based predictor?

A4: 'E' is the feedback module. It mainly consists of two convolutional layers and is a learning-based predictor. We show the pseudo code of this module below.

class E_feedback(nn.Module):
    def __init__(self, num_feat=4):
        super(E_feedback, self).__init__()
        self.conv1 = nn.Conv2d(num_feat, 3, 1, 1, 0)
        self.conv2 = nn.Conv2d(num_feat + 3, 1, 1, 1, 0)
        self.lrelu = nn.LeakyReLU(negative_slope=0.2, inplace=True)
    def forward(self, x):
        x1 = self.lrelu(self.conv1(x))
        x2 = self.conv2(torch.cat((x, x1), 1))

Q5: Suggestion on discussing the failure cases.

A5: Thanks for pointing out this issue. For the fine-grained expressive areas, the failure cases happen when hand interaction exists, which is mainly caused by incorrect SMPLX reconstruction (e.g. the estimated SMPLX model that drives the reconstruction has implausible interpenetration between hands). Another failure case is the existence of floaters in the reconstruction which is a common issue of 3DGS modeling. We show some representative examples in Figure 2 of the rebuttal pdf.

Q1: Clarification on technical contributions.

A1: Our proposed three technical contributions are well-motivated and well-formulated to meet the requirement of expressiveness in human avatar modeling, which are validated by extensive experiments and have been acknowledged by Reviewer 4SGk ('the paper introduces three significant innovations').

Our proposed fitting aims at mitigating the misalignment issues, especially on the fine-grained hands, which designs effective loss terms leveraging multiple pseudo ground truths (meshes and 2D keypoints predicted by off-the-shelf methods). As shown in Figure 1 of the rebuttal pdf, our fitting results are clearly better than both SOTA regression-based method SMPLer-X and optimization-based fitting (SMPLify-X). Its effectiveness has also been acknowledged by Reviewer g7vJ ('significantly improves the alignment between SMPL-X and the RGB frames').

Q2: Reasons for not reporting results on ZJU-Mocap.

A2: The main reason for not choosing ZJU-Mocap is that it is not suitable for evaluating expressiveness. It has limited diversity on hand gestures/facial expressions (only clenched fist or open palm), and insufficient hand resolutions following the practice of previous methods. Meanwhile, we also prioritize the experiment on ZJU_Mocap and have spent over 24 hours trying to solve it.

1. The first option (proposed by the reviewer) is that we use the same setup as UPB and still evaluate using predicted SMPL-X parameters, the input SMPLX quality will be worse than the SMPL counterpart given in ZJU-Mocap, which will lead to an unfair comparison.
2. Another option is that we try to convert the given SMPL annotation to the form of SMPL-X. However, we still find there remains the projection discrepancy which will also lead to an unfair comparison.

We have conducted the experiments on two benchmarks to perform a fair comparison with previous methods, which is acknowledged by Reviewer ZKQ6 ('It is commendable that the authors added SMPL-X to existing methods for fair and thorough comparisons').

Q3: Clarification on the empirical usefulness of the proposed method.

A3: The main empirical usefulness is that our method could build the expressive avatar from monocular RGB video while getting rid of time-consuming SMPLX annotation. From the application perspective, it could be utilized in sign language production. We have demonstrated this application by conducting experiments on the UPB benchmark (videos collected from the Web).