Pose Modulated Avatars from Video

We sincerely thank the reviewer for reading our paper and giving us thoughtful feedback. We carefully respond to each of the summarized concerns and questions. All these feedbacks will be incorporated into our revised manuscript.

Q1: In Figure 1, the frequency distributions appear to be quite similar.

We respectfully clarify that these two frequency distributions are actually very different. In Figure 1, we can see that the avatar on the bottom exhibits much more small STD values than the top one: 52 (bottom) vs 23 (top). In Figure 10, we provide a more detailed frequency illustration. For the number of STD values larger than 0.4, the top avatar in Figure 1 contributes much more than the one on the bottom. In contrast, the top avatar in Figure 1 provides much less STD values smaller than 0.4.

Q2: When a subject is in a similar pose, the frequency distributions can still be distinct.

As mentioned in the abstract, we hypothesize that different poses necessitate unique frequency assignments. And neglecting this distinction yields noisy artifacts in smooth areas or blurs fine-grained texture and shape details in sharp regions. So we agree with the reviewer that when a subject is in a similar pose, the frequency distributions should be close as well. If there are some misunderstandings / errors in our main text, please let us know.

Q3: Are the point locations clearly separated by the learned weights for each part?

As the per-part weights {$w_i$} are a continuous feature vector, instead of being binary. So when two bones overlap, the parts mixed around the joints as the query points are influenced by different bones.

Q4: Table 1 (b) caption seems to be incorrect.

Thank you for pointing this out. The ablation studies are performed on Human3.6M S9. We will correct it in the revision.

Q5: Is there a setup difference for the MonoPerfCap results compared to DANBO?

Yes, we use tighter masks to highlight the foreground differences.

Q6: The ablation should be performed on the complete setup.

Thank you for your suggestions. We follow DANBO’s settings to perform the ablation studies on Human3.6M S9. To verify our motivations comprehensively, we train our OnlySyn, noGNN, OnlySyn ablation models over the whole Human3.6M sequences and report the scores below. Being consistent with the conclusions in the main text, each component does not suffice to provide the optimal performance. Due to the constraints of response phase and computation resources, we will provide other ablation study results over all Human3.6M sequences in the future revision.

I appreciate the authors for addressing all my concerns through clear explanations and additional experiments within the limited rebuttal time. The idea of using frequency modulation to model wrinkles is interesting, and the results seem convincing, so I'm willing to raise my rating.

We sincerely thank the reviewer for reading our paper and giving us thoughtful feedback. We carefully respond to each of the summarized concerns and questions. All these feedbacks will be incorporated into our revised manuscript.

Q1: It is unclear whether or not GNNs are actually needed for this task.

As an example, Vid2Avatar takes the whole pose information as a condition, but it is only applied for novel view synthesis, not for animation with novel driving poses. GNNs enable learning separate feature spaces for different parts, which was shown to improve disentanglement for animation (Su et al., 2022). Plus the sparse nature of our learnable window functions, the output feature of one query point only attends to a few joints. Thus a more accurate animation with out-of-distribution poses can be captured than directly using the whole skeletons .

To further highlight the importance of GNN, we report the novel view synthesis && novel pose rendering results of an ablation model by removing graph neural networks but directly feeding the joint parameters to one single Multi-Layer Perceptron (MLP) with comparable trainable parameters to our full model. The detailed architecture of the GNN ablation model is illustrated in Figure 15 of the revised paper. Similar to Vid2Avatar, processing the whole joints together disenables modeling part-level feature space. We train the ablation model over all Human3.6M sequences. As listed in the table below, our full model (the right value in each entry) outperforms the ablated model across all metrics. Especially, being consistent with the aforementioned discussions, the usage of GNN enables us to better generalize to novel poses, proving the effectiveness of GNNs and part-based feature learning. Together with other ablation study results (e.g. no window) over full sequences, these quantitative results will be included in future revision.

Q2: Video results have very low FPS.

In the comparison sequences, for ARAH, we utilized the generated images provided by its authors. These video frames were sampled from the entire sequence at a fixed frame rate (extracting one frame per 30 frames), which is a common strategy. From the video, we can observe that the frame sequences obtained through sampling have adequately approximated the complete input videos.

To further validate the advantages of our method in terms of temporal smoothness, we present the results of DANBO, TAVA and ours on the complete image sequences. In the newly uploaded video, we insert the full frame comparison clip into a section named Temporal Smoothness Comparisons after the section Comparisons for Animation Sequences. As shown in the attached video, our method demonstrates superior temporal smoothness with consistently adaptive textures.

In addition to the video clips used for comparison (named as Comparisons for Animation Sequence in the video), we utilized the full image sets to synthesize the subsequent video results (starting from More Results for Animation Sequence). We can observe that our method constantly showcases structured features (e.g., the stripe patterns on the shoulder area) and smooth body contours. Please refer to the results under the video sections titled More Results for Animation Sequences and Motion Retargeting.

Our technical contribution is also evident in the ablation study section. In comparison to the OnlySyn ablation model, our full framework can synthesize a more complete shape contour (e.g., hands) and produce realistic marker textures without artifacts.

Due to the constraints of rebuttal phase and computation resources, the results of ARAH on the complete sequence will be included in the future revision stage.

Q3: No experiments on loose clothing.

Our goal is to verify the effectiveness of pose modulated frequency learning. When the deformed avatar wears soft clothes (e.g. T-shirt), our method can successfully reconstruct the desired wrinkle textures; please refer to Figure 1 and Figure 11 of the main text as an example. Although we have already presented our advantages across different popular datasets, how to extend our method to more challenging datasets with loose clothing is an interesting direction and lies in our future work. Please note that loose clothing calls for modeling dynamics physically, e.g. swinging, which is orthogonal to our current work.

May we ask if our explanation satisfies you?

We thank Reviewer SAAP for the feedback. We are glad to see that the temporal smoothness issue has already been addressed. In the following, we will summarize your concerns and our corresponding response point-by-point.

Q1: The provided quantitative results do not show substantial improvement.

As mentioned in the Common Response to All Reviewers and our response to Q2 of Response to Reviewer McVe, our empirical improvement can be recognized to be significant according to the most recent literature. For example, in Table 2 of Vid2Avatar, it improves over HumanNeRF by 0.001 in SSIM and 0.3 in PSNR. Similarly, in the Table 4 of MonoHuman, the full model improves over the baselines by <0.001 in SSIM and <0.2 in PSNR. And in the Table 4 of DANBO, the full model improves the ‘Softmax-OOB` baseline with 0.002 in SSIM and 0.38 in PSNR. Please check their papers for details. Correspondingly, for the provided comparisons over ZJU-Mocap dataset, we improve the best baseline (MonoHuman) with 0.002 in SSIM and 0.34 in PSNR. Thus, in terms of reported scores, our method has already achieved the typical improvements over prior work. As mentioned in the past response, we also list the comparisons for model complexities and geometry reconstruction. In Table 4 of the revised paper, our method has the about 45.5% and 121.3% relative improvements over HumanNeRF and MonoHuman respectively. We incorporate the metrics of rendering results, model complexities and geometry reconstructions into one single table below for more convenient comparisons.

We emphasize that our method outperforms the baselines across almost all metrics. PSNR and SSIM metrics are known to differ in magnitude from dataset to dataset, but relative improvements remain very consistent and quantify the overall improvement well, even if small in absolute numbers. Thus we respectfully disagree with the reviewer that the improvement is not statistically substantial (c.f. data size discussion below).

Q2: The provided qualitative results do not show substantial improvement.

Following the conventions of former neural avatar papers, we illustrate to exemplify our conceptual advantages. Specifically, the pose guided frequency modulation can facilitate synthesizing adaptive patterns with reduced artifacts. Specifically, in Figure 7 (a), the noGNN ablation fails to synthesize the structured stripe with blurry artifacts. For evaluations on ZJU-Mocap, only our method can successfully synthesize the avatar fist with accurate arm contours and capture the cloth colors in Figure 13. In contrast, all baselines either distort or blur the fist patterns (top row) and clothing textures with incorrect cloth colors (bottom row). To better illustrate our advantages, we present the frequency error maps which are introduced in Figure 11. It is clear that the pose guided frequency modulation can remarkably reduce the unwanted errors than all baselines. The computed F-Dist scores further highlight this point: our method has relatively 47.4% (top row) and 73.7% (bottom row) improvements over the best baseline.

For the authors, both qualitative and quantitative results clearly demonstrate the effectiveness of pose guided frequency modulation. But we are also happy to further discuss with detailed comments or arguments.

Q3: Sequence number for ablation studies.

Following the settings of former baselines, we think we have used sufficient data to achieve a fair comparison and conclusion. For example, DANBO uses one Human3.6M sequence (S9) as described in Section 4.4. Both HumanNeRF and MonoHuman apply six ZJU-Mocap sequences for ablations as shown in Table 3 and Table 1-2 of their papers respectively. And in our paper, we have used seven sequences. Overall, we have made use of 7 Human3.6M sequences, 2 MonoPerfCap sequences and 4 ZJU-Mocap sequences to pinpoint the effectiveness of the proposed framework, each containing dozens to hundreds of frames. It can be seen to be a very thorough experimental setting and prior work validated that the magnitude of improvements we attain is a consistent and significant improvement.

We sincerely thank the reviewer for reading our paper and giving us thoughtful feedback. We carefully respond to each of the summarized concerns and questions. All these feedbacks will be incorporated into our revised manuscript.

Q1: The improvement in texture quality of the wrinkles are subtle.

We point out that it is more challenging to capture adaptive textures under different input poses when more frequency variations are presented in different video frames. There are two clips in the attached video for comparisons. Compared to the second clip where the highlighted pants region almost always presents wrinkles, the marked T-shirt region in the first clip shows wrinkles with much more variations and thus can better reveal our improvements in terms of wrinkle quality.

We pick up two representative frames to highlight our wrinkle synthesis capability as it is difficult to directly evaluate the dynamic wrinkles. As exemplified in Fig. 1 and Fig. 10 of the submitted file, our model can faithfully enable pose-dependent details across different frames and pose contexts. Given that it is subjective to visually assess the produced wrinkles, we compute the matched histogram distances (F-Dist) between generated images and corresponding ground truths for quantitative comparisons. In Fig. 10-13, our proposed method can synthesize closer frequency distribution to the ground truth, exhibiting our conceptual advantages in producing adaptive textures, including wrinkles.

Q2: Marginal numerical improvement in the reported metrics.

In Tab. 2 of the main text, we outperform DANBO by ~0.4 dB in PSNR and 0.04 in SSIM with a comparable number of trainable parameters. According to the recent literature (e.g. Tab. 2 in Vid2Avatar and Tab. 3-4 in MonoHuman), this can have already been recognized as a significant improvement. When moving to more challenging scenarios of novel pose rendering, our empirical advantages become more remarkable: 0.41 (PSNR) and 0.05 (SSIM).

Our effectiveness can be also demonstrated by the generality to the monocular video dataset. On the MonoPerfCap dataset, our method improves DANBO with about 0.45 dB in PSNR and 0.008 in SSIM for novel pose rendering.

Lastly, the goal of our paper is to highlight the importance of pose modulated frequency modeling, which can also be demonstrated through the ablation studies. In Tab. 1 of the main text, our full model significantly outperforms the ablation baselines, clearly supporting the effectiveness of the pose modulated frequency learning.

Q3: Overall the results in the supplementary videos are quite blurry.

With the pose modulated frequency learning, our goal is to capture the desired textures with reduced artifacts for animated avatars. The sequences in the supplementary show that the proposed method can consistently outperform baselines with better preserved patterns in various scales. One possible explanation for the blurry issue is that the highlighted close-ups for comparisons have relatively low resolutions and the background is in black. Note that another visual artifact is the masks used to extract foregrounds introducing some jagged edges.

Q4: Geometry comparisons on modern baselines.

In the following table provides the geometry comparison results with the settings presented in the Sect. 4.2 of ARAH (Wang et al., 2022). Specifically, we also apply NeuS [1] to obtain the pseudo geometry ground truth and compute quantitative metrics with Chamfer Distance. Our method possesses better geometry reconstruction than the template-free methods including HumanNeRF and MonoHuman. The geometry scores are worse than Vid2Avatar which is a template based baseline and relies on SMPL parameters as a prior. The SDF-based surface rendering also helps Vid2Avatar achieve smoother shapes compared to our density-based learning.

Besides the quantitative comparisons, we also report the visualizations in Fig. 14 of the updated paper. As a template-free method, HumanNeRF & MonoHuman introduce obvious floating artifacts in empty space while our method generates a much smoother body surface.

Please note that our method focuses on how to capture high-frequency texture details for the view rendering. How to accurately reconstruct shapes is another exciting yet non-trivial research problem, and we leave it for future exploration.

[1] NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction, NeurIPS 2020.

We thank all reviewers for their input again. Besides our response submitted before, we add some clarifications on why we train Vid2Avatar+mask on ZJU-Mocap sequences.

Joint learning of separating humans from arbitrary background and reconstructing detailed avatar surfaces is the key claim in the original Vid2Avatar paper. However, the background of ZJU-Mocap sequences is black and some parts of the foreground look dark, thus making it more challenging to accurately extract foreground and synthesize the texture details. Please see Fig. 13 and Fig. 14 of the uploaded revision as an example. After discussing with the authors of Vid2Avatar, we add the mask to impose the groundtruth supervision and explicitly extract the foreground as the Vid2Avatar+mask model. However, as shown in the comments in Common Response to All Reviewers, the pose-dependent frequency modulation successfully improves these strategies used in Vid2Avatar paper. Plus our model complexity and the results on geometry reconstruction, we have comprehensively revealed our advantages over the most recent baselines.

Please let us know if you have further questions during the reviewer-author discussion period.

I have carefully read the authors' responses. I would like to thank the authors for diligently answering all my questions and for the additional experiments that they conducted. My doubts have been satisfied. I don't have any further questions. My outlooks towards this paper remains positive.

Dear Reviewer McVe,

We thank you very much for your positive and constructive feedbacks! We are especially glad that our qualitative and quantitative results as well as the improvements over baselines are acknowledged. We really appreciate your time and knowledge on guiding us to advance our paper from these key aspects.

Best, The Authors