Locality Sensitive Avatars From Video

We thank the reviewer for reviewing our paper and providing valuable comments. We address the concerns in the common responses and below, and are open to further discussion and questions.

Q1: Concerning the generalizability of the pose-conditioned GNN

Thank you for your insightful comments. However, please note that this point is well aligned with our key motivation. As summarized in Table 1 and discussed in the Introduction Section (L47-L52), some state-of-the-art methods (e.g. NPC and PM-Avatar) encode locality information with GNN to enhance detail synthesis but struggle to generalize well and thus yield artifacts with sparse input data. In contrast, other methods propose to learn the geometry and appearance in canonical space to preserve more consistent outputs across different frames but are inferior in producing fine-grained signals. Figure B provides an example that unifying these two ideas can best adopt their strengths and eliminate their weaknesses, while the GNN-based baseline (NPC) generates significant unwanted black artifacts. The results of PoseVocab shown below also evidence this shortcoming.

Our ablation study results further qualitatively and quantitatively verify this finding in Figure 8 and Table 3 (b). After removing the canonical feature learning, the "w / o canonical" baseline fails to maintain reasonable body contours and significantly distorts the vertical strip textures. In contrast, our full model equipped with locality encoding and canonical space rendering can faithfully reconstruct the ground truth image. In Figure 5, Table F and the supplementary video, our empirical improvements in in-the-wild sequences additionally display our generalization to real-world scenarios.

Q2: Results on more challenging clothes, such as THUman2.1 dataset

Thank you for your suggestions. As presented in the Common Response to Reviewers, we have shown additional results on one ActorsHQ sequence and three MVHumanNet sequences. Together with results on three ActorsHQ sequences shown in the original manuscript, we have utilized at least six sequences to highlight our ability to handle loose clothing. We have also contacted the authors of the THUman2.1 dataset for download access but found that it focuses on high-quality human scans rather than images captured by multiple cameras which do not fit our task.

Q3: The major contribution of this paper can be seen as incremental

As summarized in Table 1 of our manuscript, our method makes the first attempt to incorporate both locality encoding and canonical space rendering into one single framework while previous methods only model one of these two objectives with inferior output quality. We emphasize that our focus is to verify the effectiveness of our observation rather than designing fancy components. One key advantage of following the existing methods, including NPC and PM-Avatar, to apply GNN for locality encoding is that we can enable a simple and neat network and rule out the effects of irrelevant factors such that our key motivation can be clearly supported.

Empirically, the significance of our method can be clearly shown via the relative improvements upon the previous methods and our extensive ablation studies, highlighting our technical contribution.

Q4: Comparisons with PoseVocab

Thank you for your suggestion. As recommended, we have evaluated PoseVocab with its default settings on the MVHumanNet dataset. The quantitative metrics are provided below and our model consistently outperforms PoseVocab across all metrics.

Consistent with your comments on the generalizability of pose-conditioned GNNs, PoseVocab tends to overfit and struggles to generalize effectively to monocular videos. As a result, it significantly distorts continuous shape contours and warps the desired foreground patterns, as shown in Figure F, Figure G and Figure H. In contrast, our method can successfully reproduce signals in multiple scales, demonstrating the effectiveness of integrating locality encoding with canonical space rendering.

Q5: Experiment setting on the ActorsHQ dataset

As described in the “ActorHQ Dataset” part of Section B, we use the images captured by ‘camera 127’ and ‘camera 128’ as training and evaluation data respectively as they can cover the whole human body. Specifically, we utilize the first 75% images for training and the remaining 25% images as novel pose references. The overall and per-sequence metrics for both novel view synthesis and novel pose animation are listed in Tab. 3 (b) and Tab. E respectively. The video results in the updated supplementary material are rendered for novel pose animation.

We thank the reviewer for reviewing our paper and providing valuable comments. We address the concerns in the common responses and below, and are open to further discussion and questions.

Q1: The model struggles with accurate reconstruction of detailed areas such as the face or hands

Thank you for pointing this out. Our method uses SMPL parameters to drive skeletal deformation and represent the skeleton utilized in the GNN, which does not account for fine-grained details of the hands and face. We note that this limitation has been largely overlooked by most methods until the recent publication of EVA [Hu et al. 2024] which leverages the more advanced SMPL-X model for the modeling of hand and face. We would like to emphasize that our proposed approach is not limited to the SMPL model, thus how to more faithfully capture the expressive details of hand and face (e.g. by SMPL-X models) in our framework lies in our future work.

[Hu et al. 2024] Expressive Gaussian Human Avatars from Monocular RGB Video, NeurIPS 2024

Q2: The visualization of non-rigid motion is insufficient

Thank you for your comments. The effects of non-rigid motion are widely researched in recent neural avatar modeling literature. For example, in Section 3.1 of Vid2Avatar (Guo et al., 2023), the authors interpret the non-rigid deformations as dynamically changing wrinkles on clothes. To better demonstrate the effect of non-rigid motion, we visualize the normal maps across different frames in Figure M. It is shown that our method successfully animates the wrinkles near the knee when the subject bends her legs.

To further demonstrate the influence of non-rigid motion in our framework, we remove the offset and denote the ablated model with only skeletal deformation as "w / o locality". We present the visual images and quantitative scores in Figure 8 and Table 3 (b) respectively. As illustrated in Figure 8, the vertical stripe texture is considerably blurred by the "w / o locality" model because that texture varies across different frames and only rigid motion does not suffice to faithfully capture these variations. With non-rigid deformations, we can better reproduce the pose-dependent effects without introducing artifacts. However, please note that, how to fully accurately capture the complicated non-rigid deformation of clothes under the monocular setting is still an open problem for all existing methods, which calls for future research.

Q3: The differences between the proposed model and GauHuman or HUGS?

There are three key differences between our method and GauHuman or HUGS: (1) Similar to other 3DGS based methods summarized in Table 1, GauHuman and HUGS render images in observation space instead of in canonical space, thus being vulnerable to sparse data (e.g. monocular setting). (2) Both GauHuman and HUGS directly perform LBS to drive canonical Gaussian representations to given poses without explicitly modeling the non-rigid motion component. In comparison, our method computes the rigid skeletal deformation and corresponding non-rigid deformation simultaneously. (3) These two methods are implemented with the 3D Gaussian splatting framework while our method is based on the neural radiance field (NeRF) which can be naturally equipped with SDF representation for better surface reconstruction.

Q4: Does using SMPL pose parameters as LBS weights result in any artifacts?

Yes, when removing the non-rigid motion, using SMPL pose parameters alone as LBS weights can notably distort the body contours and desired patterns. But the appropriate non-rigid motion component can greatly alleviate this issue as in Figure 8.

Q5: What is the rendering speed of the proposed model?

As discussed in Section E, the MLPs used in our framework limit real-time applications, which is our most significant constraint. Currently, our method renders a 512x512 image with about ten seconds on a RTX 3090 GPU. However, please note that there is always a speed-accuracy tradeoff and our emphasis is on detail reproduction and model generalization. How to advance our inference speed with grid-based methods (e.g. triplane or hash grids) or 3D Gaussian Splatting framework is our future work.

Q6: Can you show the qualitative and quantitative results if the non-rigid Δx component is omitted?

As discussed in Q2, we denote the ablated model without Δx as "w / o locality" and present qualitative and quantitative results in Figure 8 and Table 3 (b) individually. Please see Q2 here and the ablation study part (Section 4.4) of our manuscript for more details.

Q7: Does considering non-rigid deformations help in accurately representing effects like the flapping of clothes?

Yes. As mentioned in Q2, considering non-rigid deformations can facilitate the pose-dependent effects which in turn improve the complicated clothing motion like the flapping of clothes.

We are looking forward to hearing more feedback from the reviewer.

Q3: Reconstruction difference with Vid2Avatar is hard to differentiate

We discuss in Figure I and in L1068, that imperfect pseudo-ground truth hinders our ability to achieve fully accurate geometric evaluations quantitatively for ZJU-Mocap dataset as it smooths out the surface details and introduces unwanted artifacts (e.g. the ground surfaces). Thus the imperfect pseudo ground truth shapes benefit Vid2Avatar in most sequences with slightly higher Normal Consistency metrics. How to more accurately evaluate quantitative geometric results is still an open problem.

Despite the marginal improvements due to inaccurate ground truth shapes, our method presents more faithful geometric details than Vid2Avatar visually. Specifically, our method can generate more clear facial structures in Figure 7 and Figure A, and can reconstruct more cloth wrinkles in the first row of Figure F. Our advantage in synthesizing fine-grained details is further enlarged when evaluating on the ActorsHQ dataset where clothes are more textured and actors are partially invisible. As shown in Figure N, our method still can reproduce vivid facial patterns and pant wrinkles under this challenging case. In comparison, Vid2Avatar simply smoothes out all these signals . We additionally present video results of the estimated normal maps in the 'Comparison on SynWild' section of the supplementary video, providing a comprehensive qualitative assessment. All these results demonstrate that our method significantly enhances Vid2Avatar by incorporating more geometric details.

Q4: Methods are missing in the comparison or discussion

Thank you for raising this question. Following your suggestions, we have included the results of MonoHuman into the table in “Common Response to Reviewers” for comparisons. Like the findings elsewhere, our method clearly surpasses MonoHuman by a large margin. Results of the newly added two baselines, GauHuman and PoseVocab, are also reported; please see the “Common Response to Reviewers” and the uploaded manuscript for more information. With results on presented datasets, our method outperforms the representative methods of canonical space rendering, locality encoding, and 3DGS driven modeling.

Q5: It would be better to show driven poses that are out-of-distribution

Thank you for your suggestion. Actually, as stated in L423, we have already shown the visual comparisons for out-of-distribution pose animation in Figure 5 and the supplementary video. Moreover, the animation results for the ActorsHQ sequence can also be regarded as out-of-distribution pose animation as the testing poses significantly differ from training poses. We have reorganized the supplementary video to make the demonstration more clear.

Q6: Motivation of the method

Inspired by previous methods, we leverage GNN as a simple way to process human skeletons and extract local features while we agree that there are other feature extraction paradigms as alternatives. But compared to other options to extract local features, such as time-dependent voxel planes [Wu et al. 2024], the GNN based feature extraction paradigm has the following benefits: (1) GNN propagates features within neighboring regions and can adaptively capture local patterns in multiple scales. (2) As inputs to GNN, the human skeleton is sparse and is able to yield decent expressiveness with limited computation complexity. (3) Pose estimation models can also work for testing poses and enable out-of-distribution pose animation.

In comparison, time-dependent voxel planes struggle to model complex local relationships and cannot extend to novel pose animation due to fixed space-time resolutions. They are also dense in memory storage, which limits their capability in processing fine-grained details over time.

Empirically, the effectiveness of GNN can be revealed through an ablation study. Specifically, we feed each skeleton node into its own MLP without feature propagation to extract features for each part, denoted as the “onlyPose” baseline. As listed in the table below, our full model which explicitly propagates nearby pose features constantly improves the “onlyPose” baseline.

However, note that our core technical contribution lies in integrating locality encoding into canonical space rendering and making solid conclusions through extensive experiments, rather than using a GNN for local feature extraction. As such, our conceptual approach can be adapted to other implementation frameworks, enabling potential extensions.

[Wu et al. 2024] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering, CVPR 2024

We thank the reviewer for reviewing our paper and providing valuable comments. We address the concerns in the common responses and below, and are open to further discussion and questions.

Q1: Video comparisons with SOTAs

We agree that the recent state-of-the-art methods make significant progress in neural avatar modeling and HumanNeRF provides a strong baseline for comparison. However, there are two key advantages of our method over HumanNeRF: 1) The locality encoding helps our method generalize better to challenging poses with improved details; 2) Our method can yield more plausible shape reconstruction.

Given our conceptual advantages, it could be tricky to recognize our visual improvements in a playing video, especially in ZJU-Mocap sequences where the foreground is relatively dark. To this end, we pick six representative frames for more explicit visual comparisons in Figure J. Being consistent with the observations in L406, our method can yield better body shapes with less pattern distortion. Similar to Figure 7 of GaussianAvatar paper, we collect the challenging poses from AIST++ dataset as out-of-distribution animation inputs to further widen our visual improvements. Compared to HumanNeRF, our method can generate more consistent body parts while HumanNeRF distorts the left hand into discontinuous shape as shown in (2’09’’). Another example of out-of-distribution animation is illustrated in Figure 5.

We also present novel pose animation results of loose cloth modeling, showcasing the complex interactions between the cloth and the human body. As shown in the “Comparison on ActorsHQ” and “Comparison on MVHumanNet” sections of the supplementary video, our method can output more satisfactory images while state-of-the-art baselines fail to preserve continuous shape contours.

To illustrate our temporal consistency in producing shape geometry, we compare the normal map sequence with Vid2Avatar under the SynWild dataset. It is clear that our method can reproduce much sharper structured patterns (e.g. facial area) and delicate outlines (e.g. the pant boundary).

Lastly, we want to emphasize that visually evaluating each method could be subjective due to different approaches having different failure modes and their weighting might differ from person to person. For example, Reviewer nejY acknowledges that our model demonstrates superior rendering performance compared to other approaches, particularly evident through enhanced normal rendering results. Thus to reach a more objective judgement, we perform comprehensive quantitative comparisons across totally 24 sequences of 6 datasets with 5 metrics and report detailed numbers in the appendix. Plus our advantages in geometry reconstruction, our method can clearly demonstrate its empirical effectiveness qualitatively and quantitatively.

Q2: Samples on more datasets are expected in video

Thank you for your suggestions. Please see the “Comparison on ActorsHQ”, “Comparisons on MVHumanNet” and “Comparison on SynWild” sections of the attached video for details. Due to time and computational constraints, we limit our comparison to Vid2Avatar on the SynWild dataset which is supposed to be the best baseline in geometry reconstruction. Results turn out that our method synthesizes much more geometric details than Vid2Avatar. We also improve time consistency over baselines under the settings of MVHumanNet and ActorsHQ datasets. Specifically, inputting challenging out-of-the-distribution poses in ActorsHQ sequences, baselines like HumanNeRF break the human body into many small pieces while our method preserves much more realistic shape contours. Our advantage remains valid in MVHumanNet sequences, showing our universality.

Thx authors for providing additional results and details. Based on this, I have raised the score accordingly.

However, the remaining issues are:

(1) While the later experiments presented during the rebuttal strengthen the effectiveness of the proposed framework, the technical contribution remains fundamentally incremental, appearing more as an engineering improvement. I am not opposed to engineering-focused work, but to enhance the impact, the authors could explore the universality of the framework components further in the future, like whether the contribution components still work upon other base scene representations, what if the training number decreases, etc.,

(2) A potential concern is that all samples used seem to have pure color or relatively simple textures, without coverage of more complex textures. Given the authors' claim of outperforming the sotas in nearly every aspect, testing with real-world, complex textures would better substantiate the claims. Additionally, the reconstruction quality in these scenarios could be further evaluated.

(3) Be precise with claims. For example, the sample in Fig. 5 is not strictly out-of-distribution, as the target video includes a black lift-foot motion, which is not significantly dissimilar to the source.

Results on more challenging clothes with more baselines (Reviewer p8GG, Reviewer nejY, Reviewer cJiu)

Thank you for the constructive comments. As suggested by reviewers, we evaluate our method over three additional sequences (102107, 200173 and 204129) of the latest MVHumanNet (CVPR 2024) dataset to consider the varying degrees of clothing looseness. Additionally, as suggested, we include another 3DGS-based baseline, GauHuman (CVPR 2024), and a locality encoding method, PoseVocab (SIGGRAPH 2023), to further validate the importance of non-rigid deformation learning and the integration of canonical space with locality encoding. For MVHumanNet sequences, since the subjects only turn around near the end of each sequence, we use all frames for training to capture the whole body and thus omit the novel pose evaluation. For each baseline, we achieve results through their public source code with hyper-parameters as specified in their papers.

We illustrate the comparison examples in Figure F, Figure G and Figure H of the uploaded revision. As reported in the table below and detailed in Table G, our method achieves better quantitative results than all baselines in terms of novel view synthesis.

Specifically, our method achieves over a 17.5% relative improvement in LPIPS compared to the second-best baseline (HumanNeRF). Together with the results assembled in the ActorsHQ dataset, we have utilized seven sequences to convincingly demonstrate our robustness to loose-fitting clothes. Please note that our method improves HumanNeRF with a much smaller network size: 1.2M (ours) vs 60M (HumanNeRF). Similarly, we achieve better results than MonoHuman whose number of trainable parameters is over 74M.

Additionally, note that, we outperform the monocular video focused methods (e.g. HumanNeRF and 3DGS-Avatar) that in turn outperform the remaining multiple view based baselines (e.g. NPC, PM-Avatar) as shown in Table 2 and Table 3 (a). Additionally, as summarized in Table 2 of the 3DGS-Avatar paper, this baseline achieves a better balance between quality and speed compared to other 3DGS-driven algorithms. It also incorporates pose-dependent cloth deformation, making it conceptually the most effective among 3DGS-based methods.

Thus we categorize our baselines into three classes: 1) NeRF based methods with only canonical space rendering, including HumanNeRF, MonoHuman and Vid2Avatar; 2) NeRF based methods with only locality encoding, including PoseVocab, NPC and PM-Avatar; 3) 3DGS driven methods, including 3DGS-Avatar, GoMAvatar and GauHuman. Given video inputs, these three categories of methods can best represent the existing neural avatar modeling pipelines.

[Qian et al., 2024] 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting, CVPR 2024.

Comparison of Experimental Settings for 3DGS based baselines (Reviewer p8GG, Reviewer nejY, Reviewer cJiu)

The tables detailing experimental settings clearly demonstrate that our method conducts the most comprehensive evaluations compared to all the chosen baselines. Specifically, we compare against the largest number of baselines across the most extensive and recently released datasets and sequences. Furthermore, apart from GauHuman, ours is the only method that leverages publicly available datasets released after 2023 for evaluation. Combined with the significant empirical improvements achieved on each dataset, we can confidently conclude that our method improves state-of-the-art approaches in novel image rendering and 3D shape reconstruction which highlights the effectiveness of our proposed method.

Comparison of Experimental Settings for NeRF based baselines (Reviewer p8GG, Reviewer nejY, Reviewer cJiu)

As mentioned above, we have incorporated sequences from the MVHumanNet dataset into our testbed. To further validate the comprehensiveness of our evaluation protocol, we provide a detailed comparison of the experimental settings for the NeRF based baselines below.

We further paste the comparisons of evaluation protocols for 3DGS-based methods in the following section.

Dear reviewers,

We have updated Table G in the appendix and the table here to include quantitative results for MonoHuman on sequences from MvHumanNet (CVPR 2024).

Please note that this can further reveal our universal advantages over baselines across varying degrees of cloth looseness. Alongside the visual comparisons in Figures F, H, and I, and the results from ActorHQ (SIGGRAPH 2023) presented in Tables E, F, and Figures N, R, the comprehensive evaluation demonstrates that our method consistently outperforms all three classes of state-of-the-art methods:

• NeRF-based methods with canonical space rendering: HumanNeRF, MonoHuman, and Vid2Avatar.
• NeRF-based methods with locality encoding: PoseVocab and PM-Avatar.
• 3DGS-based methods: 3DGS-Avatar and GauHuman.

These results underscore the superior performance of our method across diverse scenarios.

We deeply value reviewers' comments and strongly encourage reviewers to share any additional feedback or insights regarding these updates. We eagerly look forward to the reviewers' detailed review and constructive discussions.