LIFe-GoM: Generalizable Human Rendering with Learned Iterative Feedback Over Multi-Resolution Gaussians-on-Mesh

Thanks for your time and feedback. We appreciate the comments regarding the novelty and effectiveness of our approach. We answer questions next:

QD1. In the related works section, the authors categorize human rendering into "per-scene optimized" and "generalizable" approaches. However, there is no discussion on "large reconstruction model-based" human rendering approaches [1,2]. It would be valuable to compare the proposed method’s performance (in terms of inference time, training time, resolution and PSNR) with such large reconstruction models.

In the original submission, we categorize the large reconstruction model-based methods for humans into diffusion-based methods in the related works (Line 129) and cite [1]. We thank the reviewer's suggestion and revise the term to large-reconstruction models to include a broader set of related works in this direction and cite more works (e.g. [2]). Such approaches usually use a two-step pipeline: 1) they first sample multiview images from diffusion models, and 2) they reconstruct and render the human utilizing the generated multiview images. Discussion: Note that the task differs from the sparse-view generalizable approach including ours. Large reconstruction model-based methods operate on a single input image, i.e., the key is to hallucinate unseen regions. In contrast, our method focuses on a sparse input setting, and the focus is to align the sparse inputs and reconstruct a canonical T-pose representation. As mentioned in our limitation section, our method cannot hallucinate large unseen regions. Therefore it is only useful to use our method in step 2) of large reconstruction model-based methods. Comparison: Unfortunately, since neither [1] nor [2] has released their codes yet, we cannot conduct a fair comparison in PSNR. We alternatively contrast inference speed and training speed based on information provided in the paper:

In terms of inference speed, [1] mentions that it takes 0.7s for reconstruction from multiview images and 1.3s for rendering at a resolution of $256\times256$ on one NVIDIA A6000 GPU. Our approach uses a higher resolution, i.e., $1024\times1024$ for all stages. Meanwhile, our reconstruction needs 0.9s and the rendering runs in 10.52ms on an NVIDIA A100 GPU. We assume the reconstruction speed of our approach and [1] is close while our rendering speed is faster. For training, [1] trains the coarse single-view model and multiview reconstruction model with 16 A100 GPUs for 7 days, while our model is trained for 4 days on a single A100 GPU. [2] does not report the reconstruction speed and inference speed. For training, it takes 20h on 4 NVIDIA A6000 GPUs while ours takes 4 days on a single A100 GPU. Since we use different resolutions and different GPU types, a fair comparison in terms of training speed is challenging.

[1] Weng, Zhenzhen, et al. "Single-view 3d human digitalization with large reconstruction models." arXiv preprint arXiv:2401.12175 (2024).

[2] Chen, Jinnan, et al. "Generalizable Human Gaussians from Single-View Image." arXiv preprint arXiv:2406.06050 (2024).

Thanks for your time and feedback. We appreciate the comments regarding the efficiency and the rendering quality of our approach. We answer questions next:

QC1. The experimental comparisons are incomplete without recent works[1,2,3] that report better metrics. A comparative analysis or discussion of technical differences is needed.

Thanks for the suggestion! We compare our method to GoMAvatar and 3DGS-Avatar in Table 5 and Table 6 of the revised appendix. The comparisons are as follows: On THuman2.0, taking three source images as input, we obtain

On AIST++, taking five source images as input, we obtain

In both settings, our approach significantly outperforms GoMAvatar and 3DGS-Avatar in both speed and rendering performance.

To provide more details about the baselines, we adapt GoMAvatar and 3DGS-Avatar to work with SMPL-X priors on THuman2.0. In both settings, we train GoMAvatar for 100K iterations and 3DGS-Avatar for 2K iterations to avoid overfitting.

[1] 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting, CVPR2024 [2] GoMAvatar: Efficient Animatable Human Modeling from Monocular Video Using Gaussians-on-Mesh, CVPR2024 [3] HUGS: Human Gaussian Splats, CVPR2024

QC2. The cross-domain generalization shows concerning facial artifacts despite the proposed iterative feedback. What causes these failures? If it's a data bias issue, why showcase this case? Consider discussing limitations and potential solutions (e.g., face-aware regularization).

The distortion in facial regions is mainly due to imperfect poses provided by DNA-Rendering. When the poses are accurate (e.g., for XHuman), we do not observe distortions, as shown in the “Cross-domain generalization - XHuman” section of the supplementary website. We also stress-test our method when human poses and masks are predicted via off-the-shelf tools. Results can be found in the “Cross-domain generalization - PeopleSnapshot” and “Cross-domain generalization - ExAvatar” sections of the supplementary webpage. Note, to demonstrate robustness, on PeopleSnapshot, we do not use the poses and masks provided by the dataset but use predictions instead. Even for predicted poses, our approach shows promising cross-domain generalization. We quantitatively study the sensitivity to input pose accuracy, compare to GHG and discuss the potential solutions in Sec. D.3 of the revised appendix. On THuman2.0, we randomly add Gaussian noise of different standard deviations to the SMPL-X poses. We observe that our approach improves upon the GHG baseline for all noise levels, though all methods are sensitive to the accuracy of input poses:

Additionally, we quantitatively compare our method to the GHG baseline on the XHuman dataset and add those results in Table 7 of the revised appendix. Our method achieves a PSNR/LPIPS*/FID of 25.32/99.32/42.90, improving upon baseline GHG’s 23.53/112.91/50.51. The results show the effectiveness of our approach in cross-domain generalization.

QC3. The ablation study focuses only on hyperparameters rather than analyzing the effectiveness of core components. Need evaluation of: Iterative feedback mechanism; Coupled-multi-resolution representation; Module-wise contribution analysis.

We would like to clarify that Table 3 and Table 4 showcase the effectiveness of the core components rather than hyperparameters only. “T=1” in Table 3 means no iterative feedback and “#subdivision=0” in Table 4 means w/o coupled-multi-resolution representation, i.e., Gaussians use the mesh resolution. Based on the reported results, both iterative feedback and coupled-multi-resolution representation are important.

Thanks for your time and feedback. We appreciate the comments regarding the effectiveness and efficiency of our approach. We answer questions next:

QB1. Missing comparisons to GoMAvatar: as mentioned in line 121-123, the presented method is derived from GoMAvatar. It therefore makes sense to compare against GoMAvatar.

Thanks for the suggestion! We compare our method to GoMAvatar and add those results as Table 5 and Table 6 in the revised appendix. On THuman2.0, taking three source images as input, we obtain the following result:

On AIST++, taking five source images as input, we obtain:

On both datasets, our approach significantly outperforms GoMAvatar. As a scene-specific method, GoMAvatar tends to overfit the training images in the sparse input setting. In contrast, our method does not suffer from overfitting and works well with only 3 or 5 input images.

QB2. The iterative-feedback bears a close similarity to [1], which also utilizes pixel/texel align features to improve fidelity. It would be great if the paper could discuss about these relevant works.

Iterative feedback iteratively updates in a feed-forward manner the canonical representation, given the prediction from the last step. The paper mentioned by the reviewer uses encoder-decoder structures to generate a 3D representation (texels) from input images. Note, the encoder-decoder architecture is not repeated several times, i.e., making it different from our iterative feedback. We highlight the differences in Line 132 of the revised related work section.

[1] Drivable Volumetric Avatars using Texel-Aligned Features, SIGGRAPH 2022

QB3. The writing can be further improved: for example, the sentence in line 143-145: As a fix, to regularize the Gaussians and to ease the animation, pairing with parametric models like FLAME or SMPL helps may not be as straightforward as saying Prior work [cite] regularizes the Gaussians and enables animation using parametric models such as FLAME and SMPL. Also, in line 180-181 given as additional input the target camera ... seems unnecessary, as line 175-179 already described the input.

Thanks for the suggestions to improve writing! We revised Line 143-145 accordingly. The poses defined in Line 180-181 and Line 175-179 differ. Specifically, in Line 175-179, we describe the inputs for reconstruction (Eq. (2)), including the camera poses $\\{K\_i, E\_i\\}\_{i=1}^N$ and human poses paired with the $N$ source images. To render the canonical 3D representation, we need to specify the camera pose and human pose. Therefore, in Line 180-181, we formally define the desired camera pose $K\_{tg}, E\_{tg}$ and human pose $P\_{tg}$ for rendering, which can be specified by the user. We apologize for the ambiguity in the notation.

QB4. Mesh topology limits the geometry that can be captured, as shown by the hairs in Figure 4 (also pointed out in Appendix C).

Note, SMPL or SMPL-X priors only serve as a mesh initialization in our method. As shown in Eq. (9), the method deforms the mesh underlying our Gaussians-on-Mesh representation. Results corroborate this ability. As shown in the first and the third example of the “Freeview Rendering and Comparison to GHG” section of the supplementary webpage, our method can render clothing which has a topology different from the SMPL topology used for initialization, e.g., long coats, though the connectivity of the underlying mesh is wrong. We defer topology updates to future work. Thanks for the great suggestion!

QB5. Missing memory profile: perhaps I have missed it, but I did not see discussions related to the memory/GPU vram consumptions for the proposed method. How much VRAM does it take to train/render the avatars? Does it work on commodity-level GPUs?

Training requires 30842 MiB VRAM. For reconstruction from given images, we subdivide the mesh once more for higher-quality results, and use 30638 MiB VRAM. Importantly, upon reconstruction, the canonical representation and rendering only uses 2804 MiB VRAM, i.e., rendering runs on commodity-level GPUs once the canonical representation is reconstructed.

Thanks for your time and feedback. We appreciate comments regarding the novelty of our approach and the more accurate reconstruction. We answer questions next:

QA1. The model does not appear to have been trained on two datasets and can support both tasks simultaneously, limiting its ability to handle mixed input types effectively.

We can handle both tasks simultaneously. Although our model is trained on only one dataset format/task at a time (multiview for THuman2.0 or multipose for AIST++), inference is not limited to the training dataset format/task. In the updated supplementary material, we apply the model trained on THuman2.0 multiview images to inputs having different poses: In the last two examples of the “Cross-domain generalization - XHuman” section of the revised website, we use input images exhibiting different poses. In the “Cross-domain generalization - PeopleSnapshot” and “Cross-domain generalization - ExAvatar” sections of the website, we sample three frames from the monocular video as input. Although the input type differs from the training data, our method shows compelling rendering quality. To further improve results, we agree with the reviewer that joint training on all datasets (THuman2.0 and AIST++) is desirable. However, it is challenging to combine THuman2.0 and AIST++ data, as the two datasets use different body priors: THuman2.0 uses the SMPL-X prior while AIST++ uses the SMPL prior. Nonetheless, combining both datasets for training is an interesting next step.

QA2. The method's heavy dependence on SMPL priors poses a risk; if the estimations are inaccurate, especially for loose clothing, it can lead to significant errors.

SMPL or SMPL-X shape priors only serve as an initialization in our method. As shown in Eq. (9), our method learns to deform the mesh underlying the Gaussians-on-Mesh representation. Results corroborate this ability. As shown in the first and the third example of the “Freeview Rendering and Comparison to GHG” section of the supplementary website, our representation can model clothing which differs from SMPL-X shapes used for initialization, e.g., long coats. However, we also observe that this isn’t always robust, e.g., for dresses, and mention this in the revised limitations section. Importantly, we think this issue is not caused by SMPL/SMPL-X initialization, as our approach deforms the mesh (e.g., long coats). Instead, we think the issue is due to limited training data: we only trained on 426 subjects, among them no subjects wearing dresses.

QA3. The method shows significant performance degradation when applied across domains, particularly affecting the facial region, as in the video.

To study cross-domain generalization, we quantitatively compare our method to the GHG baseline on the XHuman dataset and add those results in Table 7 of the revised appendix. Our method achieves a PSNR/LPIPS*/FID of 25.32/99.32/42.90, improving upon baseline GHG’s 23.53/112.91/50.51.

We further demonstrate cross-domain generalization in the revised supplementary website (see “Cross-domain generalization - XHuman” section in the supplementary website).

Note, the distortion in facial regions is mainly due to imperfect poses provided by DNA-Rendering. When the poses are accurate (e.g., for XHuman), we do not observe distortions, as shown in the “Cross-domain generalization - XHuman” section of the supplementary website. We also stress-test our method when human poses and masks are predicted from off-the-shelf tools. Results can be found in the “Cross-domain generalization - PeopleSnapshot” and “Cross-domain generalization - ExAvatar” sections of the supplementary webpage. Note, to demonstrate robustness, on PeopleSnapshot, we do not use the poses and masks provided by the dataset but use predictions instead. Even for predicted poses, our approach shows promising cross-domain generalization.

QA4. The novel pose representation displays black hands in the video.

The black hands observed for novel pose synthesis are due to occlusions: note that the hand is in the pocket in the reference source images and is therefore entirely invisible. As mentioned in the original limitation section, our method doesn’t hallucinate large unseen regions.

QA5. Minor issues: line 156 "estimate" to "estimation", line 234 translation -> scale?

Thanks for pointing this out, we updated the paper accordingly.