SignAvatars: A Large-scale 3D Sign Language Holistic Motion Dataset and Benchmark

We thank the reviewer for their considerate feedback. We appreciate that they recognize the impact and significance of the SMPL-X and MANO annotations in our dataset as well as its multi-prompt and cross-language nature. We are delighted that the reviewer finds our system design novel and paper exceptional. We confirm that the reviewer has a good grasp of the contributions of our paper.

We agree with the reviewer that some of the de-facto metrics used ubiquitously do not reflect the performance very well for the task at hand. $**FID**$ could be one of them. However, rather than omitting these metrics, we find it important to have them available as they are used across a wide range of works and make cross-checking easier. Nonetheless, we only employ FID to evaluate the first stage of VQVAE training. For this part, FID provides hints on whether our VQVAE is generating reasonable and plausible motions with a similar distribution to our ground-truth annotation.

New videos in $**our project webpage (see general remarks)**$, correlate well with our low-FID scores and demonstrte the realism and stability of our motion generation. Following reviewer's suggestion, in the project webpage we included character-driven animation to produce more visible motion patterns. In addition, we have $**enriched our Appendix**$ with Figures $\textcolor{red}{10-13}$, where the benefits are qualitatively shown.

Evaluating the impact of biomechanical constraints is indeed a good idea. To this end, we have included a version of our method ablating the biomechanical constratints in Table $\textcolor{red}{6}$ of our $**revised Appendix**$. This table can also be found in response to Reviewer qy3Y. Furthermore, Figure $\textcolor{red}{12}$ qualitatively depicts the the impact of these constraints, where the reconstructed poses are essentially more realistic and plausible.

We thank the reviewer for the recognition, especially regarding the impact and significance of our work. Your precious comments are indeed encouraging. Please feel free to follow up on the updates of our process, evaluation analysis, and applications on our anonymous GitHub page.

All the best,

We thank the reviewer for your detailed comments and careful perusal. We appreciate that the reviewer acknowledges the usefulness of our method and finds our SLP performance to be compelling, and multi-source prompting to be interesting especially when combined with 3D mesh generation.

Importantly, we appreciate their warning on the use of terminology. We have now updated the manuscript to use either $**``Deaf" or `Hard of Hearing' everywhere**$.

$**General.**$ We thank the reviewer for the suggestions. Our new supplementary video on our project page (see general comments) now contains narration. Rapid motions are still difficult for us to handle. Handling ambiguities in the SL videos is in fact an important future work.

$**About Dataset.**$ We kindly disagree with the reviewer. There are several components to a dataset, two of which are the input and the labels or annotations. While we source our videos from various online repositories, we do compute our annotations (crucial for supervised learning) which indicates a new dataset. We will in fact be releasing these annotations upon publication.

We agree with the reviewer that "large scale" is an ambiguous term used in reference to varying sizes for varying problems. Note that, we are a "3D motion" dataset and not a 2D video dataset. For example, a subset of our data, How2Sign, is already claimed to be large-scale for this setting. Nevertheless, our dataset is the largest as far as 3D meshes, SMPL-X and MANO annotations are concerned, making it a significant contribution in this field as confirmed by Reviewer MwWu.

$**About Pose Estimation Framework.**$ We thank the reviewer for bringing this up. Note that our annotation pipeline (pose estimation or reconstruction algorithm) with its temporal-consistency, biomechanical constraints and hierarchical initialization is novel. To demonstrate its advantage, we have now included more qualitative and quantitative results, both regarding our dataset and EHF dataset [1]. Specifically, $**Appendix B.1**$ provides new evaluations of our reconstruction in comparison to the state-of-the-art. In the existing benchmark of EHF, our annotation method shows a clear advantage (see Table 6, which we also include in response to Reviewer qy3Y). We further illustrate this qualitatively in Figure $\textcolor{red}{10}$.

[1] Pavlakos, Georgios, et al. "Expressive body capture: 3d hands, face, and body from a single image." CVPR 2019.

We thank the reviewer for their feedback.

$**1. Are there other labeling methods that can be compared?**$ We agree with the reviewer that other annotation methods can be used and thank the reviewer for bringing this up. We have considered this suggestion and have evaluated our method against other potential annotation strategies, on a dataset where precise ground truth is available, specifically the EHF dataset [1]. In particular, we choose the state-of-the-art methods of OSX [2], PyMafX [3], and Pixie [4], and present a comparison in $**Table 6**$ of our $**revised Appendix**$. We also include this table below:

We further present in Figures $\textcolor{red}{9}, \textcolor{red}{10}$ and $\textcolor{red}{11}$ of our $**revised Appendix**$, that our method provides significantly better quality regarding $**pixel alignment**$, with more natural and plausible hand poses. Please find more results in our updated videos on our anonymous project page, found in the general comments.

$**2. Structure of autoregressive Transformer and code-index vector.**$ Code index vector is the (discrete) latent feature of the motion. Let us consider Motion VAE in Figure 4. Any SL motions can be encoded into a continuous latent space, which can be discretized into a sequence of codebook indices via quantization using a learnable codebook. This is denoted as $X = [x_{1}, ..., x_{T/w}, x_{\mathrm{EOS}}]$, where $\mathrm{EOS}$ (end of sequence) is an extra learnable end token representing the stop signal. By projecting $X$ back to their corresponding codebook entries, we can obtain the $*code index vectors*$ as the latent feature $F^m_{1:(T/w)}=(f_{1}^{m}, \dots , f_{1:(T/w)}^{m})$ and $f_{i}^{m} \in \mathbb{R}^{d_{z}}$.

Autoregressive Transformer serves as an autoregressive code index generator and the objective for training the code index generator can be seen as an autoregressive next-index prediction task. The process can be formulated as follows: given the previous $i-1$ indices and the conditioning code $c$ embedded by fusing the linguistic feature $E^{l}$ and the codebook index vectors, the Transformer predicts the distribution of possible next indices $p(X_{i},c\mid X_{<i})$ as shown in Figure 4.

$**References**$

[1] Pavlakos, Georgios, et al. "Expressive body capture: 3d hands, face, and body from a single image." CVPR 2019.

[2] Lin, Jing, et al. "One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer." CVPR 2023.

[3] Zhang, Hongwen, et al. "Pymaf-x: Towards well-aligned full-body model regression from monocular images." TPAMI 2023.

[4] Feng, Yao, et al. "Collaborative regression of expressive bodies using moderation." 3DV 2021.

[5] Rong, Yu, Takaaki Shiratori, and Hanbyul Joo. "Frankmocap: A monocular 3d whole-body pose estimation system via regression and integration." ICCV 2021.

[6] Moon, Gyeongsik, Hongsuk Choi, and Kyoung Mu Lee. "Accurate 3D hand pose estimation for whole-body 3D human mesh estimation." CVPRW 2022.

[7] Lin, Jing, et al. "Motion-x: A large-scale 3d expressive whole-body human motion dataset." NeurIPS 2023.

We thank the reviewer for their constructive feedback and appreciate that the reviewer acknowledges the impact of our SL motion dataset on sign language production as well as the soundness of our baseline and the quality of our manuscript. We have incorporated the minor corrections and additional citations. We address the rest of the concerns below.

• $**Evaluation metrics.**$ We absolutely agree with the reviewer that, due to the novelty of the task and our benchmark, evaluation needs to be rethought. Unfortunately, we are not aware of a direct 3D back-translation model for motion SLP, which we plan to investigate in a future study. In this work, instead of re-inventing the wheel, we indeed contribute to the available repertoire of metrics. In particular, inspired by the studies in human-motion-generation, we introduce MR-precision, which evaluates how much $*generated motion*$ resembles the $*ground truth motion*$. As we describe in our Appendix C1, it measures the average accuracy at top-$k$ positions of sorted Euclidean distance between a motion feature generated from a sequence joint-rotations and 16 motion samples from dataset. We argue that, as opposed to mean 3D joint position error, this metric is especially suited to the 3D motion SLP task since: (i) the orientation of hand joints carries particular semantic meaning, (ii) such comparison in the latent space parallels a $*semantic evaluation*$ of signs, without having the need for an explicit continuous-mesh to sign classifier. In the future, we will work on designing more explicit, physical evaluation-models. We have now updated the paper for further clarity.

• $**Baseline semantics input.**$ $**In our SignVAE**$, the semantics input to our method is the linguistic feature $E^{l}$. As in Figure 4, our conditioning code $c$ is formed by concatenating this semantics input and the codebook index vectors of $Z^{l}$, whose dimension is reduced into the transformer dimension by a linear projection. We subsequently concatenated this feature with the current $*code index vector*$ for a positional embedding. $**In our baseline in Table 5**$, SignVAE (Base), we remove the PLFG, and replace the conditioning code $c$ with a canonical CLIP-style encoder feature with 512 dimensions.

• $**PLFG**$ is our parallel linguistic feature generator, which is the VQVAE based structure illustrated at the top of Figure 4. It is a linguistic feature generator jointly optimized with our SL motion generator (bottom of Figure 4) in a parallel manner. We have polished the description of this part in our revised version.

• $**Ablation on VQVAE.**$ We agree with the reviewer that ablations on our VQVAE are important. This is the reason why Table 5 of our main paper ablates on the essential components of our VQVAE, namely the parallel linguistic feature generator (PLFG) and the quantized semantic representation.

• $**Comparison to SignBERT+ on reconstruction.**$ We have properly cited and discussed SignBERT+ in our revised paper.

Dear Maria,

We acknowledge the oversight in our paper regarding the use of HamNoSys. Specifically, while SGNfiy does not directly require HamNoSys, it does necessitate labels for training a group classifier. These labels are typically derived from HamNoSys annotations, as stated by Forte et al. in the parsing of the Corpus-based Dictionary of Polish Sign Language (CDPSL). Though, to the best of our understanding, the paper does not provide alternative methods for obtaining these labels beyond HamNoSys parsing. We have clarified this solely implicit dependence on HamNoSys in our revised manuscript.

We now reflect the recent update that the SGNify 3D SL holistic dataset was released back in June, and mention that SGNify, enhanced by the methodologies of Renz et al., can process continuous signs, despite the dataset comprising only isolated signs (the primary focus of SGNify).

We would kindly like to point out that our work does not distinguish between isolated or continuous videos, and treats continuous videos naturally within its pipeline. This seamless integration removes the necessity for external sentence-level segmentation, allowing us to bypass the computation of group labels. Moreover, the SignAvatars dataset includes both continuous and isolated sign language videos.

It is important to stress that our SignVAE significantly departs from SGNify. We introduce a generative model for 3D sign language videos, enabling the exploration of new tasks such as 3D motion sign language production.

We are committed to adhering to all reviewing guidelines and look forward to further communication with you post-review stage.

All the best,