Thank you for recognizing the evaluations, engineering, and openness of our work. We will address your concerns below.

Weaknesses

W1: Since the training involves multiple separately stages, extending the pipeline to more datasets—especially in a multilingual setting—can significantly complicate the task.

R1: Although our SignViP framework consists of three training stages, only the final stage (Step III: Multi-Condition Token Translator) is directly language-dependent. The first and second stages are entirely language-agnostic, as they operate solely on videos and associated conditions without considering language categories.

Consequently, when extending to multilingual datasets, the training of these two stages can be shared across languages. Specifically, the Sign Video Diffusion Model and FSQ Autoencoder can be pre-trained on large-scale human video datasets, even if these videos are not for sign language. Subsequently, for each target language, only the Multi-Condition Token Translator needs to be trained separately.

Thus, our pipeline remains efficient and does not introduce significant complexity when scaling to multiple languages.

Questions

Q1: Could you use more references for SLVG (e.g., PGMM [1])?

A1: Thank you for pointing out the omission of one relevant work on SLVG. As this work has not released its code and pre-trained weights, we are unable to reproduce it from scratch and include it as a baseline in our experiments within the limited timeframe. However, we will reference and introduce this work in Section 2 (Related Works).

---

Q2: Do you use Sign Language Transformers (SLT) [2] for video back-translation and the method from [3] for pose back-translation?

A2: No. For both pose and video back-translation evaluations, we use SLT as the back-translation model. Specifically, for video back-translation, video frames are input into the SLT model. For pose back-translation, we follow the evaluation protocol of ProTran [4] and modify the original SLT architecture to accept pose points as input.

---

Q3: What is the resolution of the output video? Is it the same as input?

A3: For RWTH-2014T and How2Sign, the output resolutions are 272×224 and 512×512, respectively. The resolution of output video is the same as that of input. Although we have noted the resolution in Appendix B.1, we will include them in Section 4.1 to ensure clarity.

---

Q4: In Figure 2, what is a Mixer?

A4: The Mixer module is a lightweight MLP that aggregates tokens from the same frame into unified hidden states, which is described in Appendix C.

---

Q5: Please provide a short explanation for Figure 2. It is difficult to understand at first glance.

A5: Thanks for your suggestion. We will provide a detailed caption for Figure 2, as follows:

"Figure 2. Framework of our SignViP for sign language video generation (SLVG). (1) The spoken language text is translated into the multi-condition tokens by Multi-Condition Token Translator. (2) These tokens are decoded by FSQ Autoencoder into multi-condition embeddings, which are equivalent to the embeddings of multiple fine-grained conditions (i.e., fine-grained poses and 3D hands) generated by a multi-condition encoder. (3) The embeddings are injected into Sign Video Diffusion Model to guide the generation of sign language videos."

---

Q6: In Section 3.6, the term "mixed" is vague. It is important to explain what the mixer is in the paper and not the appendix.

A6: Thanks for your suggestion. We realize that not describing the structure and function of Mixer in the main text may cause confusion for readers, and we will include a detailed description of Mixer in the main part of the paper.

Paper Formatting Concerns

C1: In line 257, "SPL" is not defined before.

R1: Thank you for your reminder. This is a typographical error. It should be "SLP (Sign Language Production)" rather than "SPL". We will correct this mistake.

References

[1] Pose-Guided Fine-Grained Sign Language Video Generation, ECCV 2024.
[2] Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation, CVPR 2020.
[3] Neural Sign Language Synthesis: Words are Our Glosses. WACV 2020.
[4] Progressive Transformers for End-to-End Sign Language Production, ECCV 2020.

We sincerely appreciate the recognition of the quality, clarity, significance, and originality of our work. We will address your concerns below.

Weaknesses

W1: Sign language translation models still do not perform too well in this domain, thus it makes it difficult to judge if the videos are generating correct sign language. Human evaluations would have strengthened the paper, though understandably it is difficult to find sign language experts that can evaluate the results.

R1: Thank you for your suggestion. To facilitate human evaluation for the SLVG task, we further adopt a compromise human evaluation strategy that does not require the sign language experts. Specifically, we present human evaluators with the ground-truth video alongside several corresponding anonymized videos generated by different methods. The evaluators are given the following instruction:

"Please choose the video where the signer's actions appear most similar to those in the ground-truth video."

We invited 10 evaluators, each of whom received 25 group samples for the RWTH-2014T dataset and 50 for the How2Sign dataset, resulting in totals of 250 and 500 votes, respectively. We calculate the proportion of votes for each method, with the results shown below. Notably, our SignViP receives the highest proportion of votes, indicating that SignViP generates more accurate sign language videos.

---

W2: Some figures could be improved. (1) Figure 1 is a bit difficult to follow while reading the text, since you almost have 4 different figures lumped into one. (2) Some figures are very difficult to see (the ones with example video frames of signers) as they have been made so small.

R2: Thank you for your suggestions. We will enhance the presentation of the figures. Specifically, Figure 1 will be reorganized to improve readability. In addition, all other figures containing video frames will be enlarged to make them easier for readers to view.

Questions

Q1: Will you share the details of Hand SSIM to enable others to utilize it?

A1: Yes, we are committed to sharing the implementation details of Hand SSIM by publicly releasing our code repository.

We sincerely appreciate your recognition of the paper's clear writing, as well as the quality and diversity of its evaluation results. We will address your concerns below.

Weaknesses

W1: This paper lacks novelty, whose core innovation is more like combining existing concepts (tokenization, diffusion, multi condition control) in a reasonable way in SLVG tasks.

R1: We respectfully disagree with your opinion and would like to further clarify the novelty and unique contributions of our paper. Our primary innovation is a discrete tokenization paradigm through the construction of a discrete multi-condition token space to bridge fine-grained conditions with the sign language video frames. While prior works have explored the use of multiple [1] or fine-grained [2, 3, 4] conditions for diffusion-based human video generation, there are significant distinctions compared to our method:

1. In the SLVG task, conditions are limited to serving as intermediary that bridge spoken language text and sign language videos. While incorporating multiple fine-grained conditions can enhance the generative process, it also increases the complexity of the translation process. To address this challenge, we introduce a compression strategy (i.e., FSQ Autoencoder) for conditions, enabling an efficient yet expressive conditional space. This strategy facilitates a balanced trade-off between translation and generation.
2. As a compression strategy, although quantization-based tokenization has been widely explored [5, 6, 7, 8], existing methods focus primarily on tokenizing pixel space. To the best of our knowledge, our work is the first to introduce tokenization on multiple fine-grained conditions for human video generation.

Therefore, we believe that our core contribution is indeed novel and advances the state-of-the-art in SLVG.

---

W2: Some baselines are not for sign language.

R2: The baselines in Tables 1 and 2 correspond to Sign Language Video Generation (SLVG) methods and Sign Language Production (SLP) methods, respectively. In Table 3, "ControlNet" and "AnimateAnyone" are trained on sign language video datasets to evaluate their condition-to-video generative capabilities.

However, we acknowledge that our paper lacks a comparison of video quality among SLVG methods. To further address the gap, we introduce two novel metrics in addition to FID and FVD, which are CLIP-FID and Identity Similarity (IDS), since temporal alignment cannot be fully preserved.

Specifically, CLIP-FID is a variant of FID that leverages CLIP embeddings as frame representations. IDS assesses identity consistency between generated and ground-truth videos by calculating the cosine similarity of face embeddings extracted using YOLO5Face and Arc2Face.

The following experimental results validate the state-of-the-art performance of SignViP.

---

W3: Lack of direct evaluation of token translation accuracy.

R3: It is challenging to directly compute the accuracy of token translation. Because of the temporal shifts and uneven scaling, the translated tokens during inference may not perfectly align with ground-truth tokens. As a replacement, we employ normalized DTW distance as the metric.

Dynamic Time Warping (DTW) distance is a similarity measure that computes the optimal alignment between two sequences, even if they differ in length or are not aligned one-to-one. The DTW distance is normalized to accommodate variations in sequence length.

For our method, we decode the translated tokens into condition embeddings using the decoder of the FSQ Autoencoder, and compare these embeddings with the ground-truth condition embeddings using the normalized DTW distance.

To validate the effectiveness of normalized DTW distance, we follow the experimental settings in Section 4.3 (Effect of Compression) and Appendix D.7 (Effect of Scheduled Sampling Strategy). The following experimental results show that the trend of the normalized DTW distance closely matches that of previously reported metrics, confirming its reliability as an evaluation metric for token translation.

Questions

Q1: The model relies on the accuracy of the discrete multi-condition tokens. Have you measured the accuracy of these tokens?

A1: Please refer to our response to Weakness 3.

---

Q2: Could you justify the use of FSQ over more established alternatives like VQ-VAE?

A2: In Appendix D.8 (FSQ vs. VQ), we have compared FSQ and VQ as quantization strategies for the FSQ Autoencoder. Specifically, we evaluated both methods in terms of codebook usage efficiency and multi-conditional reconstruction loss.

The results are summarized as follows: the codebook usage experiments demonstrate the stability and scalability of FSQ, while the reconstruction loss experiments show that FSQ more effectively preserves the original conditional structure.

Additionally, in Section 4.3 (Effect of Quantization), we abalated quantization from FSQ Autoencoder, treating this as a special quantization strategy. The qualitative comparison in Figure 7 demonstrates that ablating quantization leads to weak semantic alignment and low video quality. It highlights the importance of quantization for our SignViP. Here, we present the detailed quantitative results on RWTH-2014T.

---

Q3: Would you please include more SLVG works as baselines?

A3: Thank you for your suggestion.

We add new baselines that combine SLP methods (i.e., MDN and MoMP) with pose-to-video methods (i.e., ContorlNet and AnimateAnyone). Specifically, we first use an SLP method to convert spoken language text into skeleton poses, and then employ a pose-to-video method to synthesize sign language videos from these poses.

The video back-translation results are presented below, which demonstrate that our SignViP still outperforms the newly added baselines.

References

[1] Follow-Your-Pose V2: Multiple-Condition Guided Character Image Animation for Stable Pose Control, arXiv:2406.03035.
[2] Magicanimate: Temporally Consistent Human Image Animation Using Diffusion Model, CVPR 2024.
[3] Magicportrait: Temporally Consistent Face Reenactment with 3D Geometric Guidance, arXiv:2504.21497.
[4] Vlogger: Multimodal Diffusion for Embodied Avatar Synthesis, arXiv:2403.08764.
[5] Finite Scalar Quantization: VQ-VAE Made Simple, ICLR 2024.
[6] Language Model Beats Diffusion–Tokenizer is Key to Visual Generation, arXiv:2310.05737.
[7] Image and Video Tokenization with Binary Spherical Quantization, arXiv:2406.07548.
[8] Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis, arXiv:2412.04431.

We sincerely appreciate your recognition of the clear motivation and SOTA performance demonstrated in our work. We will address your concerns below.

Weaknesses

W1: The method introduction is a bit complicated, without emphasizing the novel part of the paper.

R1: The main novelties of our paper are detailed in Sections 3.2 and 3.3. Our key innovations are as follows:

1. Discrete Tokenization Paradigm (lines 144–145): To mitigate errors in directly translating high-dimensional conditions, we propose a discrete tokenization approach to integrate and represent these conditions;
2. Novel Module Design (lines 146–154): We present new modules specifically designed for SignViP, enabling more robust processing of multi-condition inputs;
3. Multi-Stage Training Strategy (Section 3.3): We introduce a multi-stage training strategy to construct a comprehensive multi-condition token space for our discrete tokenization paradigm.

We will improve the method section to more clearly emphasize these core innovations and improve its clarity and readability.

---

W2: The performance evaluation mainly depends on back-translation techniques, which highly relate to the quality of the back-translation model.

R2: We agree that the evaluation protocol is highly dependent on the quality of the back-translation model, which may introduce bias. While human evaluation maybe provide a more reliable resolution, it is challenging to recruit sign language experts.

To address this, we further adopt a compromise human evaluation strategy that does not require the sign language experts. Specifically, we present human evaluators with the ground-truth video alongside several corresponding anonymized videos generated by different SLVG methods. The evaluators are given the following instruction:

"Please choose the video where the signer's actions appear most similar to those in the ground-truth video."

We invited 10 evaluators, each of whom received 25 group samples for the RWTH-2014T dataset and 50 for the How2Sign dataset, resulting in totals of 250 and 500 votes, respectively. We calculate the proportion of votes for each SLVG method, with the results shown below. Notably, our SignViP receives the highest proportion of votes, indicating that SignViP generates more accurate sign language videos.

---

W3: While the method achieves SOTA on Table 1, several metrics in Table 2 fall behind the MoMP method.

R3: In Table 2, we compare our SLVG method with SLP (Sign Language Production) approaches rather than other SLVG methods. These SLP baselines translate text directly into pose sequences, which aligns with our pose back-translation evaluation pipeline. However, for our SLVG method, we need to detect poses from the generated videos, which may introduce additional errors and impact the evaluation results. Consequently, the state-of-the-art SLP method (i.e., MoMP) outperforms our method on certain metrics in Table 2, although our results remain comparable.

To enable a fair comparison in the SLVG setting, we further combine MoMP with a pose-to-video generation approach. The following video back-translation results demonstrate that our method is better suited for the SLVG task compared to MoMP-based methods.

---

W4: Sign Video Diffusion Model is initialized with an existing diffusion model, and it lacks ablation to show the effect of such initialization, making whether the comparison is fair is not sure.

R4: The comparison is fair, because all the diffusion-based SLVG methods (i.e., SignGAN+AnimateAnyone, SignGen, and our SignViP) are initialized with parameters from existing pretrained diffusion models.

Training a video generation diffusion model from scratch is extremely challenging due to the high computational costs and slow convergence. Therefore, using pretrained diffusion parameters to initialize the customized diffusion model is a fundamental strategy to significantly accelerate training [1, 2, 3, 4, 5].

To further validate the effectiveness of pretrained parameter initialization, we conducted an ablation study, evaluating the quality of generated videos both with and without pretrained initialization under the same training steps. The results below clearly demonstrate the substantial advantage of initializing with pretrained parameters.

Questions

Q1: Is there any other evaluation metric to determine the quality of video generation? As BLEU and ROUGE scores have already shown not perfect in language generation tasks.

A1: Thank you for your suggestion. To provide a more comprehensive evaluation of back-translated texts, we further employ the COMET [6, 7] metric, which is specifically designed to predict human judgments of machine translation quality. COMET is widely used for machine translation tasks [8, 9, 10] and is considered more suitable than BLEU and ROUGE.

As shown below, for video back-translation, our SignViP still outperforms other SLVG baselines according to the COMET metric.

As shown below, for pose back-translation, SignViP achieves performance that is slightly lower but comparable to the state-of-the-art SLP method (i.e., MoMP). However, when MoMP is combined with pose-to-video generative models (i.e., ControlNet or AnimateAnyone) for the SLVG task, the resulting MoMP-based methods yield lower COMET scores than our approach. These results on the COMET metric indicate that SignViP is more suitable for the SLVG task compared to MoMP-based methods.

References

[1] Latent Video Diffusion Models for High-Fidelity Long Video Generation, arXiv:2211.13221.
[2] Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, arXiv:2311.15127.
[3] Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Models, ECCV 2024.
[4] AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning, ICML 2024.
[5] Follow Your Pose: Pose-Guided Text-to-Video Generation Using Pose-Free Videos, AAAI 2024.
[6] COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task, WMT 2022.
[7] COMET: A Neural Framework for MT Evaluation, EMNLP 2020.
[8] Hallucinations in Large Multilingual Translation Models, TACL 2023.
[9] Tower: An Open Multilingual Large Language Model for Translation-Related Tasks, COLM 2024.
[10] Exploring Human-Like Translation Strategy with Large Language Models, TACL 2024.