Thank you for acknowledging our work in terms of writing, architecture design, experimental discussion, analysis, and the significance of the bounded-duration predictor for talking head translation. We appreciate your valuable feedback, and we have thoroughly considered your comments, making important clarifications in response to the raised issues:

1.Details of bounded-duration-predictor

We apologize for the typos in the initial paper. Allow us to re-describe them:

For instance, when U={u1,u2,u3,u4}, D'={2.2,1.8,2.3,2.7} and T=10, the weight of each frame can be denoted as U'={1.0*u1, 1.0*u1, 0.2*u1, 1.0*u2, 0.8*u2, 1.0*u3, 1.0*u3, 0.3*u3, 1.0*u4, 1.0*u4, 0.7*u4}, where the weight of 0.2*u1 is only 0.2. And after enough 10 frames have been selected in order of highest to lowest weight([1.0*u1,1.0*u1,1.0*u2,1.0*u2,1.0*u3,1.0*u3,1.0*u4,1.0*u4,0.8*u2,0.7*u4,0.3*u3,0.2*u1]), 0.2*u1 is discarded. The sequence of input discrete units can be represented as U′={u1, u1, u2, u2, u3, u3, u3, u4, u4, u4}.

We will add the revised description and pseudocode to the latest version of the paper.

2.Why discrete units are extracted from mhubert? (component selection)

We carefully considered the choice of method for extracting discrete units. As a translation task, our objective is to be multilingual, encompassing not only language pairs with English as the target language (such as es-en and fr-en) but also translations involving other languages as the target language (en-es and en-fr, with some results showcased in the demo-page). This broader scope aims to further demonstrate the effectiveness of our model across diverse languages.

Among various discrete unit schemes (encodec, avhubert, hubert, mhubert, etc.), only mhubert is publicly available and widely used as a multilingual discrete unit extractor. Hence, we chose mhubert to extract discrete unit representations. As demonstrated in other research [1], if we want the model to encode languages other than English, it must be pre-trained on the speech of these languages.

Here, we also present a comparison of the clustering effect of avhubert and mhubert on different languages. Notably, there is no difference in performance between the two in English, but their effectiveness varies significantly in other languages. Furthermore, in this paper, we showcase that discrete units based on acoustic-only feature can also be effectively utilized for visual speech synthesis.

3.More issues:

• About Table 4: Thank you for your valuable suggestions. Initially, our goal was to assist readers in visualizing the quality of the translation. However, we believe your suggestion is highly valuable, and we will thoughtfully select the more crucial elements to be included in the main paper.
• About typos: We appreciate your careful reading of our work, and we will promptly correct these typos.
• About Open Source: We have organized the code and model and are committed to releasing it upon acceptance. We hope that our paper will be successfully accepted.

We believe that these clarifications and enhancements address the concerns raised by the reviewers and further strengthen the credibility and contribution of our work. We appreciate the opportunity to engage in this dialogue and are committed to refining our paper based on your feedback. If you have any other concerns, we look forward to discussing them further with you.

[1]. Le M, Vyas A, Shi B, et al. Voicebox: Text-guided multilingual universal speech generation at scale[J]. arXiv preprint arXiv:2306.15687, 2023.

Thank you for recognizing our work in terms of presentation, task significance, experimental adequacy, practicality, and reproducibility. We are grateful for your valuable input and have taken your comments into full consideration and made important clarifications in response to the issues raised:

1.The isometric feature is crucial for talking head translation

The isometric feature is pivotal for talking head translation. We acknowledge the concerns raised by the reviewers regarding equal-length translation, and indeed, we extensively deliberated on this matter during our research. In contrast to speech translation, video translation encounters a relatively fixed limit on video frame length, especially evident when dubbing a translated movie. In such instances, the voice actor for translated movies must synchronize the translation to match the original video's duration.

The absence of a duration-bounded module in the video results in noticeable frame skipping, leading to a significant loss of realism (refer to the demo page for results without the bounded-duration predictor). This limitation renders the approach unsuitable for professional scenarios like online meetings and movie translating, where the number of generated video frames must align with the original reference video. The introduction of the bounded-duration predictor becomes imperative in such cases, despite tradeoffs in other factors, as it effectively satisfies the fundamental requirements of video translation.

Acknowledging the reviewers' observation that this approach may cause excessive speedups and slow reads, we plan to address this concern in our future work. Specifically, we intend to investigate the vocabulary length of the generated content to further enhance the realism and authenticity of the translated videos.

2.The detailed evaluation process of MOS

Our comprehensive MOS scoring process for talking head translation tasks involves gathering scores across four dimensions: translation quality, image quality, synchronization, and overall sensation. For the unit-based talking head generation, we streamline the evaluation to two dimensions: image quality and synchronization. Translation quality assesses the consistency of the translated content with the original sentence, while image quality evaluates the presence of artifacts in the generated image. Synchronization measures the coherence of audio and visual speech, and overall sensation indicates the evaluation of the video's authenticity. Each sample is randomly scrambled and presented to 15 participants for scoring. A composite MOS is then calculated by averaging the scores for the corresponding dimensions, with each dimension scored individually on a scale of 1 (lowest) to 5 (highest).

3.Details of bounded-duration-predictor

We apologize for the typos in the initial paper. Allow us to re-describe them:

For instance, when U={u1,u2,u3,u4}, D'={2.2,1.8,2.3,2.7} and T=10, the weight of each frame can be denoted as U'={1.0*u1, 1.0*u1, 0.2*u1, 1.0*u2, 0.8*u2, 1.0*u3, 1.0*u3, 0.3*u3, 1.0*u4, 1.0*u4, 0.7*u4}, where the weight of 0.2*u1 is only 0.2. And after enough 10 frames have been selected in order of highest to lowest weight([1.0*u1,1.0*u1,1.0*u2,1.0*u2,1.0*u3,1.0*u3,1.0*u4,1.0*u4,0.8*u2,0.7*u4,0.3*u3,0.2*u1]), 0.2*u1 is discarded. The sequence of input discrete units can be represented as U′={u1, u1, u2, u2, u3, u3, u3, u4, u4, u4}.

We will add the revised description and pseudocode to the latest version of the paper.

We express our sincere gratitude to the reviewers for their thorough consideration of our work. In order to further underscore the significance of isometric translation in video translation tasks, we conducted an investigation into several websites that have already implemented video translation:

• Video Translation of HeyGen: https://labs.heygen.com/video-translate
• AI Dubbing & Video Translator of ElevenLabs: https://elevenlabs.io/dubbing
• Video Translator of Rask AI: https://www.rask.ai/tools/video-translator

Remarkably, we found that all of these platforms have adopted isometric translation as a fundamental requirement for video translation. As demonstrated by the outcomes of experiments of our own demo page, while variable-duration translation may yield viable results, it unmistakably compromises authenticity. Therefore, isometric translation is not only advisable but has also been widely recognized as a fundamental prerequisite for video translation applications.

We trust that this additional survey provided above will aid in clarifying the original intent of isometric translation and contribute to your better understanding and recognition of our work. If you have any further questions or require clarification, please feel free to engage in a discussion with us!

Dear Reviewer Qjzu,

Thanks again for your comments. We would like to kindly remind you that we tried our best to address the concerns you raised.

As the end of the author-reviewer discussion period is approaching, we would be grateful if we could hear your feedback regarding our answers to the reviews. We would be happy to discuss in detail if you have additional comments about our paper.

Best regards, Authors

Thank you for acknowledging our work in terms of its soundness, presentation, contribution, and experimental adequacy, and for providing valuable feedback. We have thoroughly considered your comments and have made significant clarifications and improvements in response to the raised issues：

1.The isometric feature is crucial for talking head translation

The isometric feature is pivotal for talking head translation. We acknowledge the concerns raised by the reviewers regarding equal-length translation, and indeed, we extensively deliberated on this matter during our research. In contrast to speech translation, video translation encounters a relatively fixed limit on video frame length, especially evident when dubbing a translated movie. In such instances, the voice actor for translated movies must synchronize the translation to match the original video's duration.

The absence of a duration-bounded module in the video results in noticeable frame skipping, leading to a significant loss of realism (refer to the demo page for results without the bounded-duration predictor). This limitation renders the approach unsuitable for professional scenarios like online meetings and movie translating, where the number of generated video frames must align with the original reference video. The introduction of the bounded-duration predictor becomes imperative in such cases, despite tradeoffs in other factors, as it effectively satisfies the fundamental requirements of video translation.

Acknowledging the reviewers' observation that this approach may cause excessive speedups and slow reads, we plan to address this concern in our future work. Specifically, we intend to investigate the vocabulary length of the generated content to further enhance the realism and authenticity of the translated videos.

2.The introduction of discrete units and bounded duration predictor meets the need for talking head translation.

Thank you for acknowledging our proposal to integrate discrete units and a bounded duration predictor into talking head translation. We take pride in being the first to introduce these two designs to the talking head translation task, effectively addressing the challenging issue of the need for equal-length translation in this context. This innovation enables the parallel synthesis of audio-video-speech, thereby enhancing translation speed and synchronization. Instead of implementing numerous intricate changes to other modules, we have opted to reuse the most basic model of talking head generation. This decision aims to underscore the modules we propose specifically for the talking head translation task, providing the community with a concise and easy-to-understand framework.

3.More issues:

• Synchronicity Loss: The synchronization loss requires a pre-trained expert model [1] to compute the synchronization-match (P_sync) between audio speech and video speech. This computed synchronization metric is then used to constrain the synchronization between the two during the training of the talking head generation model.

• Top frame part:

  In the talking head generation task, the majority of methods utilize the upper half of the face as a reference for the current frame. This approach exhibits slight pixel changes during the generation process but generally maintains consistency with the original image.

• Confusing Expressions: Predicting discrete units, not phoneme units. We decode discrete units, not phonemes, and we apologize for the misleading expression used here. In the decoding process, there is a specific mapping correlation between the discrete unit and the pronunciation, leading us to use the term "phoneme" in this context. We acknowledge the need to revise this expression in the latest version of the paper.

We believe that these clarifications and enhancements address the concerns raised by the reviewers and further strengthen the credibility and contribution of our work. We appreciate the opportunity to engage in this dialogue and are committed to refining our paper based on your feedback. If you have any other concerns, we look forward to discussing them further with you.

[1]. Chung J S, Zisserman A. Out of time: automated lip sync in the wild[C]//Computer Vision–ACCV 2016 Workshops: ACCV 2016 International Workshops, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part II 13. Springer International Publishing, 2017: 251-263.

Thank you for acknowledging the Talking Head Translation task in our paper and providing valuable feedback. We have thoroughly considered your comments and implemented important clarifications and improvements to address the raised issues:

1. The First Audio Discrete Units Based Talking Head Generation:

We appreciate the recognition of our work as the first exploration into talking head translation using audio discrete units:

Talking head translation has many practical applications but has not been studied by prior work.

As correctly highlighted, prior works you mentioned primarily focused on speech-to-speech translation [1] or only utilized visual speech as input[2]. Our contribution lies in introducing a novel approach that employs audio discrete units specifically for talking head synthesis. The parallel generation of highly synchronized audio-visual speech, coupled with the introduction of a bounded duration-predictor for enhanced realism, distinguishes our work from existing research in this domain.

2. Prioritizing Translation Quality Over Voice Cloning:

It is noteworthy that the speech-to-speech translation works [1] you mentioned did not incorporate voice cloning. In the realm of translation tasks, I contend that maintaining the consistency of the translated content is more crucial than introducing additional engineering operations for voice cloning (please also note that the speech part is not our primary contribution).

Additionally, our approach is highly extensible, facilitating seamless integration into various speech-to-speech translation tasks, including [1], among others. Should these efforts subsequently incorporate voice cloning, our work can be synergistically combined with them to achieve parallel audio-visual speech translation with voice cloning.

3. Evaluation Metrics:

• Authenticity of BLEU Testing Texts:

  We want to clarify that the texts used for BLEU testing are not machine-translated. English, being the only language possessing audio-video speech in our study, is paired with Es-En and Fr-En for experimentation. The ground-truth text of the translation results are the text from the original LRS3 dataset, professionally annotated and verified by the official TED, ensuring credibility. Because of the same concern as yours, we purposely did not choose en-es and en-fr for qualitative validation as discussed in Appendix D.1, and only showed the corresponding results in the demo-page.

• FID Metric Details:

  To assess the generation quality of the unit-based face synthesizer, we use the FID metric, following the evaluation process of wav2lip[3]. Real video content from LRS2 serves as a reference to gauge the quality of video synthesis on LRS2, employing the InceptionV3 model.

• Comprehensive Evaluation of Talking Head Translation:

  We have provided detailed MOS scores for talking head translation tasks, evaluating on four dimensions: translation quality, image quality, synchronization, and overall sensation. Despite the high translation quality achieved by ASR+NMT+TTS, it is evident that the overall satisfaction is compromised due to jitter caused by repeated references to the reference frames. With the incorporation of the bounded-duration predictor, TransFace+bounded attains competitive results, notably outperforming other methods in terms of synchronization and overall sensation.

  	translation quality	image quality	synchronization	overall sensation	Mean
  ST+TTS+Wav2Lip	3.78±0.05	4.03±0.08	3.66±0.04	3.57±0.03	3.76±0.05
  ASR+NMT+TTS+Wav2Lip	4.23±0.06	4.11±0.07	4.12±0.08	4.18±0.11	4.16±0.08
  Translatotron2+Wav2Lip	2.79±0.09	3.98±0.12	3.68±0.07	2.91±0.20	3.34±0.12
  S2ST+Wav2Lip	4.03±0.08	4.05±0.04	4.02±0.09	3.86±0.07	3.99±0.07
  TransFace(ours)	4.19±0.08	4.08±0.07	4.28±0.04	3.93±0.05	4.12±0.06
  TransFace(ours)+bounded	4.17±0.06	4.16±0.06	4.28±0.05	4.39±0.11	4.25±0.07

We believe that these clarifications and enhancements address the concerns raised by the reviewers and further strengthen the credibility and contribution of our work. We appreciate the opportunity to engage in this dialogue and are committed to refining our paper based on your feedback. If you have any other concerns, we look forward to discussing them further with you.

[1]. Lee et al., 2021 Textless speech-to-speech translation on real data.

[2]. Hsu et al., 2023 ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement

[3]. KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. A lip sync expert is all you need for speech to lip generation in the wild.

Hello reviewer hwnr, we've noticed that there may be some inherent impressions regarding the S2ST task, particularly with a focus on voice cloning. To ensure a fair assessment and prevent any potential bias, we've curated demo pages from past speech-to-speech translation works for your consideration:

1. Textless Speech-to-Speech Translation on Real Data [1]: https://facebookresearch.github.io/speech_translation/textless_s2st_real_data/index.html
2. Direct Speech-to-Speech Translation With Discrete Units [2]: https://facebookresearch.github.io/speech_translation/direct_s2st_units/index.html
3. TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation [3]: https://transpeech.github.io/
4. SeamlessM4T-Massively Multilingual Multimodal Machine Translation [4]: https://seamless.metademolab.com/

These demos highlight the emphasis on improving the consistency of the translated speech content, prioritizing this aspect over the realization of voice cloning. We trust that these demonstrations will provide valuable insights for your evaluation. If you have any further questions or require clarification, please feel free to engage in a discussion with us!

[1] Lee A, Gong H, Duquenne P A, et al. Textless speech-to-speech translation on real data[J]. arXiv preprint arXiv:2112.08352, 2021.

[2] Lee A, Chen P J, Wang C, et al. Direct speech-to-speech translation with discrete units[J]. arXiv preprint arXiv:2107.05604, 2021.

[3] Huang R, Liu J, Liu H, et al. Transpeech: Speech-to-speech translation with bilateral perturbation[J]. arXiv preprint arXiv:2205.12523, 2022.

[4] Barrault L, Chung Y A, Meglioli M C, et al. SeamlessM4T-Massively Multilingual & Multimodal Machine Translation[J]. arXiv preprint arXiv:2308.11596, 2023.

Difference between bounded-duration-predictor and naive duration-predictor.

1. The naive duration predictor directly rounds up the predicted duration sequence after predicting the duration length for each discrete unit. This approach often leads to a mismatch between the length of the synthesized translation result and the original audio.
2. Making adjustments to the naive duration predictor by normalizing it and then rounding it (as in step 1 of the bounded-duration-predictor) allows for some degree of duration control. However, this method still encounters issues when dealing with more or fewer frames. For instance, in scenarios like:
   • Case 1: D′=[1.566,1.666,1.766],T=5 (Predicted frames = 6, one extra frame)
   • Case 2: D′=[1.55,1.6,1.55,1.65,1.65],T=8 (Predicted frames = 10, two extra frames)
   • Case 3: D′=[2.2,1.8,2.3,2.7],T=10 (Predicted frames = 9, one frame less).
3. Our proposed bounded-duration-predictor module addresses these challenges by implementing an efficient duration allocation method as detailed in Appendix A. This method ensures effective control of the total duration of the discrete unit sequence, resulting in 100% isometric translation.

An Ablation on Keeping the Repeating Units

I'm not sure I fully understand what you're saying, but I'm guessing that you might want an ablation experiment of TranFace without bounded-duration-predictor in Table 3. In fact, its length metrics are all essentially the same as the S2ST+Wav2Lip approach, as both use the same naive duration-predictor, which does not allow for overall modulation of duration:

If my understanding deviates from your question in any way, feel free to continue asking, and I'm here to address any further doubts you may have.

The synchronization of A-to-AV results is consistent with the unit-based talking head generation model.

Throughout the process of talking head translation (A-to-AV), the synchronization of audio-visual speech in the generated results is dependent on the specific audio-visual speech synthesis model used. Put simply, the synchronization metric can be cross-referenced with the outcomes from different talking head generation models detailed in Table 1. To address your curiosity, we promptly conducted an assessment of the synchronization in all the generated audio-visual speech and present the comparison in the following table:

Summary

Thank you for your prompt response and your interest in this paper. Your insightful questions have contributed to a more comprehensive understanding of the details associated with this work. I trust that my responses have addressed all of your inquiries. If you have any further questions, please do not hesitate to ask.

Thank you for addressing my comments.

1. On real data vs. synthesized data, the real concern is about the extremely high BLEU score: 97.04 in using ground-truth video. For such high BLEU score, the mapping between source speech and target text is almost deterministic, which isn't the case for real translation (e.g., the BLEU of Whisper for Es-En is typically between 20 and 40). As is already pointed out either in the rebuttal or the paper (Table 2), there exists a trade-off between translation quality and synchronicity for cascaded models (ASR+NMT+TTS+Wav2Lip) vs. the proposed approach (Transface). As the measurement of translation quality is directly impacted by the evaluation data, It is a bit unclear to me how such trade-off looks like when using a more challenging dataset. In addition, there does exist real multilingual data for Es-En and Fr-En such as Multilingual TEDx [1], which has already been used in audio-visual setting in several prior works [2] (recognition), [3] (translation). Is there a reason of not using that? As to the contribution on speed, I did acknowledge that in the review.
2. On duration prediction, it's a natural extension to the duration prediction of Lee et al. 2021 when the length of target speech is given. This isometric assumption makes sense in using the LRS3-T data. However, in general, the length of target speech can vary much from the length of source speech. Under a setting where its length is unknown, how does one predict the target length and how does the prediction error impacts the performance? Or simply how much value does the isometric assumption add compared to a simple cascaded system (e.g., ASR+NMT+TTS+Wav2Lip) without such constraint? Again this is related to the data issue mentioned above.
3. The question on synchronicity is resolved, thank you.

[1]. Salesky, E. et al. The Multilingual TEDx Corpus for Speech Recognition and Translation. In Proceedings of the 22nd Annual Conference of International Speech Communication Association.

[2]. Pingchuan M et al., Visual Speech Recognition for Multiple Languages in the Wild. Nature Machine Intelligence

[3]. Mohamed A et al., MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation.

Thank you for your response. Allow me to provide some clarifications in response to your inquiries:

Firstly, I want to underscore that the experiment with ID=0 in Table 2 (id=0, BLEU=97.04) involved calculating BLEU directly from the audio-visual speech recognition results of the target English audio-visual speech in LRS3-T. Our intention was to demonstrate the additional errors that may be introduced by utilizing the audio-visual speech recognition model. It's crucial to note that this process represents a mapping from target language speech to target language text, rather than which you described "mapping between source speech and target text."

Additionally, there are Three primary reasons for choosing LRS3-T over Multilingual TEDx:

1. The average length of Multilingual TEDx is 6.5 seconds, while all current audio-visual speech datasets are approximately 3 seconds. Performing translation on this heavily excessive length of speech could result in significant reference frame reuse or audio acceleration. This not only diminishes the authenticity of the translation result for the speaking head but also introduces interference that can adversely impact the subjective evaluation of the translation outcome.
2. LRS3-T has real target audio-visual speech, it can provide better quantitative comparison.
3. LRS3-T allows for the simulation of the real talking head translation scenarios (the results of En-Es and En-Fr) with higher persuasive power.

Furthermore, it's worth noting that you mention muavic, which is another multilingual audio/video speech dataset. We have indeed examined muavic but found its performance to be less impressive. Even when evaluated on the dataset used for training, its Audio-Visual Speech Recognition (AVSR) WER are only 16.2% (Es) and 19.0% (Fr). In contrast, the ASR and AVSR models utilized for validation in speech-to-speech translation are required to perform well on their respective training datasets. For instance, the ASR model in [1] achieved a Word Error Rate (WER) of 2.0% on LJSpeech, and the AVSR model we employed achieved a WER of 1.5% on LRS3. Consequently, we believe that utilizing the AVSR model of language X from muavic could potentially introduce a significant amount of error into the results presented in this paper.

[1] Lee A, Gong H, Duquenne P A, et al. Textless speech-to-speech translation on real data. ACL 2021.

Thank you for recognizing our proposed bounded-duration-predictor, who brings more novel features to duration-predictor. I think your current problem can be summarized as the meaning of isometric translation, which has been discussed in discussions with other reviewers (https://openreview.net/forum?id=71oyMJiUm2&noteId=FZva2rAT19) and in the general response (https://openreview.net/forum?id=71oyMJiUm2&noteId=rNyqlgFpnm).

In contrast to speech translation, video translation encounters a relatively fixed limit on video frame length, especially evident when dubbing a translated movie. In such instances, the voice actor for translated movies must synchronize the translation to match the original video's duration.

The absence of a duration-bounded module in the video results in noticeable frame skipping, leading to a significant loss of realism (refer to the demo page for results without the bounded-duration predictor). This limitation renders the approach unsuitable for professional scenarios like online meetings and movie translating, where the number of generated video frames must align with the original reference video. The introduction of the bounded-duration predictor becomes imperative in such cases, despite tradeoffs in other factors, as it effectively satisfies the fundamental requirements of video translation.

We also conducted a survey of existing video translation platforms (https://openreview.net/forum?id=71oyMJiUm2&noteId=uV5tsVEcTc). Remarkably, we found that $\textcolor{red}{\text{all these platforms have adopted isometric translation as a fundamental requirement for video translation}}$. As demonstrated by the outcomes of our experiments on our demo page, while variable-duration translation may yield viable results, it unmistakably compromises authenticity. Therefore, isometric translation is not only advisable but has also been widely recognized as a fundamental prerequisite for video translation applications.

Acknowledging the reviewers' observation that this approach may cause excessive speedups and slow reads, we plan to address this concern in our future work. Specifically, we intend to investigate the vocabulary length of the generated content to further enhance the realism and authenticity of the translated videos. We have incorporated a discussion on the limitations of isometric translations in Appendix D.1 to underscore the significance of isometric translations while also acknowledging the potential associated limitations.

3.Details of bounded-duration-predictor

We apologize for the typos in the initial paper. Allow us to re-describe them:

For instance, when U={u1,u2,u3,u4}, D'={2.2,1.8,2.3,2.7} and T=10, the weight of each frame can be denoted as U'={1.0*u1, 1.0*u1, 0.2*u1, 1.0*u2, 0.8*u2, 1.0*u3, 1.0*u3, 0.3*u3, 1.0*u4, 1.0*u4, 0.7*u4}, where the weight of 0.2*u1 is only 0.2. And after enough 10 frames have been selected in order of highest to lowest weight([1.0*u1,1.0*u1,1.0*u2,1.0*u2,1.0*u3,1.0*u3,1.0*u4,1.0*u4,0.8*u2,0.7*u4,0.3*u3,0.2*u1]), 0.2*u1 is discarded. The sequence of input discrete units can be represented as U′={u1, u1, u2, u2, u3, u3, u3, u4, u4, u4}.

In the specific implementation, we meticulously devised algorithms to attain the bounded duration goal:

1. After normalization, the duration D can be computed as the allocated duration D’ corresponding to each token based on the target length T.

   D′ = Normalize(D)×T

2. Following the rounding method, it is converted to an integer predicted duration PRED.

   PRED = Clamp(Round(D′),min = 1)

3. Calculate the difference DIFF between the predicted duration PRED and the allocated duration D’ for each token.

   DIFF = D′−PRED

4. Determine whether the predicted duration PRED still needs adjustment in the number of frames. If an increase is required, select the highest-weighted difference DIFF from the sequence for its duration+1. Conversely, if a decrease is needed, select the lowest-weighted difference DIFF for its duration-1.

   ADD =Zeroes()

   if Sum(PRED) > T then

   INDEX =TopK(−DIFF,k =Sum(PRED)−T)

   ADD [INDEX ]= −1 ;

   else

   INDEX =TopK(DIFF,k =T −Sum(PRED))

   ADD [INDEX ]= 1

   end

   OUT =PRED +ADD

Let's revisit the previous example for illustration, when U={u1,u2,u3,u4}, D'={2.2,1.8,2.3,2.7} and T=10:

Step 1: D'=[2.2,1.8,2.3,2.7], T=10

Step 2: PRED=[2,2,2,3]

Step 3: DIFF=[0.2,-0.2,0.3,-0.3]

Step 4: Since SUM(PRED)=9<10, for the largest T-SUM(PRED)=1 corresponding token in DIFF, its duration+1, resulting in OUT=[2,2,2+1,3] = [2,2,3,3]

We will add the revised description and pseudocode to the latest version of the paper.

We express our sincere gratitude to all the reviewers for their diligence and invaluable suggestions. We look forward to any further insights or inquiries that may arise from our work.