TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. The detailed responses are listed below.

R1. About presentation (Weakness 1)

We will reorganize the paper, particularly by dividing section 3 into two main subsections and adding pointers from the introduction to improve clarity. With an additional page allowed in the main content of the final version, we will also consider adding visualizations to further illustrate the proposed methods.

R2. About inference speech (Weakness 2, Question1)

Your concern about inference speed is valid. Generally, the inference speed for AR models is slower than that of non-AR models. To address this, we have conducted an analysis of inference speed. The RTF (real-time factor) for our model architecture is close to 1, while that of seamless expressive (with NAR T2U modeling) is 0.3, as measured on an Nvidia A6000 GPU with fp16. We believe that the inference speed could be improved by 2 to 4 times by leveraging Grouped Code Modeling, as proposed in VALL-E 2 (https://arxiv.org/pdf/2406.05370). Our current application scenario is video dubbing, which can be done on the cloud and offline. Therefore, inference speed has not been thoroughly investigated in this study.

R3. Clarifying target clip encoding (Question 2)

The pooled representation of the target clip serves as the acoustic prompt that the generation of the first layer codec can condition on. It is similar to the acoustic prompt in Vall-e, except we use only one token instead of a token sequence for acoustic prompting. The benefit of using a single token is that it creates an information bottleneck, preventing too much semantic information from passing through. This way, we hope the model can learn purely acoustic/speaker-related information. We will clarify this in the revised version.

In response to your question about what happens during inference when such clips do not exist, our ablation study results presented in Table 3 indicate that the speaker similarity suffers. Without this prompt, the first layer codec will produce a neutral voice, and the NAR model will struggle to convert it to the target voice.

We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. The detailed responses are listed below.

R1. About improving writing and clarity (Weakness 1)

We will reorganize the paper, include ablations in the main section, revise the unclear writing and conduct a thorough proofread. These revisions will appear in the updated version.

R2. About few ablation study (Weakness 2)

We have presented the ablation studies, such as w/ and w/o acoustic embedding, the choice of different codec, w and w/o text BPE and Layer Bean Search in NAR acoustic model. Due to the limited space for the main content, we had to present the ablation studies in the appendix. We will also add an ablation study related to isochrony preservation in the revised version. In this study, we compared our model with the following three baselines. We will report the ASR-BLEU, SLC_p (Speech Length Compliant, as defined in the paper), and Overlap ratio (i.e., speech overlap between the reference and the hypothesis) as follows.

where

1. No Isochrony control (No IC).

2. Isochrony control on the decoder (Dec IC). This involves adding the Isochrony embedding to the input of the encoder as another positional embedding. We implemented the method from ref [1] in our system.

3. Isochrony control on the decoder with future pause information (Dec IC + FPI). This is an improvement over above 2. In addition to the distance to the global end and VAD information, two extra pieces of information are encoded: the distance to the next pause and the number of pauses in the future. We implemented the method from ref [2] in our system.

Ref: [1] Y. Wu, et al. “VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing,” AAAI, 2023.

Ref: [2] P. Pal, et al. “Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters”, Interspeech, 2023.

Please let us know if more ablation studies should be added.

R3. About evaluation on more language-pairs (Weakness 3)

We will validate the method with an additional language pair in future work. Our model is built on a multi-lingual Seamless model, and we believe the proposed methods can be extended to other language pairs, including those between English and non-European languages. The main hurdle to this kind of extension investigation is the availability of publicly accessible data.

R4. About the discussion on isochronic translation (Weakness 4)

We will add discussions regarding isochronic translation as follows in the revised version.

The conventional method for isochronic translation involves first translating as usual and then adjusting the speech rate to match the length of the source speech. This approach ensures that the translation quality is not compromised by isochronic control. However, for long videos with multiple utterances, an inconsistent speaking rate can significantly affect the naturalness of the translated speech.

We aim to use isochrony control to translate optimally by considering both the timing boxes and the speech rate in real application scenarios. Both should align with the source speech. In our proposed method, the generation of both text and speech is conditioned on global isochrony information. Our experimental results also show that this approach can improve ASR-BLEU score compare to Isochrony control on decoder, meaning the model is more confident and accurate in the generation and make less error like repetition and truncation.

Finally, we would like to express our gratitude once again for your time and effort in reviewing our paper. Considering the interesting ideas, SOTA performance, adequate ablation studies, and improved presentation of our paper, we would greatly appreciate it if you could consider increasing your score.

Dear Reviewer ujH4,

Thank you for your suggestions. We appreciate you are considering raising your score. However, due to the review rule, we cannot send you a revised pdf on the system. We also fully understand your reluctance. Let’s try our best to include the revision within this 5,000-character comment box. Otherwise, we will have to ask AC how we can send you an updated version anonymously.

With an additional page allowed in the main content of the final version, we will include the following on that page.

Ablation Studies

1. Acoustic Embedding

We compared the inference with and without acoustic embedding and presented the results in Table 3. Without the acoustic embedding, speaker similarity scores decreased by 0.040 in French-English (Fr-En) translations and by 0.032 in English-French (En-Fr) translations. Additionally, there was a slight decline in the AutoPCP scores.

Tabel 3. Ablation on the acoustic embedding in the joint translation model.

2. Choice of Codec

We compared training TransVIP using different codecs: SpeechTokenizer [9] and our SASC. In this study, the joint translation model was trained with a subset containing only CVSS-T Fr-En uni-direction data. For the NAR acoustic model, SASC uses 16 codec layers, while SpeechTokenizer uses 8 layers, as it only has an 8-layer version. With both codec we have trained a full system(AR+NAR) with CVSS-T data only. The results are presented in Table 4. Compared to SpeechTokenizer, the model trained with SASC exhibits superior performance in all aspects. Most notably, the speaker similarity improved by 0.04, from 0.226 to 0.264, aligning with the improvement in codec re-synthesis results.

Tabel 4. Ablation on the Choice of Codec

3. NAR Acoustic Model

We conducted two comparisons. Firstly, we compared the performance of a NAR acoustic model with and without text input, i.e., using BPE as input. Secondly, we assessed the inference results with and without the utilization of the Layer Beam Search (LBS) algorithm to determine its impact on performance enhancement. The results are presented in Table 5, where it indicates that the textless model consistently outperforms the model with text input across all metrics of ASR-BLEU, speaker similarity, and naturalness. Moreover, employing LBS yields superior results compared to greedy decoding.

Tabel 5. Ablation on the BPE and Layer Beam Search

4. Isochrony Control

We compared our proposed Isochrony control method with and without using and other strategies, and presented the results in Table 6, where it demonstrates that our approach achieves the best performance in terms of BLEU score and isochrony evaluation metrics.

Tabel 6. Ablation on the isochrony control strategy

where

a. No Isochrony control (No IC).

b. Isochrony control on the decoder (Dec IC). This involves adding the Isochrony embedding to the input of the encoder as another positional embedding.

c. Isochrony control on the decoder with future pause information (Dec IC + FPI). This is an improvement over (b). In addition to the distance to the global end and VAD information, two extra pieces of information are encoded: the distance to the next pause and the number of pauses in the future.

Furthermore, we have made several improvements to the paper. We employ a professional proofreading service to fix typos and improve the writing. We add a discussion on isochronic translation, as shown in our previous response. We also rewrite several paragraphs such as the acoustic encoder to make it easier to understand the design and to make the structure clearer

Please let us know if you have any further concerns.

best regards,

Authors

Dear Reviewer ujH4,

We have checked with AC, and there isn't a way to send an updated paper. Therefore, we can only use the official comment box for any updates. In fact, most of the ablation studies have already been included in the appendix of the submitted version (please refer to the page14-16). We have revised them and moved them to the main content in the updated version.

Further suggestions and concerns are welcome. Each time, we can leverage this 5,000-character comment box to address them. Thanks!

Best,

Authors

We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. The detailed responses are listed below.

R1. About empirical evaluation (Weakness 1)

We will validate the method with one more language pair in future work. Additionally, to compare to cascaded solutions, we will add the results of cascaded ST + TTS solutions as follows,

where

1. ST is Seamless Expressive Speech-to-Text translation model

2. We evaluated cascaded system by using TTS model from StyleTTS (open sourced) and VALLE-X (implemented) and reported the better one in terms of objective measurements.

R2. About presentation (Weakness 2)

We will reorganize the paper, include ablations in the main section, and conduct a thorough proofread. These revisions will appear in the updated version.

R3. About prosody preservation(question1)

Our framework has not been explicitly designed to preserve prosody. Therefore, prosody is not purposefully maintained but is preserved alongside the voice feature. We have kept the metric in the paper to provide a comprehensive comparison with Seamless. Recently, we have observed an increase in the use of explicit prosody modules in zero-shot TTS. We may consider adding one in future work.

We sincerely appreciate your efforts in reviewing our paper and providing us with valuable, constructive feedback. The detailed responses are listed below.

R1. About not fully representative title (Weakness 1)

Thank you for acknowledging the numerous innovations presented in our paper. We will refine the title or add a subtitle to encompass as many aspects as possible.

R2. About limited experimental comparisons (Weakness 2)

We are always eager to compare our work with related research to verify the effectiveness of our proposed methods. However, most works, such as PolyVoice and MSLM-S2ST, neither have open-sourced models/codes nor matched language pairs for investigation. As we know, S2ST is a complicated system, and reproducing others' entire systems is both challenging and unaffordable. Moreover, we need to point out that Seamless does consider voice information. According to its technical report (https://arxiv.org/pdf/2312.05187), they explicitly encode the speaker and prosody information into the speech generation process. Additionally, the Seamless team leveraged far more data for training than we did. Therefore, we already leveraged a very strong baseline to compare with our model.

We will also add following results from cascaded system (ST+TTS) as a comparison to our model.

where

1. ST is Seamless Speech-to-Text translation model

2. We evaluated cascaded system by using TTS model from StyleTTS (open sourced) and VALLE-X (implemented) and reported the better one in terms of objective measurement.

R3. About few ablation studies and in-depth analysis (Weakness 3).

We have presented the ablation studies, such as w/ and w/o acoustic embedding, the choice of different codec, w and w/o text BPE and Layer Bean Search in NAR acoustic model. Due to the limited space for the main content, we had to present the ablation studies in the appendix. We will also include an ablation study related to isochrony preservation in the revised version. In this study, we compared our model with the following three baselines. We will report the ASR-BLEU, SLC_p (Speech Length Compliant, as defined in the paper), and Overlap ratio (i.e., speech overlap between the reference and the hypothesis) as follows.

where

1. No Isochrony control (No IC).

2. Isochrony control on the decoder (Dec IC). This involves adding the Isochrony embedding to the input of the encoder as another positional embedding. We implemented the method from ref [1] in our system.

3. Isochrony control on the decoder with future pause information (Dec IC + FPI). This is an improvement over above 2. In addition to the distance to the global end and VAD information, two extra pieces of information are encoded: the distance to the next pause and the number of pauses in the future. We implemented the method from ref [2] in our system.

Ref: [1] Y. Wu, et al. “VideoDubber: Machine Translation with Speech-Aware Length Control for Video Dubbing,” AAAI, 2023.

Ref: [2] P. Pal, et al. “Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters”, Interspeech, 2023.

Please let us know if more ablation studies should be added.

R4. About unclear writing (Question 1)

We will revise the paper as clear as possible. For example, we have refined the description as follows to address how the acoustic encoder learned from scratch can be expected to extract the desired acoustic information.

Many previous works have adopted a reversed gradient approach to remove semantic information from acoustic features. However, these approaches require an additional decoder and training objective, which increases the training burden. In our work, we use the information bottleneck to train the acoustic extractor. The sum pooling serves as the information bottleneck, preventing too much information, especially semantic information, from passing through. Additionally, we design the system to use part of the target speech as input and predict the other part, making it less likely for the acoustic encoder to learn anything semantically meaningful. This approach integrates seamlessly into the original training process, with no extra modules or loss required.

Finally, we would like to express our gratitude once again for your time and effort in reviewing our paper. Considering the multiple innovations, adequate ablation studies, added comparisons and improved presentation of our paper, we would greatly appreciate it if you could consider increasing your score.

Dear Reviewer WU4H

We hope we have addressed your questions. Please let us know if you have any further concerns, as the discussion between the reviewers and authors will end soon. Thanks!

Best regards,

Authors

Dear reviewer WU4H,

We appreciate your efforts in increasing the rating for our paper. Your suggestions and comments are all valid, and we will address them in either the final version or future work due to the short rebuttal period. Additionally, further suggestions and concerns are welcome until the end of the reviewer and author discussion period.

Thanks!

Authors