We thank the reviewer for their constructive feedback.

Updates in the revised version of the paper

We first inform the reviewer that we will update the reported results of Hibiki-M in Table 2 after fixing an issue that further improves performance. Following the reviewers' suggestions and thanks to the extra page provided for the camera ready, we will revise the paper on the following aspects:

• Clarify the model's architecture, configurations and the nature/size of the datasets used.
• Add quality/latency trade-off curves by varying hyperparameters of our contextual alignment method.
• Add COMET evaluation scores.
• Extend experiments to the English->French direction.
• Improve the references section and discuss similar works pointed out by the reviewers.

Comments

On the generalization of the contextual alignment method to many language pairs

We acknowledge that our method is only illustrated by a single language direction in the paper. However, there is no specific aspect of our method with respect to the language pair, but the fact that MADLAD -- which we used to derive contextual alignment -- performs well on these languages. We expect our method to work as strongly on pairs of languages where SOTA text translation models perform well and can thus allow us to derive reliable contextual alignments. Given the massively multilingual nature of MADLAD or even more recent systems like GemmaX2-28-9B (Cui et al., 2025), we expect this approach to be a good candidate for scaling to many language pairs. We provide the reviewers examples of contextual alignments with other languages. As a first step towards more language directions, we have extended our experiments to the English->French direction and provide experimental results that we will add to the revised paper.

On the quality/latency trade-off

As mentioned in Section 3.2.2 (l.183), we enforce a 2s delay between words associated through contextual alignment as we found it to provide a good balance between latency and translation quality. We acknowledge that this choice can be reconsidered and that trade-off curves would provide a clearer picture to the reader. We thus produced a trade-off curve by varying the delay, as asked by the reviewers. The results reported in this quality/latency study show that Hibiki provides an overall better trade-off than Seamless. We will add this figure to the revised version of the paper.

On the alignment-aware TTS

We acknowledge that a non-negligible part of the technical details inherited from Défossez et al. (2024) were not exposed in our paper as we preferred to focus on the data creation pipelines and the experimental protocol and results. We will take the opportunity of the extra page provided for the camera ready to improve the clarity of technical details such as the alignment-aware TTS. Moreover, we would like to highlight that there is no missing Appendix to our paper as we referrerd to Appendix C of Défossez et al. (2024) at Section 3.2 l.203. We will improve the formulation in the updated version of the paper.

As a rapid summary of the explanations given in Appendix C of Défossez et al. (2024), one can force text tokens directly in the text stream of a TTS model derived from the Moshi architecture. Thanks to the contextual alignment, we can ensure that a given text token is fed at the right timestamp (that is not too early) using PAD tokens instead to delay its insertion.

On text-only pretraining

The text-only pretraining phase is that of standard next token prediction, there is no adaptation to a MT model in the text-only pretraining phase. In early development, we tried alternating batches of text translation and speech translation (starting from a pretrained text model), however while this model did perform quite well in text translation, this did not result in any measurable improvement of the speech translation. We hypothesize that this lack of transferrability is due to the fact that MT samples were built by concatenating the source and target texts in the text stream which is radically different from what is seen in the text stream for a speech translation sample (where the source is audio-only and the target is time-aligned). This highlights a challenge in aligning text and speech representations in speech-text LLMs for each modality to benefit the other, which we believe will be critical to extend speech translation to more language pairs as text translation data is much more accessible than speech translation data.

We thank the reviewer for their constructive feedback.

Updates in the revised version of the paper

We first inform the reviewer that we will update the reported results of Hibiki-M in Table 2 after fixing an issue that further improves performance. Following the reviewers' suggestions and thanks to the extra page provided for the camera ready, we will revise the paper on the following aspects:

• Clarify the model's architecture, configurations and the nature/size of the datasets used.
• Add quality/latency trade-off curves by varying hyperparameters of our contextual alignment method.
• Add COMET evaluation scores.
• Extend experiments to the English->French direction.
• Improve the references section and discuss similar works pointed out by the reviewers.

Comments

On the generalization of the contextual alignment method to many language pairs

We acknowledge that our method is only illustrated by a single language direction, however there is no specific aspect of our method with respect to the language pair but the fact that MADLAD -- which we used to derive contextual alignment -- performs well on these languages. We expect our method to work as strongly on pairs of languages where SOTA MT performs well and allows for reliable contextual alignments. Given the massively multilingual nature of MADLAD, it is a good candidate for scaling to many language pairs. We provide the reviewers examples of contextual alignments with other languages. As a first step towards more language directions, we have extended our experiments to the English->French direction and provide experimental results that we will add to the revised paper.

On the quality/latency trade-off and controllable latency

As mentioned in Section 3.2.2 (l.183), we enforce a 2s delay between words associated through contextual alignment. We acknowledge that this choice can be reconsidered and produced a trade-off curve by varying the delay, as asked by the reviewers. The results reported in this quality/latency study show that Hibiki provides a better trade-off than Seamless. We will add this figure to the revised version of the paper.

We also acknowledge that the proposed version of Hibiki does not allow for inference-time latency control. We could rely on conditional training, as we did for the speaker similarity, to simultaneously train the model on multiple latency levels making it possible to control the latency at inference by changing the conditioning. We will add this mention to the limitations section.

On references

We acknowledge the contributions made by Papi et al. (2023), Wang et al. (2024) and Yu et al. (2025). We also acknowledge the progress made in streaming and speech translation for complex language pairs such as English-Japanese as highlighted in Ahmad et al. (2024). We will add these references to the related work in the updated version of our paper.

On the usage of a single text translation model

We used a single model for translation and alignment as we expect this model to be the most appropriate to derive a reliable likelihood-based alignment. However we acknowledge in Section 4.6.2 that we may overfit MADLAD and diversifying the models used to generate and align data may improve the robustness of our system.

On neural evaluation of quality

We added COMET evaluations and executed the comet-compare script which gave the following system ranking: MADLAD-3B > Hibiki > Seamless with a p-value < 0.05.

On statistical significance of human evaluation

Indeed we may get more robust estimation from more samples, however the gap between approaches (as demonstrated by error intervals) is wide enough such that we consider these results trustworthy. We also encourage the reviewer to listen to the example webpage.

Answer: What is the noise augmentation techniques used in Hibiki?

We use samples from freesound.org that are randomly added with various intensities to the input audio during training. We will add details about it in the revised version of the paper and the associated code will be released with the training code.

Answer: Is Hibiki able to generalize to unbounded speech?

The updated version of Hibiki that we will release is trained with windowed attention to extrapolate beyond a few minutes.

Answer: Is Hibiki still better than StreamSpeech using only CVSS training data?

We acknowledge the fact that we never trained Hibiki on CVSS data only as we aimed to handle real-world use cases with longer and diverse speech inputs, while CVSS only contains single sentences of a few seconds.

We thank the reviewer for their constructive feedback.

Updates in the revised version of the paper

We first inform the reviewer that we will update the reported results of Hibiki-M in Table 2 after fixing an issue that further improves performance. Following the reviewers' suggestions and thanks to the extra page provided for the camera ready, we will revise the paper on the following aspects:

• Clarify the model's architecture, configurations and the nature/size of the datasets used.
• Add quality/latency trade-off curves by varying hyperparameters of our contextual alignment method.
• Add COMET evaluation scores.
• Extend experiments to the English->French direction.
• Improve the references section and discuss similar works pointed out by the reviewers.

Comments

On the generalization of the contextual alignment method to many language pairs

We acknowledge that our method is only illustrated by a single language direction, however there is no specific aspect of our method with respect to the language pair but the fact that MADLAD -- which we used to derive contextual alignment -- performs well on these languages. We expect our method to work as strongly on pairs of languages where SOTA MT performs well and allows for reliable contextual alignments. Given the massively multilingual nature of MADLAD, it is a good candidate for scaling to many language pairs. We provide the reviewers examples of contextual alignments with other languages. As a first step towards more language directions, we have extended our experiments to the English->French direction and provide experimental results that we will add to the revised paper.

On the quality/latency trade-off and controllable latency

As mentioned in Section 3.2.2 (l.183), we enforce a 2s delay between words associated through contextual alignment. We acknowledge that this choice can be reconsidered and produced a trade-off curve by varying the delay, as asked by the reviewers. The results reported in this quality/latency study show that Hibiki provides a better trade-off than Seamless. We will add this figure to the revised version of the paper.

We also acknowledge that the proposed version of Hibiki does not allow for inference-time latency control. We could rely on conditional training, as we did for the speaker similarity, to simultaneously train the model on multiple latency levels making it possible to control the latency at inference by changing the conditioning. We will add this mention to the limitations section.

On incorporating real interpreter speech in training or fine-tuning

Real interpreter speech would indeed be an ideal source of data. However, given the scarcity of such data in terms of volume, number of speakers, covered languages, etc. we believe that developing pipelines for synthetic paired data generation is the best path towards scaling speech translation to more languages and conditions.

On speaker similarity and Classifier-Free Guidance (CFG)

Accent transfer is indeed a limitation that we mention in the comments on the ablation on classifier-free guidance in Section 4.6. We expect that both a high speaker similarity and a reduced accent can be achieved by labeling our data with a speaker identification system that is invariant to accent and more accurate in terms of identity.

While CFG offers fine-grained control on the strength of conditioning, it also doubles the computational cost at inference. Cideron et al. (2024) have proposed distilling the post-CFG logits into a student model. Since our submission, we have experimented this method to distill the logits with $\gamma = 3$ into a student model such that the latter can run without CFG. In our long-form evaluations, while Hibiki-M without CFG reaches a speaker similarity of 0.33, after CFG-distillation it reaches 0.38 (without CFG), close to the 0.39 obtained with CFG. This suggest we can use distillation to remove the need for CFG at inference.

On decoding with stochastic sampling

Stochastic sampling indeed induces variability in the output, some of which is desirable (e.g. acoustic diversity) while some is undesirable (unreliable translation). We thus use a lower top-k inference parameter on the text stream compared to the audio streams to disentangle acoustic and linguistic diversity, keeping the former high while lowering the latter.

On the alignment of text and speech tokens

As described in Section 3.4.4 of Défossez et al. (2024), special PAD and EPAD (End of PADding) tokens are inserted in the text stream to account for the difference between the constant framerate of the audio tokens and the variable rate of text tokens.

We thank the reviewer for their constructive feedback.

Updates in the revised version of the paper

We first inform the reviewer that we will update the reported results of Hibiki-M in Table 2 after fixing an issue that further improves performance. Following the reviewers' suggestions and thanks to the extra page provided for the camera ready, we will revise the paper on the following aspects:

• Clarify the model's architecture, configurations and the nature/size of the datasets used.
• Add quality/latency trade-off curves by varying hyperparameters of our contextual alignment method.
• Add COMET evaluation scores.
• Extend experiments to the English->French direction.
• Improve the references section and discuss similar works pointed out by the reviewers.

Comments

On the generalization of the contextual alignment method to many language pairs

We acknowledge that our method is only illustrated by a single language direction in the paper. However, there is no specific aspect of our method with respect to the language pair, but the fact that MADLAD -- which we used to derive contextual alignment -- performs well on these languages. We expect our method to work as strongly on pairs of languages where SOTA text translation models perform well and can thus allow us to derive reliable contextual alignments. Given the massively multilingual nature of MADLAD or even more recent systems like GemmaX2-28-9B (Cui et al., 2025), we expect this approach to be a good candidate for scaling to many language pairs. We provide the reviewers examples of contextual alignments with other languages. As a first step towards more language directions, we have extended our experiments to the English->French direction and provide experimental results that we will add to the revised paper.

On the quality/latency trade-off

As mentioned in Section 3.2.2 (l.183), we enforce a 2s delay between words associated through contextual alignment as we found it to provide a good balance between latency and translation quality. We acknowledge that this choice can be reconsidered and that trade-off curves would provide a clearer picture to the reader. We thus produced a trade-off curve by varying the delay, as asked by the reviewers. The results reported in this quality/latency study show that Hibiki provides an overall better trade-off than Seamless. We will add this figure to the revised version of the paper.

We also acknowledge that the proposed version of Hibiki does not allow for inference-time latency control. We could rely on conditional training, as we did for the speaker similarity, to simultaneously train the model on multiple latency levels making it possible to control the latency at inference by changing the conditioning. We will add this mention to the limitations section.

On the release of code and data

We acknowledge that some critical parts of our framework are particularly challenging to reproduce, in particular the synthetic data generation using contextual alignment and we will release our code for these steps along with training and inference code, trained models and around 900h of synthetic paired data with voice preservation corresponding to our speech translation fine-tuning dataset introduced in Section 4.2. In order to build the speech translation training dataset, we relied on various data sources and will release the portion that we are allowed to release by their license.

On references

We thank the reviewers for their suggestion and acknowledge the contributions made by Papi et al. (2023), Wang et al. (2024) and Yu et al. (2025) that are particularly relevant with respect to our work. We also acknowledge the progress made in streaming and speech translation for complex language pairs such as English-Japanese as highlighted in Ahmad et al. (2024). We will add these references to the related work in the updated version of our paper.

Answer to: Section 3: "X is padded" - What happens when X is longer than Y ? Is Y padded in that case?

At this level of explanation (l.106), we also assume that the modeling of Y knowing X should be causal. This implies that Y is longer than X.