Dear Reviewer,

Thank you for your insightful comments and constructive feedback on our work. We have carefully addressed each of your concerns as follows:

Addressing Weaknesses

1. Latency Evaluation: We have incorporated numerical evaluation results on the CoVoST2 testset using LAAL, as the latency metric, presented in Table 1. The results confirm our original conclusion that SimulMEGA continues to marginally outperform baseline methods. In the final version, all quality-latency curves will be systematically re-plotted using the LAAL metric for comprehensive comparison.
2. Baselines and presentation clarification:
   • While AlignATT did not function effectively with our base model (due to persistent attention fixation on particular frames leading to over-generation), we successfully integrated EDATT (Papi et al., 2023), which shares conceptual similarities. As shown in Table 1, EDATT outperforms both DIG-SST and wait-K baselines, further validating our approach's competitive advantage.
   • Publicly available implementations of StreamSpeech and NAST-S2X differ significantly from SimulMEGA in three critical dimensions: model architecture scale, training data volume, and multilingual capability scope. To provide meaningful comparison while acknowledging these differences, we present results on the CVSS FR→EN benchmark in Table 2, where SimulMEGA achieves more than 10 BLEU points higher than these public implementations. We maintain AL (Average Lagging) as the latency metric to ensure comparability with the original publications. These comparative results will be included in the appendix of the final version.
   • Regarding the "Seamless" baseline: this refers to the integrated system described in Section 6 of the Seamless documentation, which combines Seamless Expressive with Seamless Streaming components. We will expand our baseline descriptions in the appendix to prevent any potential misinterpretation.
3. Real-world streaming processing: Our supplementary materials include a deployed web demonstration that processes streaming input using a VAD (Voice Activity Detection) system to segment audio into optimal 10-20 second chunks for model processing. While standardized long-form evaluation benchmarks remain scarce in the field, we constructed a specialized test set following Whisper's methodology, comprising 17 TED-LIUM talks with an average duration of nine minutes. We translated the reference transcripts to Chinese using ByteDance's Doubao translation service for evaluation. The S2TT performance metrics are presented in Table 3.
4. Reproducibility on standard pretrained ST model: Unlike attention-based policies (EDATT and AlignATT) or divergence-based approaches (DIG-SST), SimulMEGA introduces two features: (1) it fine-tunes the base model without freezing parameters, and (2) it learns the read/write policy without relying on empirical patterns. Our training process demonstrates stable convergence in both loss values and routing score distributions. Theoretical analysis suggests minimal impact from the chunk streaming mask design, which is empirically validated by consistent performance across different mask configurations. Furthermore, SimulMEGA's successful adaptation to streaming TTS demonstrates the method's robustness across both architectural variants and task domains.

Responding to Questions

1. Evaluation framework: While we did not utilize SimulEval as our primary evaluation framework, we implemented a rigorously controlled evaluation pipeline where all models undergo identical processing conditions. Our latency metrics directly implement the same computational definitions as SimulEval to ensure measurement consistency with established SimulST literature.
2. Effect of streaming mask: Although comprehensive ablation studies on architectural variations were not our primary focus, we conducted targeted experiments evaluating different mask configurations. Our findings indicate that the 640ms chunk mask configuration yields performance equivalent to processing without chunk masking. However, we observed measurable performance degradation with the 320ms configuration, suggesting an optimal balance between temporal resolution and contextual completeness at the 640ms threshold.

We appreciate your thorough review and welcome any additional questions you may have regarding these clarifications. All revisions outlined above will be incorporated into the final manuscript to enhance both methodological transparency and experimental rigor.

Table 1: CoVoST2 Evaluation

Table 2: Evaluation on CVSS FR-EN

Table 3: Long-form S2TT result on TED-LIUM

Thank you for your prompt and thoughtful response. We appreciate the opportunity to clarify a few points where we believe there may be some misunderstanding.

in the original EDAtt paper, we have results spanning $1-1.5s$

Actually, in the original EDAtt paper, results are reported in 1 - 2.5s spanning of AL. At 1.5s AL , the method suffers a degradation of approximately 5 BLEU points (around 25%) compared to full-context performance. In contrast, in our experiments, EDAtt incurs only a 3 BLEU drop (10%) at 1.8s LAAL , indicating that our implementation of EDAtt serves as a stronger baseline than the original. This suggests that our overall framework achieves better latency-accuracy trade-offs.

Attention "fixation" can be easily solved by a frame-wise normalization.

We respectfully disagree with the claim that attention fixation "can be easily solved by frame-wise normalization." Our implementation of EDAtt strictly follows the official codebase from FBK-FairSeq , where frame-wise normalization is already applied. However, neither the AlignAtt paper nor its public code includes such normalization. Moreover, it is unclear how normalization—applied before an argmax operation—would fundamentally resolve fixation issues, as the argmax is invariant to scale. We would welcome further clarification on this point.

while direct approaches to long-form (without the need for an additional segmentation step) also exist

While it is true that direct long-form streaming methods exist, their practical applicability is limited. For instance, StreamAtt introduces up to 2 seconds of computational latency , which exceeds typical real-time requirements. InfiniSST, though innovative, still relies on crafted alignment data and exhibits performance degradation. In contrast, our VAD-based segmentation enables processing of up to 30-second segments with stable performance in practice. We fully acknowledge that automatic segmentation of unbounded speech and preservation of history context information remains challenging, and we have noted this as a direction for future work. And we will add the current reliance on VAD to the limitation.

it is important to notice that EDAtt and AlignAtt are policies that do not require any retraining as they just rely on the cross-attention scores of the original model, while the proposed method requires specific architectural choices and specific fine-tuning.

We emphasize that our method does not require specific architecture . Its successful application to a decoder-only architecture (e.g., Whisper-based models) demonstrates its architectural flexibility. Regarding fine-tuning, we only use the same data as in the original pre-training—no additional labeled data is introduced. The results show that our approach significantly outperforms EDAtt, particularly in low-latency regimes. Thus, the choice boils down to whether one prefers to improve streaming performance through lightweight fine-tuning (as we propose) or accept higher latency and quality degradation with attention based policies.

the claim of minimal modification does not hold, as the architecture chosen for the experiments`

Finally, the base model used in our experiments was adapted from Whisper, which originally formed part of a "wait-K" streaming system. The modifications we made were aimed at balancing efficiency and performance, not introducing architectural novelty. There is no evidence that our method is restricted to non-standard architectures. In fact, we are among the few works to validate our approach across two fundamentally different architectures (encoder-decoder and decoder-only). For your concern, we have conducted a quick experiment with full-parameter finetuning of Qwen-omni encoder + Qwen_3_0.6B on en-zh pair. The result is shown below. SimulMEGA(whipser) is the model in the paper and SimulMEGA(SpeechLLM) is the model with Qwen architecture.

1. As you request we have extend the regime to around 1 s LAAL. However, for practical applications, we still do not recommend using this specific regime.
2. We also successfully implement AlignATT after adding the frame-wise normalization. Thank you for suggestion on the implementation. Experiment shows that EDATT and AlignATT is similar in performance. And SimulMEGA shows even more advantage in low latency regime.
3. "2 seconds of computational latency" comes from Table 2 of StreamAtt paper. We get the extra computation latency by (LAAL_CA - LAAL_NCA). Please clarify if this interpretation is incorrect.
4. We can slightly widen the 1.5 s LAAL boundary as translations between Mandarin and Latin languages are included in the evaluation.
5. Our commitment is to explicitly state the current reliance on VAD for real-world deployment as a limitation. All evaluation details will be covered and we have tested that our evaluation pipeline perfectly matches the result of SimulEval.
6. We have shown versatility of our method by training on streamTTS on top of CosyVoice2. We have also switched the architecture to QwenAudio-style SpeechLLM and shows it reaches better result. SimulMega success on each attempt and always converges to the best policy given the training dataset. We believe this hold for strong grounding that our method is not picky on starting base. We hope our effort invested in this not to be dismissed.

Evaluation result in CoVoST X2EN, BLEU score of base model is 37.04

The computational latency it's the latency computed considering the actual elapsed time during simultaneous translation (therefore including latency dependent on, for instance, hardware and an efficient codebase), while the ideal latency it's computed considering the latency introduced by the method itself, which is the latency that the authors are considering there. This follows the literature on simultaneous translation; see the SimulEval paper for further reference.

I don't think that I'm too defensive of attention-based methods (and I am not referring to attention-based methods in my response with standard architecture, rather to autoregressive models, as I already mentioned), but assuming that the underlying architecture of the model should already implement inference-efficient approaches (i.e., chunk-AR and NAR) is not realistic, as almost all of them are not based on that.

Lastly, I agree that the discussion is not constructive anymore with such behavior, as I don't get why pointing out a weakness in a work should be disrespectful, especially since my rating is still positive and I'm not penalizing it, but, conversely, I'm trying to point out possible problems and points of improvement.

We don't mean any disrespect. And we really appreciate your engagement in the conversation, which benefits us a lot. We have already addressed may weakness in our paper under your suggection.

And we are juist trying to prove that architectual choice should not hinder the utility of our method. We can direct fine-tune it from a standard model like Qwen2Audio. And we have given some prove. We realy hope we can reach an agreement on that so everyone can try our on their own base model of any kind. But we respect your judgement on that.

And we have taken your suggection and experiment on a standard QwenAudio architecture. That's all we can do by now. We have removed the potentially controversial words and sorry for your trouble.

Dear Reviewer

Thank you for your valuable feedback. We have carefully addressed each of your concerns as follows:

Addressing Weaknesses

1. Evaluation on specific language pair: Comprehensive numerical results for each language pair in the CoVoST2 testset are currently available in our supplementary materials. In the final version, we will expand these into a complete table presented in the appendix for easier reference. Additionally, the supplementary materials include detailed translation examples that illustrate linguistic phenomena affecting performance. For instance, we observe higher translation latency between English and Mandarin due to their inverted grammatical structures—a finding that aligns with established linguistic research on language directionality in translation systems.
2. TTS evaluation: Our current TTS implementation supports two languages, which explains the more focused evaluation scope. Nevertheless, our results demonstrate that SimulMEGA-TTS achieves lower streaming latency while maintaining the high-quality output standards of CosyVoice 2. We invite you to experience the practical performance through our audio samples and video demonstration in the supplementary materials, which showcase real-time translation capabilities across various speech patterns and speaking rates.
3. Benchmark Clarification: While CVSS [1] is derived from CoVoST2, it specifically focuses on speech-to-speech translation pairs. In the final version, we will revise our benchmark description to accurately reflect this distinction and better align with the S2ST task framework.

Responding to Conceptual Questions

1. MoE router weights learning: As the MoE mechanism, the router will automatically select the best expert for a given input hidden state. The router weights are learning through gradient decent -- weights for the optimal expert will increase during training as the gradient suggests. Under this mechanism, we turn the MoE router into a read/write decision maker:
   • Training Setup: For a given source sentence, the model only accesses the first half during training.
   • Expert Roles:
     • A prefix expert analyzes the visible partial input ($h^{prefix}$).
     • A global expert infers high-level context from the full sentence ($H^{global}$)
   • Policy Learning: The MoE router learns to select between experts when predicting each target token (e.g., preferring the prefix expert for early tokens and the global expert for later tokens). Crucially, no explicit policy labels are used—the router implicitly learns when input is sufficient for generation.
   • At inference (streaming mode), the global expert is unavailable (since future input is unknown). If the router "attempts" to use the global expert (i.e., assigns it high weight), this signals insufficient input—trigerring a wait decision. Thus, the MoE mechanism naturally emerges as a read/write policy without supervision.
2. Difference to CTC-based policy or attention-based policy:

• CTC based policy [2] These require learning CTC force alignments between source speech and target translations. In multilingual settings, this becomes problematic as the system must establish alignments across multiple target languages—a challenge with no demonstrated successful implementations in multilingual SimulST to date as far as we know.
• Attention-based policy ( [3]) These leverage attention patterns from offline models for streaming decisions. While innovative, they typically require using the offline model without streaming-specific fine-tuning, potentially compromising performance in real-time scenarios.

[1] Jia, Yeting et al. “CVSS Corpus and Massively Multilingual Speech-to-Speech Translation.” International Conference on Language Resources and Evaluation 2022.

[2] S. Zhang, Q. Fang, S. Guo, Z. Ma, M. Zhang, and Y. Feng, “StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning,” 2024.

[3] S. Papi, M. Negri, and M. Turchi, “Attention as a Guide for Simultaneous Speech Translation,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, 2023, pp. 13 340–13 356.

Dear Reviewer,

Thank you for your thoughtful feedback regarding the accessibility of our work for readers unfamiliar with simultaneous translation. We acknowledge that space constraints limited our ability to include extensive background in the main text, and we sincerely appreciate your engagement with this challenge. Below, we clarify the core concepts to address your concerns.

The fundamental challenge in streaming (simultaneous) translation is the read/write decision problem: as the system processes speech in continuous chunks, it must dynamically decide whether to generate output tokens or wait for future input. Suboptimal decisions lead to excessive latency (waiting too long) or over-generation (outputting incorrect tokens due to incomplete input).

While a robust read/write policy is essential, supervised training data for such policies is scarce, and existing data may not reflect optimal decisions. We thus propose an unsupervised approach where the model learns the policy intrinsically. Our solution leverages a Mixture-of-Experts (MoE) gating mechanism to enable this learning:

1. Training Setup: For a given source sentence, the model only accesses the first half during training.
2. Expert Roles:
   • A prefix expert analyzes the visible partial input ($h^{prefix}$).
   • A global expert infers high-level context from the full sentence ($H^{global}$)
3. Policy Learning: The MoE router learns to select between experts when predicting each target token (e.g., preferring the prefix expert for early tokens and the global expert for later tokens). Crucially, no explicit policy labels are used—the router implicitly learns when input is sufficient for generation.

At inference (streaming mode), the global expert is unavailable (since future input is unknown). If the router "attempts" to use the global expert (i.e., assigns it high weight), this signals insufficient input—trigerring a wait decision. Thus, the MoE mechanism naturally emerges as a read/write policy without supervision.

Regarding your query about "native" baselines: all comparative methods inherently implement some read/write policy (e.g., fixed latency rules or learned triggers). There is no "policy-free" baseline for simultaneous translation, as the task requires such decisions. We have added a new baseline per other reviewers’ suggestions to further strengthen comparisons.

For streaming TTS, the underlying mechanism is identical to simultaneous translation (as noted in Appendix X), differing only in modality-specific implementations. We kept this section concise to prioritize core contributions but are happy to expand it if needed.

We believe all notations (e.g., $h^{prefix}$, $H^{global}$, $L^{prefix}$) are clearly defined in Figures 1–2 and Equations 4. Should any term remain unclear, we will gladly refine explanations in revision. Thank you for your time and valuable critique—we welcome the opportunity to improve the manuscript further.

Thank you for your response. I think the author solved my concerns. I will keep my score.

Dear Reviewer,

Thank you for your insightful comments and constructive feedback on our work. We have carefully addressed each of your concerns as follows:

Addressing Weaknesses

1. Latency Evaluation: We have incorporated numerical evaluation results on the CoVoST2 testset using LAAL (Average Lagging) as the latency metric, presented in Table 1. The results confirm our original conclusion that SimulMEGA continues to marginally outperform baseline methods. In the final version, all quality-latency curves will be systematically re-plotted using the LAAL metric for comprehensive comparison.

2. Baseline Comparison: While AlignATT did not function effectively with our base model (due to persistent attention fixation on particular frames leading to over-generation), we successfully integrated EDATT (Papi et al., 2023), which shares conceptual similarities. As shown in Table 1, EDATT outperforms both DIG-SST and wait-K baselines, further validating our approach's competitive advantage.

3. Threshold Documentation: The final version will include comprehensive tables in the appendix detailing numerical results alongside thorough explanations of threshold selection methodology. For your immediate reference, Table 1 presents our threshold selection rationale, with additional implementation details available in the supplementary materials.

4. Notation Correction: We sincerely appreciate your careful reading—the reference to $L^{\text{trunc}}$ in line 285 should indeed be $L^{\text{prefix}}$. This was an oversight in notation consistency, and we will correct it in the final version.

5. Real-world streaming processing: Our supplementary materials include a deployed web demonstration that processes streaming input using a VAD (Voice Activity Detection) system to segment audio into optimal 10-20 second chunks for model processing. While standardized long-form evaluation benchmarks remain scarce in the field, we constructed a specialized test set following Whisper's methodology, comprising 17 TED-LIUM talks with an average duration of nine minutes. We translated the reference transcripts to Chinese using ByteDance's Doubao translation service for evaluation. The S2TT performance metrics are presented in Table 2.

Clarifying Conceptual Questions

1. Architecture Design: Our decision to apply chunk masking selectively rather than across all layers stems from preliminary experiments showing that universal application degraded translation performance. While we acknowledge the value of ablation studies, training additional base models on this data scale would be computationally prohibitive. We note that architectural choices serve as implementation details rather than core contributions of this work.

2. Routing Score Normalization: The normalization mechanism is designed to align routing scores proportionally with available input information. Since information content is intractable, we approximate it using input duration. For instance, when processing half a sentence, the average routing score should approach 0.5 (with ideal scores transitioning from 0 in the first half to 1 in the second half). To account for boundary uncertainty, we slightly adjust the normalization target beyond the strict duration proportion.

3. Generation Process: Upon receiving each text chunk, we initiate a new autoregressive generation process incorporating all previous text chunks and generated speech tokens as context. Generation terminates when the routing score reaches the predetermined threshold. This approach maintains full compatibility with vLLM's CosyVoice implementation, achieving real-time factors (RTF) of 0.1 in deployment.

4. Pseudo-Labeling: We employed Google Translate for pseudo-label generation, collecting translation pairs between November 2024 and March 2025 to ensure temporal consistency in the translation service's behavior.

5. Router Score Interpretation: The notation $p^{router}$ represents the router probability score. For visual clarification, Figure 6 in the appendix (supplementary materials) illustrates the dynamic behavior of these scores.

We would like to highlight that our supplementary materials contain complete implementation code, comprehensive numerical results, extended appendices, and a video demonstration of our deployed system, which may provide additional context for evaluation.

We sincerely appreciate your thorough review and valuable suggestions. These improvements have significantly strengthened our manuscript, and we welcome any further feedback you might have regarding our responses.

Table 1: CoVoST2 Evaluation

Table 2: Long-form S2TT result on TED-LIUM

While standardized long-form evaluation benchmarks remain scarce in the field

There are two options you can consider.

1. One is ACL 60/60 dev set used in IWSLT 2025 Simul Track https://iwslt.org/2025/simultaneous
2. The other is RealSI https://github.com/byteresearchcla/RealSI

Table 2

The latency for long-form speech is too large to use in real life. A usable system should keep at least < 5 seconds latency. This is achieved by many prior methods [1, 2, 3].

[1] Siqi Ouyang, Xi Xu, and Lei Li. 2025. InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3032–3046, Vienna, Austria. Association for Computational Linguistics.

[2] Sara Papi, Marco Gaido, Matteo Negri, and Luisa Bentivogli. 2024. StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3692–3707, Bangkok, Thailand. Association for Computational Linguistics.

[3] Cheng S, Bao Y, Huang Z, Lu Y, Peng N, Xu L, Yu R, Cao R, Han T, Li Z, Liu S. Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice. arXiv preprint arXiv:2507.17527. 2025 Jul 23.

Thank you for your suggestion and meaningful discussion. We would like to clarify our latency evaluation methodology:

First, our AL and LAAL are computed over the entire long-form audio. Both AL and LAAL assume a uniform information distribution along the input, which rarely holds in long-form speech. As a result, these metrics can be misleading in practice.

Second, the results reported in [1] and [2] use the StreamLAAL metric, which computes LAAL on each pre-segmented audio clip. StreamLAAL is sensitive to the segmentation method, making fair comparisons between systems difficult. In our experiment, segments are produced automatically by a VAD (without known transcripts); therefore, we are unable to report StreamLAAL results.

As suggested by Papi et al. [4], long-form evaluation should rely on automatically segmented speech rather than human segmentation. For instance, Cheng et al. [3] state that “AL and FLAL measure translation latency at the segment level,” which implies use of the original human-segmented RealSI test set; strictly speaking, those are not long-form results. In [5], paragraph-level RealSI results likewise exhibit very large latency figures.

Regarding Table 4, Seamless outperforms our system primarily because it clips speech into shorter (~5 s) segments, whereas we use longer (20–30 s) segments to maximize context. This longer context leads to noticeably better translation quality in our system.

We fully acknowledge that establishing a standard, intuitive, and fair evaluation protocol for long-form translation remains an open challenge. Our long-form experiments were conducted under tight time constraints, and we included them here to demonstrate real-world applicability with VAD-based segmentation. The aforementioned works[1,2,3], though, are inherently more suitable for unbounded speech; they have their drawbacks (Either relying on human-crafted data or suffering high computation costs). In future work, we will pursue semantic-aware automatic segmentation and develop comprehensive, fair long-form evaluation metrics, which we will also add to our limitations section.

[1] Siqi Ouyang, Xi Xu, and Lei Li. 2025. InfiniSST: Simultaneous Translation of Unbounded Speech with Large Language Model. In Findings of ACL 2025.

[2] Sara Papi et al. 2024. StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection. In ACL 2024 (Long Papers).

[3] Shanbo Cheng et al. 2025. Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice. arXiv:2507.17527.

[4] Papi, S., Polak, P., Macháček, D., & Bojar, O. 2025. How “Real” is Your Real-Time Simultaneous Speech-to-Text Translation System? TACL, 13:281–313.

[5] Shanbo Cheng et al. 2024. Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent. arXiv:2407.21646.

StreamLAAL is sensitive to the segmentation method, making fair comparisons between systems difficult.

StreamLAAL is at least better than VAD + LAAL.

segments are produced automatically by a VAD (without known transcripts); therefore, we are unable to report StreamLAAL results.

You are testing on TED-LIUM, which has meta data detailing the start and end timestamp of each gold utterance, so you can actually calculate StreamLAAL.

Cheng et al. [3] state that “AL and FLAL measure translation latency at the segment level,” which implies use of the original human-segmented RealSI test set; strictly speaking, those are not long-form results.

I consulted the authors and they confirm they evaluate on unsegmented test set and segment long-form hypothesis into segments by human annotators and then do segment-level AL, so it is long-form result.

The following is StreamLAAL result on TED-LIUM with the provided segmentation. We also add another result on EN-FR pair. The previous result was just for showcasing the capability of our deployed online system. And we have acknowledged that it will be better to follow IWSLT standard for evaluation.