Overcoming Non-monotonicity in Transducer-based Streaming Generation

Thank you for your thoughtful review! We will make every effort to respond to your concerns.

1. However, the paper could be improved by providing more detailed theoretical analysis of why the proposed method works well, especially in comparison to other methods. Additionally, while the experimental results are convincing, more comprehensive ablation studies could strengthen the paper's claims.

Thank you for your suggestion. In the rebuttal period, we have designed an additional analysis to validate the rationale of our approach.

The core technical aspect of this paper is dynamically computing the expected contextualized representation of monotonic attention through inference of the posterior alignment during training. This posterior alignment requires a prior alignment for calculation. Naturally, there are two potential questions here: 1) Is posterior alignment absolutely necessary, or can the expected contextualized representation be computed directly using a prior alignment? 2) How does the choice of prior alignment affect the result of the posterior alignment, and is it robust?

For the first question, we attempted to directly calculate the expected contextualized representation using the prior probability, without inferring the posterior alignment. In Table 2, we present the results obtained by using a diagonal prior for this direct calculation. We found that this approach leads to a significant performance drop, particularly when the chunk size is small, where finer alignment is required. This addresses the first question. Here, we will focus on the second question: the robustness of the posterior alignment. We examine the impact of different prior choices: diagonal prior and uniform prior.

As shown, MonoAttn-Transducer's performance demonstrates robustness to the choice of prior alignment. We visualize the posterior alignment when using different priors in this annoymous URL. We have observed that, even with significant differences in the prior distribution, the posterior remains fairly robust. This nice property reinforces the robustness of using the inferred posterior to train monotonic attention.

2.Could the proposed method be applied to other types of streaming generation tasks beyond speech translation, such as real-time text summarization or dialogue generation?

Yes, the proposed method is a general framework that can be applied to many streaming generation task, as long as the target sequence consists of discrete tokens. We note a recent trend in research indicating that audio, images, and even video data can be tokenized into discrete sequences with minimal information loss [1,2]. Therefore, we believe this research has broad applicability to various real-time streaming tasks.

[1] SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models
[2] Finite Scalar Quantization: VQ-VAE Made Simple

3.How does the performance of MonoAttn-Transducer scale with larger datasets and more complex language pairs?

Thank you for your suggestion. We are currently working on scaling our approach with a larger dataset (GigaST) to further evaluate its performance. However, due to time constraints during the rebuttal period, the experiment has not been completed. We plan to include the results in the revised version.

4.What are the limitations of the current implementation, and what are the plans for future work to address these limitations?

Currently, we have implemented the forward-backward algorithm and posterior inference using Numba. However, we have observed that GPU usage during training occasionally is around 60%. We suspect that some inefficient operators in our implementation may be contributing to this. We believe that optimizing the implementation may benefit better scaling this approach.

Thank you for your acknowledgement of this work. We will make every effort to address your remaining concerns.

1. For Table 2., why would MonoAttn still out-perform Transducer with an infinite chunk size? I would anticipate that in such settings, alignment is not as important, as the context includes everything that may be relevant for genderating output?

Yes, when the chunk size is infinite, the posterior alignment would be that all predictor states align with the last encoder state with a probability of 1. This means that the context in the cross-attention would encompass the entire source. However, it is important to note that the naive Transducer does not include such an attention mechanism. Instead, its encoder and predictor are loosely coupled through a joiner (typically a simple MLP), which may limit its modeling capacity compared to our approach.

2.This paper investigates a less-studied problem of non-monotonic alignment, which is a true problem in real-world applications. For the same reason, the paper is limited in that its impact is limited to these specific areas of real world problems.

Yes, the scope of this paper is limited to streaming sequence generation scenarios, specifically where the target sequence consists of discrete tokens. Given that an increasing number of studies have pointed out that data from audio, images and video modalities can be tokenized into discrete sequences in a manner similar to text [1,2], we believe this research can be widely applied to various real-time streaming tasks.

[1] SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models
[2] Finite Scalar Quantization: VQ-VAE Made Simple

3.Would recommend adding a reference for forward/backward algorithm.

Thank you for your suggestion! We will fix it.

Thank you for your efforts in reviewing! We will make every effort to respond to your concerns.

1.However I would need some more information from the authors before I could make a good assessment.

We will address your questions about the method point by point.

1. It appears that the overall architecture of Mono-Attn-Transducer is very similar to TAED except that previous decoder states are not updated when new source features become available. This however is only my best guess.

   Yes, in the inference of Mono-Attn-Transducer, the generated predictor states are not updated when receiving new source features. When the predictor encodes the $u$-th target state, it depends on previous predictor states and the currently available source:

   $s_{u} = f_{\theta}(s_{0:u-1},h_{1:g(u)},y_{u-1})$

2. $c_u$ is the approximate average cross attention output based on my interpretation of Equation (8). However $c_u$ depends on $s_u$, the predictor state which in turn depends on the actual alignment path $g(\cdot)$. It is not clear to me which alignment path is used to produce $s_u$.

   In fact, the core idea of the approach lies in the uncertainty of the alignment path used to produce $s_u$ in training. Therefore, a posterior approximation is required to guide the learning process. To analyze this issue, we can further examine Eq. 4 in detail. In fact, the predictor producing $s_u$ primarily relies on two steps:

   A. Self-attention with earlier predictor states ($s_{0:u-1}$): This is deterministic.

   B. Cross-attention with encoder states ($h_{1:g(u)}$): Since the specific value of $g(u)$ is unknown in training, we instead estimate the posterior alignment (Eq. 7) and use the resulting alignment probability to approximate the expected context representation $c_u$ in the cross-attention (Eq. 8).

3. Further, it is not clear to me how $c_u$ is used. My guess is that it's passed to the joiner to join with each $h_t$ to produce the Transducer sum of alignment path probabilities.

   As mentioned earlier, $c_u$ represents the expected context representation (in other words, $c_u$ is the output of the cross-attention module). This $c_u$ is then passed to the next module in the predictor, typically an FFN. We refer to the final output of the predictor as $s_u$ (Eq. 4), which is subsequently passed to the joiner.

2.Key details are missing that prevent the readers from fully understanding or reproducing Mono-Attn-Transducer.

We hope the previous clarifications help better understanding. We also provide the source code in this anonymous URL to ensure reproducibility.

3.Concerns on Baseline Results

1. Clarification of CAAT results.

   We used the code provided by CAAT for replication but did not obtain the results reported in their paper. Since CAAT did not provide the distillation data used to train their model, we trained with the same data we used for training the Transducer, which could be the source of the discrepancy. The official CAAT config (audio_cat) does use fewer params; however, we argue that this is a compromise made due to its $O(T)$ memory usage in training.

2. Clarification of S2S results.

   In fact, Zhao et. al specifically augmented the speech synthesis module, while the Transducer remains unchanged. They use an acoustic LM to refine semantic tokens generated by Transducer and then generate the waveform. However, their design is not directly related to the Transducer itself, but rather to improving waveform generation. Our focus is on the Transducer, and we opt for a standard approach [1]: directly converting generated semantic tokens to waveforms using a vocoder.

   [1] Speech Resynthesis from Discrete Disentangled Self-Supervised Representations

4.Comments on definition of $\alpha(t,u)$ and $\beta(t,u)$.

We noticed you argued that $\alpha(t,u)$ is the sum of probabilities of partial alignments where $y_u$ being produced at $x_t$ is the last alignment label.

However, this is not correct. $\alpha(t,u)$ does incorporate the probabilities where $y_u$ is followed by zero or more blanks. $\alpha(t,u)$ is precisely defined as the probability of generating $y_{1:u}$ from $x_{1:t}$ in the original Transducer paper (see Section 2.4 of the paper). Similarly, $\beta(t,u)$ is the probability of outputting $y_{u+1,U}$ from $x_{t:T}$. There is no need for $y_u$ being produced at $x_t$.

5.Suggestions on Equation (4) and Section 3.2.1.

Thank you! We will fix it.

6.Comments on Equivalence between monotonic attention and Transducer.

We quite agree with this statement. Our approach actually leverages the DP algorithm of the Transducer to assist in the computation of monotonic attention. We will include Variani et al. (2020) in the discussion of the paper. Thank you for the reminder.

Thank you for your thoughtful review! We will make every effort to respond to your concerns.

1. Concerns regarding latency metric that are biased toward over-generation: The AL in Table 3 is abnormal, both 118 and 153 ms are abnormally low, which I find hard to believe. One possible cause is over-generation. The average lagging (AL) evaluation metric is problematic, since AL is not reliable if the over-generation happens. It is better to use length adaptive average lagging (LAAL).

Thank you for your suggestion. Actually, we present the S2T results using LAAL in Table 6, App. C. Apologies for the confusion. For clarity, we have included the results from Table 6 here and added the S2S results using LAAL and Start Offset.

L: EN-ES R: EN-DE

FR-EN S2S

We have realized that LAAL is a more robust metric for assessing latency (especially in S2S), and we will replace the metrics in the main body of the paper with LAAL for clearer presentation.

2.One concern is equation (11). It is possible to obtain a better prior by either leveraging the confidence of pretrained MT models or word alignment methods. Another concern is that the expected contextual representations still leads to training-inference mismatch.

Leveraging the confidence of pretrained MT models or word alignment methods for prior design is a possible approach. However, it will further complicate the training pipeline. In practice, we find that the posterior alignment is relatively robust to the choice of prior. Specifically, we examine the impact of different prior choices: diagonal prior and uniform prior.

As shown, MonoAttn-Transducer's performance demonstrates robustness to the choice of prior alignment. We visualize the posterior alignment when using different priors in this annoymous URL. We have observed that, even with significant differences in the prior distribution, the posterior remains fairly robust. This nice property relieves concerns about the imperfect design of the prior.

3.The author misses one important baseline AlignAtt [1], which is far better than EDAtt compared in Figure 2.

Thank you for the reminder. I have checked AlignAtt's paper and found that it performs particularly well at medium-to-high level latency. We will include it in Fig. 2 to facilitate a comparison between different streaming generation methods.

4.Can you elaborate on why MonoAttn-Transducer does not improve over Transducer at chunk size of 320ms?

On one hand, when the chunk size is 320ms, the Transducer's prediction window is very small, which limits its flexibility in managing reordering. On the other hand, we found that the small improvement observed with a 320ms chunk size is mainly reflected when using BLEU as a metric. However, results on COMET show that MonoAttn-transducer still demonstrates significant improvement.

5.Could you please also report speech offset for simultaneous speech-to-speech translation?

Thank you for your suggestion. We have incorporated the results into the table above.