Aligner-Encoders: Self-Attention Transformers Can Be Self-Transducers

Thank you for the close review of our work.

Thank you for the suggestion to revise the section on the long-form inference modification. In earlier drafts of the paper, this has been the most difficult part to write clearly. Space permitting, a figure or perhaps a pseudo-algorithm would be helpful, we will continue to revise to try to improve this section.

Good idea to highlight the computational efficiency / latency results in a table.

It is true we make no claims toward streaming capability. It is possible that our model is capable of performing streaming recognition like RNN-T models trained for this purpose (although perhaps not with as low of latency). Or it might require adaptations similar to what has been done to make Whisper streaming-capable, for example: https://aclanthology.org/2023.ijcnlp-demo.3.pdf. Since our model decodes non-streaming with much lower latency than AED (owing to our small decoder), it is likely that such adaptations could produce a much lower latency in our model. It would definitely be worthwhile to investigate streaming capabilities of models based on our Aligner Encoder. Since this potential limitation is not only for our model, but also applies to AED, and it could require significant further study and reporting, we hope that it does not diminish too much the significance/relevance of our submission, where we have included many other detailed analyses and ablations.

Yes, we have cited https://arxiv.org/pdf/2307.14132 in our manuscript: [34].

Another reviewer also asked about why AED could be worse than RNN-T on LibriSpeech. We believe this reversal happened with the introduction of the Conformer, as that paper shows RNN-T achieving better performance than transformer-AED. In researching this question we also discovered a reference that reports a conformer-AED on LibriSpeech (https://arxiv.org/abs/2210.00077), which is still not quite as good as RNN-T (so the SOTA we are comparing against remains the same). It is better than our conformer-AED result, but it also used more learnable parameters. Still, we should include it as a point of comparison and double-check all the settings. We welcome any other suggested references or explanations.

Good question about RoPE versus relative position encoding. In the RNN-T baselines we ran, the training run with relative positional embedding produced very slightly better results than with RoPE (e.g. 2.1 versus 2.2, 4.6 versus 4.7), so we reported the relative positional embedding results. However relative positional embedding is significantly slower to train than RoPE, so for the remaining LibriSpeech models (including ours) we used RoPE.

Thank you for considering our work closely, your summary is accurate to what we intended.

It is a worthwhile question whether pre-training can be combined with our model. As it stands, our model seems to use several of the same layers in the conformer for 1) encoding and 2) alignment, whereas existing pre-training methods will only train encoding. It would be interesting for example to take an existing good encoder (e.g. from RNN-T) and see if it can be fine-tuned for a small number of steps to learn the alignment, rather than needing to start from scratch. We may attempt this experiment, thank you for the suggestion.

Thank you for your careful consideration of our work.

Important question about the requirement for U <= T, especially for machine-translation. One possible solution to extend to U>T would be to pad the input with some (fixed) number, P, of learnable input frames, which would provide the model the ability to write T+P output tokens. (This is similar to the "registers" for vision transformers: https://arxiv.org/abs/2309.16588.) Another possibility, for U >> T, would be to train the model to decode two (or more) tokens per embedding frame. We did not need these changes for any of our ASR experiments, but would like to include them as suggestions in a revised discussion section.

We only tried Conformer because it tends to perform better and was already in use for many pre-existing baselines, so we could expect the encoder quality to degrade without the convolutions (even if the alignment is still possible to perform solely with the transformer).

In Figure 2, we plotted attention weights for a single head. The patterns do look different for different heads in the early layers (this also happens in RNN-T), but we found them all to express the alignment as seen in Layers 14 & 15. So in Figure 3 we were able to average across all heads when showing the alignment in Layer 15, which is probably more accurate than relying on a single head.

Unfortunately we do not have expertise or infrastructure for MT or AST---we welcome any correspondence from future readers over future work!

Regarding sections 4.5.3 and 4.5.4 figures, we placed images for these ablations into the appendix so they could be printed as large as possible. On re-reading, we did not adequately describe the figure from 4.5.4 for the text to serve as a standalone result--will revise!

On hallucinations, after looking at many, many examples, we did not observe this issue in any of our models, and did not see abnormally high insertion errors. In other work, we have observed hallucinations when using a larger language model in AED-style, so it seems this is a characteristic of LLMs. It's possible that we have some small number of tokens hallucinated during silence only, which RNN-T models may also do.

It is unfortunate that we are unable to release some of the datasets and our actual code--our hope is that the result on LibriSpeech (public) is sufficient for reproduction, paired with the fact that our method is only a simplification over previous models. We are happy to correspond with anyone re-implementing in a public code repository.

Thank you for reviewing our work closely.

The convergence rates are similar among CTC, RNN-T, and our model (good checkpoints are between 100k-150k training steps on LibriSpeech). Interestingly, AED models sometimes did converge faster (as fast as 25k training steps for the best checkpoint). We did not investigate this further other than to run AED again with a lower learning rate in case that was too high, but we weren't able to find a better result. So in this regard our model is more similar to RNN-T.

In Table 3 Dev is Dev-Clean only, we will edit the column label.

This is an interesting question about AED versus RNN-T. A possible explanation is the use of the conformer encoder. "A Comparison of Sequence-to-Sequence Models for Speech Recognition" uses attention architectures that predate transformers. "On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition" finds an improvement using transformer-AED over RNN-AED, but it only reports RNN-T in its Table 1. Our results more closely match those of the Transformer-AED and Conformer-Transducer given in the Conformer paper https://arxiv.org/abs/2005.08100 (Table 2) which also experiments on LibriSpeech. In searching around for this question, we found a paper we could cite which gives a score for Conformer-AED on LibriSpeech is: https://arxiv.org/abs/2210.00077, which reports test-clean 2.16% and test-other 4.74%. This is better than our AED results (with conformer), although they use more learnable parameters: 148M versus ours at 128M. It is still less good than our RNN-T (conformer), so it would not change the SOTA against which we are comparing. Still, it seems worth citing and including in the comparison, and investigating their other settings.

This is a very interesting question about running our model with limited recurrence in the decoder. It is true that RNN-T models can often perform well using a history of only the 2 most recent tokens. Time permitting we will attempt to launch this experiment, thank you for the suggestion.

Unfortunately our implementation cannot be disentangled from a code base which cannot be shared. We hope that the fact that our model is a simplification over previous models, and uses more standard deep learning components, will make it relatively easier to reproduce the results, and we have attempted to include extensive hyperparameters.