HAINAN: Fast and Accurate Transducer for Hybrid-Autoregressive ASR

• Contribution and relation to prior works

Thanks for bringing our attention to those papers that explored combining a fast and a more accurate model for better modeling performance, and we’d be happy to include discussions in our paper regarding the difference between our and their work.

1. Reranking vs refining (Comparing with H. Inaguma et all)

A key difference is that those works are of the framework of using the faster model to generate multiple hypotheses, which are reranked by the more accurate model. Our work, however, uses the AR model to “refine” the NAR hypothesis directly.

This difference has a significant impact on the speed of models. For the “refinement” technique, the SAR rescoring steps add around 5% - 10% computation time overhead on top of the NAR models; however, with the “reranking” technique, since it involves generating multiple hypotheses, the approach adds significant computation overheads, e.g. in H. Inaguma et al., it is at least 2X slower than NAR CTC models according to Tables I and II.

Theoretically, our method is orthogonal to the reranking approach. In the future, we plan to investigate combining our SAR refinement and also the k-best hypotheses reranking to better improve models.

2. Other "semi-autoregressive" methods (S. Arora, et al.)

Although with similar names, our work is very different from S. Arora, et al. Their work proposes a structure where speech is processed in blocks, and within each block, they use NAR processing, but between different blocks, they utilize AR processing.

3. Fast-slow method in other network components (cascaded encoders by J. Mahadeokar et al.)

The work by J. Mahadeokar et al. is an application of fast inference and slow rescore methods done at the encoder level. They use a cascaded encoder design in the model, each with separate joint networks for prediction, and the whole model is jointly trained, balancing the loss computed from the outputs of two joint networks.

In their case, the slower model has extra parameters in its encoder and a separate joint network. Therefore, the slower model still adds considerable extra computation to the processing on top of the fast model.

In comparison, in our case, aside from the decoder network (with very small #params) that is unique to the slow model, all other parameters are shared among the fast and slow models, offering greater flexibility for our deployment model. Furthermore, with our SAR methods, our “slower” model only adds less than 10% computation time compared to the “fast” model, due to the parallelized nature of the added computation.

Our primary contribution is that we propose a single model that operates in both NAR and AR modes, and each mode achieves better or on-par accuracy with alternative methods while maintaining efficiency. This means adopters of CTC/RNNT/TDT models can switch to using our method safely without risk.

Additionally, our SAR method, in particular, is a fast-slow method that gets to invent new hypotheses directly, rather than the commonly used reranking method where the fast model rescores a set of hypotheses generated by the slower model. This has the benefit of adding minimal computational cost to the pipeline while consistently improving accuracy. We also hope this paradigm can inspire new works in the research field, allowing better speed/accuracy tradeoffs in speech/language-related models. Also, this method is orthogonal to existing reranking methods, so it’s possible to combine them to further improve results.

Thank you for the answers.

• Do the authors plan to upload the revised paper by the communication deadline (11/26)? I believe the authors can update the paper (I saw an updated version in other assignments), and I’d like to check the revised version if possible.
• To clarify, I suggested moving Viterbi decoding into the main methods section because the authors "propose" it in both the introduction and the experiment sections. If the authors believe it is just a method to better analyze the model, they could avoid using the term "propose" and just conduct an analysis with Viterbi decoding. That being said, I think it makes sense to "propose" it and write it in the main methods section. Finally, just to make it clear, I am not suggesting including the Viterbi decoding results in the main "table" because it is not an apples-to-apples comparison, as the authors mentioned.
• Regarding the reranking vs. refining discussion, could the authors clarify the difference compared to the deliberation model (e.g., Ke Hu’s work)? I'm still a little cautious about calling the "refining method" a new "paradigm" (as written in the Introduction), although I now have a better understanding of the differences from prior works.
  • Ke Hu, et al. Deliberation model based two-pass end-to-end speech recognition. ICASSP 2020.
• I cannot agree with the argument that "It is generally reported in the literature that beam search on Transducer-based models only brings marginal improvement." While it is okay to use beam=1 in your experiment for simplicity, the improvement from beam search is usually not "marginal," and I believe it should not be used as an excuse. (Again, I am fine with the setting itself, just in case.)

• comparison with Hu. et al

We thank the reviewer pghP for bringing our attention to the paper “Deliberation model based two-pass end-to-end speech recognition” by Hu, et al. This work is indeed an improvement upon the regular "reranking" method. It first uses the fast model to generate multiple hypotheses and then, and uses a much stronger 2nd pass model, that takes in both the acoustic input, as well as the set of hypotheses, to generate the final hypothesis. In particular, in this work, the 2nd pass (slow) model has its additional encoder layers for processing speech input, bi-direction LSTMs for processing hypotheses, and is trained using a different loss function (Listen-Attend-and-Spell model instead of the RNN-T used in the 1st-pass fast model). Because the 2nd pass model has access to the whole list of k-best hypotheses, I would imagine that this approach can "invent" new hypotheses by recombining elements from different individual hypotheses. So in that aspect, this work has similarities with our work. However, Hu et al.'s 2nd pass is significantly slower, since it requires additional encoder computation, additional decoder computation, and cross-attention computation between the encoder/decoder outputs.

• Regarding uploading a revision of the paper by the discussion deadline.

Yes, we are working on a revision of the paper and will upload a new version by 26th. Given the tight timing, we will not be able to include additional experiment results in the revision. We will try to focus more on improving the descriptions in the paper. Some of the things we will try to accomplish in this revision are,

1. including additional discussions with related works, as suggested by reviewers
2. including discussions on the terminology of NAR/AR decoding to make our description more rigorous
3. re-organize the structure of the paper and bring the Viterbi method from the analysis to the method section.
4. improving the algorithm block to make the interfaces/variables more clear.
5. improving the discussion in Sec. 6.3, by including the points suggested by reviewer pghP23, as well as further explaining our point of "the ability to skip frames ... " by referencing Sec. 6.1.
6. including a discussion on why 0 duration is suppressed by HAI-T model.
7. fix formatting issues, 404 URL links, notation explanations for [1-8], claim the AR-HAI-T is "on par" with TDT, etc, as suggested by reviewers.

While we're working on the revision, are there any particular points that reviewers would suggest that we prioritize addressing in this revision, given that we have limited time?

We thank reviewer jHSk for the thorough review and the positive feedback.

• Incremental innovation

We understand that the method we propose is simple, that only a one-line change on existing code is required for training such models. We believe the simplicity of the approach is a strength of the work, since the method can be easily implemented to take advantage of the more flexible speed/accuracy tradeoff brought by our model.

That being said, our SAR method is a novel innovation. Unlike alternative rescoring-based methods, which first use a fast model to generate a set of hypotheses, and then rerank them by the slower but more accurate model, our model directly uses the more accurate model to “correct” the output of the fast model, inventing new hypotheses in the process in parallel. This is a novel methodology that we believe can inspire new approaches to further the development of speech models.

• Notation of [1-8]

We follow the notation from [Xu et al. 2023] where [1-8] means the model supports duration predictions of 1, 2, 3, …, 7, 8. We will revise our paper to make this more clear.

• Comparison with joint-CTC-RNNT decoding

Can we ask the reviewer to clarify which research work the joint-CTC-RNNT decoding refers to, since it’s not a well-established term?

One work we know of that is related to that is the "4D-ASR" method by Sudo et al. We’d be happy to include a discussion of the relation to that work.

Or if the reviewer suggests a more generic way of joint-CTC-RNNT decoding, e.g. by first performing CTC decoding, and passing those non-blank emitting frames and tokens to the RNNT-decoder to generate the RNNT hypothesis. We feel this method might require some careful designing to work. Firstly, a jointly trained CTC-RNNT model doesn’t necessary make sure CTC and RNNT share the same alignment, and due to CTC allowing token repetitions in its text augmentation rules, there are different boundary cases we need to consider if we pass CTC outputs to RNNT decoding.

• Results for the HAI-T model without duration settings in Table 2

We assume the reviewer meant the HAI-T model with config without 0 duration for the German model? Sure, we can add those experiment results.

• Viterbi decoding

We purposefully put the Viterbi in the analysis section rather than the main results. The aim of including those results is to show that the HAI-T models can learn more diverse representations among different time indices. Using Viterbi decoding is intended as an empirical validation of this claim, which we see from consistent accuracy improvements. It’s not an apple-to-apple comparison if include Viterbi results with our main table, since the nature of the search algorithms is different.

• Add x and y labels in Figure 2

Thanks for noticing this issue. We will improve Figure 2 with clear and informative labels on axes.

• RTF and latency metrics

We agree that information of RTF and latency metrics will provide a clear picture of how HAI-T works compared to baselines. We will add those metrics.

• Break down of error types

This is an excellent suggestion. We will include both theoretical analysis of error patterns and also experimental results.

• Discussion on why HAI-T suppresses 0 duration

We will include a more detailed discussion on why duration 0 is suppressed. To explain it, let’s call encoder output E and predictor output D. The following key points are causing this phenomenon.

1. By performing stochastic masking of predictor output during training, the model is learning to make the output distribution of joint(E_t, D_u) similar to joint(E_t, D_u * 0).

2. If duration 0 is predicted from joint(E_t, D_u), then the next step of processing would be joint(E_t, D_{u+1}), where t remains the same but u gets incremented by 1.

3. From 1, we know that model is learning to make joint(E_t, D_u) similar to joint(E_t, D_u * 0); also it’s learning to make joint(E_t, D_{u+1}) similar to joint(E_t, D_{u+1} * 0). Consequently, the model would make joint(E_t, D_u) similar to joint(E_t, D_{u+1}), since joint(E_t, D_u * 0) is the same as joint(E_t, D_{u+1} * 0), both of which has predictor masked out.

4. However, we know that joint(E_t, D_u) and joint(E_t, D_{u+1}) should predict different tokens since the text histories are different. This causes a contradiction.

Therefore, the model would greatly suppress 0 duration prediction.

We thank reviewer n6nQ for the thorough and positive review of the paper.

• Incremental innovation

We understand that the method we propose is simple, and that only a one-line change on existing code is required for training such models. We believe the simplicity of the approach is a strength of the work, allowing the method to be easily implemented to take advantage of the more flexible speed/accuracy tradeoff brought by our model.

That being said, our SAR method is a novel innovation. Unlike alternative rescoring-based methods, which first use a fast model to generate a set of hypotheses, and then rerank them by the slower but more accurate model, our model directly uses the more accurate model to “correct” the output of the fast model, inventing new hypotheses in the process while running in parallel. This is a novel methodology that we believe can inspire new approaches to further the development of speech/language models.

• Lack of detailed training/inference specs

We apologize that the code link we provided in the paper links to a 404 error. It seems the official NeMo repository has been updated since our submission and the config file has been moved to https://github.com/NVIDIA/NeMo/blob/main/examples/asr/conf/fastconformer/fast-conformer_transducer_bpe.yaml. We will make sure to include the latest link in the revised version of the paper.

For inference time, we measure the wall time. We take the reviewer’s suggestion and plan to include more details, including machine/GPU specs for inference in our evaluations.

• On the use of initialization from existing checkpoints

We use initialization to speed up the experiments. At the beginning of the development, we also ran experiments training from scratch and noticed the same performance/speed patterns with those models, and the only difference is that many more training steps are required for models to converge.

This faster convergence is also seen when we initialize the German model with an English checkpoint. While those languages are different, it’s reasonable to assume at least in the early layers where the network mostly performs feature extraction, an English model checkpoint can still benefit German model training. In practice, the converge speed gain isn’t as great as English models, but still faster than from scratch.

• On the limitation of the model in noisy/hard scenarios

The reported results on librispeech test-clean and test-other touched on this aspect, as test-other is a noisier version of the librispeech dataset, which corresponds to the slightly degraded performance observed among all models. And from the numbers, there are no clear patterns in terms of which model is significantly more noise-robust than others, and the existence of noise impacts all models to similar degrees.

• On Viterbi decoding as an after-thought

Yes, we purposefully put the Viterbi in the analysis section rather than the main results. The aim of including those results is to show that the HAI-T models can learn more diverse representations among different time indices. Using Viterbi decoding is intended as an empirical validation of this claim, which we see from consistent accuracy improvements. It’s not an apple-to-apple comparison if include Viterbi results with our main table, since the nature of the search algorithms is different.

• Table 1 and 3 are too wide.

Thanks for noticing this. We will fix it.

• Notations in Algorithm box.

We will take the advice and improve the clarity of the algorithm.

We thank reviewer rxjr for the thorough review of our paper.

• On the terminology of NAR

We appreciate the reviewer rxjr's careful analysis of our NAR terminology. Our decoding process does maintain some sequential dependencies through duration-based frame selection, so strictly speaking, the decoding procedure is not NAR.

We would be happy to revise our writing and emphasize that when we call the model NAR, it specifically regards the model’s neural network computation. Note, that the AR/NAR categorization for models usually refers to the neural network computations but not the complete procedure. As a point of comparison, even CTC models - widely accepted as non-autoregressive (e.g many papers from https://scholar.google.com/scholar?q=CTC+non+autoregressive), have implicit sequential dependencies in their decoding (e.g., blank token handling and repeated token resolution would depend on prior predicted tokens), yet are considered NAR due to their parallel neural computations.

• On the limitation of SAR methods

We agree that there are limitations in terms of the errors SAR methods can fix. We had planned a section for that but was later removed due to space constraints. It’s quite interesting that despite limitations at the subword level, SAR can correct all three types of errors (deletion/substitution and insertion) at the word level.

1. From the subword level, the SAR methods can correct substitution errors and insertion errors (by replacing an original token with blank). It can't correct deletion errors, since as the reviewer correctly points out, the SAR output can't be longer than the original hypothesis.

2. From the word level though, SAR can actually correct all three types of errors, including deletion errors (where hypothesis is shorter than the reference). E.g. if the reference text is “black board”, and the initial hypothesis is “blackboard”. This results in a substitution error and a deletion error; it’s possible for SAR to replace the subword “board” with “_board”, thus reducing both the substitution error and deletion error.

Since this issue was brought up by reviewers, we’d be happy to include the discussion on this topic as well as stats on the distribution of those error corrections in our revision of the paper.

• On comparison with top-k + rescoring methods

We point out that SAR is orthogonal to methods like k-best rescoring. With k-best rescoring, it uses a stronger model to rerank and pick the best among existing hypotheses, and this process is usually quite computationally extensive; with our SAR approach, it can invent new hypotheses based on old ones. As we’ve shown in the paper, SAR can be done extremely fast.

It is possible to combine k-best restoring methods with SAR techniques, which we set as our future work.

• On the claim of HAI-T being on par with TDT

We thank the reviewer and take the suggestion to claim that the models are on-par in terms of performance.

• On the importance of initialization from existing checkpoints

We use initialization to speed up the experiments, and yes, all experiments reported in the paper including the baselines are trained with checkpoint initialization, so they're all fair comparisons. At the beginning of the development, we also ran experiments training from scratch and we noticed the same performance/speed patterns with those models, and the only difference is much more training steps are required for models to converge.

• On the implementation of the stateless predictor

We use the official stateless predictor implementation at https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/asr/parts/submodules/stateless_net.py. And yes, the stateless predictor stores a set of embedding tables per location; when it takes input, it does an embedding look-up, and outputs the concatenated embeddings as its output.

• On NAR VS NAR-Viterbi

There is no extra hyperparameter we use to combine the token and duration probabilities. They are simply added in the log space with 1:1 weight.

NAR results are shown in Table 1, and in Table 3, we didn’t repeat those results of NAR due to space constraints, since readers can still compare them across tables. If space permits, we’d be happy to duplicate rows in Table 3 for better comparison. That being said, we include the Viterbi results mainly to demonstrate that the HAI-T model can learn diverse representations dispersed among different time stamps. The reported improved accuracy of the Viterbi method is to prove that the representation learned by HAI-T models is indeed better. We understand that by including the Viterbi results, a comparison between that and the NAR/AR method is inevitable but those aren't really fair comparisons due to the different nature of those inference algorithms.

Thank you for the update.

• The authors still seem to be sticking with the statement of "present a novel paradigm" even after including many prior works in the related work section, which I am really hesitant to agree with. As I have repeatedly mentioned, the notions of slow-fast decoding, deliberation, and semi-autoregressive methods are all known concepts, and I don’t see the proposal as a novel "paradigm" nor a novel "approach," although I acknowledge it is a novel "method." I’d recommend the authors change "paradigm" to "method" in line 50. I also recommend fixing line 528 to avoid the wording of "approach," which also implies the proposal is more than a method.

• I didn’t recommend including the Viterbi result in Table 1. (As I mentioned in my prior comment, what I suggested was just moving the method explanation into the main method section.) However, given the suggestion was made by another reviewer, I am fine with keeping the current form.

• Given that the paper has been updated, I have raised the soundness and presentation scores as well as overall rating. I still rate the overall score as marginally below acceptance, given that I think the novelty is incremental and not all comments have been addressed. My rating may be too severe, and I’d like to follow the decision by a meta-reviewer in this regard.