Align With Purpose: Optimize Desired Properties in CTC Models with a General Plug-and-Play Framework

We thank the reviewer for the valuable feedback, which assisted us clarify some aspects of the work and update the text accordingly. We value the acknowledgement of the strengths of our work, in particularly its thorough comparison to previous work and its generality. In the following, we did our best to address every concern or question raised by the reviewer.

Strengthening table 2 by adding more experiments with an online Conformer

Table 2 presents an extensive research, to allow the different methods to work under the same settings as AWP, and we thank the reviewer for acknowledging it as a strength of the paper. The wide array of experiments conducted with a Conformer weren’t fully matured by the ICLR deadline. However, we enhanced table 2 by adding an online Conformer with AWP and updated the text accordingly. We kindly refer the reviewer for the ‘common response’ part for the details.

While the latency reduction part presents extensive experimental results, there are fewer results on minimum WER training, and their gain is marginal.

The evidence for the successful application of AWP to the low latency application is stronger and wider. Yet, although more modest, the WER part presents 2 model architectures, and another method for optimizing mWER was added to table 3 for additional reference (see the ‘common response’ part). The gain in this part is moderate compared to prior work. Graves & Jaitly, 2014 reported a relative improvement ranging from 3.7% to 9% on the Wall Street Journal dataset. Prabhavalkar et al,. 2018 reported an improvement ranging between 4.1% to 7.4%, and [1] reported a 5% improvement, both on non-public datasets. We believe that although the gains for the latency application are more prominent than the WER application, the relative improvement in the WER application demonstrates the versatility and the generality of AWP.

Can results be improved by increasing the number of shifted frames (tokens), or by applying AWP together with Trim-Tail, for example?

As demonstrated in table 2, AWP achieves better results than the other methods both in latency and in accuracy, including reaching a negative DL. Increasing the number of shifted frames yielded a similar trend, but our empirical findings suggest that gradual changes were favored. We added appendix C to discuss the variations of the property function, we kindly refer the reviewer to the ‘common response’. Exploring the combination of AWP with other methods like Trim-Tail can be very promising. However, it can be expected that the trade off between latency and accuracy may still play an important role, as decreasing the latency even further is expected to increase the WER.

WSR is figured 6 and 7

WSR is the word success rate, 100-WER. We modified figure 6 to present WER, and changed the caption of figure 7.

How many alignments (N) do you sample for each step? How does AWP affect the overall training time/resource usage?

We thank the reviewer for the question, we added this information to appendix A. Under the low latency setting, we set N=5 and for the WER setting we set N=10, based on empirical observations. As mentioned, sampling occurs every training step, and training with AWP is, empirically, 2-3 times slower than training a baseline model.

[1] M. Shannon, “Optimizing expected word error rate via sampling for speech recognition”

We thank the reviewer for the feedback, for noting that our method is clear and effective, and for finding our paper to be well written.

It seems that start epoch is a sensitive parameter, which is different in different models. So how is start epoch defined? By grid search?

Empirically we found that applying AWP works best after CTC training loss reaches 80-90% accuracy, which may vary across different settings (i.e. models, datasets, etc). We elaborate more on the start epoch parameter in the 'common response' part, we kindly refer the reviewer to this part.

We thank the reviewer for the valuable feedback. We did our best to address the concerns raised by the reviewer.

Absence of comparison with prior works

For the latency application, we’ve conducted an extensive comparison to prior works. To ensure a fair comparison, we implemented or adjusted public code to fit our setting. Additionally, we enhanced the text with an additional mWER optimization for comparison, we kindly refer the reviewer to the ‘common response’ for details.

complexity of hyper-parameters

This work introduces three additional hyperparameters: $\lambda$ and $\alpha$, standard in the context of hinge loss and additional loss term, and start-epoch. Our rationale for the latter is detailed in the 'common response'.

Missing details on the implementation. Why WER and not CER? Is language model (LM) used? We optimize WER rather than CER as WER is the primary metric for evaluating ASR systems. Furthermore, CTC is more closely related to CER than WER, given its indifference to whether errors are localized within a single word or distributed across multiple words (see fig 2.b in the text). In response to the reviewer’s comment, we added the code for WER optimization to the supplementary material. Concerning the LM, our implementation uses an LM in the beam search decoding (we kindly refer the review to appendix A1).

Sampling vs. top-N alignments

Empirically, we found no evidence in favour of top-N alignments. We elaborated on this issue in the ‘common response’, we kindly refer the reviewer to this section.

Comparing models in table 2

Each of the top 2 sections in table 2, along with the Conformer section, exhibit comparability. It presents an offline model, an online model and an online model + AWP, and shows clear trend: the offline model is superior in terms of WER but also has the highest TL, the online model improves the TL on the expense of WER, and the online + AWP improves the TL even further in the cost of a WER. The third section presents these 3 models, alongside additional losses or input manipulation. We can see that AWP performs better in both DL and WER.

Results in table 3 are within std on Librispeech. Report results on Greedy Decoding.

Although not reported, we repeated all experiments with several seeds. Based on this, we found that our results are exceeding 1 std. We didn’t want to overload the paper with results, therefore we reported after beam search decoding, which is not unusual in the area of mWER optimization.

I found it overall hard to formulate the proper reweighing between different alignments, rather than simple way of controlling the latency by restricting context or optimising directly mWER. Reported results are marginal.

We will endeavor to respond to the best of our understanding. AWP only requires a definition of a property function, which given an alignment, yields an improved alignment w.r.t to the property. Thus, it does not explicitly mandate the user to formulate a reweighting scheme. As for the advantages of AWP over alternatives - We discuss in the paper that context restriction allows control of the latency, but it also results in a drift (DL) in the online model. With AWP we were able to reduce this drift at the expense of WER. In table 2 we show that AWP can have a negative drift. Overall, we present compelling evidence that AWP can be successfully applied to low latency applications across, and yield moderate gains in WER optimization.

CTC posteriors

We agree, text was updated accordingly.

Shifting several tokens

We kindly refer the reviewer to the ‘general response’ part, where we elaborated on variations of the property function.

35K hours curated from LibriVox

English only.

LM usage

LM is used for decoding. We provide details in Appendix A1.

Conformer Performance

We use a different decoder and a different LM than the ones used in the Conformer paper. This could explain the difference. We kindly refer the reviewer to the ‘common response’ for details on the Conformer.

WSR

WSR is the word success rate. Figure 6 now shows WER.

multilingual w2v

We aimed for the base size model, and as far as we know, this model was trained on the largest training set for such a model.

Gumbel softmax (GS)

We show that sampling and GS yields similar results. Since softmax is applied to the CTC matrix, the derivatives of the hinge loss affects all the entries, and not just the entries correspond to the sampled alignment, which can explain the effectiveness of the sampling method.

The relation between Tables 2 and 4

Both present other applications to mitigate the DL. To compare table 4 to AWP, we see that AWP achieves the best working point in terms of both DL and WER. Other approaches may yield better WER or latency, but not simultaneously both.

32ms window, stride of 16ms

It is optimized for STFT that works in multiples of 2.

We thank the reviewer for the valuable feedback, which assisted us clarify some aspects of the work and update the text accordingly. We are particularly thankful for the recognition of the strengths of our work. In the following, we did our best to address the comments and the concerns raised by the reviewer.

Comparison to discriminative sequence training (DST)

We thank the reviewer for the comment. The focus of this work is CTC, and thus we focused on comparing AWP to other optimizations that are suitable for CTC. As noted by reviewer 4pZf, we conducted an extensive comparison in the latency application. For the WER application, to the best of our knowledge, there is a limited body of work on DST with CTC. Graves & Jaitly, 2014 is one example, yet, it required extensive optimization. Thus, we adapted another approach by Prabhavalkar et al,. 2018 to work in a CTC setting. With respect to the reviewer’s suggestion for the property function, we believe it will be an interesting direction to explore and see if it’s closer to other DST baselines. We kindly refer the reviewer to the ‘common response’ for more details.

N-Best beam search (BS) hypotheses Vs. Sampling

We thank the reviewer for this comment. We have conducted experiments with both sampling and n-best BS hypotheses, which didn’t indicate that n-best BS is superior to the sampling method. Moreover, running a BS decoder at every training step requires more resources and training time. We kindly refer the reviewer to the ‘common response’ for elaborated explanation and a speculation as to why n-best BS method is not superior in this case.

Latency WER optimization compete with each other

WER and Latency were two property examples for AWP. We optimized each property separately in order to compare with previous results. Yet, the simplicity of AWP allows us to combine the losses together, or to use a single property function that improves both properties simultaneously, and test how the two properties compete with each other. We added this comment to the discussion in section 5.

Is the optimization sensitive to a specific choice of the property function?

We experimented with multiple implementations of the property functions, and eventually reported the simplest (and evidently the most effective) implementation. Empirically, we found that other implementations worked as well. We added appendix C in the paper to shed more light on that matter. We kindly refer the reviewer to the ‘common response’ for more details on the different variations of the property function.

An Alternative approach for the latency application which samples ground truth (GT) alignments and compares them to an alignment obtained by a force aligner

We presume the objective is to derive a forced alignment from an offline model. Otherwise, drawing it from the concurrently trained online model introduces uncertainty regarding its time accuracy and latency compared to the random GT alignment. While one might expect the online model's latency to match the offline model, negative values in table 2 indicate our property function implementation encourages the online model to drift beyond the offline. Additionally, our implementation diverges from prior works like Tian et al., 2022 by extending beyond perfect alignments. The choice not to limit the property function to perfect alignments aligns with the notion that streaming ASR systems should emit fast even if the transcript is inaccurate.

AWP with an offline Conformer

We appreciate the reviewer's feedback. The Conformer experiments were not fully matured by the ICLR deadline. We chose to present the offline model with AWP as a proof of concept, assuming success with an offline model implies potential success with an online model. To adapt the offline model to an online setting, at inference we restricted its context (both right and left) so it works in a streaming manner. We updated table 2 with the online Conformer, and show that although it wasn’t trained as an online model, but inferred as one, the performance of the Conformer with AWP shows a significant DL reduction and only a mild reduction in WER.

Start Epoch as a tunable hyperparameter for balancing between latency and quality. How transferable is the optimal start epoch to different settings?

Figure 6 right indicates that tuning alpha is a better practice for balancing between latency and accuracy. However, figure 6 left shows that tuning start-epoch allows some balancing, as there’s a ~11% WER deterioration if applied after 0.9 epochs instead of 16.7. Empirically we found that applying AWP works best after CTC training loss reaches 80-90% accuracy, which may vary across different settings (i.e. dataset, etc). We kindly refer the reviewer to the ‘common response’ for more details.

WSR in figure 6

WSR is word success rate, 100-WER. We changed figure 6 to present WER.