T2V2: A Unified Non-Autoregressive Model for Speech Recognition and Synthesis via Multitask Learning

We thank the reviewer for the detailed feedback and constructive questions. We are encouraged by the recognition of the strengths of our method and clarity of writing.
Below we first list the revision plan based on the queries and suggestions of the reviewer, followed by detailed responses to each query.

---

Revision Plan
(Please check detailed responses for more information):

1. Update and simplify Fig 1.
2. Add inference runtime comparisons (Tables 4, 7, and 8).

NOTE: The changes are colored in blue text in the revised manuscript for easy visibility.

---

Weaknesses

Figure 1 is too complicated, with the arrows flowing in interlocking directions, so that readers may feel difficult to follow it.
Thank you for pointing this out, we will revise the figure to make it simpler to understand.

---

The efficacy of the speech synthesis process is largely dependent upon the performance of SoundStorm.
The performance of the T2V2 model depends partly on the SoundStorm model for content-to-acoustic token conversion. SoundStorm was chosen for its efficiency as an iterative NAR method, which aligns well with our framework. However, we acknowledge that alternative models, such as Grouped Masked Modeling (predicting grouped acoustic levels) [1] or PolySpeech detokenizer (using a VITS-like structure with adversarial loss) [2], could potentially offer further improvements. Exploring these alternatives is part of our future work.

Moreover, we have included inference runtime comparisons with the baselines (Tab. 4), especially comparing only the text-to-content tokens stage of T2V2 with that of WhisperSpeech, which employs AR modeling for this stage. We can see that our method is significantly faster.

Please also refer to the “Inference Runtimes” comment at the top.

---

The models being compared in the TTS experiment should have models that only predict content tokens + SoundStorm, which is the most similar setting to T2V2 + SoundStorm.
The models we compared in the TTS experiments include dual-stage methods like Hierspeech++ and WhisperSpeech, which also predict content features before acoustic features. While WhisperSpeech uses autoregressive modeling, it shares the dual-stage tokenized modeling paradigm with our approach. Additionally, it is trained on the same dataset (LibriLight), making the comparison relevant. The other baselines are the current state-of-the-art methods, thus demonstrating the competitiveness of our method.

---

In Tables 1 and 6, when comparing the performance of (with SMLM, without COR) and (without SMLM, without COR), it appears that SMLM harms both TTS and ASR tasks.
The slight degradation observed in single-pass inference with SMLM is marginal, as iterative refinement (not used in single-pass inference) fully utilizes SMLM’s benefits. During iterative inference, CFG (enabled by SMLM) significantly improves TTS performance, and the iterative CTC correction (enabled by the CORR task) compensates for the slight single-pass degradation. Thus, emphasizing that SMLM is a crucial component of our proposed framework.

---

The change in Corr. thresholds and iterations in Table 7 doesn't seem to make the CER change much.
While average CER changes are minimal, small character corrections can significantly affect WER, as correct words (coarser units) lead to greater overall improvements as seen in the improved WER.

---

Continued in the next comment ...

We thank the reviewer for the detailed feedback and constructive questions. We are encouraged by the recognition of the strengths of our work.
Below we first list the revision plan based on the queries and suggestions of the reviewer, followed by detailed responses to each query.

---

Revision Plan
(Please check detailed responses for more information):

1. Include references with other AR multi-task models [1] and [2].
2. Add inference runtime measurements and comparisons.
3. Update Figure 1 and unify the terminology.
4. Update Equation 16 to include the length prediction objective.
5. Include a discussion about the CTC correction task and the independence assumption.

NOTE: The changes are colored in blue text in the revised manuscript for easy visibility.

---

Weaknesses

Comparison with Other Multi-Task Models ([1] and [2])
Thank you for pointing out these references. [1] was posted on October 24, after the ICLR submission deadline, and we were not aware of [2] as it is a preprint without many citations. Both works, however, focus on autoregressive (AR) models, whereas our work explores a non-autoregressive (NAR) approach.

Additionally, we have already included citations for AR-based unified models in the manuscript, we will add these references as well for completeness.

---

Inference Efficiency Comparison
We acknowledge the need for inference time comparisons with AR models, and we did not originally include these since it is well-known that NAR models are faster than AR models for inference. However, we now realize that it is important to demonstrate the inference efficiency our approach explicitly, and will include inference runtime measurements to the revised manuscript.

Please also refer to the “Inference Runtimes” comment at the top.

---

Mixed Terminology
We agree that the use of mixed terminologies (e.g., "CTC token," "Text token," "Content token," "Acoustic token") in Figure 1 and Sections 2/3 could confuse readers. We will unify the terminologies as well as simplify Figure 1 for easier understanding.

---

Questions

Rationale for Length Prediction Using Text Embeddings
While we recognize the potential risk of confusing the model, our rationale is that text embeddings are simply another representation of the same underlying language, albeit at a different and irregular sampling frequency., but with similar linguistic patterns, which enables the model to generalize across tasks effectively.

The length prediction task, as shown in our experiments, does not hamper optimization, and the model achieves competitive performance across tasks.

---

Length Prediction Loss in Final Training Objective
You are correct that the length prediction loss was inadvertently omitted in Eq. 16. The length prediction objective is also added to the overall objective and optimized together with all other tasks. We will update the equation to include the length prediction objective.

---

CTC Error Correction and Independence Assumption
The initial pass of CTC follows the original independence assumption. However, in subsequent passes, the intermediate CTC predictions (including blanks) are masked based on confidence (log probabilities) and combined with speech tokens. The output tokens are predicted without reduction, allowing the model to access partial text contexts.

This iterative refinement enables the model to relax the independence assumption and utilize the interdependencies among text predictions in subsequent passes, which improves performance.

Furthermore, Unlike approaches like Mask-CTC[3] and Improved-Mask-CTC[4], which operate on reduced intermediate CTC outputs and explicitly handle substitutions, insertions, and deletions, our method operates on un-reduced outputs, allowing implicit handling of all error types.

---

Thank you for recognizing the contributions of our work and for your valuable feedback. We look forward to incorporating these improvements to strengthen the manuscript.

---

References

1. Toyin, Hawau Olamide, Hao Li, and Hanan Aldarmaki. "STTATTS: Unified Speech-To-Text And Text-To-Speech Model." arXiv preprint arXiv:2410.18607 (2024).
2. Yang, Runyan, et al. "PolySpeech: Exploring Unified Multitask Speech Models for Competitiveness with Single-task Models." arXiv preprint arXiv:2406.07801 (2024).
3. Higuchi, Yosuke, et al. "Mask CTC: Non-autoregressive end-to-end ASR with CTC and mask predict." arXiv preprint arXiv:2005.08700 (2020).
4. Higuchi, Yosuke, et al. "Improved Mask-CTC for non-autoregressive end-to-end ASR." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.

We thank the reviewer for the detailed feedback and constructive questions.
Below we first list the revision plan based on the queries and suggestions of the reviewer, followed by detailed responses to each query.

---

Revision Plan
(Please check detailed responses for more information):

1. Include details of how MAS is applied in our framework (Sec. 2.3.2).
2. Update and simplify Fig. 1.
3. Add inference runtime comparisons (Tables 4, 7, and 8).
4. Include a discussion positioning the novelty of our CTC correction formulation in comparison to Mask-CTC and Improved Mask-CTC (Sec. 2.4.1).
5. Include a discussion on the advantages of discrete ASR (Sec. 1).

NOTE: The changes are colored in blue text in the revised manuscript for easy visibility.

---

Weaknesses and Questions

Clarity

It is not clear how MAS is applied to the framework.
We acknowledge that the description of how we applied MAS to our framework is inadequate. We have updated the manuscript with a more detailed description of it, as well as updated the figure to reflect the MAS.

We consider the CTC log probabilities as the alignment score by gathering the log probability corresponding to each text token in the text sequence for each time step in the intermediate CTC output. This gives us the alignment score matrix, which is then passed to the original MAS algorithm to find the best monotonic alignment.

---

How does it compare to the external alignment?
While it is possible to use external alignments, the external alignments are computed from the features of a separate model, which might not exactly align with the discrete representations that we work on. Unifying the ASR and TTS tasks provides us with alignments that are more consistent with the input representation since both tasks share the same representation.

---

How it is compared to CTC alignment, which also provides speech-text alignments for non-autoregressive ASR?
Thank you for bringing the CASS-NAT [1] paper to our attention. Upon reading the paper, here are our observations:

Both the Viterbi-alignment used in CASS-NAT and our MAS with intermediate CTC log probabilities would result in obtaining similar alignments since even though the monotonicity is not directly enforced in the Viterbi alignment, the intermediate outputs are being trained with CTC loss which ensures monotonicity.

The key difference is in the use of these alignments. We utilize a straightforward method of upsampling the text tokens by repetition for TTS task input, while CASS-NAT uses it to effectively reduce the sequence length for the ASR decoder input.

While the purposes of using the alignments are different, we believe a discussion about the CASS-NAT’s application of alignment and our CTC correction task is interesting and warranted (even though we do not use alignments for the correction task). CASS-NAT’s application of the alignment can be seen as a means of length-adaptive soft-copy of encoder features for the decoder input (since, in their framework, the NAR decoder cannot be trained with teacher forcing) to further enhance the intermediate CTC outputs. However, since the decoder input is dependent on the predicted reduced sequence length, CASS-NAT requires special tricks like Error-based sampling (Sec. 2.6) during inference to give the decoder some flexibility. In contrast, our CTC correction mechanism has more room to maneuver since we perform the iterative masked correction on un-reduced intermediate CTC outputs, essentially enabling the handling of insertion and deletion errors implicitly in addition to substitution errors.

---

It is suggested to clarify the loss for each head in Figure 1.
Thank you for pointing this out, we will revise the figure to make it simpler to understand and all inputs and outputs clearly demarked.

---

The experiments of hyperparameter tuning can be presented in the appendix.
We acknowledge the suggestion to include hyperparameter tuning details in the appendix. However, we did not perform extensive tuning due to the significant computational costs involved. The main tuning efforts focused on the auxiliary tasks, and these results have already been presented in the ablation studies.

For other hyperparameters, such as learning rate and batch size, we relied on fairly standard values commonly used in the literature and did not conduct additional tuning experiments. We will clarify this in the revised manuscript.

---

Continued in the next comment ...

Thanks for providing the explanations and revision plan. I still have several concerns on the paper.

1. I am thinking about the benefits of unifying the ASR and TTS tasks in one framework. After reading the MAS explanations, I just realized that one of the roles of CTC loss here is to provide alignment for TTS as a replacement of external alignment. Hence, it looks like the main task here would still be the TTS task. From my experience, CTC alignment is not always accurate, and it should be worse than the external alignment, e.g. obtained from hybrid models. This is why I was asking a comparison to using external alignment. On the other hand, it is very hard to see if the TTS task can help the ASR task as there is no such ablation (ASR loss only) and probably would not. I would be very cautious about the motivation and the storytelling of unifying ASR and TTS in the framework, especially an emphasize on the competitive ASR performance.

2. There is many non-autoregressive ASR work based on refining CTC outputs, some of which have similar ideas, e.g.

• Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignment
• Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions especially for the second paper, concatenating CTC logits embedding with intermediate representations. These papers are not well referenced. This is also why I was mentioning CTC error correction is not that novel in the paper.

3. For comparison of latency, did you consider the speech discretization time for ASR and codec decoder time for TTS? If yes, please mention it or ignore if you already did it. If no, please add this time to make a fair comparison especially for ASR when comparing with non-discrete system in Table 9.

Overall, I can raise my score to 5 but I cannot give a high value to the paper. The paper can be a good candidate for other conferences but should be lower to the acceptance broadline for ICLR in my mind.

We thank the reviewer for the detailed feedback and constructive questions. We are encouraged by the recognition of the strengths of our work.

Below, we first list the revision plan based on the queries and suggestions of the reviewer, followed by detailed responses to each query.

---

Revision Plan
(Please check detailed responses for more information):

1. Simplify Figure 1.
2. Include references and discussion about multi-task learning challenges and optimization of losses.
3. Include references and discussion about the choice of Conformers with RoPE for ASR and TTS tasks.
4. Include clarification about training from scratch on the large subset for the scaling experiment.
5. If possible, include preliminary results of the discrete Zipformer-CTC baseline.

NOTE: The changes are colored in blue text in the revised manuscript for easy visibility.

---

Weaknesses

Contribution of Each Task to Overall Performance
Instead of positioning our work as “applying multi-task learning to improve individual tasks,” we position it as “the first work to unify the ASR and TTS tasks in a NAR framework by utilizing multi-task learning and without using external alignments.” Given this, the ASR and TTS tasks are integral to our framework.

While the ASR task could be trained on its own (basically our Conformer-CTC baseline), the TTS task cannot be trained without the aid of the ASR task, as the alignment is deeply linked to the ASR task. Regarding the auxiliary tasks, the speech MLM and CTC correction tasks aim to improve the performance of the respective TTS and ASR tasks. We performed ablation studies to analyze the effects of these tasks and the results are in Tables 1 and 6. These show that the auxiliary tasks help improve individual task performances.

---

Analysis of TTS Attributes (Pitch, Volume, Speed)
The Soundstorm model influences the pitch and volume of the synthesized speech, thus that is not the focus of our work. As such, analyzing these attributes would not directly reflect the advantages or disadvantages of T2V2. Regarding the speaking rate (speed), it relies on the token-wise length prediction task, once the lengths are predicted, it is possible to scale the speed by multiplying each length with a factor, to make it faster or slower, similar to how it is done in prior work like VITS [1].

---

Questions

Multitask Learning Challenges and Optimization of Losses
Multitask learning, particularly multi-objective optimization (MOO), is known to pose challenges, such as balancing performance across tasks, especially for conflicting objectives. Prior works, such as [2], detail these challenges and common approaches like Pareto optimality, genetic algorithms, and scalarization.

Our framework adopted the simplest weighted-sum scalarization approach, assigning equal weights to all task-specific losses. This choice was based on the assumption that the tasks are cooperative rather than conflicting, especially in our single-language scenario. The equal weighting reflects the equal importance of the tasks in our unified model and the experimental results demonstrate that this simple equally weighted scalarization helps achieve state-of-the-art performance.

---

Simplification of Figure 1
We acknowledge that the figure is a bit complicated and will update it in the revised manuscript.

---

Choice of Conformer with RoPE
The Conformer architecture is well-suited for speech processing tasks, capturing both local and global dependencies. Adding Rotary Positional Embeddings (RoPE) enhances its ability to handle variable-length sequences, benefiting both ASR and TTS. Conformers with RoPE have demonstrated effectiveness in ASR [3] and TTS [4].

---

Continued in the next comment ...