Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs

Thank you for your detailed review and time. Below we address the key concerns raised.

How is the weight of the teacher model in self-supervised pretraining initialized? Is it initialized randomly or with pretrained weight on another task?

The teacher model is randomly initialised in pre-training and improves throughout training via an exponential moving average operation through bootstrapping [44], as in related works [15, 17, 18]. We will emphasise this more in the paper.

Did the author make a comparison between the teacher-student self-supervised pretraining in the paper with masked-autoencoding training of audio and/or visual features? Is the proposed pretraining method superior?

We are not aware of works in audio-visual self-supervised learning for speech recognition that employ masked autoencoding (MAE) for pre-training, i.e., predicting raw audio and/or visual features as in [45]. Although the investigation of MAE in the context of audio-visual speech recognition is outside the scope of our work, we agree that it is an interesting direction for future research. Our pre-training task, which combines masked prediction (similarly to MAE) with teacher-student training, aligns more closely with recent successful models in this area, such as RAVEn [17], AV-data2vec [15], and BRAVEn [18].

Did the author investigate the effect of different masking ratios?

Initially, we conducted a set of preliminary experiments and chose to use the masking ratio from [18]. However, we agree that exploring different masking ratios is a valuable ablation study, and thank you for prompting us to investigate this further. We have now conducted experiments with different masking ratios for pre-training (see Table 2a in the rebuttal PDF). We find that the model performs well with a masking probability of 0.4-0.6 and is, interestingly, quite robust to the exact choice of masking ratio. We added this ablation in the Appendix.

Thank you for your detailed review and time. Below we address the key concerns raised.

Line 104 states a single FC layer on top of the encoder for vocabulary predictions, while line 107 states to use the decoder output sequence, which is subsequently not used as 1−λctc=0. So the decoder is not actually used during fine-tuning? How is inference actually done?

We apologise for the confusion; this is a typo. The intended value for the CTC loss weight was 0.1, not 1, and the decoder is indeed used during fine-tuning. Inference is then performed with both CTC and attention scores (see Appendix C.6). We have now fixed this typo in the paper.

I see no mention of a fixed random seed for running experiments, are all model initialized equal?

Thank you for raising this point. We indeed use a fixed random seed (42) for our experiments, and have now clarified this in the paper. Due to high computational demands and in line with previous studies [13-18, 20], we do not include error bars for our main results. However, we do show error bars for a subset of our main experiments in Table 13, where we observe that the results are consistently stable around the mean.

Minor editing comments

Thank you for the editing suggestions, which have now been incorporated into the paper. Specifically, we made the following changes: we placed the captions above all tables; we separated Table 2a and the rest of Table 2 into distinct figures and tables; we added a description of the datasets in the Appendix; and we fixed the typo you pointed.

In which settings/experiments is the transformer decoder used? What is the CTC attention in Table 2 (C)?

We apologise again for the confusion caused by the typo regarding the CTC loss weight, which should be 0.1. In Table 2d, CTC-attention (i.e., including the Transformer decoder) is our default loss. The CTC-only configuration, which corresponds to a CTC loss weight of 1, is used solely for the purposes of this ablation study.

In table 3 (A), is there a reason for not trying targets A + V + AV, as during fine-tuning?

Please note that during fine-tuning, we use only the AV targets (see Figure 1, top-left) because they provide the richest information and allow us to amortise the pseudo-labelling cost across the three modalities. Incorporating all three targets during fine-tuning is challenging, as it is unclear how to efficiently combine / predict the per-modality pseudo-label sequences, which are generated auto-regressively by the decoder. However, during pre-training, which involves only the encoder, we can more easily combine the targets from the three modalities. In Table 2b of the rebuttal PDF, we compare audio-visual targets (our default) with the sum of the per-modality targets. We observe that combining the per-modality targets results does not outperform predicting only the AV targets. Additionally, predicting all targets is more computationally expensive because the teacher encoder must process auditory, visual, and audio-visual features rather than just the audio-visual ones. We added this experiment in the Appendix.

You state in line 103 that features from the 3 modalities are concatenated along the batch dimension for efficient processing. However, Table 1 (B) shows that random sampling of modalities performs much worse, requiring 3x more epochs for similar performance. So it seems to me it's not only done for efficient processing, but also for effective optimization?

Randomly sampling the modalities means that, on average, the model is exposed to only one-third of the modalities at each iteration compared to concatenating all three modalities along the batch dimension. As a result, it would require approximately three times more epochs to achieve similar performance. However, an advantage of our approach is that it allows us to use the same pseudo-labels for all three modalities at each iteration, thus amortising the pseudo-label generation cost across modalities, which would not be possible with random modality sampling. We have updated the caption of Table 1b to explain why random sampling is trained for longer.

Also, do [13, 15] in line 179 share parameters for each task or not? According to Table 4 they do not, but if you use random sampling of modalities, how does this explain their relevance to Table 1 (B)?

[13, 15] do share parameters during pre-training, hence their relevance to Table 1b, but they separately fine-tune the resulting model on ASR, VSR, and AVSR, resulting in a separate model for each task during inference. In contrast, USR yields a single unified model that is capable of performing all three tasks during inference (section 2, “Single model for multiple modalities” includes a relevant discussion).

The methods requires all data to be audio-video paired. An interesting future direction could be the inclusion of audio-only data in the framework.

Indeed, thank you for this interesting suggestion, which we have now added to our Conclusion.

Thank you for your detailed review and time. Below we address the key concerns raised.

The complexity of training current SSL-based VSR or AVSR systems remains a challenge.

We recognise that VSR and AVSR systems present unique challenges compared to audio-only systems, and one of our future goals is to improve the computational efficiency of multi-modal speech recognition. Despite these challenges, we believe that audio-visual speech representation learning is an exciting and promising area of research. Leveraging visual modalities like lip movements offers significant advantages in disambiguating difficult utterances, particularly in noisy environments or when audio is unavailable (see Table 12, Appendix). VSR and AVSR can also improve communication for individuals who have difficulty producing voiced speech. Additionally, evidence suggests that incorporating multiple modalities during pre-training can enhance the performance of audio-only systems (e.g., see [13, 17]).

Performance is highly sensitive to certain configurations, such as the ratios of pseudo labels and the use of EMA.

While self-supervised learning systems can indeed be sensitive to hyperparameters, we believe that USR significantly reduces this sensitivity. Our semi-supervised pseudo-labelling framework is sensitive to extreme threshold values (see Table 2a) and somewhat sensitive in terms of VSR to the weighting of labelled and unlabelled losses (see Table 2b). In Section 4.2, we hypothesise that this sensitivity arises from the inherent trade-off between label quantity and quality, which must be balanced. However, as shown in Table 3, USR’s performance remains stable across a range of pre-training hyperparameter configurations. We attribute this stability to our semi-supervised method, which leverages abundant unlabelled samples during fine-tuning. Appendix E demonstrates that our method is more robust to pre-training target types than supervised fine-tuning (compare Table 11b and Table 3a) and does not require additional training tricks for strong performance (see Table 11a), unlike other works (e.g., [13, 17, 18]). We have updated the paper to better highlight USR's reduced sensitivity to pre-training hyperparameters.

The results do not consistently achieve state-of-the-art performance.

We have indeed carefully tuned hyperparameters, such as the learning rate, to obtain our final results. In the well-established and highly competitive LRS3 high- and low-resource benchmarks (see Table 4), our model surpasses (sometimes by a large margin) or matches the previous state-of-the-art modality-specific models in 16 out of 18 cases. In the remaining two cases, we are marginally behind (1.6% vs. 1.4% and 2.4% vs. 2.3% WER). Crucially, we achieve these results using a single model for ASR, VSR, and AVSR, while other methods require separate models for each modality, resulting in ~3x the number of weights during inference. We also achieve SotA results on WildVSR (Table 9) and LRS2 (Table 10).

Failure cases were not discussed too much.

Thank you for the suggestion. We added a discussion on failure cases in the Appendix. See Table 3 in the rebuttal PDF. We observe that, while VSR tends to produce more errors than ASR and AVSR, these errors are often related to phonetically similar sounds, such as "this" vs. "these" or "disguised"" vs. "denies." Additionally, using both visual and auditory modalities (AVSR) can improve the model's ability to distinguish challenging samples, such as "Mali Wear" vs. "malware".

During pretraining, have you explored using audio-only targets? If so, what was the performance like compared to AV targets? How does it compare to AV-HuBERT?

We have explored using audio-only targets for pre-training with our semi-supervised fine-tuning in Table 3a, where we observe that AV targets work best. Still, with audio-only targets, we achieve WERs of 37.3% for VSR, 3.2% for ASR, and 3.1% for AVSR, significantly outperforming AV-HuBERT's 51.8%, 4.9%, and 4.7%, respectively. Additionally, we experimented with supervised fine-tuning (see Appendix E), observing WERs of 43.9%, 4.8%, and 4.6% for the three tasks (see Table 11b). Notably, AV-HuBERT uses a separate model for each task, whereas we use a single unified model.

Why do you incorporate all three features (audio, video, audio-visual) during fine-tuning?

Our goal was to develop a single, unified model capable of performing well on audio-only (ASR), video-only (VSR), and audio-visual (AVSR) data during inference, thereby reducing the computational and memory redundancies associated with separate models per task. To achieve this, we fine-tune the model using all three types of data, enabling it to effectively perform each task. We have now emphasised this point in Section 3.1.

Consider experimenting with more advanced video encoders.

We used Transformer-based architectures with convolutional frontends to align with closely related works [13-18, 20] for fair comparisons (e.g., see Table 4). However, given that USR is agnostic to the choice of encoder architecture, we agree it would be interesting to explore other architectural variants, which could improve results even further. Still, we believe this direction lies outside the scope of our present work and therefore defer it to future research. This idea for future work has now been included in the Conclusion.

For pseudo label sampling, why opt for a greedy search? Have you considered trying soft sampling instead?

We initially chose a greedy search for its efficiency and effectiveness. Based on your suggestion, we now also experimented with a soft sampling approach, where we used weighted sampling at each generation step. The results are in Table 1b of the rebuttal PDF. We see that hard sampling outperforms this variant of soft sampling but believe that exploring more sophisticated methods to effectively increase pseudo-label variety is a promising direction for future research.

Thank you for your detailed review and time. Below we address the key concerns raised.

the technical novelty is not very strong. Most techniques are well-known or straightforward (e.g., the use of CTC, pseudo-label filtering, etc.).

While individual components of our work have been previously used in other studies (as discussed in the Related Work), we believe that the USR framework as a whole represents significant novelty. As noted by the reviewer, earlier efforts to unify ASR, VSR, and AVSR have often lagged behind modality-specific approaches. Our research demonstrates, for the first time in the literature, that a combination of self-supervised and semi-supervised learning can produce a unified model that achieves state-of-the-art performance across all tasks. This success is attributed to key technical design choices, including (but not limited to) the use of a greedy, computationally efficient attention-based pseudo-labelling approach; a multi-modal feature extraction step that enables amortisation of the pseudo-label generation costs across the three modalities; and multi-modal (audio-visual) target prediction in self-supervised pre-training, which previously proved unsuccessful with supervised fine-tuning. Furthermore, we believe that the straightforward and intuitive nature of USR enhances its utility and potential impact in the community.

Page 4, line 110: I'm a bit confused about "We set to 1." Do you mean that you always set to 1? No attention weights? Is it related to Table 2-d? Please clarify it.

We apologise for the confusion; this is a typo. The intended value for the CTC loss weight was 0.1, not 1. We always use CTC-attention training, except for the ablation in Table 2d where we study the performance of a CTC-only loss. We have fixed this typo in the paper.

Equation (4): Why didn't you prepare a different weight for a and av?

In preliminary experiments, we observed that the training dynamics for ASR and AVSR were very similar, and hence decided to use a combined weight for the two modalities in Eq. 4 and Eq. 7 in order to reduce the number of hyperparameters. We have now added this point to the paper (Section 3.1).

Section 3.2, Filtering: Did you use the same threshold for CTC and ATT? The dynamic range of c and a could be different, and I'm not sure that using the same threshold is optimal.

Similarly, we use the same threshold for CTC and attention for simplicity. However, we agree that the dynamic ranges between the two could be different and have run experiments with separate thresholds to investigate this point. The results are in Table 1a in the attached rebuttal PDF. We observe that USR's performance remains consistent across a range of different thresholds, with no clear improvement when using separate thresholds. We added this experiment in the Appendix.

Section 4: Did you only use a Transformer architecture? How about using a Conformer architecture?

We used Transformer-based architectures with convolutional frontends to align with closely related works [13-18, 20] for fair comparisons (e.g., see Table 4). However, given that USR is agnostic to the choice of encoder architecture, we agree it would be interesting to explore other architectural variants, including Conformers, which could improve results even further. Still, we believe this direction lies outside the scope of our present work and therefore defer it to future research. This idea for future work has now been included in the Conclusion.

It is not a question but a suggestion. I recommend you emphasize the results of the multiple databases in the abstract to claim the generalization of this work across the database.

Thank you for this suggestion. We have updated the abstract accordingly.