SMILE: Audio-Visual Speech Recognition with Siamese Masked Interaction Learning

Page [1/2]

I'm gratitude for your valuable feedback and suggestions on our manuscript. We list our responses to the questions below.

Questions: I find the introductory paragraph that motivates multimodal speech recognition to be unclear. The authors state that “... speech contains detailed phonetic information, while lip movement conveys emphasis or stress ...”. Phonetic information is conveyed visually too — for example, the place of articulation for bilabial, labiodental, and dental sounds can be seen. Furthermore, emphasis and stress are both also conveyed acoustically. The main benefit of multimodal speech is that when acoustic noise masks the differences between two sounds so that they sound similar, those sounds can be disambiguated visually. Similar sounding speech events look different, and similar looking speech events sound different from one another. This is the complementary nature of audiovisual speech.

ANS: I appreciate your clear and precise insights, which teach me a lot. According to your advise, we re-formulate our paper and provide the corresponding experiments in our revised paper. Please see the first part of the introduction and new Table 2. Our motivation is : Lip movements primarily contribute to the improvement of Automatic Speech Recognition (ASR) in two main scenarios. Firstly, they offer distinct visual cues that aid in differentiating between speech words that sound similar. Secondly, they provide valuable visual cues that enhance speech recognition accuracy in the presence of various types of noise commonly encountered in real-world environments.

Questions: The authors refer to “The original Siamese networks” and cite papers from circa 2020, yet Siamese networks pre-dated these works by a couple of years.

ANS: Thanks for pointing it out. We have corrected it in our revised paper.

Questions: Equation (7) and the accompanying explanation feels a little redundant as this already is clear from Equation (6).

ANS: We believe that this formula better captures the interaction process between the audio and video modalities, specifically between masked tokens and unmasked tokens.

Questions: In Equation (9), S is shared across all layers, which weighs the contribution of the visual information. What about using a dynamic, layer-dependent weight (as in Equation (10)) since the contribution of the visual information for the speech recognition task might be different for different layers (as you already note for the acoustic information: “... the shallower layers in audio models tend to prioritize speaker-related information, while the deeper layers tend to emphasize content-related information.”. The same might be true for visual features, and allowing the visual contribution to \hat{F}^{I}_{av} to vary might provide more robust features.

ANS: We agree with your viewpoint. However, when conducting experiments with specific S_i values for each layer, we observed that this specific S_i setting performed worse compared to sharing the same S value across all layers. We postulate that S may represent the global weight assigned to visual cues that need to be incorporated into speech.

Questions: In Section 4.2 you refer to extracting the per-frame ROI and then applying a horizontal flip with probability 0.5. To be clear, this flip is not applied per frame, but to the whole sequence?

ANS: For each frame in the whole sequence, it has the chance to be flipped with a probability of 0.5.

Questions: For Table 3(b), why were these four combinations of masking rate for the audio and visual selected? What about the other combinations?

ANS: We firstly apply 0.3 masking rate and 6 mask length for both audio and visual modalities as our base setting and find it works well. Then we suppose that Siamese architecture may work with more masking rate and masking length, which is benefit to improve the audio-visual interaction and learn a higher-level audio-visual temporal correspondence. Considering that audio feaures are more redundant than visual features in terms of temporal information, we raise audio mask length to 12. We compare audio-to-video mask ratio with 0.4:0.6 and 0.6:0.4 under same mask length setting, and find that 0.6:0.4 is better. We suppose that masking more audio can force the model to pay more attention to visual cues, which may reduce the “audio-predominant” modality-imbalance problem. Further increasing the masking rate may lead to significant loss of modality information, resulting in a decline in model performance.

Page[2/2]

Continue here.

Weaknesses: It is not clear to me how much the visual information contributes to the WER. The WERs reported here are on par with state of the art acoustic-only speech recognition. Why not include an ablation of the modalities — show results for audio-only, video-only and audiovisual.

ANS: We provide Table 6 in the appendix in our newly revised paper.

Weaknesses: Also, the motivation for the work is that the visual modality helps in the presence of acoustic noise. There are no experiments that actually demonstrate this, or quantify the effective gain in SNR provided by the visual information.

ANS: We newly provide Table 2 which is trained and tested on LRS2 with noise conditions.

Weaknesses: The paper describes the problem of AVSR as an “emerging research area”, but this ignores the 30+ years of prior research into multimodal speech recognition.

ANS: Thanks for pointing it out. I revise it into "AVSR is an active and evolving research area derived from the Automatic Speech Recognition (ASR) task."

Thanks for your valuable advise, i appreciate it a lot. We list our responses to the questions below.

Weaknesses: The authors state that AVSR "is primarily desgined to address ... high levels of noise and speech occlusion" but the paper does not address such scenarios.

ANS: We re-formulated our motivation in the revised paper to make it more clearer. Our motivation is revised in the first paragraph in the introduction part.

Weaknesses: Related to above points, the LRS2/3 dataset was collected using an automated pipeline using ASR. This means that the test set is biased towards examples that are easy for speech recognition systems -- the speech-only model in the data curation pipeline can achieve 100% accuracy (Sec 4 of 1809.02108 and Sec 2.1 of 1809.00496). This undermines the usefulness of this dataset for AVSR under clean conditions.

ANS: We provide Table 2 in our paper, which is experimented on noise conditions.

Weaknesses: The authors cite 2020 and later papers for original Siamese network, but the term was proposed much before by (Koch et al., 2015) for contrastive learning.

ANS: Thanks for pointing out it. We correct it in the revised paper.

Questions: Does the model work in missing modality conditions?What is the performance of audio-only ASR model trained using the same pretraining/finetuning data setup?

ANS: In the newly-revised paper, Table 6 in the appendix is added to provide the audio-only test and audio-only train&test results of our model in both clean and noise test conditions. All models are trained under noise conditions, with a fixed number of mixtures set to 1, a probability of noise set to 1, and a signal-to-noise ratio (SNR) set to 0.

Questions: How sensitive is the result to the values of alpha and beta.

ANS: In the newly-revised paper, Table 5(d) in the appendix is added.

Thank you for your reply.

In the new Table 2, the results are WER results under different SNR settings. For example, the test noise snr=-10 column represents that we add noise to the test speech with snr=-10 setting. Under this noise setting, we get the corresponding WER results. And the lower WER results means better model performance. The last column of Table 2 is the average WER result, which is the average WER for snr settings for -10,-5,0,5,10.

Thanks for the revision. In the new Table 2, why is the SNR higher for the baseline (Hsu et al., 2021) compared to the proposed method? Isn't high SNR better?

Thank you for giving us valuable comments. We present responses to the questions below.

The method described in [1] used more training data than ours, as indicated by the data presented in Table 4 of [1]. A clearer comparison is listed below.

[1]. Pingchuan Ma, Alexandros Haliassos, Adriana Fernandez-Lopez, Honglie Chen, Stavros Petridis, and Maja Pantic. Auto-AVSR: Audio-visual speech recognition with automatic labels.

We appreciate your valuable comments. We list the responses to the questions below.

In Table2, the effect of the proposed adaptive multimodal fusion (obtained with a weighted sum over different layers) seems not so appealing, and the slight improvement may come from the increase of extra learnable parameters? Have the authors checked the learned value of the weight S and wi? What are they like in the learning process?

ANS:

1. The learned value of the weight S is 0.75. And the w_i is [0.0751, 0.0751, 0.0787, 0.0733, 0.0739, 0.0727, 0.0703, 0.0735, 0.0774,0.0855, 0.1082, 0.1362] for each layer.
2. As for the weight value S, we hypothesize that the audio modality plays a dominant role in the AVSR task, while the visual modality can only serve to improve or enhance the audio modality.
3. As for the weight values w_i, we observed that in the shallow layers of audio, they primarily represent speaker information, while in the deeper layers of audio, they primarily represent speech content information. The learned values indicate that speech recognition relies more on speech content, but speaker information can also provide some assistance.
4. Additionally, to investigate whether the improvement is attributed to the increase in extra learnable parameters, we conducted additional experiments by setting specific values S_i for each layer i. The performance dropped from 4.9 to 5.0 WER.

In Table 3, the gap among different strategies can achieve as high as about 2%, which has already been significant if we compared it with the general gap of the final performance of the proposed work in Table1. This big gap, especially when compared with the improvement of the proposed modules here, states the importance of training strategies in the process, but these strategies are existing ones, not firstly proposed in this work. I think this may further weakens the contribution of the proposed method.

ANS:

In the newly revised paper version, the number for the ablation table is Table 3. In Table 3, it can be observed that the disparity in performance between Siamese architectures using different training strategies can be as high as 2%. However, it is important to note that these results are obtained under the assumption that other components of the designed model are already optimized for the 'stop gradient' setting. Furthermore, the reason for selecting these four different training settings is the existence of divergent opinions among researchers regarding the most effective approach to train Siamese networks. Different training strategies are suitable for different scenarios.