MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition

Thank you for reviewing our paper thoroughly and for recognizing the key strengths, including the suitable hierarchical MoE architecture for AVSR, a novel gating mechanism, and strong experimental results demonstrating robust performances.

We greatly appreciate your thorough evaluation from the perspective of AVSR, even though MoE lies outside your primary research field. As such, we would like to briefly highlight our contributions of MoHAVE to the MoE framework:

• Novel hierarchical gating mechanism: An inter-modal router dynamically allocates tokens to audio or visual expert groups based on input characteristics, and an intra-modal router further dispatches tokens to appropriate experts within these groups.
• Adaptive and robust MoE: Previous works for multimodal MoEs have relied on modality fusion strategies or assigning fixed roles to each expert. While they lack adaptability and robustness to dynamically changing noisy environments, our work addresses this limitation by dynamically adjusting the usage of expert groups.

We also kindly encourage you to go through other reviewers' comments as well as our responses, and if any additional concerns arise, please let us know through the discussion phase. We would be glad to provide any further clarification.

Weakness 1: The model is evaluated on synthetic data and not on real-world conditions

A1: We acknowledge your concern regarding evaluation with synthetic noise data. While standard AVSR benchmarks such as LRS3 and MuAViC typically offer curated datasets with high-quality audio and clear visual information, these benchmarks alone cannot fully represent real-world noisy conditions. Therefore, following standard practice in robust AVSR research [1,2], we have introduced various noise conditions to evaluate our MoHAVE's robustness and adaptability.

Additionally, to better reflect real-world noise conditions, we conducted further evaluations by augmenting LRS3 with realistic background audio from the DEMAND dataset [3], which contains recordings from diverse indoor and outdoor environments, e.g., cafeteria. On this enhanced benchmark (at SNR=-10~0), MoHAVE consistently outperformed AV-HuBERT across various real-world settings including cafeteria (WER: 6.4 vs. 8.6), restaurant (11.9 vs. 13.1), meeting room (4.5 vs. 5.7), and river (4.4 vs. 6.1), achieving an average WER of 3.6% vs. 4.1% across all 18 environments. These results further confirm MoHAVE’s performance under realistic audio-visual conditions.

Weakness 2: CMA and UniVPM are missing from Table 1

A2: The CMA and UniVPM models in Table 2 are all built upon AV-HuBERT-Large, matching the architecture and activated parameter count with the dense (non-MoE) baseline in Table 1. Furthermore, both Table 1 and Table 2 evaluate under identical experimental setups and noise configurations (i.e., babble, speech, music, and natural noises), while Table 2 reports average results for music and natural. For detail, please refer to the table below.

We note that Table 1 primarily demonstrates the effectiveness and efficiency of MoHAVE, by comparing different MoE variants, including standard MoE, hard routing, and hierarchical MoE. In contrast, CMA and UniVPM in Table 2 utilize special modules for cross-modality, which are orthogonal to the MoE framework, focusing instead on audio-visual fusion or feature enhancement strategies independent of expert routing mechanisms. Yet, recognizing the importance of comprehensive comparisons, we have included the result of incorporated CMA + MoHAVE in Table 2. To improve clarity, we will revise by merging Table 1 and Table 2.

Question 1: Can MoHAVE handle asynchronous speech and lip movements?

A3: We have not yet evaluated MoHAVE under audio-visual asynchronous conditions. Our current framework is optimized for scenarios where audio and video are temporally aligned, as is standard in most AVSR works. Prior works that address the audio-visual asynchrony have proposed solutions like external synchronization module [4], which explicitly model temporal offsets between audio and visual streams. While MoHAVE does not currently model asynchrony, we believe combining MoHAVE with methods that explicitly handle asynchronous inputs could be a valuable extension for future research.

Question 2: How do different languages affect expert selection and hierarchical MoE routing?

A4: Thank you for this insightful question. Our analysis indicates language-dependent differences in expert allocation within MoHAVE. For example, Arabic tokens tend to be routed more frequently toward visual experts, whereas French or Spanish tokens rely more heavily on audio experts (please see this anonymized link). However, we also note that these trends vary by layer. Also, within each expert group, the intra-modal router’s load-balancing ensures a uniform expert utilization across data samples. Thus, there is no explicit language-specific expert selection within groups, consistent with observations found in [5]. We suppose that more detailed investigation into expert load distribution across languages and its relation to linguistic/paralinguistic characteristics would be valuable future work.

---

References:

[1] Hong et al. "Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring." CVPR, 2023.

[2] Kim et al. "Learning video temporal dynamics with cross-modal attention for robust audio-visual speech recognition." SLT, 2024.

[3] Thiemann et al. "The diverse environments multi-channel acoustic noise database" Proceedings of Meetings on Acoustics, 2013.

[4] Li et al. "Unified cross-modal attention: robust audio-visual speech recognition and beyond." TASLP, 2024.

[5] Zoph et al. "St-moe: Designing stable and transferable sparse expert models." arXiv, 2022.

Weakness 1: The notation in the equations could be improved

A1: Thank you for the detailed review. We will revise the notations, especially in Eq. (11), to clearly distinguish the set of experts from its cardinality.

Suggestion 1: The computational cost analysis is only discussed in Appendix

A2: We agree with your suggestion that computational efficiency is indeed one of the main contributions of MoHAVE, especially given its sparse MoE architecture that activates only a fraction of parameters. To better highlight this contribution, we will move the computational cost comparison as well as its discussion currently presented in Appx. A.2 into the main body.

Question 1: What was the reason for the following choice? For sequences containing both audio and video, we exclude them from the load biasing loss calculation but incorporate them into the load balancing.

A3: The load biasing loss is designed specifically to encourage modality specialization for a subset of experts. Audio-visual multimodal tokens do not inherently favor one modality-specific expert group over another. Therefore, these tokens are included in the load balancing, ensuring uniform expert group loads across every multimodal token.

Question 2: Do the batches contain a mix of audio-only, video-only, and AV data points in a single batch?

A4: Yes, each batch is constructed by sampling a total of 320 seconds of data, structured as follows: 25% of the data points in the batch are audio-only (video dropped), another 25% are video-only (audio dropped), and the remaining 50% are multimodal, containing both audio and video.

Weakness 1: The proposed method and its improvement are incremental

A1: Thank you for your valuable comments. We would like to clarify that MoHAVE introduces several key innovations in both scalability and robustness for AVSR systems, which go beyond existing works:

MoHAVE is the first AVSR framework that scales up to ~1B params, through using a sparse MoE architecture to enable efficient scaling with low computational overhead. To mitigate the model’s inherent bias toward audio, we also introduced expert group specialization, followed by a novel hierarchical gating mechanism that dynamically routes tokens based on modality reliability and input characteristics. Unlike previous multimodal MoEs relying on modality fusion or assigning fixed roles to each expert, MoHAVE explicitly adjusts expert group usage dynamically—enhancing both adaptability and robustness.

MoHAVE achieves state-of-the-art results across robust AVSR benchmarks (Tables 2 and 3). While the improvements in Table 1 may seem modest on average, the gains under severe noise conditions are substantial. As shown in Appx. A.3 (Table 5), MoHAVE-Large achieves 5.0% WER on LRS3 with speech noise at SNR=-10—yielding a 56.1% relative WER improvement over AV-HuBERT-Large, 36.7% over AV-MoE-Large, and 25.4% over the hard-routing variant. This indicates that MoHAVE correctly predicts over half of the words AV-HuBERT misses.

We believe these contributions pose a significant breakthrough in AVSR scalability and adaptive learning, and our hierarchical routing design offers broad potential for other multimodal MoE applications as well.

Weakness 2: The paper should have more comparisons with hard routing

A2: We initially introduced hard routing into AVSR to utilize visual experts for noise robustness. However, it lacks adaptability since the allocation of expert groups must be manually fixed (50% audio / 50% visual in our implementation). This approach is sub-optimal depending on audio-visual input quality (as discussed in Fig. 4(b)), and the limitation becomes clearer under challenging environments. Under severe conditions (e.g., Table 5, SNR=-10, -5) hard routing cannot dynamically adjust expert usage, performing much worse than MoHAVE. In response to the reviewer’s suggestion, we additionally evaluated the hard routing model on MuAViC, as provided below. Here, hard routing substantially underperforms MoHAVE and even mAV-HuBERT.

Regarding Fig. 4 and 5: computing expert load distribution for hard routing would be trivial. By design, expert usage is manually set depending on the input: 100% audio experts for audio-only, 100% visual for video-only, and 50/50 for audio-visual (finding optimal split is heuristic). Unlike MoHAVE, there is no data-driven or noise-aware expert selection. Thus, hard routing would trivially display static distributions without dynamic behavior.

Weakness 3: Experimental comparisons with (Cheng et al., 2024; Wu et al., 2024)

A3: Direct comparisons with AV-MoE (Cheng et al.) and EVA (Wu et al.) are unfortunately infeasible due to fundamental differences in target tasks and methods. Both AV-MoE and EVA primarily address audio captioning for visual contexts (e.g., narrating sports game scenes), while our work specifically targets typical AVSR tasks, where both audio and visual inputs directly involve human speech.

Moreover, AV-MoE employs a dense MoE; unlike sparse expert structures commonly used in modern LLMs or Transformers, AV-MoE’s "MoE" is actually implemented as weighting between unimodal and cross-modal adapters, rather than selecting sparse FFN experts. Specifically, AV-MoE uses two entirely separate MoEs for audio encoder and visual encoder, infeasible for processing multimodal tokens. Our approach, MoHAVE, fundamentally differs by employing a sparse multimodal MoE, dynamically routing tokens based on audio-visual inputs.

Closer to our work is EVA, which simply applies a sparse MoE structure into an audio-visual encoder. Although exact implementation details are unavailable (code/checkpoints unreleased), EVA’s structure aligns closely with our basic MoE implementation which we evaluated as AV-MoE in Table 1 (AV-MoE-Base and AV-MoE-Large), except ours is in the decoder. As demonstrated in our study (Table 9 in Appx. B.5), applying MoE at the encoder-level—like EVA—falls behind our multimodal decoder approach. Thus, EVA likely cannot achieve comparable robustness or efficiency.

Suggestion 1: Rewrite the equations

A4: Thank you. We will revise the equations to clearly distinguish the expert set from its cardinality, defining $E_i$ as the output of the $i$-th expert.