Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers, and Gradient Clipping

Dear Reviewer R1Qm,

We deeply appreciate your positive assessment of our work and your recognition of its importance as "the first benchmark of FL+DP in large-scale ASR." We address each of your points below.

"The computational overhead of the proposed methods may limit scalability to larger models or real-time applications"

You raise an important practical concern about computational overhead. We acknowledge this limitation while emphasizing that our work addresses a more fundamental challenge:

• Prior works such as Gao et. al 2022 [20] pointed out that training FL-ASR was infeasible. Some improvements were made by Azam et al. 2023 [56] to train a 120M Conformer model, but the training of FL with DP for ASR was still infeasible. Our primary contribution is establishing that such training is possible and studying the underlying reasons why this infeasibility existed.
• We deliberately focused on the feasibility first, with efficiency optimization being a natural next step that merits dedicated research. We have already seen important insights from contemporary research in the domains of (i) parameter-efficient methods, (ii) gradient compression and quantization (iii) selective layer updates that might be helpful in this regard. However, each of these directions deserves substantial research, which we defer to future improvements that can be studied on top of our findings.

"the relatively high error rates of baseline models on open-source datasets like Librispeech suggest that more recent ASR models should be used as baselines"

We believe that this assessment might be originating from the confusion between the model performance in the centralized and federated learning cases. For context, we compare against Whisper small (244M parameters, see Radford et al. 2022 D.2.2 - similar size to our model):

• For LibriSpeech test-other, Whisper small achieves 7.6% WER. In comparison, our centralized baseline achieves 6.8% WER on LibriSpeech test-other (Table 3, Appendix). SOTA models trained only on LibriSpeech achieve 5-6% WER for CTC models w/o a beam search and LM and are close to 3% with a beam search and LM - our model is ~3.5% when the beam search and LM are integrated. Our FL baseline on LibriSpeech test-other achieves 7.1% WER (only 0.3% degradation from centralized baseline). For context, prior work, Azam et al. 2023 [56], achieves 10.29% WER for FL setting with CTC-AED model.
• For Common-Voice English, Whisper small achieves 14.5% WER. In comparison, our centralized baseline achieves 18.2% WER (Table 4, Appendix) without a beam search and LM. Our FL baseline on Common Voice achieves 25.7% WER. For context, no prior work in FL-ASR, including Gao et al. 2022 [20], uses Common Voice English. 
• Additionally, Gao et al. 2022 [20] are able to train a model using FL on Common-Voice French and achieve 20.99% WER. In comparison, we achieve 13.1% WER (Table 10, Appendix) on Common-Voice French, while Whisper small is reported as 22.7%.

"How does the proposed method scale to even larger models?"

While we were not able to train a larger model within the rebuttal timeframe, we hope that the following table comparing against a model with 114M parameters can shed some light on the phenomenon. We can observe that (i) the larger 250M parameter model is more affected by uniform clipping, and (ii) it also benefits more from the per-layer clipping.

This is in line with our theoretical analysis (Theorem 2) which emphasizes that per-layer clipping benefits model with greater depth H and layer-wise gradient heterogeneity (Ψ$_h$ terms) which is more prominent in deeper models (Chan et al. [23], Wang et al. [24]; discussed in Sec. 3). This confirms that larger models benefit more from our approach. This is intuitive since more gradient heterogeneity across layers can be expected in deeper models, thus benefiting from per-layer intervention, i.e., per-layer clipping and layer-wise adaptive optimization.

"Could you provide more insights into how different DP budgets affect WER in practical scenarios?"

We provide a detailed analysis in Table 17 (Appendix). There exists a privacy-utility trade-off between DP noise and WER, wherein larger DP noise leads to worse WER. However, we point to the trend that better privacy-utility trade-offs are realized by per-layer clipping when compared to constant clipping

Practically, the target epsilon value often depends on the application, wherein a smaller epsilon ε ≤ 3 (i.e., more privacy) is preferred for training with sensitive data such as healthcare, and a slightly larger epsilon ε = 3-10 might be acceptable in other domains such as object detection, etc.

Furthermore, population size dramatically impacts achievable privacy (Table 1). This is in line with prior works such as McMahan et al. 2018 [46].

"Does the FL+DP approach ever match centralized training performance, or is there always a gap?"

This touches on fundamental aspects of privacy-preserving ML. In general, if the amount of data is similar between the centralized and FL training, then centralized training acts as an upper bound on FL training performance. This is due to the following:

• FL training observes more heterogeneous training as data cannot be shuffled, similar to centralized training.
• DP adds random noise to the training, which creates an irreducible gap in performance.

Summary

We thank you for recognizing this work as making "an important contribution" with "high originality." Your questions about practical deployment are exactly the right ones to ask as we move FL+DP from research to production. While computational challenges exist, this work addresses the fundamental challenge of the feasibility of training.

Dear Reviewer TNaa,

We thank you for the thoughtful and detailed feedback. We greatly appreciate the points raised and would like to clarify a few aspects of our work, as some key elements of our contributions and the FL+DP setting may have been misunderstood. We address each point below with the hope of resolving these potential concerns.

"the use of a transformer with CTC—while effective in earlier work—is relatively outdated given recent advances in end-to-end ASR, including models utilizing both CTC and sequence-to-sequence (S2S) losses"

We offer a different perspective that CTC is still a highly relevant and widely adopted approach in the literature:

• NVIDIA's FastConformer-CTC achieves competitive results on the OpenASR leaderboard (2024) (see HuggingFace Open ASR Leaderboard)
• Google's USM employs CTC for their 2B parameter universal speech model [Zhang et al., 2023; Wang et al., 2024]
• Meta's MMS uses CTC for their massively multilingual model covering 1,100+ languages [Pratap et al., 2023]

Additionally, our choice of ASR model is deliberate since we aimed to solve the ASR problem with on-device capability. Speech LLMs may still be infeasible due to device constraints and are still behind cascaded models (see “Spirit-LM: Interleaved spoken and written language model,” Transactions of the Association for Computational Linguistics, 2025; “Qwen2. 5-omni technical report,” arXiv:2503.20215, 2025; Voicebench: Benchmarking llm-based voice assistants; arXiv:2410.17196, 2024; MMAU: A massive multitask audio understanding and reasoning benchmark, arXiv:2410.19168, 2024). Foundation models of similar size are not comparable on ASR tasks, thus making the current ASR models more suitable for on-device applications.

"it is unclear what specific contributions have been made to address speech recognition challenges beyond the integration of existing federated learning and differential privacy techniques"

Below, we clarify the aspects of our work that are beyond just the integration of existing methods:

1. We observe that a naive integration of FL with DP does not converge in ASR. We arrive at a practical recipe to circumvent the convergence problems by using additional interventions: (i) per-layer clipping and (ii) layer-wise normalization. 
2. We demonstrate a clear departure from existing knowledge about per-layer clipping — previous work [McMahan et al., 2018] found no benefit for LSTMs — by providing empirical evidence supporting its effectiveness.
3. We also theoretically analyze the effect of per-layer clipping together with layer-wise adaptive optimization on the convergence dynamics. Our theoretical analysis goes beyond existing FL literature to understand the impact of heterogeneity, depth, and optimizer dynamics on convergence.
4. We “establish the first benchmark of FL+DP in large-scale ASR” (also echoed by Reviewer R1Qm) and further study the framework to explore the impact of: (i) different initialization strategies and (ii) in-domain and out-of-domain pre-training initialization.

Thus, in addition to the engineering implementation, our work also provides fundamental insights into optimization dynamics, theoretical guarantees, and practical techniques that advance the field. Furthermore, as echoed by Reviewer UQPF, the "proposed approach can potentially be applied to other domains apart from ASR and can have a significant impact."

"the reported WERs for both datasets are relatively high, especially for LibriSpeech, where much lower WERs are typically achieved"

We believe that this assessment might be originating from the confusion between the model performance in the centralized and federated learning cases. For context, we compare against Whisper small (244M parameters, see Radford et al. 2022 D.2.2 - similar size to our model):

• For LibriSpeech test-other, Whisper small achieves 7.6% WER. In comparison, our centralized baseline achieves 6.8% WER on LibriSpeech test-other (Table 3, Appendix). SOTA models trained only on LibriSpeech achieve 5-6% WER for CTC models w/o a beam search and LM and are close to 3% with a beam search and LM - our model is ~3.5% when the beam search and LM are integrated. Our FL baseline on LibriSpeech test-other achieves 7.1% WER (only 0.3% degradation from centralized baseline). For context, prior work, Azam et al. 2023 [56], achieves 10.29% WER for FL setting with CTC-AED model.
• For Common-Voice English, Whisper small achieves 14.5% WER. In comparison, our centralized baseline achieves 18.2% WER (Table 4, Appendix) without a beam search and LM. Our FL baseline on Common Voice achieves 25.7% WER. For context, no prior work in FL-ASR, including Gao et al. 2022 [20], uses Common Voice English. 
• Additionally, Gao et al. 2022 [20] are able to train a model using FL on Common-Voice French and achieve 20.99% WER. In comparison, we achieve 13.1% WER (Table 10, Appendix) on Common-Voice French, while Whisper small is reported as 22.7%.

"Relying solely on LibriSpeech and CommonVoice is insufficient to robustly support the claims"

We understand the concern and wanted to clarify that our choice for FL simulations is constrained by the requirements of speaker metadata:

• Datasets WITH speaker IDs (suitable for FL): LibriSpeech (~2k speakers), CommonVoice (~35k speakers), VCTK (~100 speakers), TED-LIUM (~1.2k speakers), etc.
• Datasets WITHOUT speaker IDs (unsuitable for FL): People's Speech, GigaSpeech, SPGISpeech, etc.

The requirement for speaker ID to simulate realistic user partitions severely limits options. Among suitable datasets, LibriSpeech and CommonVoice provide the largest speaker diversity (see Table 2, Appendix).

This diversity ensures our methods are tested across the wide spectrum of FL challenges. Our evaluation spans 4 datasets (LS + 3 CV languages), whereas most prior FL-ASR works typically evaluate on 1-2 datasets [19, 20, 56].

"It is unclear why the proposed model is characterized as a 'large model' in the context of ASR"

The characterization of "large" is relative to FL literature, not centralized ASR. Typical FL models in literature are <30M parameters (e.g., FedAvg [1], FedProx [11]). Prior work in FL-ASR has scaled up to ~120M parameters [19] without DP. Thus, by comparison, our model with 250M parameters is more than 2x larger than the prior works in the domain and ~10x larger than conventional FL works. This scaling is particularly significant in the domain of DP because DP noise scales with model dimension. Additionally, successfully training such models in FL+DP was previously considered infeasible (Gao et. al [20]).

Summary

We thank the reviewer for their thoughtful feedback and hope the responses above address their concerns. We will incorporate these clarifications in the final draft of the paper. We request the reviewer to reconsider their score in light of these clarifications and would be happy to provide further details if helpful.

Dear Reviewer UQPF,

We sincerely thank you for your thorough and constructive review of our work. We address each of your concerns below.

"The rows marked blue in Table 1. They seem to be extrapolated... To increase confidence in the approach, it would be great to have at least one blue (ε below 10) row with non-extrapolated numbers."

We acknowledge this concern. The extrapolation to a practical ε value at the scale required for blue rows would require simulating millions of users with cohort sizes of ~200k. However, we cannot run such simulations since there are no public datasets that can support this scale. 

Our extrapolation is based on the moments accountant (Abadi et al. 2016 [8], McMahan et al. 2018 [46]) method, which is a standard practice in FL and DP (for vision, text, etc.) due to the absence of such large-scale public data. The extrapolation estimates epsilon when population and cohort size increase while preserving the privacy noise-to-signal ratio.

We validated the above assumption by simulating different cohort sizes with ~35k users in Common-Voice (Table 16, Appendix):

These results demonstrate the consistent performance trend supporting the validity of the extrapolation. 

"the proposed work can be further enriched through a more thorough ablation study and utilization of a larger variety of ASR datasets"

Our choice for FL simulations is constrained by the requirements of speaker metadata:

• Datasets WITH speaker IDs (suitable for FL): LibriSpeech (~2k speakers), CommonVoice (~35k speakers), VCTK (~100 speakers), TED-LIUM (~1.2k speakers), etc.
• Datasets WITHOUT speaker IDs (unsuitable for FL): People's Speech, GigaSpeech, SPGISpeech, etc.

The requirement for speaker ID to simulate realistic user partitions severely limits options. Among suitable datasets, LibriSpeech and CommonVoice provide the largest speaker diversity (see Table 2, Appendix).

This diversity ensures our methods are tested across the wide spectrum of FL challenges. Our evaluation spans 4 datasets (LS + 3 CV languages), whereas prior FL-ASR works typically evaluate on 1-2 datasets [19, 20, 56].

"The per layer clipping in the context of DP-FL does not seem specific to ASR. Are there any characteristics that are especially more useful for ASR?"

You raise an excellent point about the generality of our approach. We view this generality as a strength that enhances rather than diminishes our contribution:

• We demonstrate that large Transformer models benefit from per-layer clipping when trained using FL with DP unlike the observations of McMahan et al. (2018) [46] who report no significant improvement for LSTMs and LMs.
• We provide insights by analyzing the ASR models and observing that layer-wise gradient heterogeneity is extreme in speech transformers (Figure 5: attention gradients are 10x smaller than LayerNorm). 

Given that our theoretical analysis is not specific to ASR, these insights should help other applications as well. 

"It would be beneficial to have an ablation study that demonstrates the impact of the proposed techniques on models of different sizes."

We agree with the suggestion and provide the following table summarizing a preliminary result for a 114M parameter model size that we were able to train within the rebuttal timeframe. We can observe that (i) the larger 250M parameter model is more affected by uniform clipping, and (ii) it also benefits more from the per-layer clipping.

This is in line with our theoretical analysis (Theorem 2), which emphasizes that per-layer clipping benefits models with greater depth H and layer-wise gradient heterogeneity (Ψ$_h$ terms), which is more prominent in deeper models (Chan et al. [23], Wang et al. [24]; discussed in Sec. 3). We will include this table in the camera-ready version.

"What are the author's thoughts on such an impact (and on σ) if the number of steps is increased."

This is an insightful observation that touches on fundamental FL+DP tradeoffs:

• Based on our theoretical analysis, it can be seen that privacy budget ε must be scaled as √T for T steps.
• Given such a dependence of noise on the number of training steps, we make the design choice of limiting to T=2k steps, thus keeping the noise manageable. All FL deployments face similar constraints, i.e., longer training requires accepting either higher ε or lower utility.
• We did not explicitly study extended training with higher noise levels since most FL studies aim to reduce the number of training steps (Azam et al. [16]) owing to the practical time limits for real-world federated learning deployments.

"Can you give an example of user level D and D' sets that are adjacent?" Each user has their own unique speech characteristics and adding/removing all their data can potentially impact the information contained inside a dataset.

Certainly. In our user-level DP setting, if we consider a population comprising 3 users X (with 50 data samples), Y (with 30 data samples), and Z (with 45 data samples), then adjacent datasets D and D’ can be defined as:

• Dataset D: Contains data from all users, i.e., X, Y, and Z, and
• Dataset D': Contains speech data from only 2 users, i.e., X and Z.

These datasets are adjacent because D' can be formed by removing all utterances from a single user Y. This DP guarantee ensures that an adversary observing the trained model cannot reliably determine whether User Y (with all 45 data samples) participated in training, protecting individual user privacy comprehensively.

In practice, we train on the actual dataset (e.g., dataset D with all users), but add carefully calibrated noise during training such that the resulting model could plausibly have come from training on D' (without User Y). The unique speech characteristics you mention make this particularly challenging - say, User Y's distinctive speech patterns could otherwise be detectable in the model, which is why we need sufficient noise to mask their contribution while still maintaining model utility.

Summary
We appreciate your recognition that our work presents a "well-structured approach" with "good quality proposal and analysis." We have identified areas where we can improve the manuscript based on your inputs. We will include these changes in the final draft of the paper.