Advancing Energy Efficiency in On-Device Streaming Speech Recognition

This uses a Pixel 5 without any neural hardware acceleration. Most modern mobile devices have some neural accelerator chip, and such a study on power consumption should use it. This will heavily influence all the results presented in the paper here. The presented results are not really so relevant nowadays that we have such neural accelerators.

The use of Pixel 5 in this work is for profiling workload runtime characteristics such as model invocation times and component invocation times. These characteristics are device-independent and consistent across platforms. The actual results in the paper are obtained using setups that include hardware accelerators, with parameters derived from authoritative circuit literature. All this information is clearly detailed in Section 2.2 of the paper, and we encourage the reviewer to refer to it.

Additionally, the Pixel 5, released only three years before this submission, is undoubtedly a “modern mobile device.” The reviewer’s comment suggesting otherwise is factually incorrect.

 

Adam-pruning to introduce sparsity to make the model smaller is the only method used here to vary the model size. There are many other ways to vary the model size, even more straightforward, to just change the number of layers and/or number of dimensions (then train from scratch or via knowledge distillation from a big model). When I see different model sizes, I think this is much more expected and relevant. But you can also compare different methods (Adam-pruning vs just changing num layers/dims vs maybe some other methods). But such comparison is not done here.

The primary goal of this work, as explicitly stated in Sections 4 and 5, is to identify model components whose compression yields the greatest power savings with minimal accuracy degradation. The compression method used to achieve this is an existing technique and is clearly noted in Section 5.2. Exploring alternative model size variations, such as changing layers or dimensions, falls entirely outside the scope of this study.

Moreover, the suggestion to retrain models from scratch by altering layers or dimensions is impractical in this context. We are working with an existing model, and retraining from scratch introduces significant overhead without serving the objectives of this work. The reviewer’s suggestion is not only irrelevant but also misaligned with the paper’s scope.

 

Sparsity is probably a suboptimal choice for neural accelerator chips. So this is even more an argument for using other methods to vary the model size.

This comment is inaccurate. Structured pruning, a specific type of sparsity, is already supported by numerous neural accelerators. The work in this paper explicitly uses structured pruning, which is highly suitable for modern hardware. The reviewer’s blanket statement about sparsity being suboptimal reflects a lack of understanding of recent advancements in neural accelerator designs.

 

The base model is not specified at all.

This is incorrect. The base model is explicitly specified in Section 5.1. We urge the reviewer to revisit that section for clarity.

 

There is no code.

The paper leverages existing methods for compressing model components, and the methodology is already well-documented in the literature. Releasing additional code would serve no purpose and is unnecessary for reproducing the results presented here.

 

The relevant properties are WER, RTF and power consumption, and you would want to see them being put directly into relation to each other. This is not done here. It's always only indirect via model size.

Model size is a critical metric in this study as it directly impacts memory consumption. To analyze the relationships among WER, RTF, power consumption, and model size, we chose model size as the shared reference point for consistency and clarity. This approach is entirely valid, and there is no issue with presenting the relationships in this manner.

 

Comparison of different encoders (Emformer vs Conformer vs maybe others, e.g. Zipformer) is missing.

This paper is not a study on encoder architecture design. Comparing Emformer and Conformer, or other architectures, is entirely irrelevant to the scope of this work. The reviewer’s suggestion for such experiments is misplaced and beyond the intended focus of this paper.

Lack of clarity around power consumption data in experiments. The power consumption results presented in Figures 6 & 7 are labeled ‘Model Power Consumption’ — are these model based estimates, or real measurements of power consumption? If these are model-based estimates then what indication is there to suggest these will correlate with real measurements?

As noted in Section 2.2, the power consumption analysis builds on established power modeling techniques referenced in [1][2][3][4]. These methods, either developed by major memory manufacturers like Micron or published in prestigious computer architecture and speech processing conferences, are widely recognized and validated.

[1] Micron. Technical Note TN-47-04: Calculating Memory System Power for DDR2. Technical report, 2006.

[2] Architecting Phase Change Memory as a Scalable DRAM Alternative. In ISCA, 2009.

[3] Utility-Based Hybrid Memory Management. In CLUSTER, 2017.

[4] Folding Attention: Memory and Power Optimization for On-Device Transformer-based Streaming Speech Recognition. In ICASSP, 2024.

 

The WER vs model size graphs seem slightly worse for the proposed approach as compared to baseline in Figures 6 & 7. This is understandable as the focus was not on optimizing for model memory as a function of WER. However, would the proposed approach be effective if the focus was on minimizing model memory footprint?

The proposed technique focuses exclusively on power optimization, not memory optimization. It has no impact on the model's memory footprint, as clearly illustrated in Figures 6 and 7. Model WER varies due to the randomness introduced by pruning. Therefore, it is unrealistic to expect the memory footprint to remain exactly the same before and after applying the technique. However, as Figures 6 and 7 demonstrate, the memory footprint after applying the technique remains highly similar to the original, with minor variations observed in both positive and negative directions.

We sincerely appreciate the reviewer’s effort in evaluating our paper.

The investigation taken in this paper is mainly empirical, using well-known techniques, with minor contributions in models and methods.

This paper presents a new ASR compression method that delivers up to 47% reduction in energy consumption and 29% improvement in real-time factor (RTF) while maintaining accuracy on par with state-of-the-art approaches. These results demonstrate significant advancements in energy-efficient on-device ASR.