BAP: BRANCH-AWARE PARALLEL EXECUTION FOR FASTER DNN INFERENCE ON MOBILE CPUS

Q1: Thank you for the comment. We thoroughly reviewed the suggested works and acknowledge their contributions to scheduling and optimization. While these approaches are effective for static models, they do not address the dynamic properties we focus on, such as runtime-determined control flows and tensor sizes in transformer-based and ASR models.

The key challenges with dynamic models are twofold:

• Incompatibility with Static Scheduling: Pre-determined schedules fail when tensor shapes or control flows change at runtime, often leading to crashes.
• Impact of Memory Reallocation: Frequent memory reallocations disrupt static scheduling plans, limiting parallelization feasibility.

With the rise of transformer-based architectures, these issues are becoming increasingly common. Our work addresses this gap by reorganizing memory allocation to reduce contention and enable safe parallel execution. While our execution scheduling remains straightforward, the focus is on tackling the critical research gap left by SOTA frameworks like TFLite, MNN, and PyTorch Micro, which do not handle these dynamic properties [1, 2]. We believe this direction is valuable for inspiring future research on accelerating models with dynamic workloads and guiding the design of future frameworks or mobile-compatible models.

[1] TensorFlow GitHub Issue #48553: https://github.com/tensorflow/tensorflow/issues/48553
[2] Arm NN GitHub Issue #773: https://github.com/ARM-software/armnn/issues/773

Q2: Thank you for the insightful suggestion. In our experiments, the TFLite baseline uses intra-operator parallelization, with the number of threads explicitly set through the API TfLiteStatus SetNumThreads(int num_threads);. Despite this, our results show that branch-wise parallelism in BAP achieves significantly better latency improvements compared to TFLite's intra-operator parallelization, as shown in Table 3.

A relevant discussion in TensorFlow GitHub Issue #20187 [1] highlights that the current intra-operator parallelism implementation often provides limited benefits. Moreover, intra-operator parallelism is insufficient for models with complex branching structures, as it focuses on individual operator acceleration without leveraging broader graph-level parallelism. These limitations make branch-wise parallelism a more effective and widely applicable solution.

[1] TensorFlow GitHub Issue #20187: https://github.com/tensorflow/tensorflow/issues/20187

Dear Reviewer XcJw,

As the discussion period is coming to an end, we would like to follow up to see if there are any remaining concerns or questions we can clarify. We have thoroughly addressed your feedback, highlighting the limitations of static scheduling for dynamic models, the challenges of runtime-determined control flows and the inefficiencies of intra-operator parallelism, which reflects the contribution and novelty of our work. These points are supported by updated results and discussions, emphasizing BAP’s effectiveness in addressing these critical gaps.

If there are no further questions, we kindly ask you to consider updating the score to reflect these clarifications and the contributions of our work. Please let us know if further discussion would be beneficial.

Best regards,

Authors

Dear Reviewer XcJw,

Today is the final day for score updates. We have carefully responded to your comments in the rebuttal and hope you can review it and consider increasing our score.

Thank you for your time and effort.

Authors

W1: Thank you for the detailed feedback.

Longer Audio Samples:
We conducted 15-second audio testing on a Google Pixel device to evaluate our approach. The results demonstrate up to 42% latency improvement in some cases, with average improvements shown below:

From the table, we observe that while absolute latency improvements vary, relative improvements remain consistent across different audio durations. This stability arises because longer audio inputs are subsampled into smaller segments (e.g., 10 or 15 seconds) to accommodate device memory constraints. Transformer-based models like Whisper and Conformer require significant memory for query-key matrices, making very long sequences infeasible on mobile devices.

Impact of Sequential Nodes:
We appreciate the suggestion to analyze the relationship between sequential nodes and performance metrics. Table 1 already shows the number of parallelized branches, reflecting the proportion of sequential and parallelizable layers. Additionally, as mentioned in our response to Reviewer rvAe, the percentage of parallelizable operators varies by model: MobileViT (small or XS) achieves 17.93%, Whisper 61.43%, and Conformer 8.46%. These percentages have been added to Table 1 for clarity.

Code Availability:
We plan to release our code base after publication to ensure reproducibility and facilitate further research.

Q1: Thank you for the insightful questions.

Justification of CPU Execution:
While many devices include GPU-like units, dynamic-sized tensors and control flow, as seen in transformer-based and ASR models, cannot be accelerated using current delegation methods and default to CPU execution. This limitation is discussed in TensorFlow GitHub Issue #48553 [1]. These dynamic properties, particularly prevalent in token-by-token predictions, make CPUs the most reliable option. Additionally, CPU execution remains critical for many IoT devices.

Comparison with Other Baselines:
Methods targeting CPU inference acceleration, such as TFLite's memory optimization tool [2], Arm NN [3], and Qualcomm's delegation work [4], are designed for models without dynamic control flows. Consequently, they cannot address the complexities of models like Whisper or ASR systems (as discussed in our responses to Reviewer rvAe Weakness 3 and 1oc7 Question 6). Using TFLite's default CPU runtime ensures a robust and fair baseline, given its widespread usage and reliability.

Impact of Existing CPU Usage:
In our experiments, only the testing application was running to ensure consistent measurements. We acknowledge that existing CPU workloads can impact inference performance. Addressing this would require further research to handle resource contention dynamically, and enhancing BAP for multi-task scenarios is part of our ongoing work.

[1] TensorFlow GitHub Issue #48553: https://github.com/tensorflow/tensorflow/issues/48553
[2] Nicholas D. Lane. "Neural networks on microcontrollers: saving memory at inference via operator reordering." arXiv preprint arXiv:1910.05110 (2019). https://github.com/eliberis/tflite-tools
[3] Arm NN: https://github.com/ARM-software/armnn
[4] Qualcomm Blog: https://www.qualcomm.com/developer/blog/2020/11/tensorflow-lite-inference-edge

Dear Reviewer MhbN,

As the discussion period nears its conclusion, we wanted to follow up to check if there are any additional points or concerns we can address. We have provided detailed responses to your feedback, including latency benchmarks for longer audio samples, an analysis of sequential versus parallelizable nodes, and clarified the justification for CPU execution and baseline comparisons. These updates, along with the percentages of parallelizable operators and their implications on performance, are reflected in the revised manuscript and updated tables.

If there are no further questions, we kindly ask you to consider increasing the score based on the improvements and clarifications provided. Please let us know if any further discussion is needed.

Best regards,

Authors

Dear Reviewer MhbN,

Today is the final day for score updates. We have carefully responded to your comments in the rebuttal and hope you can review it and consider increasing our score.

Thank you for your time and effort.

Authors

W1: Thank you for the suggestions. To address the overlap and ensure clarity, we already rephrased the contributions as:

1. "We introduce BAP, a CPU-specific optimization system for ASR and transformer-based models with dynamic control flows and tensor operations. BAP combines branch-aware memory allocation and multithreading to optimize parallel execution and reduce latency."

2. "We develop a branch-aware memory allocation strategy that reduces data contention by isolating memory arenas for parallel branches, enabling safe and efficient parallel execution."

3. "BAP delivers substantial improvements in latency, memory efficiency, and energy consumption without requiring model refactoring, supporting practical real-time inference on edge devices"

W2-1: Thank you for the suggestion. Regarding the term "branch," it is synonymous with "subgraph" in our context. To enhance readability, we will ensure consistent terminology by using "branch" throughout the paper. Regarding the phrase "provided there are no inter-branch dependencies," we agree it may be redundant given the earlier definition and will streamline this part for clarity.

W2-2: Thank you for pointing this out. After Line 15, we have added the following lines (See the updated Algorithm 1):

16:     if $v_i$ is visited then
17:         continue 
18:     end if

W3-1: Thank you for the question and for highlighting the works of Arm NN and Qualcomm. We carefully reviewed their repositories and implementations to address this point. As mentioned in our paper, many proposed methods, including vendor-specific delegates, do not handle dynamic-sized tensors or control flow operations effectively. This limitation is particularly relevant with the increasing deployment of transformer-based and ASR models, which frequently exhibit these characteristics on mobile devices.

In the Arm NN GitHub issue #773 [1], this limitation is explicitly discussed, where the Whisper TFLite model fails with Arm NN due to its reliance on dynamic-sized tensors. Additionally, TFLite is one of the most widely adopted frameworks for embedded and mobile systems, with deployment on over 4 billion devices globally [2]. This further supports our choice to use TFLite’s default CPU implementation as the baseline, as it remains the most reliable option for such models.

[1] Arm NN GitHub Issue #773: https://github.com/ARM-software/armnn/issues/773
[2] TensorFlow Lite – Computer Vision on Edge Devices [2024 Guide]: https://capalearning.com/2024/10/21/tensorflow-lite-computer-vision-on-edge-devices-2024-guide/

W3-2: Thank you for the question. TFLite’s arena-based memory allocation strategy is widely recognized for its efficiency in handling resource-constrained environments using greedy heuristics, as noted in [1]. Regarding latency, TFLite integrates multi-threading and optimized operators for CPU execution, making it a strong baseline for CPU-only performance, as evidenced in TensorFlow's official documentation and benchmarks [2]. We will clarify these points in the revised manuscript.

[1] TelaMalloc: Efficient On-Chip Memory Allocation for Production Machine Learning Accelerators, Martin Maas et al.
[2] TensorFlow blog, "Optimizing TensorFlow Lite runtime," https://blog.tensorflow.org/2020/10/optimizing-tensorflow-lite-runtime.html.

W4: Thank you for the question. The TFLite implementation in our experiments supports multithreading, which we enabled using the TfLiteStatus SetNumThreads(int num_threads) API to fully utilize all available CPU cores, ensuring a fair comparison with BAP. The improvements observed in our paper are attributed to BAP’s graph-level parallelization. We will clarify this in the revised manuscript.

Dear Reviewer 1oc7,

As the discussion period nears its conclusion, we would like to ask if there are any remaining questions or concerns we can address. We have carefully incorporated your feedback, refining our contributions, terminology, and algorithm details, as well as clarifying key aspects like our baselines and multithreading comparisons with TFLite.

If there are no further points of clarification, we respectfully ask Reviewer 1oc7 to consider increasing the score based on the addressed concerns and improvements. Please let us know if further discussion would be helpful.

Best regards,

Authors

Dear Reviewer 1oc7,

Today is the final day for score updates. We have carefully responded to your comments in the rebuttal and hope you can review it and consider increasing our score.

Thank you for your time and effort.

Authors

W1: We appreciate the suggestion and have enhanced Figure 1 to illustrate how runtime uncertainties in tensor sizes necessitate frequent memory reallocations, disrupting branch-level isolation. See the updated paper.

W2: We appreciate the suggestion and clarify that CPU-specific optimizations are addressed in the Introduction and Related Work sections:

• CPU and GPU Performance: As mentioned, on mobile devices, the performance gap between CPUs and GPUs for inference tasks is minimal, making CPUs a flexible and viable choice (See citations in our paper Zhang et al., 2023; Bordawekar et al., 2010).

• Dynamic Tensor Limitations: Hardware-specific delegates (e.g., GPUs, NPUs) often fail with dynamic tensor sizes in ASR and transformer models, leading computations to fall back to the CPU (TensorFlow GitHub Issue #48553 [1]).

• IoT and Edge Devices: CPUs dominate IoT and embedded systems. The embedded processor market is projected to grow from USD 19.36 billion (2019) to USD 32.53 billion (2028) [2], while the IoT market is expected to grow from USD 98.06 billion to USD 336.64 billion within five years [3], with many devices relying solely on CPUs for cost-effective, power-efficient applications.

1. TensorFlow GitHub Issue #48553: https://github.com/tensorflow/tensorflow/issues/48553
2. Allied Market Research: https://www.alliedmarketresearch.com/embedded-processor-market
3. Mordor Intelligence: https://www.mordorintelligence.com/industry-reports/iot-devices-market

W3: We appreciate the suggestion to incorporate the operator reordering benchmark tool. However, after attempting to apply it to models, we found several limitations:

• Unsupported Operations: The tool does not support all operations in our models, leading to errors (e.g., KeyError).
• Single Output Limitation: The tool assumes single-output operations, making it incompatible with our models, which involve multi-branch architectures and multiple output nodes (e.g., AssertionError).

Additionally, our work focuses on accelerating inference through branch-aware memory allocation, not on memory optimization. By isolating parallel branches, we minimize interference and enable efficient multithreading. As shown in Table 2, while our memory usage is slightly higher than TFLite’s optimized runtime, this trade-off achieves faster inference and higher energy efficiency for models with dynamic control flows.

Q1: BAP improves cache performance by isolating memory arenas for parallel branches, reducing contention and ensuring efficient memory reuse. We acknowledge that "cache locality" may not fully capture this mechanism. The abstract has been refined to:
"...reducing contention and optimizing memory reuse within each branch."

Q2: We thank the reviewer for this insightful question. The percentage of parallelized operators varies by model: MobileViT (small or XS) achieves 17.93%, Whisper 61.43%, and Conformer 8.46%. Table 1 already shows the number of parallelized layers, and we have updated it to include these percentages for clarity.

Q3: Thank you for the question. The quantified benefit is shown in Figure 2, demonstrating overall latency improvements, and in Table 3, where parallelizable layers like Layer 1 in Whisper (67.7%) and Layer 2 in MobileViT-XS (58.1%) achieve significant reductions. These results highlight the efficiency of parallel execution in multi-branch layers, and the updated Table 1 includes these values.

Q4: Thank you for your suggestion. To clarify how BAP addresses workload imbalance, we will revise the explanation to describe the process in two steps:

• Workload Quantification: During graph structure analysis, we approximate each branch's workload by counting operations. Lightweight operations (e.g., reshape) are assigned a weight of 0.5, while heavier operations (e.g., dense layers, convolutions) are assigned a weight of 1. This provides a quantified workload for scheduling.

• Scheduling: Based on the quantified workload, branches with balanced workloads are grouped for parallel execution. Heavier or lighter branches are scheduled sequentially, either before or after the parallel branches, depending on their position in the computation graph. This approach ensures efficient execution without bottlenecks.

Q5: Thank you for highlighting asymmetric core designs in mobile CPUs. While BAP does not currently account for core heterogeneity, future work could assign tasks to core types (e.g., big cores for heavy workloads, LITTLE cores for light ones) to further optimize energy and performance. We appreciate this insightful suggestion and plan to explore this avenue in future iterations of our method.

Dear Reviewer rvAe,

As the discussion period is nearing its end, we wanted to check if there are any follow-up points requiring clarification kindly. We have thoroughly addressed all raised concerns in our response, including enhancing Figure 1, incorporating additional benchmarks and data, and refining key explanations throughout the manuscript. Furthermore, we conducted extensive experiments and updates, such as detailing parallelized operator percentages, latency improvements, and workload scheduling mechanisms, as reflected in Tables 1–3.

If there are no further questions or concerns regarding these updates, we would respectfully ask Reviewer rvAe to consider increasing the score to reflect these clarifications and improvements. Please let us know if further discussion would be helpful.

Best regards,

Authors