MoteS: Memory Optimization via Fine-grained Scheduling for DNNs on Tiny Devices

(1) The effectiveness of the proposed model compression and learning strategy is closely tied to the processing capabilities of the underlying hardware. It is important to consider that not all commodity processing chips can support specific int8 operations or efficiently handle them. The true acceleration of sparse computing heavily relies on specific chips. It would be interesting to investigate whether the proposed methods can be generalized to other commodity devices, such as the NVIDIA Jetson or mobile phone neural chips, rather than being limited to the specific STM series.

(2) The experiments primarily focus on simple image classification with a limited number of training epochs, it would be valuable to assess the performance of the proposed methods on segmentation and detection tasks as well.

(3) Timely handling of the learning procedure is crucial. It is suggested to present the performance of training/inference speed or time cost when applying the proposed methods to downstream tasks. This would provide a more comprehensive understanding of the practical implications and efficiency of the approach.

1. The writing and presentation of this paper have huge room for improvement. Issues like incorrect figure references, grammatical errors, and lack of clarity in explaining key insights/definitions suggest the authors need to refine their work.

2. Operator/model partitioning during inference is not a novel idea. A referenced paper ((https://ieeexplore.ieee.org/document/9546452)) demonstrates such approach has been explored before in cloud/edge DNN settings.

3. Details of how operators actually execute during scheduling using the ACG are not fully explained.

(1) There seems to be a fundamental discrepancy between the parameters of the ACG graph (which is partitioned on the basis of peak memory usage) and the algorithm 1 objectives which is latency. There is a complex and unknown correlation between latency and peak memory usage, latency is affected by many other independent factors. Optimizing for peak memory usage is not the same as optimizing for latency. Thus many existing accelerators work on optimizing memory communication overhead i.e., memory traffic which has a direct correlation with latency. The approach followed in this paper towards optimizing latency through minimizing peak memory storage ignores many other factors. (i) Reducing peak storage by increasing the number of partitions could increase memory traffic and therefore latency. (ii) Other works look at local optimization criteria when considering optimal tile sizes (partitions) such as cache constraints, increased hit ratios etc. but none of that is considered in this paper. (iii) The authors estimate latency of partitions through a run-time simulator which also adds overhead and could be incorrect. (iv) What about the properties of the tensor, Memory usage is assumed the same whether it is a dense tensor or sparse tensor. Again existing works tile and do run time resource allocations based on sparsity.

(2) The definition of an Axis Connecting Graph is rather vague, it consists of a set of sufficient conditions. It is not clear how this definition actually allows you to get efficient partitions. It sems to be an exercise in definitions without an application. What, if any, is the rule for generating successively effective partitions using this definition of ACG?

(3) The paper seems to have been written hastily without proof reading, there are several critical omissions. For example, (i) · In Section 3, the authors reference Figure 2 (c) and Figure 2 (d) when introducing an example, but these figures are missing in the paper. This example is very crucial to understanding their concept of peak memory usage and memory bottlenecks but since it is missing one has to guess. (ii) The definition of memory bottleneck in Sec. 3 (bottom of page 3) is ambiguous and possibly incorrect. They seem to say that $start(i)$ is when $i-1$ is completed which can be assumed to be $end(i-1)$. However $end(i) = max_{j \in succesors(i)} j$, which is a recursive definition, i.e., i ends when its last successor ends which itself ends when its last successor ends and so on. Such inconsistent and badly presented notation makes it hard to make sense of the rest of the paper. This needs to be proof read and cleaned up.

(4) Authors state that existing methods propose only coarse grained scheduling, however many such methods are based on solid optimization criteria that exploit algorithmic substructures for reducing the optimization search space, not just heuristics, e.g,, dynamic programming [Ahn2020].

(5) Authors state that techniques like depth tiling [Stahl 2023] are just special cases of their approach, which is technically true since their heuristic can generate all possible partitions through parameter $n$. Unfortunately this partitioning algorithm is brute force exhaustive, there is no exploitation of substructure to drive the algorithm towards an optimal partitioning solution. The authors make no claim as to whether they can even find the optimal.

(6) Note that every set of contiguous elements of a tensor is a trivial partition, thus the total number of partitions can be combinatorially explosive, it is not clear how they can reduce the search space in an algorithmically efficient manner to find the optimal.

• The paper does not highlight limitations of the framework/approach and areas for future work in a detailed section, which can benefit the larger research community.
• While the latency improves compared to other approaches, it is still non trivial, and future work can be explored to reduce it even more.