The paper addresses the challenge of on-device training for deep learning models, particularly focusing on the memory bottleneck caused by the storage of activation maps during backpropagation. The authors propose a method to compress activation maps using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and its tensor variant, High-Order Singular Value Decomposition (HOSVD).

This paper proposes a method to compress activation maps in deep neural networks using tensor decomposition techniques, specifically Singular Value Decomposition (SVD) and Higher Order SVD (HOSVD). The goal is to reduce memory requirements during backpropagation, enabling on-device learning for resource-constrained environments. The authors provide theoretical analysis of their method's impact on memory usage, computational complexity, and error bounds. They demonstrate the effectiveness of their approach through experiments on various tasks, architectures, and datasets.

[Editing to reflect my score increase from 6 to 7 after the author discussion phase.]

The authors tackle the problem of memory consumption due to needing to keep realized activation tensors available between their use in the forwards pass and backwards propagation in training neural networks. They propose the use of tensor decomposition, specifically SVD and higher-order SVD, to compress activation maps and show that the compressed activations can be used in backpropagation without first decompressing. An error analysis is performed, and experimental results confirm the hypothesis that significant compression can be applied while maintaining significant explained variance. To validate the method's success on real workloads, the authors apply it to image classification and semantic segmentation tasks, fine-tuning pre-trained models for both tasks, as well as training from scratch for image classification.

Tensor low-rank decomposition is used to compress the backpropagation process of full connection and convolution in neural networks. The main process is to decompose input activation tensors into Tucker structures using HOSVD algorithm and truncate subtensors at each mode. Experimental result find an efficient trade-off between the desired explained variance and compression rates.

First of all, we would like to thank you and express our appreciation for the time and effort you have invested in reviewing our work. We are especially grateful for the highlights regarding the novelty of the proposed method (HEKd, dqZf), the theoretical groundings (HEKd, dqZf, WgAz) and the experimental results showing its efficiency (f1cQ, HEKd). Below, we have provided detailed responses to your questions and concerns, together with the PDF Rebuttal file containing additional figures and a table. We look forward to further feedback during the discussion period. Thank you once again for your valuable input.

Note:

• Wi: Weakness number i
• Qi: Question number i

General answers

#1: Related works comparison (answer for Q3 of f1cQ and W5 of dqzf)

Regarding tensor decomposition, methods introduced in the related works section either focus on weight compression for accelerated inference and lighter networks [13, 50, 52], or on gradient compression for accelerated backpropagation [48] whereas our approach compresses activation maps for efficient and lightweight backpropagation, thus making them not directly comparable. In a similar fashion most current Activation Map Compression strategies focus on accelerated inference [10, 12, 22]. [9] apply their method for large GNN training. We considered this setup to be too far remote from our own experimental setup and we leave comparison with this work for future research. In the context of on-device learning, we compared our technique to the state-of-the-art method, Gradient Filtering [49]. Other methods introduced in this paragraph focus on efficient subgraph selection [23, 25, 33] which is orthogonal to our proposition and thus compatible for joint implementation. We leave this research area for future work.

#2: Discussion on results (answer for W3.1 - Q2 of HEKd and W4 of dqZf)

• Discussion: SVD is only slightly better than HOSVD in terms of accuracy but is significantly worse in terms of memory usage. Specifically, Figure 4 in our paper shows the Pareto curves for SVD, HOSVD, and Gradient Filter. It is clear that HOSVD has a significantly better Pareto curve. To be clearer, we will update Figure 4 by Figure 1 in the PDF Rebuttal file showing the full pareto curve of HOSVD and SVD.
• Explanation: HOSVD essentially performs SVD on each mode of a tensor. Specifically, applying HOSVD to a tensor $\mathcal{A} \in \mathbb{R}^{B \times C \times H \times W}$ involves performing SVD on four unfolded versions of $\mathcal{A}$ with sizes $B \times CHW$, $C \times BHW$, $H \times BCW$, and $W \times BCH$, respectively. In contrast, SVD is applied only to the first mode, which is a tensor of size $B \times CHW$ (lines 180 - 181 in our paper). Thus, with SVD, the explained variance threshold is applied to retain an $\varepsilon$ fraction of the information in the first mode, without affecting other modes, meaning that the three remaining modes correspond to the raw uncompressed activation. In comparison, HOSVD allows us to retain an $\varepsilon$ fraction of the information across all modes. This explains why, in the same context, SVD generally offers higher accuracy but is more memory-intensive, while HOSVD is less accurate but more memory-efficient.

#3 Compression overhead analysis (Answer for W2.1 - Q4 of HEKd and W1.1 of WgAz)

In section A3 in supplementary, we will add description for overhead as follows:

" A.3 Details of Overhead, Complexity and Speed-up Computations

Following the notation in our paper, we can compute the overhead, space complexity and the speed-up of HOSVD.

Overhead: During the forward pass, while Vanilla BP does not perform decomposition, HOSVD does. Therefore, the overhead of the training process is the computational complexity of HOSVD, which can be calculated as follows:

The computational complexity of SVD for a matrix of size $m \times n$ with $m \geq n$ is $\mathcal{O}_{\text{time}}(m^2n)$. HOSVD essentially performs SVD on each mode of the tensor. During the forward pass, at each convolutional layer, given the activation map of size $BCHW$, HOSVD involves performing SVD on four matrices of sizes $B\times CHW$, $C\times BHW$, $H \times BCW$, and $W\times BCH$. Therefore, the computational complexity for decomposition in the forward pass is:

Space complexity: (refer to $complexity$ - formula (23) in our paper)

Speedup: (refer to $speedup$ - formula (24) in our paper) "

Additionally, in Figure 2 in our paper we will add another sub-figure showing the predicted overhead as shown in Figure 3c of the PDF Rebuttal file.

#4: Details on rank selection (answer for W3 - Q3 of dqZf and W1.2 of WgAz)

We do not focus on rank but rather on how much information is retained after projecting tensors into a lower rank space, and we do that by manipulating the explained variance threshold $\varepsilon$, which is 0.8 or 0.9 in our experiments. We are the first method allowing for explicit control over the information loss. In lines 185 - 187, we wrote "The larger the explained variance, the closer $\tilde{A}$ will be to $A$, intuitively allowing for better estimation when performing backpropagation." We use the explained variance threshold $\varepsilon$ to control how much information can be retained after compression. This value goes from 0 (loss of all information) to 1 (retention of all information). Moreover, as shown in Figure 2c in our paper and Figure 2d in the PDF Rebuttal file , as $\varepsilon$ is closer to 1, $SNR$ becomes bigger resulting in improved performance.

Dear reviewers,

Authors submitted rebuttals, which should be visible to you now.

Please read the rebuttals carefully and start discussions with the authors now.

The reviewer-author discussion period will end on August 13, 2024. Since authors usually need time to prepare for their responses, your quickest response would be very appreciated.

In case you had requested additional experiments / analysis and the authors provided in the rebuttal, please pay extra attention to the results.

Thank you,
Your AC

The authors submitted rebuttals, and all the reviewers here acknowledged them and had discussions with the authors. I requested clarifications about their comments, and some of them responded. Variance of the ratings was somewhat high as one of the reviewers (Reviewer HEKd) raised their score to 7, following multiple discussions with the authors. As they suggested, this is an interesting tensor decomposition research with great clarity and writing in general. They also value originality and its potential impact and interest of data center users.

I should clarify that I decided to put my focus on Reviewers HEKd, WgAz, and dqZf as I found none of the weaknesses claimed by Reviewer f1cQ's concrete, but more like questions. The reviewer argued that they had found authors' rebuttal satisfactory, but didn't provide any justifications (including their initial review) to support the rating.

Reviewer WgAz gave the lowest rating among the four reviewers, and key weaknesses they claimed in the initial review and follow-up are

1. "why not search for a more appropriate model architecture using NAS?"
2. "Rebuttal was not convincing because the experiments are on small datasets with small models. [1] is more efficient and doesn't affect model performance ".

For 1, I saw that the reviewer pushed NAS through interactions, but NAS is a resource-intensive approach. This work discusses on-device learning, and I believe that not using NAS cannot be a weakness.

For 2, I agree with the reviewer to the point in general, but again this work is about on-device learning. I also checked [1], but it doesn't discuss model performance (accuracy) but FLOPs, latency, and memory consumption.

Reviewer dqZf pointed out multiple weaknesses of this work, and the authors rebuttal addressed their concerns. But the reviewer didn't explain rationale of the rating 4. I requested a follow-up from the reviewer, but the reviewer did not answer my question. Having reviewed the authors rebuttal, I found the rebuttal satisfactory. E.g. (1), "Severe performance degradation relative to memory reduction" should be observed as part of trade-off discussions, which also shows better trade-off by the proposed method than those of the baselines. E.g. (2), "Limited comparison with other activation map compression techniques" <- The authors provided additional experimental results using Swin Transformer model (a significantly different architecture) in the supplementary material, which seems like a reasonable choice to me. Overall, it seems to me that the reviewer is looking for reasons to reject.

Considering all these points, I found my position literally on on fence, but towards acceptance. My recommendation is based on the fact that the review by Reviewer HEKd shared very concrete suggestions and relevant concerns, and the authors successfully addressed the concerns through the discussions, thus I would like to recommend acceptance for this work.

[Suggestions] While my recommendation is acceptance, I still highly recommend that the authors take into account all review comments and address their concerns in the camera-ready (e.g., clarifying motivation of this work).

• Replace "AI model" with "deep learning model". AI is not the right term in this scientific paper, but just a distraction.
• My recommendation is partly conditioned on the authors' code release.