Maestro: Uncovering Low-Rank Structures via Trainable Decomposition

We thank the reviewer for their feedback and appreciate their comments. Let us reply point-by-point.

Wrt scaling to the ILSVRC results, we have conducted experiments on ResNet-50 trained on ImageNet and showcase how our method performs. However, due to unforeseen technical difficulties with the filesystem of our GPU infrastructure, we only have preliminary results, up to epoch 60. We will continue the training process and can update the manuscript upon termination. For what it's worth, current results show similar convergence dynamics with Pufferfish, with training reaching 70.05% of accuracy at 47% of MACs and 32% of the original parameters. Moreover, we have results for TinyImageNet and Multi-30K, both of which should hint the scaling ability of our technique. Last, while we did our best to have the results in time, we would like to point out that similar works have been published in ICLR without scaling their evaluation to ImageNet levels (Khodak et al., ICLR'21).

Moreover, we did not have the time to conduct extra training for ViTs on ImageNet due to limited time and resources. Since the ResNet-50 on ImageNet experiment was also requested by Reviewer ZdCx, we have prioritised this. Nevertheless, we have had no indication of scalability issues so far, therefore, we anticipate similar behaviour.

Wrt GMACs, they correspond to inference cost on Tables 2-4. However, we have quantified the training cost in GMACs on Tables 11-14 of the appendix. We have now clarified that in the manuscript and included a pointer to the appendix. We hope that these answer the reviewer's raised points and can raise their score accordingly.

[1] Khodak, M., Tenenholtz, N., Mackey, L., & Fusi, N. (2021). Initialization and regularization of factorized neural layers. ICLR'21.

We would like to thank the reviewer for their valuable feedback and insightful comments.

In the new manuscript, we have fixed all of the indicated typos and have further changed Figure 1 to better illustrate the inner workings of Maestro without the need for prior knowledge of OD.

As requested, the current version of Figure 1 is readable even when printed in black/white. Regarding the various DNN layers, we do not specifically visualize those as their implementation is equivalent to linear layers, as discussed in section 3.2, where the main trick is that we can represent these types of layers as a special case of linear mapping that we then decompose using our technique. If the reviewer believes that our presentation would benefit from further visualization of CNNs, RNNs, and Transformers, we are happy to include such figures in the camera-ready version.

We appreciate both of these suggestions and believe that they ameliorate the quality and legibility of our manuscript.

We would like to thank the reviewer for their time and effort in reviewing our manuscript.

First, we have fixed the indicated typos. Thank you for raising these.

Please allow us to reply to the rest of the reviewer's concerns below:

The only novelty seems to be the use of ordered dropout

We appreciate the reviewer's perspective, but we do not think that this renders our contribution as "marginal." We have non-trivially extended ordered dropout to low-rank decompositions and have further allowed for non-uniform ranks across layers, contrary to FjORD. This has been acknowledged and mentioned as a strong point by Reviewer ZdCx. Moreover, we have used the lasso penalty specifically to optimally consolidate knowledge so that the network can leverage this structure and more efficiently compress the model. As a result, we believe this is a significant step in this area of research.

The proposed method is obviously very sensitive to the choice of the Lasso coefficient lambda, but there is no theory behind how it can be chosen effectively.

Thank you for raising this important point. We acknowledge that while our method can operate with various values of the Lasso coefficient lambda, there is indeed a degree of sensitivity to its selection. Our experiments have shown that Maestro can function effectively even when the Lasso coefficient is absent (set to zero), demonstrating the flexibility of our method. However, we do agree that the choice of lambda can influence the performance and efficiency of the method to some extent. To address this, we discuss some guidelines for selecting an appropriate lambda value. We provide an empirical way for hyperparameter tuning.

Specifically, one can select the value of $\lambda_{gl}$ by starting with a large lambda value and gradually decrease the Lasso coefficient by a factor of 2 until the required criteria (footprint, accuracy) are met. This heuristic finds a close-to-optimal value for the hyperparameter, while the total cost of training remains comparable below the baseline of training the whole model. Please see Tables 11-14 for our lambda sweeps and the optimal values per dataset. The theoretical underpinning behind the selection of $\lambda_{gl}$ is that we want the loss component that relates to hierarchical lasso to be lower than the learning component, so that it does not dominate the learning objective, but not too low that it does not sparsify the network.

Nevertheless, it is important to highlight that Maestro does not specifically require the group Lasso penalty for operation; we utilize it primarily to enhance training efficiency. As demonstrated in Figure 4, we can implement post-training pruning (or even during training, which is not currently shown in our manuscript) even when the Lasso coefficient is set to zero, and still achieve competitive performance across a wide range of computational and memory constraints due to the nature of ordered dropout. Please let us know whether we have sufficiently answered your concern.

How is the initial factorized mapping performed without SVD? How is the initial maximal rank r chosen?

In our manuscript, we describe initializing our network using standard neural network initialization techniques, followed by factorization using SVD to obtain the Maestro layer. Additionally, we explore an alternative approach where each layer is pre-factorized as illustrated in Figure 1, with each matrix initialized using standard neural network initialization methods (e.g., the default initialization in PyTorch). Both approaches lead to the same final results.vWe decided to use SVD as the default initialization method for Maestro in future tasks that we have in mind. For instance, this is particularly relevant when starting with pre-trained model weights. By decomposing weights using SVD, the initial output of the full network remains unchanged. Moreover, SVD can be viewed as an approximation to Maestro that only considers current weights and is computationally inexpensive to compute once, while Maestro also accounts for both data and the full network dynamics.

Regarding the initial maximal rank, for a linear mapping from $\mathbb{R}^m$ to $\mathbb{R}^n$, we set $r = \min$ {$m, n$}. This ensures that our network initially has the same capacity as the original network.

We would like to thank the reviewer for their valuable feedback and time invested in reviewing our manuscript.

Wrt the manuscript presentation, we have now changed the citation style to conform to the ICLR style.

Wrt the title, we did not feel that the name needed further explanation since Maestro does not constitute an abbreviation. However, to give some context to the reviewer, our interpretation of the name is that it coordinates the ranks that need to be used for training/inference of a model, much like a Maestro conducts an orchestra. (Like orchestrator components in distributed systems)

Wrt to running experiments on ImageNet to validate the scalability of our contribution, we have started training ResNet-50 on ImageNet. However, due to unforeseen technical difficulties with the filesystem of our GPU infrastructure, we only have preliminary results, up to epoch 60. We will continue the training process and can update the manuscript upon termination. For what it's worth, current results show similar convergence dynamics with Pufferfish, with training reaching 70.05% of accuracy at 47% of MACs and 32% of the original parameters. Moreover, we have results for TinyImageNet and Multi-30K, both of which should hint at the scaling ability of our technique. Last, while we did our best to have the results in time, we would like to point out that similar works have been published in ICLR without scaling their evaluation to ImageNet levels (Khodak et al., ICLR'21).

Last, we have positioned our work against the OTOv2 framework, as requested. Indicatively, OTOv2 involves a much more involved (and costly) training process because the HSPG solving step does not offer a method to trade off accuracy for model size further. In contrast, Maestro enables a more efficient training method along with flexible gains upon deployment without the need to retrain (one-shot). Last, wrt the mentioned figures, let us first note that we are comparing VGG-16 (OTOv2) vs. VGG-19 (Maestro). Be that as it may, our original accuracy of vanilla VGG-19 is 92.94% (vs. 93.2%). Compared to OTOv2's performance of 26.8% of MACs, 5.5% of parameters at +0.1 pp of accuracy, the closest operating points of ours are at:

• 33% of MACs, 11% of parameters at +0.16 pp of accuracy
• 20% of MACs, 6% of parameters at -0.24pp of accuracy

While competitive, we may not be beating this baseline on the end accuracy. However, we believe the benefits of training overhead and deployment flexibility make Maestro an advantageous approach in many cases.

[1] Khodak, M., Tenenholtz, N., Mackey, L., & Fusi, N. (2021). Initialization and regularization of factorized neural layers. ICLR'21.