MCNC: Manifold-Constrained Reparameterization for Neural Compression

We thank the reviewer for their time and consideration. Please find questions to your questions and comments below.

The benefits of achieving extreme compression rates under the high accuracy loss are not clear to me. Using table 1's result as an example, with 1% percentage of model size, the proposed approach can achieve an accuracy of 49% compared to the baselines which are 33%-39% on ViT-Ti model. However, compared to the baseline model accuracy 76.2%, it is a huge accuracy drop (more than 20%). Under this context, one might consider using a different model variant that is smaller but achieves a higher accuracy than compressing the ViT-Ti model. Although the proposed method can achieve better accuracy compared to baselines under high compression rates, I am concerned about how likely such advantages would be useful in practice due to the significant accuracy drop compared to the non-compressed or lightly-compressed model.

The accuracy loss of extreme compression rates are an understandable concern. We first highlight our results at 10% compression rate of ViT-Ti. Here, we show that we are able to outperform pruning methods with performance only dropping ~6% accuracy. While there is significant degradation at the more extreme compression rates, we show that we outperform pruning in this setting. Although training a smaller architecture appears to be a plausible alternative, prior work PRANC (ECCV 2023) showed that their method significantly outperforms a knowledge distilled LeNet (60k parameters) with a compressed ResNet-56 (5k parameters) on CIFAR-10 classification. In our experiments, we show that we out-perform PRANC in a similar setting.

Table 4 highlights the same concern. The trainable parameters from the LoRA model are already minimal compared to the number of parameters in the LLaMA-2 7B and 13B. With the quantization from NOLA and MCNC, both of them slow down the training throughput of the model. I agree with the advantages of MCNC compared to NOLA, but neither approaches are appealing compared to running the original LoRA method.

When compared to the number of parameters in LLaMA-2 7B and 13B, the number of LoRA parameters is indeed small. However, despite being relatively smaller, the parameter count for LoRA remains significant. For example, implementing LoRA at rank=1 for LLaMA-2 7B requires 2.5M parameters, and for LLaMA-2 13B, it requires 3.9M parameters. Typically, higher ranks are needed for effective PEFT with LoRA, which can pose challenges depending on the application. For instance, consider a scenario where an LLM must be personalized for each user of a system. To prevent private data leakage, a reasonable solution would involve fine-tuning the LLM independently for each user. Supporting 1,000 users on LLaMA-2 13B under this approach would demand an additional 3.9GB of storage (assuming rank=1). In contrast, MCNC would require only 77MB for the same use case.

Can the author clarify how does the benefit of the proposed approach over baselines translate to practical computational performance gains?

As MCNC directly generates model parameters, it does not provide performance improvements in terms of FLOPS. However, MCNC can significantly decrease a model's memory footprint. This reduction is particularly impactful in scenarios where a model is too large to fit within GPU memory, as the time required to transfer model parameters from CPU to GPU can dominate inference latency. This issue is already a significant bottleneck for LLMs. For instance, the recently proposed quantization method, SqueezeLLM[1], specifically targets memory optimization to improve inference latency rather than computation. Beyond LLMs, memory-constrained devices can benefit from running compressed versions of larger models. Additionally, applications that require hosting multiple models on a single GPU can leverage MCNC's improved CPU-to-GPU transfer times, as demonstrated in Table 9.

[1]: Kim et al., "SqueezeLLM: Dense-and-Sparse Quantization," arXiv, 2023. arXiv:2306.07629.

I appreciate the authors' comments and responses to my questions. However, my question regarding the extent of the benefits provided by the proposed approach still stands. The citation to SqueezeLLM is not particularly relevant in this context. Note that while SqueezeLLM focuses on compressing the LLM model itself, this work compresses the LoRA parameters within an LLM. The memory pressure caused by an LLM with 13 billion parameters is far more apparent than one caused by just a few million LoRA parameters, making the latter scenario less compelling by comparison. With the additional computation overheads from generating the parameters, it is not obvious that the approach will bring inference speedups, even if memory is a bottleneck.

With that said, I do see a compelling case when the approach is applied to the entire model with a high compression rate (e.g., ResNets, ViT). The compressed models can fit into resource-constrained devices with limited memory.

We thank the reviewer for their time and consideration. Below please see our answers to the raised questions.

the paper claims that a randomly initialized generator is enough to span the entire space of weights in a higher dimension, but never shows much empirical evidence beyond an experiment in low dimensions. It is not clear if this result also holds in higher dimensions. I would have ideally liked to see a theoretical result around the expressivity of the manifold constrained reparametrization, but even an empirical investigation showing that one does not lose out on expressivity would be great.

We first wish to clarify that we do not expect a randomly initialized generator to span the entire space of weights in a higher dimension. This is infeasible without assuming access to infinite precision values. Otherwise, we are inherently limited by the quantization of the input space. Instead, we claim that a randomly initialized generator is not significantly worse than a trained generator. We do agree, however, that we were not thorough in our evaluation of this claim. To further evaluate our claim, we provide additional comparisons between random initialization and trained generators for higher dimensions for $k=2$ below:

We compute the above using the sliced wasserstein distance with 100,000 samples and 10,000 slices. We use the sliced-Wasserstein distance instead of Wasserstein here due to its: 1) superior computational speed, and 2) better sample complexity. We use the same set of target samples and slices for each to ensure a fair comparison. We also include the distance between a random dirac distribution on the sphere as well as a separate set of 100,000 points sampled from the ground truth distribution to further contextualize the reported distances. Note that the decrease in distance as dimensionality increases is due to the likelihood of sampling an informative slice decreasing exponentially with respect to dimensionality. In addition to the sliced-wasserstein distance, we also report $100*\exp(-\tau SW_2^2(\hat{\mu}, \nu))$ where $\tau$ is set such that the value of the dirac distribution is less than 1e-2. Lastly, we emphasize that, due to the over-parameterization of neural networks, full coverage of high dimensional space might not be necessarily needed.

Continuing on my previous point, from table 10, it appears that a trained generator is better for larger models. However, this result is down-played in the text description, with the narrative claiming that the improvement is "marginal"

While a trained generator does show an improvement over our randomly initialized generator, it is our opinion that the degradation is not too severe given the additional benefits of using a random initialization. By using a random initialization, we are able to communicate/store the generator using only a random seed. If we were to use a trained generator instead, it would be necessary to either include the entire generator in the compression budget or assume the generator has been pre-installed in the downstream device. Additionally, a randomly initialized generator allows greater flexibility as it allows generators of varying compression rates to be easily generated on demand. However, we do agree that if an application can utilize a trained generator, it is preferable to do so.

How was the wasserstein distance computed for Fig 2? If I understand correctly, there is no closed form expression to compute this metric in higher dimensions.

You are correct that there is no closed-form expression for computing the Wasserstein distance between arbitrary distributions. In our approach, we randomly sample $N$ points from a uniform distribution over $[-L, L]$, pass them through the generator to produce samples from its output distribution, and compare these to $N$ points sampled uniformly from the sphere (obtained by sampling from a $d$-dimensional normal distribution and normalizing the samples). The Wasserstein distance between these two empirical distributions is then computed using linear programming. However, the computational complexity of this approach becomes prohibitive for large $N$ due to the cubic complexity of linear programming. Alternatively, given the improved sample efficiency of the sliced Wasserstein distance, we opt to report results using this metric, following the procedure outlined above.

I thank the authors for their response.

FLOP computations

The computation of FLOPs makes it clear as to exactly where the savings come for MCNC on top of NOLA. I would encourage the authors to add this to the appendix for better exposition.

Accuracy on ViT-S

I am curious as to what changed in the experiments to produce this version of the table. Here, it appears that MCNC is much better than all the baselines, and more interestingly, the 25% sized model produced by MCNC is also better than the 100% trained base model! I would appreciate any insights of the authors here.

Training Time

I appreciate that the method has only small overheads above LoRA despite strong performance and much smaller memory overhead.

We thank the reviewer for their time and comments. Below see our response to the raised questions and concerns.

In the exposition of the idea, the actual architecture of the generator network comes very late; with actual numbers/settings coming in appendix! I think, the simplicity of the approach must be showcased from the beginning, I highly suggest to add simple pytorch code showing how easy is to plug and play with the method

We thank the reviewer for this suggestion. We have added a short snippet of PyTorch code showing how to perform the reparameterization for a PyTorch Linear layer. Unfortunately, page constraints make it difficult to include this in the main paper so this has been added to the top of the appendix.

Similar to the previous weakness, I think authors should consider recommending some bulletproof settings that would work for most of the cases (like number of layer in, number of dimensions, some rules of thumb, how long to train, optimizer settings) to easy any potential adoption: at the end of the day, the ideas have very limited impact if not used in practice.

This is a good point. As we use the same generator configuration for all experiments when training from scratch, we have migrated these settings into a table so that readers can quickly find the recommended configuration. In this table, we also include recommendations for optimizer and learning rate.

Theoretical guarantees of any sorts would be welcome as well: how universal is such a generator network? can it be used to train/finetune any neural network or it has limitations?

This is an area of great interest to us as well. While we do not yet have concrete theoretical results, we believe there may be connections between our work and recent theoretical advancements on gradient descent with large learning rates[1]. By reparameterizing a network as the output of a sinusoid-activated generator, we constrain the weight space to a highly dynamic manifold. This allows small changes in the generator's inputs to induce substantial changes in the model's weights. This behavior suggests potential links between our approach and the theoretical insights into optimization dynamics with large learning rates. These connections and a more theoretical exploration of the MCNC framework will be addressed in future work.

While the papers literature review is extensive, I think the connection to hashing must be explored in depth and more papers included into review (some suggestions: Structured Multi-Hashing for Model Compression, Lossy weight encoding for deep neural network compression)

We thank the reviewer for drawing our attention to these works. We have added them to our related works section. As hashing methods are heavily related to weight sharing, we have added them to this section.

[1]: Cai et al., "Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization," arXiv, 2024. arXiv:2306.07629.