In defense of parameter sharing for model-compression

We appreciate your time and efforts to evaluate our paper. We incorporate your suggestions in the updated version and address other concerns below.

"Is pruning the correct method to reduce model parameters?" What does correct mean?

We are raising the question that, given an architecture and a parameter memory budget whether "pruning" is the best method to reduce parameters in the model to achieve the memory budget. We have reworded the sentence to read, "Is pruning the optimal method to reduce model parameters?", where optimality is measured in terms of model memory and quality trade-off.

what does pruning beyond dx mean?

Pruning beyond dx implies we are compressing more than a factor of $d\times$ ( where 2x means reducing the model to half the parameters ).

why is not it obvious that pruning reduces model capacity

The model capacity is reduced with any model compression technique. We argue that pruning affects the model capacity more adversely than other methods, such as RPS. The embedding table is a perfect example because compression with pruning leads to degenerate embeddings, which will not happen with RPS, even at very high compression. We have also added datasets in experiments on compressing embedding tables, corroborating this argument.

why pruning baselines, why not other baselines such as low-rank, distillation, quantization, etc? In this paper, we talk about model compression of a given architecture at the start of training. Pruning is the most advanced and well-studied baseline in this setting.

1. Low-Rank: Low-rank methods for model compression are inferior to global-mask pruning. We have added a low-rank baseline to our new results on graphs to show this.

2. Distillation: In this setting, distillation is not the correct baseline as it needs first to train a big model and then use it to train smaller models.

3. Quantization: Quantization is popularly applied in model compression as either post-quantization or training-aware quantization - both of which are not compression-at-start techniques. To apply quantization at the start, we can train low-precision models, which can be used in conjunction with other techniques, such as pruning and RPS. Also, the compression range of quantization is very limited.

Diversity of Datasets and models for evaluation

1. We added more datasets from different domains : We understand the concern from reviewers about restricted domains of evaluation. To provide more diverse evidence, we have added two more domains of "Graphs with Attention-based Networks" and "Deep learning Recommendation Models (DLRM) " in Appendix H. The DLRM models (~540M parameters) and the associated dataset are relatively large (40M samples). The observations are consistent in that RPS outperforms other methods. In fact, pruning is especially bad in DLRM models due to the pruning of embedding tables. Pruning of embedding tables at high compression (beyond d\times ) will give degenerate embeddings.

2. Why not train vit on imagenet? Creating a single data point by training a vit model on imagenet needs 300 GPU hours if using A100 GPUs. We do not have access to the kind of computing it would need to create Pareto curves for this experiment in a reasonable time.

How does STABLE-RPS compare with neural architecture search-based methods? Can RPS still outperform search methods in different compression settings?

At this stage, we cannot compare the two methods. NAS finds the best architecture given standard modules. STABLE-RPS and pruning methods find the reduced parameters for a given architecture. From works on pruning, it is clear that once a model is decided, it can be sparsified further and still keep matching accuracies in some cases. However, we still need to choose a good model to begin with, which is where NAS is useful. In the future, NAS can be combined with methods such as STABLE-RPS to obtain even better architectures. But, this direction is out of the scope of the current paper.

How does the proposed method affect latency? Why does Theorem 3.2 involve cache line fetches?

In theorem 3, we proved STABLE-RPS introduces minimal cache-inefficiency over ROAST mapping. We add latency numbers to the appendix section G. We measure the training time for different-sized matrices using different mappings in these experiments. We observe that STABLE-RPS mapping introduces a minimal increase in the training time of the model over ROAST.

Thank you for the reply. The rebuttal does not convince me for the following reasons:

1. The authors revised the problem formulation as "Is pruning the optimal method to reduce model parameters?" However, this problem statement seems to be made based on the assumption that pruning is the optimal method for reducing model parameters, which is a false premise. To the reviewer's knowledge, there is few work that claims that pruning is the optimal method for compressing DNN models. In fact, many existing works have demonstrated that there are alternative compression methods that can achieve better capacity-vs-accuracy than pruning, e.g., several existing works have demonstrated that LLMs are better compressed with quantization, as https://arxiv.org/abs/2210.17323. The reviewer is happy to be convinced if the authors can provide concrete evidence that shows pruning is the optimal method for reducing the model parameters.

2. The authors claim that "We argue that pruning affects the model capacity more adversely than other methods, such as RPS.". However, this does not seem to make logical sense. Even if existing pruning methods perform empirically worse than RPS, it does not prove convincingly that pruning will in theory lead to worse accuracy than the parameter-sharing method or parameter-sharing method will lead to better pareto optimality, e.g., it seems possible that the model architectures, the datasets, and the training methods used for the tested pruning methods are sub-optimal. The authors do not seem to convincingly exclude the possibility of those factors.

3. "In this paper, we talk about model compression of a given architecture at the start of training. Pruning is the most advanced and well-studied baseline in this setting." The authors might be referring to zero-shot or single-shot pruning, where pruning is done at the beginning. While this is an interesting line of research, to the reviewer's knowledge, few work has shown that this scheme can lead to better capacity-vs-accuracy results than post-training compression methods or real training speedups. Therefore, the reviewer is not fully convinced that this is a method model scientists can benefit from in practice. Therefore, showing improvements over this baseline limits the generalizability and applicability of the proposed method in practice. Would appreciate it if the authors could provide examples where real-world models are trained when pruning is applied at the beginning of training.

Thank you for identifying the thoroughness of theory and experiments and the value and soundness of the proposal. Your comment on the work's overall quality, motivation, and clarity is encouraging.

We have incorporated many of your suggestions, as we describe below:  

1. Additional Experimental validation

   1. We have added more datasets. : We understand the concern from reviewers about restricted domains of evaluation. To provide more diverse evidence, we have added two more domains of "Graphs with Attention-based Networks" and "Deep learning Recommendation Models (DLRM) " in Appendix H. Specifically, DLRM experiments are relatively large, with the dataset having 40M samples and the model having 540M parameters. The observations are consistent in that RPS outperforms other methods.
   2. Why not train larger models/datasets such as vit on imagenet? Another reviewer suggested training vit on imagenet. Creating a single data point by training a vit model on imagenet needs 300 GPU hours if using A100 GPUs. We do not have access to the kind of computing it would need to create Pareto curves for this experiment in a reasonable time.

2. Computation overhead in the proposed STABLE-RPS method compared to other methods like ROAST

In theorem 3, we proved STABLE-RPS introduces minimal cache-inefficiency over ROAST mapping. We add latency numbers to the appendix section G. We measure the training time for different-sized linear layers using different mappings in these experiments. We observe that STABLE-RPS mapping introduces a minimal increase in the training time of the model over ROAST.

3. Some discussion around what are there any specific scenarios where the proposed method might not perform well?

The effectiveness of RPS standalone will depend on how much redundancy exists inside the model given a specific task. For some models, it might be very effective (for example, see the newly added embedding table compression in DLRMs example). In contrast, for some tasks, it will show degradation with compression like the vision tasks we show. Compared to methods such as pruning, as pointed out by theory and empirical validation, RPS is superior concerning the memory-accuracy tradeoff.

Thank you for identifying the thoroughness in theory and experiments, and the value and soundness of the proposal. We would like to let you know that we have added more diverse datasets, latency experiments and validation of theorems to further improve our paper. (details of which are in official comment "Summary of Rebuttal")

We thank you for your encouraging support of the paper.

Yes Quantization can be used in conjunction with other methods including RPS and Pruning to obtain further compression.

Thank you for identifying the thoroughness in theory and experiments, and the value and soundness of the proposal. We have incorporated your suggestions into the updated version of the paper.

Experimental verification for theorems

Thanks for the great suggestion. We have added empirical verification for theorems in the appendix F. As you can see, the theoretical predictions and empirical computations match in all presented cases.

Add parameters to the figures

We have added parameter counts to both existing and newly added figures.

Would it be possible to explore this method on a contemporary large-scale model?

1. We added more datasets from diverse domains: We understand the concern from reviewers about restricted domains of evaluation. To provide more diverse evidence, we have added two more domains of "Graphs with Attention-based Networks" and "Deep learning Recommendation Models (DLRM) " in Appendix "MORE DOMAINS." The DLRM models (~540M parameters) and the associated dataset are relatively large (40M samples). The observations are consistent in that RPS outperforms other methods. In fact, pruning is especially bad in DLRM models due to the pruning of embedding tables which is expected since pruning of embedding tables at high compression (beyond d\times ) will give degenerate embeddings.

2. Why not train larger models/datasets such as vit on imagenet? Another reviewer suggested training vit on imagenet. Creating a single data point by training a vit model on imagenet needs 300 GPU hours if using A100 GPUs. We do not have access to the kind of computing it would need to create Pareto curves for this experiment in a reasonable time.

Limitation of random parameter sharing methods versus pruning methods, When would RPS really be competitive?

We mention the current limitations of RPS in the section "Discussions and conclusion," Reducing the number of flops in RPS is currently an open problem. We will also mention it in the introduction to avoid any confusion.

There are many situations in which RPS can be potentially beneficial right away.

1. Embedding heavy models such as DLRMs. We added new results comparing pruning and RPS, and pruning is terrible when compressing embedding tables. As embedding tables do not have any flops, RPS is equally efficient in computation.

2. MoE models. A mixture of expert models is generally memory-heavy, and their computation is sparsified by design.

3. Other scenarios exist where higher compression can help us eliminate model parallel execution, which can have latency advantages.

Having said that, this paper focuses on memory-accuracy tradeoff compression techniques. By establishing this, we want to argue for further investigation of RPS to solve existing open problems.

For the same parameter budget there are now many network architectures that perform much better. In other words, where would MobileNet or EfficientNet be placed on Figure 3?

Given a memory budget, finding the best architecture is a challenging problem and comes under NAS. At this stage, we cannot compare RPS on one architecture with other architectures (alternatively, we cannot compare the RPS and NAS) . NAS finds the best architecture given standard modules. STABLE-RPS and pruning methods find the reduced parameters for a given architecture. From works on pruning, it is clear that once a model is decided, it can be sparsified further and still keep matching accuracies in some cases. However, we still need to choose a good model to begin with, which is where NAS is useful. In the future, NAS can be combined with methods such as STABLE-RPS to obtain even better architectures. But, this direction is out of the scope of the current paper.

Unfortunately, as the compression ratio increases, the network quickly enters regions where the accuracy is not worthwhile.

In practical scenarios, what tradeoff benefits a particular use case depends on the existing resource constraints and quality requirements. Thus, it is impossible to comment on whether RPS will solve the issue to satisfaction. But, in this paper, we want to rigorously compare RPS and pruning paradigms to see which method is better, given the resource constraints, and we find that RPS has better expressivity than pruned models at given memory budget.

Writing Issues

1. We have rewritten the background on parameter sharing to improve the readability of the section with a focus on the ROAST method. In the appendix, we can add details on other methods, such as ROBE and Hashednet, if needed.

2. The colors used in the full model denote the different ROAST chunks in the model, and the color in the linear array shows one instance of the mapping -- mapping of one chunk of ROAST or one fold of STABLE-RPS into the array showing which parameter goes where.