Sparse Iso-FLOP Transformations for Maximizing Training Efficiency

While the experimental results are quite strong, the work itself lacks scientific novelty and insights. Sparsity models appear to improve upon dense models in terms of performance / FLOPs tradeoff, but the reason or at least intuition behind the advantage of sparse model is not clear. I think the work could greatly benefit if one showed that sparse models are more expressive than dense or have faster convergence. Even on a toy example.

Sparsity patterns in this work are adopted from the prior work and cannot be considered as novel.

1. This paper lacks some crucial ablation studies. While it asserts that, given a fixed FLOPs budget, we can enhance the accuracy of the current network capacity with Sparse-IFO, it's important to note an alternative approach, which involves pruning a larger dense model to the same FLOPs level and then training it with sparse training techniques. For instance, when considering the task of training a dense ResNet-18 on ImageNet within a specific FLOPs budget, this paper showcases how accuracy can be increased from 70.9% to 74.4% at a sparsity level of 0.9 using Sparse-IFT. However, an alternative strategy would be to prune a ResNet-50 or a Wide ResNet-18 to a certain sparsity level (e.g., maybe 80% sparsity) such that these pruned networks share the same FLOPs as the dense ResNet-18. Subsequently, we apply sparse training techniques to train this pruned networks. The comparison between the accuracy of the pruned ResNet-50/Wide-ResNet-18 and the scaled ResNet-18 could provide valuable insights, which could help the community understand both direction better.

2. A more in-depth exploration of GPU memory usage are preferred. The methods introduced in this paper, such as Sparse Wide, Sparse Parallel, and Sparse Factorized, have the capability to effectively double the dimensions of weight matrices, potentially increasing GPU memory demands during training. Thus, it is better to provide an analysis that elucidates the trade-off between memory utilization and varying sparsity levels.

3. There appears to be a pattern where the proposed method consistently yields more significant improvements on relatively smaller networks. For instance, when examining Table 7, at a sparsity level of 90%, we observe a substantial 3.5% improvement on ResNet-18 but only a modest 1.0% enhancement on BotNet-50. This raises the question of whether these findings hold true for larger and deeper networks. Ideally, the focus should be on demonstrating that increasing model capacity can push the limit of state-of-the-art models, rather than solely improving the accuracy of smaller models.

Overall, it is still very difficult for me to fully grasp the benefit of the proposed method over the existing techniques. In particular, here are some concerns:

• Novelty. It is quite difficult for me to clearly tell apart the proposed 'sparse-wide' method from the standard approach of pruning the over-parameterized model (at initialization). In fact, the main message of this paper---having a wider, sparse model is better than smaller, dense model---has already been confirmed at the classic paper of "To prune or not to prune" by Zhu & Gupta (2017).
• Baseline. Related to the previous question---if authors indeed think that the proposed method is any different from random pruning or performing pruning-at-initialization to a widened model, I strongly suggest adding these as a baseline and comparing the performances of such methods. Currently, there is no baseline method in all the tables, so it is really difficult to know the difference.
• Practical benefit. I am not sure these claimed iso-FLOP will indeed have a similar computational footprint, in terms of wall-clock latency or memory requirement. First, the widened model may have more number of activations, requiring more memory space to be loaded and trained. Second, sparse models often need more memory space to load their sparse weights (e.g., in CSR), and loading these through limited memory bandwidth incurs a lot of delay and power consumption. Third, there is no satisfactory hardware that can handle sparse operations with the same speed as iso-FLOP dense operations. Thus, I am not really sure if these can be really considered iso-computation.
• Training Speed Comparison. Sparse models are known to require more number of SGD steps to converge to the minimum (see, e.g., RigL). I wonder if this is true for the proposed iso-FLOP model.

My major concerns include three aspects:

1. The paper's contribution is limited, the only novel components are the four FLOP-invariant transformations. However, a similar idea of improving exploration in sparse training processes has been considered in existing works, e.g. [1][2].

[1] Jayakumar, Siddhant, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. "Top-kast: Top-k always sparse training." Advances in Neural Information Processing Systems 33 (2020): 20744-20754.

[2] Huang, Shaoyi, Bowen Lei, Dongkuan Xu, Hongwu Peng, Yue Sun, Mimi Xie, and Caiwen Ding. "Dynamic sparse training via balancing the exploration-exploitation trade-off." In 2023 60th ACM/IEEE Design Automation Conference (DAC), pp. 1-6. IEEE, 2023.

2. Although the authors claim training with sparse weights often results in accuracy degradation or necessitates prolonged training schedules to attain performance similar to the original dense models, the experiments of the paper are mainly focused on unstructured sparsity, which means the exact wallclock running time will be longer than dense models. Even the authors also conduct experiments on structured N:M sparsity. However, to the best of my knowledge, currently, the Nvidia Ampere GPU only supports acceleration on static 2:4 sparsity.

3. Besides, the performance of the model equipped with Sparse Iso-FLOP Transformations improvement compared to the dense model is also marginal.