Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Models

1. Poor writing. Some example: 1) Wrong citation format in contents, like “Alizadeh et al. Alizadeh et al. (2023)”; 2) “No Model Retraining or Parameter Modification” and “Simplified Implementation and Broad GPU Compatibility” are features, not contributions; 3) The first line of text on page 3 is indented incorrectly.

2. The evaluation on inference is unproper. This work claims the contribution on enabling small device to run large models for both training and inference. However, all the models used in inference part are smaller than the GPU memory. Can you either use larger model size or batch size to show the benefits of the proposed method on inference? Can you clarify whether LLMs or other models can be inferenced in parallel and be accelerated by the proposed method? If not, this paper should only claim the contribution on training.

3. From the training comparison results shown in Figure 2, the proposed method is much slower than standard GPU processing. Can you judge why it is deserved to train ViT-BigG slower to use less GPU memory? Is this method only beneficial to huge LLMs? If so, this paper should only claim the contribution on LLMs instead of “universal”.

4. The comparison of training is not fair. Can you add the training speed results in table 2 for fair comparison? How much slower is the proposed method than standard GPU processing? Based on the results, can you prove that the proposed method can outperform standard GPU training by processing more tokens at the same time on the same device? Can the conclusion be extended to multiple GPU system with distributed training (including pipeline parallelism, data parallelism, and tensor parallelism)?

5. Lack of innovation. To my knowledge, the key idea of swapping between GPU and CPU memory has already been proposed and researched in many works. The innovation and contribution of this work is a little weak to me.

The paper has several critical shortcomings that limit its contribution:

Lack of Comparison with State-of-the-Art Solutions: The evaluation section only compares Superpipeline to a baseline approach without benchmarking against more advanced or recent solutions. This omission is problematic, given that many existing methods—some of which are cited in the paper (e.g., FlexGen, PipeDream)—are designed to address similar memory challenges. Without such comparisons, it is difficult to assess whether Superpipeline offers any real advantage.

Use of a Trivial Cost Model: The memory management strategy, which involves swapping data between CPU and GPU based on a simple partitioning scheme, is neither new nor innovative. Similar strategies have been implemented in other works cited in the paper (e.g., GPipe, SuperNeurons). The proposed dynamic transfer mechanism offers no significant improvement beyond these established solutions, raising questions about the novelty of the approach.

Lack of Technical Depth: The paper does not provide any substantial innovation in the cost model for data transfer. The approach relies on well-known techniques without presenting any meaningful modifications or optimizations to improve performance. In summary, the work fails to demonstrate its novelty by omitting meaningful comparisons with existing methods and relying on a simplistic strategy that has been previously explored. These weaknesses undermine the paper’s claim of contributing a new solution to the problem of memory-constrained model execution.

• No new research contribution. Method is quite simple and well-known
• No design or implementation details are given
• Poor evaluation without comparison to other works

Thanks for submitting to ICLR. This work attempts to tackle the major problem of AI training/inference, the lack of GPU memory. It did a reasonable work on summarizing SOTA, but the extremely short description on the design (Section 3, less than a page) did not give any insights about the design and implementation details. From the very limited words, it seems the idea is quite simple: model parallelism with a three-stage pipeline to buffer some data at the CPU side. This is already well-known and did not actually solve the problem, as then the CPU-GPU communication becomes the bottleneck, and workload balance becomes another issue. This may also explain why authors did not compare their approach to any SOTA works exploiting model/pipeline parallelism. They mention generally applicability, but they also did not test on non-NVIDIA GPUs.

1. In models or hardware configurations where this transfer is slower, the performance gain over standard methods may be reduced
2. The tuning of k and k’ parameters add complexity and require initial experimentation. Can this process be automated selected, such as triton's @autotune method (As stated in Section 5)? What's the possible solution for the automated selection algorithm, can it be solved through Linear Programming?

1. Transfer Overhead: The performance impact due to GPU-CPU transfer time, particularly for specific models (e.g., ViT-bigG), limits the efficacy of Superpipeline in certain scenarios.
2. Scalability Concerns: The dependency on effective data transfer strategies (e.g., batch vs. sequential) could affect scalability, requiring additional customization for certain model or hardware setups.