We thank the reviewer for the positive feedback and suggestions for improvement. We respond to individual points from your review below.

It would be helpful to have experimental results for long sequence lengths. 1024 and 3072 sequence lengths are too short compared to what SOTA LLMs can handle.

Thanks for pointing out this. Theoretically, increasing the sequence lengths should not significantly affect our conclusions, as our methods are flexible with different F/B/W timings. On the contrary, increasing the sequence length will increase memory pressure, highlighting the advantages of our memory-saving schedules.

We added another set of grid-search experiment using almost the same setup but with sequence length=16384. Here is a summary of experiment result. In this setup, 1F1B, ZBV and ZB1P run OOM for all combination of distributed training parameters (dp/tp/pp degrees). The experiment shows V-Half is 38% faster than 1F1B-R, which is the only baseline method that can run.

One of the problems with pipeline parallelism is that it increases implementation complexity because of its multi-program-multi-data (MPMD) nature. Do V-shaped abstractions bring any convenience in terms of implementation perspective?

Thanks for discussing this question from the implementation perspective. We believe that implementing pipeline parallelism primarily involves two components: a) a scheduler that determines the order of passes, and b) a runtime that executes the schedule.

Regarding the scheduler, we believe the V-Shaped abstractions simplify the implementation. Developers only need to implement the building blocks, and we can use a framework to handle automated repeating, squeezing, and reordering. This approach is exactly how we implemented the schedules in our experiments.

For the runtime, implementing a pipeline parallelism (PP) runtime is indeed more complex. It requires careful insertion of communications between stages to avoid performance degradation (e.g., stalled send/recv operations) or deadlocks. Note that this complication is inherent to PP and not introduced by V-shape schedules. To address this, we built a uniform runtime that automatically inserts and fuses (or refrains from fusing when better) communication operations according to the schedule produced by the scheduler. This allows the scheduler to focus solely on the ordering of computation passes. We will also open-source our PP runtime in hopes that it will benefit the community.

We thank the reviewer for the valuable feedback and suggestions for improvement. We respond to individual points from your review below.

There are quite some typos, some of which are listed here: Line 22 "tensor" -> "Tensor"; Line 189, "V-Min" -> "V-ZB" (?); Line 193, "serious" -> "series". Line 210, "while other methods' memory is similar": one exception is 1F1B-R, whose activation memory is much smaller.

Thanks for the careful reading. We have corrected them in the revised version. We also used a grammar tool to check the whole paper and correct the grammar errors.

A high-level question that might be worth a brief discussion in the manuscript is: is it true that a pipeline schedule must be repetition of a building block? Actually the answer is no. A naive example I can think of is a variant of GPipe, where the order of backward passes can be arbitrary, and thus the building block for one microbatch can be different from that of another microbatch. This extension of GPipe is designed just out of theoretical curiosity though. So the next question is whether there could be any real advantage (in terms of peak memory, throughput, etc.) in a pipeline schedule that cannot be expressed as repetition of a building block. It would be great if there is a clear answer with a formal proof. But even if the authors do not know the answer yet (and neither do I), it might be worth mentioning, so that anyone who has read this work will keep this in mind, rather than blindly constrain themselves to repeating a building block when designing new schedules.

This is very true, thank you for bringing up this insightful point. We add a sentence "Notice that repeating a building block is not the only way of building a pipeline, other methods like greedy search could generate a pipeline that has no repeating patterns." in Conclusion.

Some more discussions on the points that may favor repeating building blocks.

1. Repeating the building blocks means uniform lifespans, and intuitively it is better than unbalanced lifespans where a single point could cause a larger peak memory.
2. Mentally more tractable, the complexity of designing a "good" pipeline is reduced to designing a "good" building block.
3. Additionally, the scalability with respect to the number of microbatches is guaranteed.

We thank the reviewer for the valuable feedback and insightful suggestions for improvement. We have revised the manuscript to address most of the mentioned issues. We respond to individual points from your review below.

grammatical errors, a lack of useful captions in figures, Some plots are hard to read.

• We have used the grammar tool to check and correct the grammar errors.
• We add captions for most of the figures to summarize the key points or conclusions, explain the notations in Figure 3 and Table 1.
• We re-organize the layout of figures to make it more friendly to readers.

a lack of substantive discussion in some of the appendices.

Could you provide specific pointers?

A lot could be done to improve clarity of the discussion in section 3... the effect in low-d situations...

To make it more clear, we rewrote the Section 3.1 to include more technical details in the main paper (refer to global response). Specifically,

• We use uniform offsets between adjacent F/B passes across different devices, and leave those within the same devices flexible. This is where the asymptotic behavior comes from.
• The exact peak memory is $\lceil \frac{d+2}{3} \rceil \frac{M}{d}$ for V-Min, and $\lceil \frac{d+1}{2} \rceil \frac{M}{d}$ for V-Half. Actually we intended to mention it in the paper, somehow we forgot to do so, thanks for pointing this out.

not clear the shown pipeline is the actual setting or a high-level demonstration of the design.

The pipeline shown in Figure 4 is the actual setting with 4 devices and 8 microbatches. (Refer to the PDF in global response)

V-Half seems to be a heuristic method... It may not be an optimal... Other configurations... are not evaluated.

V-Half is not heuristically designed, all schedules are searched systematically with designated memory limit. There are indeed some other configurations from the search, e.g. a pipeline with 2/3 memory and 1/3 bubbles of 1F1B. As V-Min and V-ZB are optimal in memory and throughput, V-Half is to represent something in the middle. For the trade-off of memory and throughput, please refer to Figure 9 and 10 in the Appendix.

Section 4.4 is confusing. Table 3 is not referenced anywhere... (Table 6) do not clearly show...

It's a mistake due to duplicated label name in latex, it is fixed now, thanks for spotting it.

Why is 1F1B the baseline used for the paper? It feels like there could be some more discussion here that is missing.

Instead of saying using 1F1B as a baseline, we are more of using it as a reference point. The memory consumption of other methods are normalized to a proportion of 1F1B, simply because 1F1B is the most well known schedule that readers should be familiar with.

Is the lack of bubble in V-Half purely empirical or ... expected? ... Also, V-ZB has no bubbles ... differing runtimes, ... so maybe it is more accurate to say that the number of bubbles decreases to 0 as activation memory budget increases?

V-Half is free of repeating bubbles (in most cases), while it is not free of warmup/cooldown bubbles. As discussed in Appendix E.1, repeating bubbles occur in V-Half only when $T_W < 2\max(T_F, T_B) - 2 \min(T_F, T_B)$.

Generally, longer lifespan means looser dependency chain, making the schedule more robust to the variation of running times, and more friendly to overlap communication and computation. That's the intuitive reason why V-Min is vulnerable (minimal lifespan means tightest dependencies) and V-Half is more resilient. While at the memory budget of V-ZB, it is free of both types of bubbles.

How is the number of stages in the pipeline determined? ... what is the number of stages used in this work? How does the number of stages influence performance?

For a balanced computation load, each device is expected to host equal number of stages. That's why strategies like interleaved-1F1B use a stage number which is a multiple of the device number.

For a balanced peak memory, we need the total lifespan to be equal on each device. As shown in our paper, it can be achieved using the "V-Shape" schedule, where the number of stages is twice that of the device. Repeating the "V-Shape" into a "W-Shape" would double this ratio, and would be memory balanced as well, but it does not further decrease the memory, and on the throughput it improves at the rate of $\frac{2+V}{V}2d$ (for V-Min) where $V$ is the number of stages per device, which is not very significant. Therefore we didn't bring this extra complexity to readers in the paper.

Given the constraints of memory ..., do the authors use grid search to find an optimal schedule? Is it possible to ... quickly find a good pipeline schedule?

We elaborate a simple search approach to quickly find the optimal schedule given the memory constraint (the number of possible building blocks is $O(1)$) in Section 3.1, and introduce an adaptive scheduler with finer control granularity (the number of possible building blocks is $O(d)$). We also evaluate its trade-off between memory and bubble in Figure 9 and 10 (Appendix B).

Thank you for the rebuttal and certain rewrites. I will keep my score.