Concerns about the integrity of the evaluation.

1. Fig.2: The shapes of Fig. 2a and 2c seem identical for two vastly different models with different computation and communication patterns. Why is that the case? Furthermore, TensorFlow throughput is lower than PyTorch on 2a and 2c, but its speedup is higher on 2b and 2d. Why is it so?
2. It would be great to show an ablation study on performance improvement.

Questionable cost model design.

1. In DistPar, the computational cost is modeled as the sum of elements of the input and output tensor regardless of the operator used. This design is questionable and uncomparable, as different operators (for instance, matmul v.s. element-wise operators) have very different FLOPs. Some ops can even have a significant input/output tensor without any computation done (eg: memcopy operation).

2. The cost model uses beta to somehow model the arithmetic intensity, and the tuning of beta was presented in the evaluation (Sec4.2). The authors claim that jointly considering computation+communication cost is better than considering computational cost alone. However, this is not supported in Fig.3, when computational cost alone performs very well (where beta=max). Does it mean that the sum of input/output tensor alone can model the performance of the DNN models? Why is the performance of DNN models consistent across a very large range of beta? Why is the unit of the y-axis in MB/sec?

Poor writing

This submission contains some unjustified claims/details and grammar errors. For instance:

• "DistPar needs to dynamically derive intermediate transformation primitives." -> How is the automatic transformation performed?
• Sec 4.3 -> "The configured with 8 GPUs of NVIDIA Tesla V100, FP32".

Lack of novelty

The submission presents 4 IRs: placement, broadcast, scatter, and partial reduce. However, similar ideas have been proposed previously [1,2,3]. It would be great to explain the difference between the new IRs and prior works, and justify why the new IRs are necessary.

---

Reference:

1. Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning
2. OneFlow: Redesign the Distributed Deep Learning Framework from Scratch
3. ATP: Adaptive Tensor Parallelism for Foundation Models

-. Limited Contribution

Our unified strategy, DistPar, introduces a set of tensor partitioning attributes aimed at instructing the allocation of global logical tensors to specific physical devices—referred to as physical tensors for simplicity. DistPar merges these devices into a coherent logical supercomputer, allowing developers to handle parallel training tasks on multiple devices as simply as a single device. This enhanced accessibility for individual users, so they can focus on more top-level design.

How is this work different from TensorFlow's DTensor, Colossal-AI's DTensor, and OneFlow's Global Tensor?

-. Limited Comparision

DistPar reaches 50% higher throughput in large-scale face recognition tasks and a 20% improvement in language modeling tasks compared with data parallelism provided by PyTorch.

Why Tensor Parallelsim work is compared PyTorchDDP? For fairness, how about comparing with other Tensor Parallelism works like PyTorchFSDP, Alpa (its intra-op parallelism), Colossal-AI (its tensor parallelism part), DeepSpeed (its Automatic Tensor Parallelism), and Megatron-LM?

-. Limited Evaluation

We conducted a comparative evaluation, analyzing ResNet-50 pre-trained on the ImageNet- 2012 dataset (Heigold et al., 2013) for image recognition and the BERT-Base model (Karpathy et al., 2014) for query answering in natural language processing tasks.

Why only ResNet50 and BERTBase? They just fit a single GPU, and Tensor Parallelism is not a requirement.

How about other models like GPT and Llama that truly require Tensor Parallelism?

-. Limited Benefit

Figure 2

It seems this work doesn't improve speed over TensorFlow DP and PyTorchDDP much.

-. Unclear Method

Computational Cost in DistPar is simplified to the sum of the elements of the input and output tensors corresponding to different parallelization strategies

Is this cost model too simple to find the best parallelization strategy?

Communication Cost is defined as the total communications across multiple devices. In our implementation, communication cost is estimated using the conversion cost that results from the conversions of distributed properties.

What are the formulas and algorithms to model this communication cost?

• The paper lacks originality as most of its content, which includes the search space and search algorithm, has already been studied in [1, 2, 3, 4, 5]. Particularly, [1, 2] deal with a more complex version of the automatic search problem, which has a larger search space. Moreover, the concepts of logical view, distributed properties, and conversion have also been discussed in [3, 4, 5]. This paper fails to provide a comparison and fails to mention most of these highly related works.
• Furthermore, the experiments conducted are outdated and limited in scale. Most of the experiments are performed on small networks such as ResNet-50 and BERT with a small number of devices (<=32).

[1] Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning, OSDI 22.
[2] Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization, OSDI 22.
[3] GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, ICLR 21.
[4] GSPMD: General and Scalable Parallelization for ML Computation Graphs, arXiv 21.
[5] OneFlow: Redesign the Distributed Deep Learning Framework from Scratch, arXiv 21.