DEPrune: Depth-wise Separable Convolution Pruning for Maximizing GPU Parallelism

Thank you for the time you have taken to review our work and for the constructive feedback. If the reviewer allows us to revise this paper, we will try our best to improve the quality of this paper.

Weakness 1) [Some references are incomplete, for example, [23][39].]

We greatly appreciate your guidance on this issue. We will revise and incorporate your suggestions as follows. Additionally, we will double-check if other references are incomplete and address them accordingly.

"[23] Krizhevsky, G. Hinton, et al. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009."

"[39] P. K. Rao and S. Chatterjee. Weight Pruning-UNet: Weight Pruning UNet with Depth-wise Separable Convolutions for Semantic Segmentation of Kidney Tumors. Research Square 2021"

Weakness 2) [The proposed scheme is only evaluated on MobileNet-V2, MobileNet-V3, and EfficientNet-B0. How does this method work for other NN models?]

As the reviewer's suggestion, we conducted additional experiments with the ConvNeXt-Base model [1]. In Author-Rebuttal-Table.3, DEPrune-BH achieves a 0.46% higher Top-1 accuracy compared to FP even with an additional 20% pruning in DW-conv. DEPrune-BH is up to 3.3 times faster than the ConvNeXt-Base while achieving a Top-1 accuracy drop within 1.5%.

[1] A ConvNet for the 2020s, CVPR 2022

Question 1) [What does “X” and “O” mean in Figure 1?]

We agree with the reviewer’s comment that the caption in Figure 1 is not clearly presented. The X and O symbols indicate the absence and presence of corresponding characteristics for each method. In the case of (a) Structured Pruning, this approach forms well-structured data patterns that can benefit performance when executed on GPUs. However, it is not a fine-grained method, as the pruning unit size can extend up to a 2D matrix, which is significantly larger than the pruning unit in DEPrune, the method we propose. Therefore, in the case of (b) DEPrune, it is marked as O for both fine-grained and structured characteristics. We will include these details in the final version of the paper’s caption to prevent any misunderstandings for readers.

Question 2) [The lower parts of Figure 1 seem misleading, which do not correspond to the DW-conv example in the upper half.]

We appreciate the reviewer’s valuable comment that points out the statements that could possibly mislead the readers. The top illustration in Figure 1 depicts a typical depth-wise convolution. The bottom illustration shows the depth-wise convolution rearranged to be more favorable for GPU processing. On the left is an image with (a) structured pruning applied, and on the right is after applying (b) DEPrune. Thus, the relationship between the top and bottom illustrations indicates before and after the Diagonalwise Refactorization is applied, and the figure shows the arrangement of the weight. We will enhance the caption with these details to ensure there is no misunderstanding for readers.

Thank you for your support of our paper and the valuable feedback. We respond to your reviews as follows.[m-#] means manuscript's reference.

Weakness 1) [The overhead of pruning method is not analyzed. How much time does it need to decide pruning points, loading balance and the parameters in HSR?]

We would like to clarify few things that might not have been clearly expressed in the paper. The pruning, load balancing, and HSR mentioned are all required only during the pruning process (offline). Pre-processing (Pruning and fine-tuning) is conducted offline, while model inference is performed online. The processes you mentioned are overheads in the offline phase and do not affect model inference time. In Author-Rebuttal-Table.4, still this offline overhead accounts for about 0.6% of the total pre-processing time in MobileNet-v3-small on ImageNet. The proportion of this overhead becomes even smaller if the fine-tuning epoch increases. We will provide detailed figures based on MobileNet-v3-small to Author-Rebuttal-Table.4.

Weakness 2) [Comparing with the gain in pruning ratio, the accuracy drop seems not negligible. Can we modify the algorithm to set the HSR constraints to align the memory access and computation with tile size in pruning phase, without need of recalibration? Does it help with accuracy?]

Yes, the algorithm can be modified as you suggested. However, the absence of recalibration does not significantly affect accuracy. Without the recalibration process, users must set the pruning ratio to match the recalibration results by considering the GPU tile size and each layer's size. Recalibration is an algorithm that automates this process. The main reason there is no significant change in accuracy is as follows: In Figure 7-(C), you can see that DEPrune-BH performs fine-tuning after step 4, which is the entire pruning setting process. Therefore, whether recalibration is performed or users manually set the pruning ratio by considering the tile size, both are processes before fine-tuning, resulting in almost no change in accuracy.

Weakness 3) [Can you provide the evaluation on edge GPU?]

Certainly. We conducted additional experiments on an edge GPU and measured the inference time before and after pruning for MobileNet-v2. The edge GPU that we used for the experiment is NVIDIA Jetson Orin Nano 8GB. In Author-Rebuttal-Table.5, the DEPrune-BH pruned model inference time shows a 2.48 times speedup in inference time than baseline (unpruned model) on edge GPU. Compared to GFS [m-50] pruned model, DEPrune-BH pruned model is approximately 1.62 times faster. From an accuracy perspective, there is no change because the weights are the same as those on the laptop GPU.

We sincerely appreciate the reviewer’s time and effort in providing constructive feedback on this manuscript. We will incorporate all of the suggestions into the final version of the paper. [m-#] means manuscript's reference.

Weakness 1)

As the reviewer pointed out, it is important to compare our proposed scheme with recent prior work. In Table 4, we have compared our scheme with two of the most recent research papers [m-11 and m-48], both presented in 2024. If you have any recommendations for recent structured pruning methods, we would be eager to compare them with our DEPrune if circumstances allow.

Weakness 2)

We appreciate your comment. The study you mentioned [2 and m-44] indeed applies an existing gradual pruning method to DSConv, so it is not accurate to state that there are no previous pruning techniques that target DW-Conv. We will revise this statement in the paper if given the opportunity to make revisions: "there are hardly any previous pruning techniques that target depth-wise convolution (DW-conv)" in line 4. We also want to mention that the pruning method proposed in [2 and m-44] is focused on applying gradual pruning [m-55] to DSConv layers, which differs from our approach. The research [2] focuses on removing filters to preserve structural pattern on DW-conv. Whereas our research considers GPU kernel and maintains structural pattern while more fine-grained pruning on DW-conv, to reduce accuracy loss.

Weakness 3)

Thank you for your valuable advice. Based on your feedback, we will make the following improvements:

1. We will rewrite the captions to provide a more detailed walkthrough, ensuring that readers can easily understand them.
2. To enhance reproducibility, we will add a pseudo-code section to the appendix and plan to release our implementation in the future.
3. We will include the ImageNet dataset information in the experiment section to clearly outline the experimental procedure:

[ImageNet experiment setting] learning rate : 0.001, Fine-tuning epoch : 65epochs, learning rate epoch : dividing 10 per every 30epochs, Process : iterative pruning, Weight decay : 1e-4, Optimizer : SGD, Momentum : 0.9, Pre-trained model : PyTorch, All data are augmented with random crop and randomly horizontal flip.

Weakness 6)

Thank you for pointing out our typo. Our proposed scheme is named DEPrune, and the mention of DSPrune is indeed a typo. We will thoroughly review our paper again and correct this and any other typographical errors in the revised version. We also think that so many names can be very difficult to understand. We will double-check the Table 1 and writing of the paper so that it can provide simple and clear concept of DEPrune. As you suggested, creating a richer table will likely make it easier for readers to understand the paper.

Question 1)

We believe there are various efficiency metrics, such as peak memory usage, computation throughput, and computation utilization, that can be measured in our experiments. Among these, we consider peak memory usage to be the most important efficiency metric, thus, we will include it in Table 4 in the revised paper. If the reviewer can suggest other efficiency metrics that should be included, we will measure them and incorporate the results in the revised version.

Question 2)

Applying DCP to general group convolution can be challenging. When the number of groups and channels differ, the rearrangement method for GPU processing varies, complicating the application of DCP. However, the enhanced method HSR, proposed in our paper, can be effectively applied in these scenarios. We believe that combining conventional structured pruning with our HSR could lead to improved inference time performance on the general group convolution.

Question 3)

Thank you very much for your insightful feedback. The paper you mentioned, GKP [3], seems an excellent work. GKP divides weights into groups and applies vector-wise pruning to each group, effectively addressing the issue described in Section 4.2. As you pointed out, when GKP is applied, it does not impact the subsequent layer, thereby avoiding the problem discussed in Section 4.2. We appreciate you bringing this important paper to our attention, and we will consider incorporating this research (GKP) with our DEPrune as a potential future work.

Question 4)

According to Table 4 of paper [m-38], the extra overhead in total memory consumption due to zero-padding is approximately 0.3%. To assess the impact of DEPrune-BH, we measured and presented the peak memory usage of MobileNet-v2 before and after applying DEPrune-BH with a 50% pruning ratio, as shown in Author-Rebuttal-Table 1. Before applying DEPrune-BH, the peak memory usage is 7.22 MB, whereas after application, it decreases to 3.63 MB, representing a reduction of approximately 49.8%.

Question 5)

Research on n:m sparsity is currently very active in the field of pruning. However, this sparsity approach has two major limitations: a lack of flexibility and the requirement for specialized hardware. First, it lacks flexibility because it is fixed at a 50% pruning ratio, specifically 2:4 pruning [m-35]. As seen in Author-Rebuttal-Table.2, we conducted comparative experiments between NVIDIA's n:m sparsity and DEPrune on MobileNet-v2 using CIFAR-10.
At the same pruning ratio of 50%, DEPrune-B achieves 0.31% higher accuracy than n:m sparsity. This is because DEPrune-B achieves a 50% pruning ratio within $32 \times k \times k$ parameters, whereas n:m sparsity achieves a 50% pruning ratio within a parameter size of 4, which is $8 \times k \times k$ times smaller than DEPrune-B, leading to a accuracy drop [m-21]. Secondly, in n:m sparsity, achieving optimal performance requires specialized hardware (NVIDIA A100) that can quickly handle index processing [m-35]. In contrast, our approach requires only a customized GPU kernel for processing.

We would like to express our gratitude to the reviewer for their constructive and insightful feedback. Below, we have provided our responses to the comments. If the paper is accepted, we will incorporate these helpful suggestions into the camera-ready version.

Weakness 1) [I hope that the authors provide more background for the readers not directly familiar with the concepts of "alignment in GPUs" or refer to proper references.]

We agree with the reviewer’s comment that the concepts of "alignment in GPUs" should be presented in our paper for better understanding. Therefore, if given the opportunity to revise our paper, we will include these details, referencing prior work [1, 2].

[1] Sparse GPU kernels for Deep Learning, SC 2020

[2] GUIDE, Design. CUDA C++ programming guide. NVIDIA, July, 2020.

Question 1) [I hope the authors release their implementations so that the community can benefit from them.]

We are willing to provide our implementation once the paper is accepted to contribute to the community.