RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers

We appreciate Reviewer 8Sgy's recognition of our method's high performance and would like to address the concerns raised:

C1: Table 2 tries to show the advantages of RePaViT comparing with pruning methods. However, the number of model parameters of most of pruning methods are not provided, which means it is hard to directly compare them with the RePaViT.

A1: We acknowledge the difficulty in directly comparing RePaViT with certain pruning methods due to the unavailability of parameter counts in their publications and/or unavailability of source code:

1. X-pruner [1] does not report the number of parameters in its paper and has not released source code so far. So, we're unable to provide its number of model parameters.
2. DC-ViT [2] does not report the number of parameters in its paper and had not provide its source code before the submission deadline of ICML. As a result, we're unable to provide its number of model parameters in our submission. However, we would like to reproduce its code and report the updates in the table below. Notably, DC-ViT's model size and computational complexity depends on both the number of pruned blocks and a special MLP compression ratio. Unfortunately, the authors haven't released the MLP compression ratios for reproducing their reported results (except 0.751734186 for DeiT-Base) and searching for exactly the same ratios would require a search process exceeding the rebuttal phase. So we can only provide the number of parameters and complexity that yields the closest performance as reported.
3. LPViT [3], while its source code is available, has not released the hardware benchmarking code so far. It still employs masking to simulate pruning without reducing the actual parameter count, which remains equivalent to the original model. Our investigation on the source code confirm this. And this is also why LPViT only reports sparsity in its paper.

To facilitate a more comprehensive comparison, we have reproduced several pruning baselines using their latest released code. The results are summarized in the table below (italic for updated, bold for best).

We would like to emphasize that RePaViT still achieves the best trade-off in terms of model efficiency and accuracy.

C2: Most of baseline methods are proposed in 2021. More recent baselines may make the experimental results stronger.

A2: We respectfully disagree with this comment. All baseline methods compared in our study are published after 2021 as shown below:

And the backbones choices (DeiT/ViT/Swin/LV-ViT) follow the convention in token pruning and network pruning works.

We sincerely appreciate your review comments, hope our responses satisfactorily address your concerns, and kindly request a consideration of recommendation score.

We sincerely appreciate Reviewer NyjX for the recognition of our work, especially on

• clear novelty

• significant real-world application potential

• and thorough and convincing experiments.

We would like to answer and clarify the questions as below:

Q1: What motivated the default choice (0.75)? How sensitive is this choice in terms of generalization to other datasets/tasks beyond those tested?

A1: We thank for this insightful question.

Firstly, we would like to clarify that the channel idle ratio $\theta$ is a static architectural hyperparameter designed to control the trade-off between model efficiency and performance. It is not meant to be universally optimal but rather user-configurable based on specific hardware constraints and application requirements. More importantly, the choice of channel idle ratio is highly related to the model size. As shown in Table 4, larger models (e.g., DeiT-Base) tolerate higher idle ratios with minimal accuracy drop. In particular, when the model size grows to DeiT-Base level and beyond, 0.75 idle ratio can result in high performance, which motivates our choice.

Secondly, as for the method-level generalization across tasks, we would like to emphasize that RePaViT has been evaluated on four diverse tasks: (1) image classification on ImageNet-1K, (2) object detection on MS COCO, (3) semantic segmentation on ADE20K, and (4) zero-shot image classification using CLIP on LAION-400M. These results collectively demonstrate the method’s robustness and applicability across different computer vision problems.

Thirdly, as for the model-level generalization across datasets, we have provided zero-shot classification results in Table 9, Appendix B, where RePaViT is applied to CLIP models with different idle ratios. Since CLIP is pretrained on the large-scale unlabelled LAION-400M dataset and directly evaluated on the ImageNet-1K validation set without finetuning, this setup reflects the model's generalizability on unseen data. Notably, RePa-CLIP-ViT-B/16 with an idle ratio of 0.5 can outperform the vanilla model by 0.5%.

Finally, on the sensitivity of the idle ratio, Table 4 presents a detailed analysis. We observe that performance remains stable or improves slightly up to a certain point, and then degrades as the idle ratio increases. Specifically, when the idle ratio exceeds 0.75, a noticeable accuracy drop occurs. We hypothesize that excessive pruning may cause insufficiency of nonlinearity in the model, negatively affecting the model representation capacity.

While our current study has provided extensive experiment results demonstrating the generalizability of our method and the sensitivity of idle ratio choice, we acknowledge that further evaluation could offer deeper insights into its robustness. However, conducting such experiments would require substantial additional time that can extend beyond the current rebuttal phase. We still welcome specific task/dataset suggestions from Reviewer NyjX and are happy to include additional results in the revised version.

Q2: Training Stability: Can you discuss any observed training instabilities or convergence issues with larger channel idle ratios or model scales, and how these might be mitigated in practice?

A2: We appreciate this insightful question.

Fortunately, we have not observed instability or convergence issues when training RePaViT with large idle ratios or larger models. As shown in Tables 1 and 4, and according to our training logs, RePaViTs converge smoothly across all configurations. We have also provided our source code and detailed instructions in the supplementary material for reproducing our results.

However, reparameterizing before training can cause instability, particularly when BatchNorm is merged early. Table 5 shows that while such reparameterization reduces training time, it degrades performance and may destabilize training—especially for large models—since normalization layers are crucial for mitigating issues like internal covariate shift, ensuring consistent signal scales across layers, which is particularly important in larger models.

W1: Batch norm instead of Layer norm may limit the training efficiency.

A3: We would like to clarify that BatchNorm is a dedicated architectural choice to facilitate a further reparameterization of normalization layers and shortcuts into the backbone. It enhances computational efficiency during inference, and we consider the increase in training time a worthwhile trade-off for the efficiency benefits achieved. Nonetheless, our method is still compatible with LayerNorm.

We hope our responses address your questions and concerns, and would sincerely appreciate your consideration of raising the recommendation score.

We sincerely appreciate Reviewer d1Yn's detailed and careful review comments. We thank Reviewer d1Yn for pointing out the strengths of our work, including

• clear and understandable presentation

• solid and comprehensive experiments

• high performance

• and novel from existing methods.

We would like to answer the question as below:

W2&Q2: I am curious about the performance of the proposed method when combined with other efficiency methods (e.g., quantization, pruning).

A2: We thank for this insightful question on the combination of our method and other kind of existing efficient methods.

• Pruning

  Since our method primarily focuses on reducing channel-wise complexity for ViTs, we decide to test our method combined with a spatial-wise token reduction method, ToME, rather than network pruning methods. The results of ToMe-RePa-ViTs with different token reduction numbers (r) are shown in the Table below. In general, our method combines with token reduction method well and yields more significant improvement in the trade-off between accuracy and efficiency.

  Notably, ToMe-RePa-ViT-Large/0.75 with reduction number 8 achieves more than 200% acceleration (374.7 imgs/s vs 107.7 imgs/s) with merely 0.8% accuracy drop.

• Quantization

  For simplicity, we employed TensorRT to perform FP32-to-FP16 quantization on our pretrained and reparameterized models. TensorRT provides a comprehensive ecosystem of tools designed to deliver high-performance deep learning inference via post-training quantization. The evaluations before and after reparameterization of our RePaViT and RePaSwin models were conducted on the same hardware platform mentioned in the paper (NVIDIA A6000).

  Remakably, when quantized to FP16, our method presents more than 200% acceleration. And the acceleration even signifies with reparameterization, demonstrating the effectiveness of combining quantization with our method in real-world scenarios.

In conclusion, our method combines with quantization and pruning methods well, exhibiting more significant enhancement in performance-efficiency trade-off. And the integration of TensorRT indicates the potential of deploying on edge devices.

W1&Q1: How does RePaViT perform in deployment scenarios on edge devices?

A1: We thank the Reviewer for this question, which is indeed interesting and essential for demonstrating the real-world practicality of our RePaViT.

Unfortunately, due to strict equipment management policies in our organization and time limits, we were unable to successfully apply for access to an edge device during the rebuttal period. As a result, we could not provide real-world inference speed measurements on actual edge hardware. We would like to report detailed performance metrics upon deployment to the Jetson AGX Orin platform in the follow-up discussion phase.

We hope our responses address your concerns and would sincerely appreciate your reconsideration of raising the score.