RePaFormer: Ferocious and Scalable Acceleration of MetaFormers via Structural Reparamterization

We thank Reviewer ufAp for the valuable comments concerning the two major designs of our RePaFormer models. We would like to provide clarification on these points below:

---

1. Response to W1 (whether BatchNorm alone could improve the test performance):

This is a good question. In fact, a previous work [1] has already investigated leveraging BatchNorm instead of LayerNorm in ViT and discovered that directly applying BatchNorm in ViT would lead to irregular training crashes. To address this, [1] proposes a novel approach (FFNBN) to enable training ViT with BatchNorm. However, all the experiment results on both DeiT and Swin Transformer demonstrate that BatchNorm consistently yields worse performance than LayerNorm on ViTs. We cite the results from [1] in the table below:

---

2. Response to W2 (whether channel idle mechanism can be replaced by a single learnable linear function):

We thank the Reviewer for this insightful comment and believe this comment aligns with Reviewer nzL3's W2. We would like to interpret this weakness as comparing the RePaFormers:

1. trained with $6C^2$ weights and then reparameterized into $C^2$
2. trained with a single $C^2$ linear projection weight from scratch

The table below shows the results of RePa-DeiT, RePa-Swin and RePa-LV-ViT when trained with a single $C^2$ linear projection weight. The training settings are the same as their counterparts outlined in the manuscript. After reparameterizing the BatchNorm, the differences in the inference speed and the number of parameters for these two cases should be negligible, so we only report their accuracies. The NaN values represent that the training crashes.

As shown in the table, training with $6C^2$ parameters and subsequently reparameterizing them into $C^2$ consistently achieves better performance, in line with the findings in [2,3,4] that train-time overparameterization improves the performance.

---

In conclusion:

• Purely using BatchNorm alone does not enhance ViT testing performance.
• Relying solely on a single linear projection for the channel idle process results in lower performance.

The above discussions and experimental results will be included in the revised version. We hope our explanation can resolve the Reviewer's concerns and respectfully request consideration for a score increase.

 

[1] Yao, Zhuliang, et al. "Leveraging batch normalization for vision transformers." ICCV, 2021.

[2] Vasu, Pavan Kumar Anasosalu, et al. "FastViT: A fast hybrid vision transformer using structural reparameterization." ICCV, 2023.

[3] Vasu, Pavan Kumar Anasosalu, et al. "Mobileone: An improved one millisecond mobile backbone." CVPR, 2023.

[4] Ding, Xiaohan, et al. "Repvgg: Making vgg-style convnets great again." CVPR, 2021.

We thank Reviewer ufAp for the valuable feedback and insightful suggestions, which have helped us refine and clarify our work. We have carefully addressed all the raised concerns in our response.

We would greatly appreciate it if Reviewer ufAp could provide further feedback. Your input is invaluable to ensuring the quality and clarity of our work. Or, if our responses have satisfactorily resolved the concerns, we respectfully request reconsideration of the score based on the clarifications and improvements provided.

We sincerely thank the reviewer for the valuable suggestions and insightful comments. In our response, we have carefully addressed all the raised concerns regarding 1) the effect of BatchNorm, and 2) the comparison with the reparameterized model trained from scratch.

If the reviewer has any further questions, we are glad to join in the discussion. Otherwise, if our responses have satisfactorily resolved the concerns, could the reviewer reconsider the score based on our clarifications and responses?

We are glad to clarify this statement.

To begin with, as shown in Figure 1(b) and Equation 2, there is a shortcut for each FFN layer as

$$Z = \text{FFN}(\text{Norm}(Y)) + Y.$$

However, the vanilla FFN incorporates LayerNorm (i.e., $\text{Norm}(\cdot)$=$\text{LN}(\cdot)$) as its normalization method, which is specific to the input feature and cannot be structurally reparameterized into linear projection weights. Consequently, the shortcut component (i.e., $+ Y$) cannot be reparameterized either. To address this, we replace LayerNorm with BatchNorm (i.e., $\text{Norm}$=$\text{BN}(\cdot)$), enabling the reparameterization of both the normalization and the shortcut.

Next, as suggested by the reviewer, the new baseline model should follow the structure in Figure 1(b), but with the channel idle mechanism replaced by a linear projection. This process can be expressed as:

$

Z = \text{Act}(\text{BN}(Y)W^{\text{1}})W^{\text{2}} + \text{BN}(Y)W^{\text{3}} + Y,

$

where $W^{\text{3}}\in\mathbb{R}^{C\times C}$. We are now directly training and testing the new baseline model as formulated above.

Nonetheless, as the reviewer further required that specifically Linear projection 3 with a BatchNorm layer but without BatchNorm inference reparameterization, the shortcut $+Y$ in the above baseline model cannot be reparameterized into the projection weights $W^{\text{3}}$ without BatchNorm reparameterized during inference. So we simply keep the BatchNorm and shortcut during testing, which can lead to a bit slower throughput.

In addition, if the BatchNorms and shortcut in the above model are expected to be reparameterized during testing, we assure the reviewer that the accuracy will not drop after reparameterization while the inference speed and model size should be analogous to our model.

The experimental results will be updated soon.

In addition to resolving the above concerns, we are excited to share the significant contribution and practical value of our RePaFormer with Reviewer ufAp.

Through discussions with you and other Reviewers, and especially inspired by Reviewer wqPR, we have gained a chance to present the key advantages and application scenarios of our approach: RePaFormer can significantly speed up large ViT models while even improving accuracy. This insight demonstrates the practical value of RePaFormer in accelerating large-scale models without compromising performance, making it an effective solution for large real-world applications requiring both speed and precision.

To empirically validate the practicality above, we conducted additional experiments on large vanilla ViTs, following Reviewer wqPR's kind suggestion. Both the vanilla and RePaFormer variants of ViT-Large and ViT-Huge are trained from scratch on the ImageNet-1k dataset using the same training recipes with the idle ratio set to 75% by default. The new results, along with those for MLPMixer-l16 reported in the original manuscript, are shown in the table below:

We are thrilled to emphasize that our method not only drastically reduces model size and latency but also achieves HIGHER top-1 accuracy on large models with more than 200M parameters and computational complexities exceeding 40 GMACs. For instance, RePaViT-Large achieves a 1.7% higher top-1 accuracy (82.0% vs 80.3%) while delivering a 66.8% speed gain (207.2 images/second vs 124.2 images/second) compared to the vanilla ViT-Large. This demonstrates a transformative contribution, as many practical large-scale foundation models for computer vision tasks rely on vanilla ViT as their backbone, such as CLIP [1], SAM [2] and ViT-22B [3].

To the best of our knowledge, RePaFormer is the first novel method that achieves significant acceleration (~68%) while having positive gains in accuracy (1~2%) instead of accuracy drops, on large and huge ViTs. Considering the unprecedented results RePaFormer is getting, we want to point out that this is a disruptive and timely innovation for the community and a significant addition to the large foundation models acceleration toolkit. Since RePaFormer can be both directly applied to larger ViT architectures and combined with other acceleration techniques such as quantization, we believe RePaFormer will catalyze further research and breakthroughs on ViT's speed and accuracy. We strongly believe that the weight and impact of this work make it best-suited for the prestigious ICLR, and the community will benefit greatly by seeing it soon from this venue.

We hope our response and plenty of additional experimental results can address all your concerns and demonstrate the significance of our contributions. We kindly request your strong support by considering a score increase.

 

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." ICML, 2021.

[2] Kirillov, Alexander, et al. "Segment anything." ICCV, 2023.

[3] Dehghani, Mostafa, et al. "Scaling vision transformers to 22 billion parameters." ICML, 2023.

We sincerely thank Reviewer nzL3 for recognizing that problem studied in this paper is critical and the proposed method is interesting and technically sound. We are honoured to hear that multiple reviewers have acknowledged the importance of the problem and the novelty of our method. Regarding the issues raised by Reviewer nzL3, we address them as follows.

 

---

1. Response to W1 (lacking comparisons with network pruning):

First, we would like to qualitatively compare the differences between our method and pruning methods:

• Different motivations: Network pruning focuses on removing redundant parameters, creating a sparse network. In contrast, our method structurally combines parameters using linear algebra operations, resulting in a condensed yet structurally regular network.

• Different requirements and implementation difficulties: Network pruning methods produce sparse networks. To fully utilize the sparsity introduced by pruning, specialized hardware (e.g., accelerators that support sparse computations) or software libraries are needed. This adds complexity to deployment and maintenance. On the contrary, the reparameterized RePaFormers are structurally regular and can be efficiently adopted on general-purpose hardware without specialized support.

• Different generalizabilities: Different network architectures have varying sensitivity to pruning; some models may not be suitable for pruning or may not benefit significantly from it. In most cases, dedicated training processes or pruning strategies need to be designed for different backbones. However, our RePaFormer method is generally applicable to all MetaFormer-structured models and can be seamlessly integrated into them.

Second, we would like to provide quantitative analysis by comparing our method with state-of-the-art pruning methods for ViTs in terms of both latency drop and accuracy in the table below:

In addition, we further compare our method with some other representative pruning methods for ViTs regarding computational complexity and accuracy in the table below:

As demonstrated in the tables above, our method achieves comparable trade-offs between accuracy and complexity/inference time to network pruning methods.

Last, it is worth emphasizing that our RePaFormer is parallel to network pruning methods. As pointed out by Reviewer wqPR, comparing with those methods is out of the scope of this work. We would not claim our approach to be more advantageous over network pruning and anticipate they can be utilized together.

After all, we thank the reviewer for this suggestion and will include the comparison in our revised version.

 

---

2. Response to W2 (lacking an essential baseline):

We thank the reviewer for this contributive comment. It is important to compare the performance of

1. training vanilla FFN layers and subsequently reparameterizing them into an efficient structure and

2. directly training reparameterized FFN layers from scratch

Therefore, we train RePa-DeiT, RePa-Swin and RePa-LV-ViT with a single $C^2$ linear projection weight without the channel idle mechanism from scratch. The training settings are kept consistent with their counterparts outlined in the manuscript. The results are shown in the table below, where the NaN values in the table represent the training crashes.

As shown in the table, directly training the reparameterized model (as illustrated in Figure 1 (c)) consistently yields a worse performance than our approach, and occasionally results in training crash. This observation aligns with the findings in [8,9] that training an overparameterized network from scratch achieves a more robust training process and better performance than training a network with fewer parameters. Additionally, we would like to clarify that the crashes in the reparameterized models are primarily due to instability caused by the lack of normalization, leading to unregulated value fluctuations during training.

We will include this baseline in our revised version.

3. Response to W3 (lacking comparisons with previous structured reparameterization methods):

The structural reparameterization in our RePaFormers is different from previous reparameterization methods (e.g., RepVGG [10] or FastViT [9]). We would like to compare RePaFormer and FastViT as follows:

• Different target components: FastViT mainly incorporates multi-branch convolutions into the hierarchical ViT network and only applies reparameterization to multi-branch convolutional layers. In other words, FastViT focuses on accelerating the token mixer layers during inference. In contrast, our RePaFormer targets accelerating the FFN layers in ViTs. Our method can be adapted to all models with the FFN layer.

• Different scopes: The main scope of FastViT is to construct novel efficient architectures, while our method aims to apply structural reparameterization to FFN layers and accelerate existing ViTs/MetaFormers of all sizes.

• Different reparameterization solutions: Another key difference is that FastViT reparameterizes horizontally across parallel convolutional kernels, while RePaFormer reparameterizes vertically on consecutive linear projection weights.

  For instance, FastViT reparameterizes two parallel convolutional branches with kernels $W_1^{\text{Conv}}$ and $W_2^{\text{Conv}}$ by summing them:

  $$W_{\text{Rep}}^{\text{Conv}} = W_1^{\text{Conv}} + W_2^{\text{Conv}}.$$

  On the contrary, as demonstrated in Equation 6, RePaFormer reparameterizes two consecutive projection weights $W_1^{\text{FFN}}$ and $W_2^{\text{FFN}}$ by multiplying them:

  $$W_{\text{Rep}}^{\text{FFN}} = W_1^{\text{FFN}} \cdot W_2^{\text{FFN}}.$$

  (In the above example, we omit the BatchNorm and suppose $W_1^{\text{Conv}}$ and $W_2^{\text{Conv}}$ have been padded to the same shape.)

Considering these differences, we think directly comparing a dedicatedly designed network architecture with a universal network acceleration method is unfair. Nonetheless, we do compare a state-of-the-art reparameterization method [11] within the same scope in our paper.

We still appreciate the reviewer for this comment and will include the above comparison in our revised version.

 

---

4. Response to W4 (the proposed method still suffers from accuracy drops):

We respectfully disagree with this weakness.

• Accuracy is indeed preserved post-reparameterization: As shown in Table 1, the accuracy remains unchanged before and after reparameterization, while the inference speed significantly improves. This demonstrates that the reparameterization process does not lead to accuracy degradation. In addition, we have provided the resource codes in our supplementary materials, which can validate this as well.

• Accuracy gap explanation: The accuracy gap between our method and vanilla models arises because vanilla models do not have idling channels in activation. Training models with significantly fewer non-linearities inherently leads to a performance drop. This trade-off is common across various model compression methods, where increased efficiency often comes at the cost of some accuracy loss.

• Improved performance with reduced idle ratios: As shown in Table 4, reducing the idle ratio further improves performance. For example, with a 25% idle ratio, RePa-Swin-Base achieves 83.7% top-1 accuracy, surpassing the vanilla Swin-Base's 83.5% accuracy. This highlights the adaptability of our method to different idle ratios while maintaining competitive accuracy.

 

---

We hope our explanations adequately address the weaknesses raised by Reviewer nzL3. We sincerely appreciate the Reviewer's helpful suggestions and recognition of the significance of our work. We respectfully request reconsideration of the score.

 

[1] Wang, Guo-Hua, and Jianxin Wu. "Practical network acceleration with tiny sets." CVPR, 2023.

[2] Zhang, Hanxiao, et al. "Dense Vision Transformer Compression with Few Samples." CVPR, 2024.

[3] Tang, Yehui, et al. "Patch slimming for efficient vision transformers." CVPR, 2022.

[4] Yu, Shixing, et al. "Unified visual transformer compression." ICLR, 2022.

[5] Yu, Fang, et al. "Width & depth pruning for vision transformers." AAAI, 2022.

[6] Yu, Lu, and Wei Xiang. "X-pruner: explainable pruning for vision transformers." CVPR, 2023.

[7] He, Yang, et al. "Data-independent Module-aware Pruning for Hierarchical Vision Transformers." ICLR, 2024.

[8] Vasu, Pavan Kumar Anasosalu, et al. "Mobileone: An improved one millisecond mobile backbone." CVPR, 2023.

[9] Vasu, Pavan Kumar Anasosalu, et al. "FastViT: A fast hybrid vision transformer using structural reparameterization." ICCV, 2023.

[10] Ding, Xiaohan, et al. "Repvgg: Making vgg-style convnets great again." CVPR, 2021.

[11] Guo, Jialong, et al. "SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization." ICML, 2024.

We thank Reviewer nzL3 for the valuable feedback and insightful suggestions, which have helped us refine and clarify our work. We have carefully addressed all the raised concerns in our response.

We would greatly appreciate it if Reviewer nzL3 could provide further feedback. Your input is invaluable to ensuring the quality and clarity of our work. Or, if our responses have satisfactorily resolved the concerns, we respectfully request reconsideration of the score based on the clarifications and improvements provided.

We sincerely thank the reviewer for the valuable suggestions and insightful comments. In our response, we have carefully addressed all the raised concerns regarding 1) the comparison with network pruning, 2) the comparison with the reparameterized model trained from scratch, 3) the comparison with FastViT and 4) the accuracy drop confusion.

If the reviewer has any further questions, we are glad to join in the discussion. Otherwise, if our responses have satisfactorily resolved the concerns, could the reviewer reconsider the score based on our clarifications and responses?

Dear Reviewer nzL3,

We sincerely appreciate your time and effort in reviewing our work, especially pointing out that the problem studied in this paper is critical. In the previous rebuttal, we have made every effort to appropriately address all your concerns.

As for your question on is there any workaround to avoid using BatchNorm, we would like to clarify that LayerNorm is specific to the input feature and is therefore not structurally reparameterizable. To enable the reparameterization of both the normalization layer and shortcut, an input-agnostic normalization method (e.g., BatchNorm) is needed. However, exploring ways to reparameterize LayerNorm can be an interesting direction for future research.

In the end, we kindly request your feedback on our rebuttal. We are willing to answer any further questions during the remaining discussion period.

Best regards,

Authors

We sincerely appreciate Reviewer nzL3 for carefully considering our rebuttal responses and for raising the score. We are pleased to hear that all other concerns have been appropriately addressed, except for the practical value of our method. The further suggestion to clearly articulate the advantages of our method over network pruning, as well as its practical value, is greatly appreciated and will further enhance the clarity and impact of our contributions.

Firstly, we respectfully disagree with the statement in your reply that "the proposed method is much more complicated (than network pruning)". On the contrary, our method is both simple and effective. Unlike network pruning, which requires a detailed analysis of the network structure and parameter redundancy, our approach only involves straightforward modifications: replacing LayerNorm with BatchNorm and keeping certain channels inactivated in the FFN layer. Additionally, our method can be easily integrated into MetaFormer-structured models without significant architectural changes. Therefore, we conclude that our method is generally simpler to implement and more adaptable than network pruning.

We would like to show pseudocodes in PyTorch style:

class Mlp(nn.Module):
    def __init__(self, dim_in, dim_hidden, dim_out, act_layer, idle_ratio=0.75):
        super().__init__()
        self.norm1 = nn.BatchNorm1d(dim_in)
        self.fc1 = nn.Linear(dim_in, dim_hidden)
        self.norm2 = nn.BatchNorm1d(dim_hidden)
        self.fc2 = nn.Linear(dim_hidden, dim_out)
        self.act = act_layer()
        self.idle_channels = int(dim_hidden * idle_ratio)

    def forward(self, x):
        x = self.norm1(x.transpose(-1,-2)).transpose(-1, -2)
        x = self.fc1(x)

        # Activation with channel idle mechanism
        mask = torch.zeros_like(x, dtype=torch.bool)
        mask[:, :, self.idle_channels:] = True
        x = torch.where(mask, self.act(x), x)

        x = self.norm2(x.transpose(-1,-2)).transpose(-1, -2)
        x = self.fc2(x)
        return x

Secondly, through discussions with you and other Reviewers, and especially inspired by Reviewer wqPR, we have gained a chance to further present the key advantages and application scenarios of our approach: RePaFormer can significantly speed up large ViT models while even improving accuracy. This insight demonstrates the practical value of RePaFormer in accelerating large-scale models without compromising performance, making it an effective solution for large real-world applications requiring both speed and precision.

To empirically validate the practicality above, we conducted additional experiments on large vanilla ViTs, following Reviewer wqPR's kind suggestion. Both the vanilla and RePaFormer variants of ViT-Large and ViT-Huge are trained from scratch on the ImageNet-1k dataset using the same training recipes with the idle ratio set to 75% by default. The new results, along with those for MLPMixer-l16 reported in the original manuscript, are shown in the table below:

We are thrilled to emphasize that our method not only drastically reduces model size and latency but also achieves HIGHER top-1 accuracy on large models. For instance, RePaViT-Large achieves a 1.7% higher top-1 accuracy (82.0% vs 80.3%) while delivering a 66.8% speed gain (207.2 images/second vs 124.2 images/second) compared to the vanilla ViT-Large. This demonstrates a transformative contribution, as many practical large-scale foundation models for computer vision tasks rely on vanilla ViT as their backbone, such as CLIP [1], SAM [2] and ViT-22B [3].

3. Response to W3 (not comparing with RepVGG-style methods and other compression methods):

1. Not comparing with RepVGG-style methods: As we have explained in the response to W2 and Q1, RepVGG-style methods are quite distinct from our methods and cannot be applied to the FFN layer in ViTs. Directly comparing with methods [1,2] in different scopes is unfair and meaningless. However, we have indeed compared against a state-of-the-art reparameterization method [7] in a similar scope in our manuscript.

2. Not comparing with other compression methods: As pointed out by Reviewer wqPR that comparing with those (model compression) methods is out of the scope of this work, our RePaFormer method is parallel to existing model compression methods and can be adapted with them. Thus, we only focus on comparing with the state-of-the-art structural reparameterization method [7] for expediting ViTs.

Despite different scopes, we are willing to provide comparisons between our method and representative pruning methods for ViTs.

1. We compare with the state-of-the-art pruning method for ViT, DC-ViT [5] (CVPR24). Since DC-ViT has not released the code, we only compare the latency drop reported in the paper:

   Model	Latency drop	Top-1 acc. (%)
   DC-ViT-T	-16.5%	64.4
   RePa-ViT-T	-24.6%	64.3
   DC-ViT-S	-16.7%	78.6
   RePa-ViT-S	-35.3%	77.1
   DC-DeiT-B	-16.7%	81.3
   RePa-DeiT-B	-40.3%	81.4

2. We compare our method with other representative pruning methods for ViTs regarding computational complexity and accuracy in the table below:

   Model	Complexity (GMACs)	Top-1 acc. (%)
   DeiT-Base:
   PatchSlimming [8]	9.8	81.5
   UVC [9]	8.0	80.6
   WDPruning [10]	9.9	80.8
   X-pruner [11]	8.5	81.0
   RePaFormer	10.6	81.4
   Swin-Base:
   PatchSlimming [8]	9.8	81.5
   UVC [9]	8.0	80.6
   DIMAP2 [6]	10.2	83.4
   RePaFormer	9.0	82.6

It is worth noting again that our RePaFormer is a novel direction of accelerating ViT models, which is parallel to existing compression methods and can be adopted with them simultaneously.

 

---

4. Response to W4 (the latency improvement on dense prediction tasks is small):

We thank the Reviewer for this insightful comment. We agree that the reduced inference speed gain on dense prediction tasks is partly due to high-resolution inputs; however, we would like to kindly point out that the key factor is the increased computational demand of tensor operations.

1. We note that FFN layers still occupy a large portion of the total computational complexity:

   Using Swin Transformer as an example, the theoretical computational complexity of a single MHSA layer is $O(4hwC^2+2M^2hwC)$ while the corresponding FFN layer complexity is $O(8hwC^2)$. Thus, only when $M^2>2C$ does the MHSA layer complexity exceed that of the FFN layer.

   However, in the Swin Transformer architecture, the window size $M$ is fixed at 7 while the channel dimension $C$ is at least 96 and increases across layers. Therefore, the FFN layers consistently represent a larger portion of the computational complexity.

2. As the input resolution increases, the processing time for numerous tensor operations (e.g., reshaping, copying, and concatenating) also rises significantly. Each Swin Transformer layer operates two window-reshaping operations, one window-shifting operation and one mask-copying operation. These on-device tensor operations become non-negligible when the input resolution reaches 800$\times$800.

In fact, this reduced speed gain on dense prediction tasks is a common challenge, as evidenced in similar work. For example, our main state-of-the-art competitor, SLAB [7], achieves even smaller speed improvements on dense prediction tasks with the same predictor than our RePaFormer.

 

---

5. Response to W5 (overlapping between Tables 1 and 2):

Table 1 and Table 2 serve different purposes.

Table 1 illustrates that our RePaFormer method is lossless post-reparameterization, assuring users that our method delivers substantial inference speed gains without accuracy drop after reparameterization in practical scenarios.

Table 2 demonstrates that RePaFormer yields a more significant speed gain and a much narrower performance gap when the backbone model complexity increases.

 

---

We sincerely hope our comprehensive explanations and experimental results can address the Reviewer's doubts and respectfully request a reconsideration of the score.

[1] Vasu, Pavan Kumar Anasosalu, et al. "FastViT: A fast hybrid vision transformer using structural reparameterization." ICCV, 2023.

[2] Wang, Ao, et al. "Repvit: Revisiting mobile cnn from vit perspective." CVPR, 2024.

[3] Rao, Yongming, et al. "Dynamicvit: Efficient vision transformers with dynamic token sparsification." NeurIPS, 2021.

[4] Xu, Yifan, et al. "Evo-vit: Slow-fast token evolution for dynamic vision transformer." AAAI, 2022.

[5] Zhang, Hanxiao, Yifan Zhou, and Guo-Hua Wang. "Dense Vision Transformer Compression with Few Samples." CVPR, 2024.

[6] He, Yang, and Joey Tianyi Zhou. "Data-independent Module-aware Pruning for Hierarchical Vision Transformers." ICLR, 2024.

[7] Guo, Jialong, et al. "SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization." ICML, 2024.

[8] Tang, Yehui, et al. "Patch slimming for efficient vision transformers." CVPR, 2022.

[9] Yu, Shixing, et al. "Unified visual transformer compression." ICLR, 2022.

[10] Yu, Fang, et al. "Width & depth pruning for vision transformers." AAAI, 2022.

[11] Yu, Lu, and Wei Xiang. "X-pruner: explainable pruning for vision transformers." CVPR, 2023.

We sincerely thank Reviewer NBjZ for the valuable comments, especially the recognition that the proposed method is motivated by a comprehensive latency analysis.

 

---

1. Response to W2 and Q1 (what makes our method distinct from RepVGG)

We would like to first respond to the Reviewer's major concern about the differences between our structural reparameterization method and RepVGG-style reparameterization. The differences are threefold:

1. Different reparameterization solutions: The key difference is that RepVGG reparameterizes horizontally across parallel convolutional kernels, while RePaFormer reparameterizes vertically on consecutive linear projection weights.

   For instance, RepVGG reparameterizes two parallel convolutional branches with kernels $W_1^{\text{Conv}}$ and $W_2^{\text{Conv}}$ by summing them:

   $$W_{\text{Rep}}^{\text{Conv}} = W_1^{\text{Conv}} + W_2^{\text{Conv}}.$$

   On the contrary, as demonstrated in Equation 6, RePaFormer reparameterizes two consecutive projection weights $W_1^{\text{FFN}}$ and $W_2^{\text{FFN}}$ by multiplying them:

   $$W_{\text{Rep}}^{\text{FFN}} = W_1^{\text{FFN}} \cdot W_2^{\text{FFN}}.$$

   (In the above example, we omit the BatchNorm and suppose $W_1^{\text{Conv}}$ and $W_2^{\text{Conv}}$ have been padded to the same shape.)

2. Different target components: RepVGG and RepVGG-style methods apply reparameterization to multi-branch convolutional layers in CNNs, while our RePaFormer targets FFN layers in ViTs. Their application targets are distinct.

3. Different scopes: Although some previous works [1,2] have attempted to adapt RepVGG-style reparameterization on ViTs by incorporating multi-branch convolutions into the ViT backbone, they only reparameterize the convolutional parts. The main scope of these works is to construct novel mobile-friendly architectures. In contrast, our method is the first to apply structural reparameterization to FFN layers and accelerate existing ViTs/MetaFormers of all sizes.

Moreover, we kindly argue that our channel idle mechanism cannot be regarded as a special case of a dual-branch structure in RepVGG. In RepVGG, all branches must be linear so that they can be reparameterized, whereas in our approach, only one branch is linear while the other one is nonlinear.

Nonetheless, we would not claim our RePaFormer to be more advantageous than RepVGG, as they are parallel approaches solving different problems that can be used simultaneously. We still appreciate the Reviewer for this comment and will include a detailed comparison with vanilla RepVGG-style reparameterization in our revised version.

 

---

2. Response to W1 (scaling poorly to smaller models):

We thank the Reviewer for this comment, which gives us a chance to emphasize our main claim again. As stated in Lines 28, 106-107, 421-423, 464-466 and Tables 2, the most important characteristic of our method is that it consistently yields a more substantial speed gain and a much narrower performance gap when the backbone model complexity increases. We anticipate this method to be increasingly effective on larger Transformer/MetaFormer-based models, aligning well with the growing significance of large foundation models today.

Moreover, in Appendix A.2, we have also observed the mentioned problem and explained that after applying the channel idle mechanism with a high idle ratio (e.g., 75%), tiny models would lack sufficient non-linear transformations, which is the major reason for the performance drop on smaller ViTs. It is commonly acknowledged that smaller ViTs are less robust and suffer more severe performance declines when compressed, which holds for both token pruning [3,4] and parameter pruning methods [5,6].

We would like to empirically validate it by presenting the performance of small-size ViT models with various idle ratios:

As the table shows, our RePaFormers demonstrate narrow performance gaps on smaller models when the idle ratio is less rigorous (i.e., $\theta$ = 25%). While scaling to small or tiny-sized models is not the primary focus of this work, our method still shows effectiveness in these cases. In addition, this hyperparameter sensitivity study is insightful and will be added to the revised version.

We thank Reviewer NBjZ for the valuable feedback and insightful suggestions, which have helped us refine and clarify our work. We have carefully addressed all the raised concerns in our response.

We would greatly appreciate it if Reviewer NBjZ could provide further feedback. Your input is invaluable to ensuring the quality and clarity of our work. Or, if our responses have satisfactorily resolved the concerns, we respectfully request reconsideration of the score based on the clarifications and improvements provided.

We sincerely thank the reviewer for the valuable suggestions and insightful comments. In our response, we have carefully addressed all the raised concerns regarding 1) the distinction between our method and RepVGG, 2) performance on smaller models, 3) latency improvement on dense prediction tasks and other minors.

If the reviewer has any further questions, we are glad to join in the discussion. Otherwise, if our responses have satisfactorily resolved the concerns, could the reviewer reconsider the score based on our clarifications and responses?

We sincerely appreciate Reviewer NBjZ's reply and thoughtful suggestions. The recommendation of clearly stating the application scenario is highly valuable and will greatly enhance the clarity and impact of our work.

In our original manuscript, we have done a series of experiments to explore the adaptability of our method across various architectures, such as PoolFormers. However, through discussions with you and other Reviewers, and especially inspired by Reviewer wqPR, we have gained a chance to present the key advantages and application scenarios of our approach: RePaFormer can significantly speed up large ViT models while even improving accuracy. This insight demonstrates the practical value of RePaFormer in accelerating large-scale models without compromising performance, making it an effective solution for large real-world applications requiring both speed and precision.

To empirically validate the practicality above, we conducted additional experiments on large vanilla ViTs, following Reviewer wqPR's kind suggestion. Both the vanilla and RePaFormer variants of ViT-Large and ViT-Huge are trained from scratch on the ImageNet-1k dataset using the same training recipes with the idle ratio set to 75% by default. The new results, along with those for MLPMixer-l16 reported in the original manuscript, are shown in the table below:

We are thrilled to emphasize that our method not only drastically reduces model size and latency but also achieves HIGHER top-1 accuracy on large models with more than 200M parameters and computational complexities exceeding 40 GMACs. For instance, RePaViT-Large achieves a 1.7% higher top-1 accuracy (82.0% vs 80.3%) while delivering a 66.8% speed gain (207.2 images/second vs 124.2 images/second) compared to the vanilla ViT-Large. This demonstrates a transformative contribution, as many practical large-scale foundation models for computer vision tasks rely on vanilla ViT as their backbone, such as CLIP [1], SAM [2] and ViT-22B [3].

In addition, as asked by Reviewer NBjZ, we further summarize the guidelines for adopting our method as follows:

• More applicable to models with complex token mixers: When the idle ratio remains constant (such as 75% in our experiments), RePaFormer performs better with more complex token mixers, as witnessed in Table 2.

• More applicable to larger models: When the backbone architecture and idle ratio are fixed, our method generally achieves narrower accuracy drops, and even accuracy gains, on larger models.

• Smaller idle ratios should be leveraged on smaller models: In our Response (Part 1), we have provided results for smaller models under different idle ratios to illustrate this point. Smaller models are less robust and need more nonlinearities to improve the feature extraction capability.

To the best of our knowledge, RePaFormer is the first novel method (orthogonal to network pruning, quantization and distillation) that achieves significant acceleration (~68%) while having positive gains in accuracy (1~2%) instead of accuracy drops, on large and huge ViTs. Considering the unprecedented results RePaFormer is getting, we want to point out that this is a disruptive and timely innovation for the community and a significant addition to the large foundation models acceleration toolkit. Since RePaFormer can be both directly applied to larger ViT architectures and combined with other acceleration techniques such as quantization, we believe RePaFormer will catalyze further research and breakthroughs on ViT's speed and accuracy. We strongly believe that the weight and impact of this work make it best-suited for the prestigious ICLR, and the community will benefit greatly by seeing it soon from this venue.

We hope our response can address all your concerns and demonstrate the significance of our contributions. We kindly request your strong support by considering a score increase.

 

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." ICML, 2021.

[2] Kirillov, Alexander, et al. "Segment anything." ICCV, 2023.

[3] Dehghani, Mostafa, et al. "Scaling vision transformers to 22 billion parameters." ICML, 2023.

Dear Reviewer NBjZ,

We sincerely thank you for your continued engagement in the discussion. In our previous response, we made every effort to address your further concerns and to clarify the key contributions and significance of our work, including:

• Providing additional experimental results on large ViT models, where our method achieves both improved efficiency and increased accuracy.

• Demonstrating the transformative contribution of our work for large-scale foundation models in vision tasks.

• Summarizing the guidelines for employing our method on different model architectures and different model sizes.

We greatly appreciate your time in carefully reviewing our further response. If it satisfactorily resolves all your concerns, we would be deeply grateful for your support of our work and reconsideration of the score.

Best regards,
Authors

5. Response to Q2 (improvements in accuracy should be supported by empirical results):

We thank the reviewer for this question. First, we would like to clarify that a more precise claim should be: "Improvements with greater speed gains and narrower accuracy gaps are expected on larger models". This claim aligns with the statements in Lines 106-107, 421-423, 464-466 and Tables 2, 5, 6.

In addition, we would like to provide results of RePaFormer when using ViT-Large and ViT-Huge as the backbone. The vanilla models and their RePaFormer versions are trained from scratch solely on the ImageNet-1K dataset with the same training schema outlined in the manuscript for fairness. We set the drop path rate at 0.3 for both models. The experiments are ongoing and the results will be updated later (update: ViT-Large and ViT-Huge results have been updated).

 

---

In the end, we sincerely appreciate Reviewer wqPR for all the insightful suggestions. We will emphasize the advantages of our approach in the revised version and include the above experiments and analysis to provide further clarity. And we're willing to answer any further questions. Given this is a new and novel direction in the efficient ViT domain, we hope to get Reviewer wqPR's strong support by increasing the score.

 

[1] Yu, Weihao, et al. "Metaformer is actually what you need for vision." CVPR, 2022.

[2] Geva, Mor, et al. "Transformer feed-forward layers are key-value memories." EMNLP, 2021.

[3] Zhang, Jiangning, et al. "Rethinking mobile block for efficient attention-based models." ICCV, 2023.

[4] Wang, Ao, et al. "Repvit: Revisiting mobile cnn from vit perspective." CVPR, 2024.

[5] Zhang, Hanxiao, Yifan Zhou, and Guo-Hua Wang. "Dense Vision Transformer Compression with Few Samples." CVPR, 2024.

[6] Guo, Jialong, et al. "SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization." ICML, 2024.

We sincerely appreciate Reviewer wqPR for all the contributive comments and support to our innovative work, especially highlighting that our idea is great and the experiments are extensive.

 

---

1. Response to W1 (justification of the advantage of the proposed method):

We thank the reviewer for acknowledging that direct comparisons with other network compression methods are out of the scope of this work. And we admit that highlighting the significance of our method and demonstrating the advantages over other compression methods is necessary in our work.

First, we would like to emphasize the significance of our method as follows:

1. Pioneering the acceleration of FFN layers: The FFN layer in MetaFormer-structured models is essential and indispensable. Specifically, [1] empirically demonstrates that the token mixer (e.g., self-attention or convolution) can be replaced by simpler operations (e.g., pooling), while the FFN layer should remain indispensable. Besides, [2] investigates the critical role of the FFN layer in the general Transformer architecture.

   However, despite its importance, the FFN layer constitutes a significant portion of the inference time, as shown in our study, yet there has been little research on accelerating this component. To the best of our knowledge, this work is the first to specifically target the acceleration of the FFN layer.

2. Generalizability to various MetaFormer-structured models: The MetaFormer structure has been widely validated as the de facto architecture for computer vision tasks by various works [3,4]. Our method can be seamlessly integrated into these models without requiring specialized design modifications. This generalizability has been thoroughly demonstrated through the results presented in Tables 1 and 2.

3. Scalability to larger models: As shown in Table 2, our method achieves greater speed gains, smaller model sizes, and narrower performance gaps as the model size increases within the same architecture. when scaling from DeiT-Tiny to DeiT-Base, the accuracy drop decreases significantly from 7.9% to 0.4%, while the inference speed gain increases from 32.6% to 67.5%.

Next, when compared with other compression methods, our method has the following advantages:

1. Hardware friendly: Our reparameterized model is dense and structurally regular, making it efficient to run on general-purpose hardware without requiring specialized hardware support. On the contrary, quantization needs support for low-precision computations, and pruning usually requires support for sparse matrix operations.

2. Easy to implement: Our RePaFormer is compatible with existing training and deployment pipelines and can be seamlessly embedded into existing MetaFormer-structured models. There is no need for specific adjustments in the training framework like quantization, distillation or pruning.

In addition, we would also like to compare our method with a state-of-the-art pruning method, DC-ViT [5] (CVPR24), since pruning methods are closer to our approach. Given that DC-ViT has not released the code, we only compare the latency drop reported in the paper. As the table below shows, our method delivers comparable performance to the state-of-the-art pruning method and achieves a better trade-off on larger backbones.

In conclusion, the suggestion to justify our advantages over existing methods is very constructive and insightful. We will take this suggestion and include the discussion in our revised version. Nonetheless, it is worth noting that our method is parallel to these model compression methods and can be combined with them to achieve further acceleration. We hope our approach provides a promising direction for accelerating large ViT models.

2. Response to W2 (the performance gap with self-supervised baselines is larger):

We thank the Reviewer for this insightful comment. However, applying our method to self-supervised learning models is an extended exploration included in this paper. The primary purpose of this experiment is to validate the key characteristic of our model—the inference speed gain increases and the accuracy gap narrows as the model size grows—still holds for self-supervised learning. Thus, we simply train our RePaFormers with the same self-supervised training schema as the baselines without any optimizations. We acknowledge the importance of employing our method in self-supervised learning baselines and plan to explore this direction in-depth in our future work, with a specific focus on generalizability as suggested.

After all, the main focus of our work still remains on supervised learning, where our method achieves significant success. This alone already represents a substantial contribution of our approach.

 

---

3. Response to W3 (the performance gap at dense prediction tasks is non-negligible):

We interpret this comment as asking why the performance gap on dense prediction tasks is very different from the performance gap on classification task:

1. First, we kindly argue that dense prediction tasks and classification tasks use distinct evaluation metrics—mean average precision (mAP) for dense prediction and accuracy for classification. Directly comparing the different metric gaps is thus inherently unfair.

2. Second, even if such a comparison is made, the performance gaps for dense prediction tasks align closely with those for classification tasks. Specifically, the accuracy gaps for RePa-Swin-Small and RePa-Swin-Base on classification tasks are 1.4% and 0.9%, respectively. Similarly, the mAP gaps for RePa-Swin-Small and RePa-Swin-Base on object detection tasks using Mask R-CNN are 0.019 and 0.01, respectively. These results indicate that the performance gaps are comparable, and the trend is consistent: larger models exhibit narrower performance gaps.

3. Additionally, when incorporating RetinaNet as the dense predictor, our method achieves even higher mAP on the object detection task, further validating its effectiveness.

However, if this comment is asking why the speed improvement reduces on dense prediction tasks compared to that on the classification task:

1. First, we note that FFN layers still occupy a large portion of the total computational complexity. Using the Swin Transformer as an example, the theoretical computational complexity of a single MHSA layer is $O(4hwC^2+2M^2hwC)$ while the corresponding FFN layer complexity is $O(8hwC^2)$. Thus, only when $M^2>2C$ does the MHSA layer complexity exceed that of the FFN layer.

   However, in the Swin Transformer architecture, the window size $M$ is fixed at 7 while the channel dimension $C$ is at least 96 and increases across layers. Therefore, the FFN layers consistently represent a larger portion of the theoretical computational complexity.

2. The key factor is the increased computational demand of tensor operations. In fact, as the input resolution increases in dense prediction tasks, the processing time for several tensor operations (e.g., reshaping, copying, and concatenating) also rises significantly. For instance, each Swin Transformer layer operates two window-reshaping operations, one window-shifting operation and one mask-copying operation, which become non-negligible when the input resolution reaches 800$\times$800.

   This reduced speed gain on dense prediction tasks is a common challenge, as evidenced in similar work. For example, our main state-of-the-art competitor, SLAB [6], achieves even smaller speed improvements on dense prediction tasks with the same predictor than our RePaFormer.

 

---

4. Response to Q1 (what would be the training time of the proposed method compared to the vanilla version):

We have provided the estimated training cost of each model on the ImageNet-1K dataset using 16 NVIDIA H100 GPUs in terms of the GPU hours in the table below:

The BatchNorm in RePaFormers introduces an increased synchronization time between devices during training, leading to 16~28% longer training time on the ImageNet-1K dataset. However, the training overhead is still less significant compared to distillation methods, which can increase the training time by more than 40%.

We thank Reviewer wqPR for the valuable feedback and insightful suggestions, which have helped us refine and clarify our work. We have carefully addressed all the raised concerns in our response.

We would greatly appreciate it if Reviewer wqPR could provide further feedback. Your input is invaluable to ensuring the quality and clarity of our work. Or, if our responses have satisfactorily resolved the concerns, we respectfully request reconsideration of the score based on the clarifications and improvements provided.

We sincerely thank the reviewer for the valuable suggestions and insightful comments. In our response, we have carefully addressed all the raised concerns regarding 1) the advantages of our method, 2) the performance gaps on SSL tasks, 3) the performance gaps on dense prediction tasks, 4) the training cost and 5) the performance on even larger models.

If the reviewer has any further questions, we are glad to join in the discussion. Otherwise, if our responses have satisfactorily resolved the concerns, could the reviewer reconsider the score based on our clarifications and responses?

Dear Reviewer wqPR,

We sincerely appreciate your insightful comments and suggestions, especially the recommendation to empirically prove the effectiveness of our method on large and huge ViTs. Through the discussion with you and other Reviewers, we have gained a chance to present the key advantages and application scenarios of our approach: RePaFormer can significantly speed up large ViT models while even improving accuracy. This insight demonstrates the practical value of RePaFormer in accelerating large-scale models without compromising performance, making it an effective solution for large real-world applications requiring both speed and precision.

We would like to share the results on ViT-Large and ViT-Huge, along with those for MLPMixer-l16 reported in the original manuscript, in the table below. Our method not only drastically reduces model size and latency but also achieves HIGHER top-1 accuracy on large models with more than 200M parameters and computational complexities exceeding 40 GMACs.

To the best of our knowledge, RePaFormer is the first novel method (orthogonal to network pruning, quantization and distillation) that achieves significant acceleration (~68%) while having positive gains in accuracy (1~2%) instead of accuracy drops, on large and huge ViTs. Considering the unprecedented results RePaFormer is getting, we want to point out that this is a disruptive and timely innovation for the community and a significant addition to the large foundation models acceleration toolkit. Since RePaFormer can be both directly applied to larger ViT architectures and combined with other acceleration techniques such as quantization, we believe RePaFormer will catalyze further research and breakthroughs on ViT's speed and accuracy. We strongly believe that the weight and impact of this work make it best-suited for the prestigious ICLR, and the community will benefit greatly by seeing it soon from this venue.

We would like to sincerely thank you once again for your thoughtful review and constructive discussion. We hope to receive your strong support through consideration of a score increase.

Best regards,
Authors