Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation

W1 & Q1: Clarify the novelty of the proposed methods over Conditional Adapters and AdaMix

Thanks. We would like to clarify the novelty of our method over Conditional Adapters and AdaMix as follows:

Conditional Adapters[1]:

• The token selection strategy in the token dispatcher is novel. The token dispather in conditional adapter[1] selects top-K tokens in each layer to pass through. However, the token dispatcher in DyT learn to select an approprate number of tokens in each layer based on inputs.
• The target model is different. The conditional adapter[1] is primarily designed for the encoder-decoder langauge model T5 [3]. Our experiments below demonstrate that applying this approach to vision transformers is suboptimal. DyT’s novelty lies in proposing a token dispatcher specifically beneficial for vision transformer.
• The block to conduct token skipping is novel. In conditional adapter [1], tokens directly skip the whole layer. In DyT, we propose four model variants, explore their effectiveness, and find that skipping the MLP block is suitable for vision transformer.

To verify that the superiority of our token dispatcher in vision transformer, we conduct experiments by replacing the proposed token dispatcher with the soft top-K method from [1].

Setting: DyT-top-K denotes using the soft top-K in [1] as the token dispatcher. We report the average FLOPs on CIFAR-100.

Analysis: DyT achieves obviously better performance than "DyT-top-K". There are two reasons:

• The soft top-K operation in [1] always skips K tokens per block. In contrast, DyT’s learned token dispatcher skips fewer tokens in lower layers and more in higher layers (see Figure 5). This indicates that the number of skipped tokens should vary by layer, and skipping a fixed number of tokens, as in [1], is suboptimal.
• The computational allocation for each test sample is constant in [1], whereas DyT allocates different FLOPs for different samples, allowing the model to handle hard and easy samples differently.

Conclusion: The token dispatcher in [1] is not suitable to vision transformer, verifing the importance of DyT. We will add this experiment into the Appendix in the revision.

AdaMix [2]:

• Our routing mechanism is novel. During training, AdaMix [2] lacks a learning-based router, instead randomly selecting a down projection layer and an up projection layer for each batch of data. Consequently, all images in a batch are processed by the same experts. In contrast, the MoE-adapter in DyT employs a learning-based router to generate scalar weights for the experts based on input features, allowing features from different images to yield distinct expert weights.
• The adapter architecture is novel. During inference, AdaMix fuses all experts by weight averaging, resulting in an ordinary adapter like [4], whereas we retain all experts. Thus, the MoE-adapter in DyT has more capacity to tackle challenging tasks. Experiments below verify this.

Below, we conduct experiments by replacing our MoE-adapter with AdaMix.

Setting: "DyT-AdaMix" denotes using the AdaMix [1] as the adapter. "DyT $N=4$" denotes the DyT with our MoE-Adapter. We report the accuracy on three image datasets and two video datasets. The average FLOPs on CIFAR-100 and K400. The bottleneck dimension in the adapter is set to 64 for all models.

Analysis:

• "DyT-AdaMix" exhibits slightly better performance on image datasets due to its training strategy.
• “DyT-AdaMix” does not enhance DyT’s performance on challenging tasks, such as video classification, and even significantly reduces performance. This may be attributed to the random selection mechanism, which further diffuses learning on challenging tasks.
• As an method to improve performance in challenging tasks without introducing extra FLOPs, our MoE-adapter achieves the best performance on K400 and SSv2.

Conclusion:

The adapter in AdaMix [2] is a training strategy rather than an effective adapter architecture. It does not assist DyT in tackling challenging tasks. We will add this experiment to the Appendix in the revision.

W2: The line spacing is too narrow.

We sincerely appreciate this valuable suggestion. We find that author are permitted to add one page to the main paper in the final version. We will increase the line spacing as recommended.

Q2: DyT employs a more complex loss function and distillation. Does this increase the training time

Thanks. We employ the complete model loss $L^\prime_{cls}$ and distillation loss $L_{distill}$ to improve the performance. Although their use increases the duration of a training iteration by 1.8 times compared to not using them, this approach brings significant performance improvements (See Table 11). Given that our primary objective is to improve inference efficiency, this additional training cost is acceptable. Therefore, we believe it is valuable to include these losses during fine-tuning.

Reference

[1] Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inference. 2023.

[2] AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning, 2022.

[3] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. 2020.

[4] Adaptformer: Adapting vision transformers for scalable visual recognition, 2022.

Thank you for your response. I will carefully consider your clarifications, and if other reviewers also find the innovation of the paper to be sufficient during discussion, I will raise my score.

W1: The core technique in Dynamic Tuning seems to be token pruning without explicitly reducing the number of tokens. This paper does not well demonstrate the difference between existing dynamic token pruning methods and the proposed method.

Thanks. We would like to kindly clarify that our method is not a token pruning method but rather a token skipping method. In our approach, some tokens skip the original blocks and only pass through the adapter, preserving the feature map completely. Below, we list the differences between our method and representative token pruning methods (DynamicViT[1], EViT[2], ToMe[3]) below. We will include these in Section 2 in the revision.

Note that, our method can be seamlessly combined with ToMe (Table 5)

Analysis:

• Compared to these methods, our approach does not rely on a manually set number or percentage of tokens to prune. Instead, it learns where and how many tokens to skip, constrained by the average activation rate loss $L_{rate}$.
• Our model allocates computation in a data-dependent manner, which varies between different samples.
• Our method can also handle the dense prediction task, whereas other methods fail due to the incomplete feature map.
• Previous token pruning methods primarily focues on accelerating the model within the same dataset used for pretraining, while DyT aims to improve efficiency during cross-domain adaption.

Reference

[1] Dynamicvit: Efficient vision transformers with dynamic token sparsification, 2021

[2] Expediting Vision Transformers via Token Reorganizations, 2022

[3] Token Merging: Your ViT But Faster, 2022

Dear Reviewer Zpyf,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarize our responses here.

1. [Difference between existing dynamic token pruning methods and the proposed method]

Response:

• Our model learns where and how many tokens to skip.

• Our model allocates computation in a data-dependent manner.

• Our model preserves complete feature maps and can handle denste prediction tasks.

• Our model aims to improve efficiency during cross-domain adaption.

Since the discussion stage is already halfway through, may we know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best Regards

Authors

W1: Does introduced additional loss functions affect the speed of convergence

Thanks. The introduced additional loss functions do not impact the speed of convergence. We plot the loss values throughout the fine-tuning process and record the test accuracy at every training epoch. Please find the figure in the attachment PDF. The explanation and figures here will be included in the Appendix, with a new section discussing the convergence speed.

W2: It is valuable to explore the disparities in the calculated results of Sigmoid and Gumbel-Sigmoid.

Thanks. We would like to clarify that DyT only employs the Gumbel-Sigmoid during training and replaces it with Sigmoid in the inference. Based on the suggestion, we conduct an experiment using Sigmoid during both training and inference, as shown in the table below:

Setting: "DyT-" denotes using Sigmoid during both training and inference. We report the average FLOPs on CIFAR-100.

Analysis & Conclusion

• Replacing the Gumbel-Sigmoid with Sigmoid negatively impacts performance, as it results in a non-differentiable problem and increases the difficulty of fine-tuning (Lines 137-149). We will include this experiment in the Appendix in the revision.

W3: Certain experimental phenomena may become more transparent with additional explanations

Thanks. These valuable questions and explanations will be added to the “Frequent Questions” section in the Appendix of the revised manuscript.

Q1: Potential performance impact of token pruning on the detection task.

Thanks for this valuable suggestion. We conduct experiments on the COCO. Unlike token pruning, DyT employs a token skipping method, enabling it to effectively address the object detection task.

Setting: we adopt ViTDet[2] as the detector. We fine-tune the model for 12 epochs on the COCO dataset. The bottleneck dimension $d$ in adapters is set to 128. Here, we only measure the FLOPs in the vision transformer and feature pyramid.

Analysis & Conclusion:

• The full-tuning method performs the best, likely due to the significant gap between bounding box regression and the pretraining of the vision transformer, requring more parameters in the backbone to be updated.
• DyT demonstrates superior performance compared to the AdapterFormer[3] with fewer FLOPs, validating the effectiveness of our approach.
• DyT with the proposed MoE-adapter (DyT $N=4$) surpasses the original DyT, highlighting the importance of the MoE-adapter.

Q2: Paper [1] dynamically assesses token significance, and it would be beneficial to discuss and cite it

We appreciate this valuable suggestion. We carefully read the paper and will cite it by "Zhou et.al [1] propose assessing token significance with a ReLU function to effectively tackle point cloud analysis" in Related Work.

Reference

[1] Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis, 2024

[2] Exploring plain vision transformer backbones for object detection, 2022

[3] Adaptformer: Adapting vision transformers for scalable visual recognition, 2022

[4] Unified perceptual parsing for scene understanding, 2018

Dear Reviewer uJS2,

Thanks so much for the support. We sincerely appreciate the valuable feedback.

We will include all experimental results and analysis in the revision to facilitate future work. We will clarify the differences with other studies highlighted by reviewers in the revised paper.

We hope our response resolves your concerns. We are more than willing to answer any further questions you may have. Your support is appreciated and helps us improve our work.

Best regards

Authors

We will include all explanations and results in the revision.

W1:MoE improves performance only in video classification, not image classification, and is absent from most experiments

We appreciate this comment. We would like to clarify that the MoE-adapter is primarily designed to enhance the adapter’s capacity to address challenging tasks (Lines 180-183). In addition to video classification, we validate its effectiveness in semantic segmentation (Appendix Table 10) and object detection (In reponse to Reviewer uJS2). We do not employ it in the VTAB-1K experiments as it does not outperform the model without it, likely due to the extremely limited training data.

W2:Table 6 shows that using MoE results in fewer FLOPs, which is theoretically implausible

Thanks. We would like to clarify that it is reasonable for the model with the MoE adapter to result in fewer FLOPs:

• The FLOPs of our models depend on learned token dispather during fine-tuning and may slightly fluctuate around the target FLOPs (controlled by $L_{rate}$).
• The extra computational cost of the adapter and the MoE adapter is nearly equivalent (Lines 202-204).

W3 & Q1:MoE might be better suited for the appendix

We appreciate this valuable suggestion. In the revision, we will move the introduction of the MoE adapter and its corresponding experiments to the Appendix.

W4:The token dispatch mechanism is similar to existing dynamic neural network

Thanks. We present the differences below:

Analysis & Conclusion:

• Dynamic neural networks typically prune a certain number or percentage of tokens in fixed layers, while our token dispatcher learns the number of tokens to skip before each MLP block.
• DynamicViT and EViT maintain the same FLOPs for all samples, while DyT achieves data-dependent computation, which is more flexible and reasonable.
• Instead of token pruning, the token skipping in our token dispatcher allows the DyT to preserve complete feature maps, enabling it to perform dense prediction tasks.

W5:The paper should report the performance of DynamicViT and EViT's full fine-tuning and their combination with other PEFT methods

Sincerely thanks. We present the results below.

Setting: We combine DynamicViT and EViT with AdaptFormer. The bottleneck dimmension is set to 8.

Analysis and Conclusion:

• Combining DynamicViT and EViT with AdaptFormer enhances performance, validating the significance of exploring both parameter and inference efficiency for vision transformers
• DyT outperforms “DynamicViT+AdaptFormer” and “EViT+AdaptFormer” by learning to skip an appropriate number of tokens at each MLP block, rather than pruning a fixed number (EViT) or percentage (DynamicViT) of tokens at specific layers. Figure 5 in the main paper illustrates that different datasets benefit from varying token-skipping strategies.

W6:It is unclear why MLP Dispatch has lower FLOPs than Layer Dispatch

Thanks. There are two reasons:

• The actual token activation rate during inference depends on the learned token dispatcher, causing real FLOPs to fluctuate around the theoretical value.
• We do not strictly control FLOPs across the four variants. Specifically, we set the activation rate $r$ to 0.5 for “Attention Dispatch” and “MLP Dispatch” variants, and to 0.7 for “Attention-MLP Dispatch” and “Layer Dispatch” variants, ensuring similar average FLOPs. Figure 6 (Appendix) shows that “MLP Dispatch” consistently achieves the best performance under similar FLOPs.

W7:The authors should explain why their method outperforms traditional adapters

Thanks. We list the explanations below:

• The dynamic architecture in DyT enhances generalization. It introduces a form of disturbance in the input data, akin to Dropout. This mechanism is particularly crucial when training data is limited e.g. VTAB-1K.
• The distillation loss in DyT. We adopt the complete model as the teacher of the dynamic model, significantly enhancing performance. Such a self-distillation mechanism is only available in the dynamic architecture.
• Previous work [1] and DynamicViT also show dynamic architectures outperforming static models with fewer FLOPs.

Q2&L1:Discuss the increased training time caused by the need for two forwards

Thanks. We acknowledge that training with two forward passes takes 1.8 times longer than only one pass, but this significantly enhances performance without compromising parameter and inference efficiency. Given that our primary contribution focuses on improving inference efficiency, the additional training time would be acceptable.

Q3:The paper should include an ablation study on the loss function

Thanks. We present the results below.

Analysis and Conclusion:

Removing $L_{distill}$ or $L^\prime_{cls}$ negatively impacts performance, validating the effectiveness of loss functions in DyT.

Reference:

[1] Latency-aware Unified Dynamic Networks for Efficient Image Recognition. 2024.

Dear Reviewer fmWb,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we summarize our responses here.

1. [MoE improves performance only in video classification, not image classification, and is absent from most experiments]

Response:

• MoE-adapter is primarily designed to enhance the adapter’s capacity to address challenging tasks (Lines 180-183).
• We also validate its effectiveness in semantic segmentation (Appendix Table 10) and object detection (In reponse to Reviewer uJS2).
• We do not employ it in the VTAB-1K experiments as it does not outperform the model without it, likely due to the extremely limited training data

2. [Table 6 shows that using MoE results in fewer FLOPs, which is theoretically implausible]

Response:

• The FLOPs of our models depend on learned token dispather during fine-tuning and may slightly fluctuate around the target FLOPs.
• The extra computational cost of the adapter and the MoE adapter is nearly equivalent (Lines 202-204).

3. [MoE might be better suited for the appendix]

Response:

• In the revision, we will move the introduction of the MoE adapter and its corresponding experiments to the Appendix.

4.[The token dispatch mechanism is similar to existing dynamic neural network]

Response:

• DynamicViT and EViT prune a certain number or percentage of tokens in fixed layers, while our token dispatcher learns the number of tokens to skip before each MLP block.

• DynamicViT and EViT maintain the same FLOPs for all samples, while DyT achieves data-dependent computation.

• DyT to preserve complete feature maps

5.[The paper should report the performance of DynamicViT and EViT's full fine-tuning and their combination with other PEFT methods]

Response:

• Combining DynamicViT and EViT with AdaptFormer enhances performance.
• DyT outperforms “DynamicViT+AdaptFormer” and “EViT+AdaptFormer” by learning to skip an appropriate number of tokens at each MLP block.

6.[It is unclear why MLP Dispatch has lower FLOPs than Layer Dispatch]

Response:

• The actual token activation rate during inference depends on the learned token dispatcher.
• We set the activation rate $r$ to 0.5 for “Attention Dispatch” and “MLP Dispatch” variants, and to 0.7 for “Attention-MLP Dispatch” and “Layer Dispatch” variants, ensuring similar average FLOPs.

7.[The authors should explain why their method outperforms traditional adapters]

Response:

• The dynamic architecture in DyT enhances generalization.
• The distillation loss in DyT improves performance.
• Dynamic architectures outperforming static models is also observed in previous works.

8.[Discuss the increased training time caused by the need for two forwards]

Response:

• Two forward passes takes 1.8 times longer than only one pass
• Given that our primary contribution focuses on improving inference efficiency, the additional training time would be acceptable.

9.[The paper should include an ablation study on the loss function]

Response:

• Removing $L_{distill}$ or $L^\prime_{cls}$ negatively impacts performance.

Since the discussion stage is already halfway through, may we know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best Regards

Authors