Hierarchical Side-Tuning for Vision Transformers

Thank you for your constructive comments and suggestions. Below, we have provided a point-by-point response to address your comments and concerns.

w1: Differences among HST, SAN and ViT-Adapter.

Please refer to the (3) Differences among HST, ST, LST, SAN, and ViT-Adapter section in the general response.

w2: Scaling analysis.

We can observe that HST performs more satisfactorily when using larger models like ViT-L. There is a performance gap of 2.6 AP_b between HST and full finetune on the base model, while achieving the same or even better performance on the large model. Additionally, the LoRA method also demonstrates further improvement in performance with larger models, although it still remains slightly lower than full finetune by approximately 1.0 AP_b.

w3: It would be helpful to see a similar analysis on COCO — or at least the current ablation study broken down over classification/structured/specialized.

In the ablation study, we provided both the average precision on VTAB and the values of AP_b and AP_m using Mask R-CNN on COCO. Here, we present more detailed experimental results on VTAB.

w4: Questions about the grad-cam visualizations.

We can confidently state that these results have not been cherry-picked. In fact, due to limitations in the length of the manuscript, we were only able to temporarily provide a set of visualizations in the main text. Nevertheless, more visualizations are available in Figure 11 of the appendix. It can be observed that the visualization outcomes of Grad-CAM are as expected, owing to the hierarchical architecture of HSN employing convolutional embeddings and modeling fine-grained features in the shallow stages. Consequently, the model naturally has the ability to focus on detailed foreground representations.

Q1: I am curious how well-tuned were the baselines for LoRA, etc. Were these coped from existing papers, or did these have a similar HP search, compared to HST?

While numerous PETL methods primarily concentrate on image classification tasks, the experimental results for these methods in image classification are widely available in multiple papers. Our paper's VTAB results are consistent with those reported. However, when it comes to dense prediction tasks, it becomes challenging to find relevant and dependable results in existing literature. Therefore, we took the initiative to independently replicate the results of all relevant methods in dense prediction tasks. We ensured that all experiments were conducted with the same set of training hyperparameters to maintain consistency across the evaluations.

Q3: Why do you think the method outperforms full FT for VTAB, but not as strong for ObjDet/SemSeg?

Under the experimental setup of the VTAB benchmark with only 1000 training images, full fine-tuning can easily lead to overfitting. However, when we transition to general image classification datasets such as CIFAR-100 or ImageNet-1k (as shown in Table 6 in our paper's appendix or Table 5 in SSF[1]), we observe that full fine-tuning still maintains optimal performance. Therefore, the size of the training dataset plays a significant role.

Additionally, it is important to note that detection and segmentation tasks differ fundamentally from classification tasks. Dense prediction tasks require models to discern features at various granularities, and simply adding a few trainable parameters to the backbone does not enhance the model's ability to model multi-scale features. As a result, the performance may be inferior.

Finally, as mentioned in the scaling analysis, based on the growth trend, we believe that PETL methods will exhibit more powerful performance as the pre-trained model becomes larger. Hence, the size of the model is also one of the influencing factors.

[1] Lian D, Zhou D, Feng J, et al. Scaling & shifting your features: A new baseline for efficient model tuning[J]. Advances in Neural Information Processing Systems, 2022, 35: 109-123.

Q4: Why is the conv stem necessary? What happens when it is excluded? How is the architecture adapted for classification tasks in VTAB?

Firstly, the conv stem serves a similar purpose to the patch embedding in popular pyramid-style vision Transformer models. It enables downsampling to reduce the number of input tokens by fourfold. The conv stem consists of two 3x3 convolutional layers with a stride of 2, which introduces spatial prior knowledge related to the image. In our experiments, we observed that the conv stem performs better in terms of both performance and efficiency compared to directly using a 4x4 patch embedding with a stride of 4.

If the conv stem layer is removed, we would directly model an H x W feature map. In this scenario, whether conducting self-attention or cross-attention, the model would face training instability due to the excessively large number of input tokens.

Lastly, when transitioning to image classification tasks, there is no need to utilize intermediate [1/4, 1/8, 1/16] features. Only the 1/32 output feature requires a global average pooling operation, followed by a linear layer for classification.

Thank you for your valuable feedback. Below, we address your comments and provide our responses point by point.

w1: Differences between ViT-Adapter and HST.

Please refer to the (3) Differences among HST, ST, LST, SAN, and ViT-Adapter section in the general response.

w2: The proposed algorithm incorporates several other designs that share similarity with previous work, VPT and BitFit, and it is not clear the improvement comes from the side network or other designs.

MetaT and prompt tokens in VPT serve the purpose of introducing trainable tokens into ViT. However, they have distinct capabilities within their respective fine-tuning frameworks. MetaT is designed to facilitate efficient computation, using linear complexity for cross-attention within HSN while integrating acquired knowledge into HSN. In contrast to VPT, our approach eliminates the need to manually determine the optimal number of tokens for each task, which often requires a large quantity. Additionally, LN-tuning is employed to reduce the distribution gap between MetaT and the output features of image patch tokens, enabling efficient meta-global/fine-grained injection.

Furthermore, to better illustrate the effects of the side network, we conducted two specific experiments. HST.v0 is without MetaT and LN-Tuning, with cross-attention relying on globalT to serve as the key and value. The second experiment is to add LN-Tuning to LoRA, verifying the effect of integrating different PETL methods. The results are as follows.

We can observe that despite the removal of MetaT and LN-Tuning, this method still achieves state-of-the-art performance in classification tasks and surpasses VPT and AdaptFormer in dense prediction tasks. However, the absence of MetaT and LN-Tuning results in the loss of their crucial ability to inject meta-global/fine-grained information, which is the most critical aspect of our framework. On the other hand, even with the integration of LN-Tuning, LoRA does not exhibit a significant improvement in accuracy. Therefore, although MetaT and LN-Tuning may share similarities with other methods, they play distinct roles in our framework, and each component complements the others to achieve optimal performance.

w3: Confused by the number of tunable parameters.

We apologize for the confusion caused by the lack of detailed explanation regarding the exact architecture design. Please refer to our (2) Detailed Architecture Specifications provided in the general response, and feel free to ask if you have any further questions.

Thank you for your careful review and valuable feedbacks. In the following section, we address your comments and provide point-by-point responses.

w1: Differences between SAN and HST.

Please refer to the (3) Differences among HST, ST, LST, SAN, and ViT-Adapter section in the general response.

w2: How to implement the hierarchical dense prediction in Fig. 2?

In simple terms, by equipping ViT with a trainable side network to enable its multi-scale output capability, it can be viewed as an overarching pyramid-based architectural model. Subsequently, by incorporating neck modules (FPN, ChannelMapper, etc) and any desired detection or segmentation heads, it can effectively accomplish various dense prediction tasks.

w3: It's not convincing of the analysis of "constraining the number of trainable prompts to a few number". Besides, the ablation study shows continuous growth when the number N increases.

The quantity of MetaT in HST plays a crucial role in determining both the transfer performance and the computational complexity of the Side blocks. Increasing the number of MetaT has an intuitive link with performance enhancement, as observed in VPT as well. However, our findings indicate that using just one MetaT is sufficient to yield satisfactory results in image classification transfer tasks. Although a slight performance boost is observed with 32 MetaT, it significantly increases training and inference costs, which is similar in dense prediction tasks. Therefore, to strike a balance between speed and accuracy, it is favorable to choose varying MetaT quantities (not exceeding 8) based on specific tasks as a hyperparameter choice.

w4: 14.4% (76.0% vs. 65.6%) --> 10.4%

Thanks for pointing out this mistake in our paper. We will carefully review the paper to ensure that such mistakes are corrected in the final version.

Q4: Providing more discussion about the comparison on Table 1, since the performance is not always the best.

Firstly, let's delve into the VTAB-1k benchmark in detail. The VTAB-1k benchmark consists of 19 tasks spanning various domains, including Natural (images from standard cameras), Specialized (context-specific domains like medical and satellite imaging), and Structured (synthesized images with controlled conditions). Each task-specific dataset contains 1000 training samples, with varying sample counts per class, and is evaluated using the original test set.

It is important to note that with only 1,000 training images, different PEFT methods demonstrate their own strengths and weaknesses across diverse image domains. However, the crucial factor is whether a method can deliver impressive performance across all 19 tasks, showcasing robust and powerful transfer capabilities. Our HST method accomplishes exactly this, exhibiting outstanding performance across various domains: an average accuracy of 82.83% in Natural, 87.13% in Specialized, 64.45% in Structured, and an overall average accuracy of 76.0%, all of which currently stand as state-of-the-art performance benchmarks.

Furthermore, we conducted experiments in general image classification using CIFAR-100 and four other FGVC datasets. Our HST method outperforms other PETL methods when using both ImageNet-22k and MAE pre-trained weights, further reinforcing the robust transfer capabilities inherent in HST. For detailed results, please refer to Table 6 in Appendix B.

Q5: The performance on much large models

We can observe that HST performs more satisfactorily when using larger models like ViT-L. There is a performance gap of 2.6 AP_b between HST and full finetune on the base model, while achieving comparative or even better performance on the large model. Additionally, the LoRA method also demonstrates further improvement in performance with larger models, although it still remains slightly lower than full finetune by approximately 1.0 AP_b.

Thank you for your valuable feedback. We have carefully considered your comments and provide our responses below.

More implement details

Please refer to our (2) Detailed Architecture Specifications given in the general response.

More fair comparison

By adjusting the dimensions from [32, 48, 64, 72] to [24, 24, 32, 32], we were able to achieve similar numbers of learnable parameters. The experimental results presented in the table below clearly demonstrate that our method consistently delivers state-of-the-art performance on VTAB-1K.

w1: leads to additional computational costs

Apart from LoRA-based techniques capable of weight fusion during inference, most mainstream methods add extra computational load (our paper includes relevant analysis). However, our approach offers a potential advantage by accelerating inference speed through optimized parallel computation. Specifically, our method facilitates concurrent computation, allowing independent progress of calculations in both the ViT network and HSN. This means that the HSN can compute simultaneously using various ViT output features obtained during the forward process of ViT.

Q1: Effect of globalT

As depicted in the table below, we performed an additional ablation experiment without GlobalT, focusing solely on FG Injection. We observed a significant improvement in performance solely due to the implementation of FG Injection. However, when combining both FG Injection and GlobalT, we achieved even better results. Importantly, the introduction of GlobalT resulted in negligible overhead.

Q2: Issues about weight-sharing and number of side block in each stage

Thank you for your informative comment. We would like to say that the designs you mentioned were indeed part of our HSN early version. In that version, we experimented with setting only one Side block in each stage, resulting in a total of four Side blocks in the HSN. Furthermore, we fused the output features of different ViT blocks within the same stage, along with their respective meta tokens, and input them into the Side Block. However, this design approach did not yield satisfactory results. We speculate that the inadequate performance may be attributed to the insufficient number of Side blocks in effectively modeling the fusion of multi-scale image features and ViT feature information within the HSN. Despite having a larger number of channels, the network was too shallow. This discovery ultimately inspired us to the design of alignment of ViT blocks and Side blocks.

Q3: The effect of metaT is not proved

As indicated in the table, the performance achieved solely using GlobalT does not surpass that obtained by solely using MetaT. This discrepancy is primarily due to MetaT's ability to extract more enriched feature from each ViT block compared to GlobalT. Additionally, similar to the prompts in VPT, MetaT demonstrates the capability to transfer information to downstream tasks. We apologize the oversight in not providing an analysis of MetaT's effectiveness in the paper. We will provide more details to elaborate on the effectiveness of MetaT.

HST vs. Side Adapter Network for Open-Vocabulary Semantic Segmentation (SAN):

(a) Motivation:

SAN enhances the capabilities of the existing VL pre-trained model (CLIP) by enabling open-vocabulary semantic segmentation. It achieves this by maintaining CLIP's open-vocabulary recognition by freezing pre-trained weight while using a side adapter network to propose masks for better alignment. Essentially, SAN is a specialized design for CLIP weights, offering a unique and efficient approach to open-vocabulary semantic segmentation.

Conversely, HST is a new method in PETL, shares the same objective as LoRA and adapters. Its goal is to swiftly transfer pre-trained models to different downstream tasks, demonstrating superior performance.

(b) Architecture:

The side network in SAN resembles LST, both using plain Transformer models. However, SAN integrates trainable Query Tokens within the side network to produce mask proposals. These Query Tokens significantly differ from our Meta Tokens. The former, similar to methods like Maskformer[1], serve the purpose of generating mask proposals, while the latter, akin to prompt tuning, are fewer in number and offer global information to the side network. Furthermore, the output features from SAN's side network are injected back into the final layers of the pre-trained model, but this process is not present in LST or our HST.

However, the side network in HST differs substantially from SAN and LST. It's a pyramid-style architecture generating multi-scale features, aiding dense prediction tasks. Additionally, we employ convolutional layers as embedding layers to introduce image-related spatial knowledge.Furthermore, our side network no longer incorporates self-attention; instead, it maximizes the use of feature information learned by MetaT for cross-attention. This approach not only circumvents the significant computational burden when dealing with high-resolution feature maps but also effectively injects pre-trained information into our HSN.

[1] Cheng B, Misra I, Schwing A G, et al. Masked-attention mask transformer for universal image segmentation[C]//Proceedings of the IEEE/CVF CVPR. 2022: 1290-1299.

HST vs. Vit-Adapter:

(a) Motivation:

ViT-Adapter boosts dense prediction task performance by integrating an adapter network into the pre-trained ViT, enhancing its overall performance through joint full fine-tuning. Conversely, HST targets efficient parameter fine-tuning to achieve satisfactory results in various visual tasks like image classification and dense predictions.

(b)Architecture:

ViT-Adapter and HST share similarities in utilizing a plain ViT as their backbone and introducing a randomly initialized adapter/side network into the pre-trained ViT when transferring to downstream tasks, thereby decoupling the model structure between the upstream pre-training and downstream fine-tuning stages. However, they differ in several aspects:

• Firstly, ViT-Adapter utilizes a spatial module to obtain three distinct features of different resolutions, concatting them into a unified entity via its adapter network for multi-scale outputs. In contrast, our HSN, a pyramid-style architecture, progressively captures features of various granularities throughout the network depth, producing multi-scale outputs at different stages.ViT-Adapter's interaction blocks impose a notable computational burden due to numerous input tokens. Conversely, our HSN mitigates this burden by utilizing a limited number of meta tokens, ensuring superior computational efficiency.

• Secondly, ViT-Adapter incorporates bidirectional information flow by exchanging features between the ViT backbone and the adapter network, impacting both training and inference efficiency. In contrast, HST operates with a unidirectional information injection solely from the ViT backbone to the side network, resulting in enhanced training efficiency.

• Thirdly, while ViT-Adapter incorporates sparse attention mechanisms, it requires significant training resources. Our attempts to train the adapter network while keeping ViT frozen within ViT-Adapter led to Out of Memory errors when using V100 GPUs, a problem that didn't occur with HST.