LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning

Dear Reviewer zHY5,

Thank you for appreciating our approach. We will address your comments below.

The working reason of CSPC is not entirely clear. The author should provide a more detailed explanation of the fundamental distinctions among the three types of sources used to construct the input prompts mentioned in Sec 3.2 and elaborate on why "adding the average embedding of patch tokens to the output prompts" leads to performance improvements.

The proposed CSPC (Class-aware Spatial Prompt Coding) is strategically designed to perpetually amass class conscious features, thereby fortifying the model’s prowess in distinguishing and identifying visual categories. The role of the mean of patch tokens behind CSPC is used for enhancing classification performance, as the mean of patch tokens is commonly used for linear probing in self-supervised ViTs, such as DINO and MAE. Adding them to the prompt tokens will help improve the model’s prowess in distinguishing and identifying visual categories.

More importantly, computation overhead is minimal to keep all patch semantics. Furthermore, we conduct additional ablation studies exploring more sophisticated methods for accumulating spatial and positional information. Specifically, we investigate alternative techniques like k-means clustering on spatial patch tokens to enrich the spatial coding process and provide a comparative analysis of these methods. The overall performance get an increase on all the compared datasets when we use k-means to incorporate spatial information, while the additional computational cost leads to a trade-off. To be noticed, more exploration space on this spatial prompt coding will leave for future work.

About generalizability. The authors have exclusively conducted experiments on the standard ViT architecture under self-supervised pretraining strategies. This raises questions about how LSPT performs when applied to a supervised pretrained model and more advanced ViT architectures. Additionally, comparing LSPT with more advanced parameter-efficient fine-tuning methods [a] [b] could further substantiate its effectiveness.

[a] Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. ICCV, 2023. [b] Adapting shortcut with normalizing flow: An efficient tuning framework for visual recognition. CVPR, 2023.

To address concerns about generalizability, we extend our experiments to include supervised pre-trained models and more advanced ViT architectures on ViT-B/16 pre-trained on supervised ImageNet-21k in the Table below. This provide a broader perspective on LSPT's applicability and effectiveness.

[a] Jie et al. Revisiting the parameter efficiency of adapters from the perspective of precision redundancy. ICCV, 2023.

[b] Wang et al. Adapting shortcut with normalizing flow: An efficient tuning framework for visual recognition. CVPR, 2023.

[c] Das et al. Learning Expressive Prompting With Residuals for Vision Transformers. CVPR, 2023.

The long-term modeling ability is not supported by evidence. The visualization of attention maps do not represent the long-term modeling ability, and in addition, they can be cherry-picked. The authors should find more direct evaluation ways to show the long-term modeling ability, especially with quantitative measures.

To address the reviewer's concerns about the lack of direct evidence for the long-term modeling ability, we introduce deeper quantitative analysis on the forgetting gate in learned LSTM to further justify our design choices. Specifically, we calculate the forget gate to decide how much of the previous prompt will be forgotten and how much of the previous prompt will be used in next steps, as shown in Table below. The number in the Forgetting colomn means how much information are discarded on average before feeding to the next layer. These quantitative results further elucidate how the forgetting gate aids in reducing forgetting in LSPT compared to VPT and GaPT. This analysis include detailed examinations of the forgetting metrics across different prompt tuning methods, demonstrating the effectiveness of LSPT's approach.

To be noticed, compared to VPT and GaPT whose ratios of forgetting are fixed and decides merely by the architecture or hyperparameter, the ratio of forgetting for our LSPT is an average over all the layers and data, controlled dynamically by the learned LSTM. This flexibility and selectivity of rememberring and forgetting also contributes to the final improvement.

The proposed method achieves better performance than previous SOTAs, but it uses more learnable parameters. For FGVC, 4 times more parameters were used, and for VTAB-1K, 5 times more parameters were used. For a fair comparison, the proposed method should decrease the number of introduced prompts.

We acknowledge the concern regarding the increased number of learnable parameters in LSPT. To address this, we conduct experiments with a reduced number of prompts, ensuring a fairer comparison with previous SOTAs. Comparison results in Table below demonstrate the efficiency of LSPT in achieving high performance with the same tunable parameters.

In addition, the proposed method may require more inference time due to LSTM module. Please compare the inference time with other VPT-based methods.

Thanks for the suggestion! The impact of the LSTM module on inference time is an important consideration. We include a detailed analysis of metrics like training and inference time per batch in the table below, where we fine-tune MAE pre-trained ViT-B/16 weights on CUB dataset. Our LSPT achieves comparable time computational overhead to GaPT, the state-of-the-art VPT baseline on self-supervised ViTs.

For long-term modeling, why LSTM is used? The auto-aggressive Transformer layer can also be applied?

The choice of LSTM for long-term modeling helps introduce forget gate to enlarge the variance for prompt tokens. The auto-aggressive Transformer layer is applied among tokens, which could increase their similarity by weighted sum of token values. So, we believe there exists a risk of losing discrepancy among latent toke embeddings during training. Additionally, we include experimental results comparing the performance of LSPT in the Table below when using LSTM versus auto-aggressive Transformer layers, providing insights into the effectiveness of our approach.

Dear Reviewer HDEH,

Thank you for the detailed review. We will address your concerns below.

My major concern is the motivation. Why preserving the spatial information is crucial? For classification, proper abstraction can be more beneficial, which discards unnecessary spatial details or provides not interpretable attention maps.

We appreciate the reviewer's query about the importance of preserving spatial information in classification tasks. In our revised response, we aim to align more closely with the underlying intuition of self-supervised Vision Transformers (ViTs) and the role of Visual Prompt Tuning (VPT) in leveraging spatial information.

The core intuition behind our approach is grounded in the understanding that self-supervised ViTs inherently encode semantics in attention weights, which highlight the relevance of different patch tokens and thus, the underlying spatial information. In such architectures, the attention mechanism is adept at discerning which parts of the input are more relevant for a given task, making the spatial context of these patch tokens particularly significant. VPT, in this context, emphasizes and harnesses this spatial information. The visual prompts in VPT are designed to encode information through an aggregation process driven by learnable attention weights. This mechanism allows for a more nuanced and detailed understanding of spatial relationships within the data, which is particularly beneficial for tasks that require fine-grained differentiation between classes.

To empirically validate this concept, we have conducted experiments where the Class-Aware Spatial Prompt Coding (CSPC) module was applied in isolation. The results, as detailed in our ablation study, show a marked improvement in performance when CSPC is used. This demonstrates that the explicit encoding of spatial information via CSPC significantly enhances the model's ability to discern and classify visual data with a higher degree of accuracy.

Dear Reviewer isM7,

Thank you for appreciating our approach. We will address your comments below.

This paper appears to exhibit a substantial degree of similarity to the methodologies presented in the GaPT paper (Yoo et al., 2023). For instance, section 3.1.2 seems to be a rephrased version of Section 2, 'Visual Prompt Tuning,' from GaPT. Equations (1, 2, 3) in this paper are identical to equations (4, 5) in GaPT. Furthermore, Table (1, 2) and the corresponding experimental results are exactly the same as those in GaPT. The overall structure of this paper closely mirrors that of GaPT. It is highly irregular and unacceptable to replicate aspects of a research paper to such an extent.

We acknowledge the reviewer's observation regarding the similarities between our work and the GaPT paper. It is important to note that our work initially builds upon the foundational concepts of GaPT, intending to improve upon them. We revised Section 3.1.2 to clarify our advancements beyond GaPT's scope and ensure that the text reflects our original contributions more distinctly. Furthermore, we updated our equations and tables to better differentiate our work from GaPT, emphasizing the unique aspects of LSPT.

Including a delineated algorithmic framework would significantly improve the replicability of the proposed LSPT method.

To enhance the replicability of our method, we agree with the reviewer's suggestion to include a detailed algorithmic framework. We add a pseudo-algorithm in our revised manuscript in Appendix Section B, providing clear steps and methodologies for implementing LSPT.

The connection between the two proposed modules, CSPC and LPC, is weak. There is no strong motivation presented to justify the inclusion of CSPC, especially since the title of the paper pertains solely to LPC. Nonetheless, the authors introduce CSPC first, which suggests a greater emphasis on the importance of CSPC.

We appreciate the feedback on the connection between CSPC and LPC. Our intention is to demonstrate that both spatial and long-term aspects are crucial for addressing forgetting in visual prompt tuning. We provide a more detailed explanation in the revised manuscript to justify the inclusion and parallel importance of both CSPC and LPC within LSPT.

Additionally, our title also includes spatial prompt tuning, which refers to the proposed CSPC (Class-aware Spatial Prompt Coding) strategically designed to perpetually amass class conscious features, thereby fortifying the model’s prowess in distinguishing and identifying visual categories. Our method named Long-term Spatial Prompt Tuning in the title include both spatial and long-term aspects, reflecting the comprehensive scope of our work.

More visual prompt tuning papers should be included in the related works. There are only two papers introduced currently.

Our current related works focus on vpt for self-supervised ViTs, but we recognize the necessity of including a broader range of related works in visual prompt tuning. In the revised manuscript, we incorporated additional references [1,2,3] on VPT to both self-supervised and supervised ViTs to provide a more comprehensive background and context for our research.

[1] Das et al. Learning Expressive Prompting With Residuals for Vision Transformers. CVPR, 2023.

[2] Wang et al. Adapting Shortcut With Normalizing Flow: An Efficient Tuning Framework for Visual Recognition. CVPR, 2023.

[3] Jie et al. Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy. ICCV, 2023.

The scope of the proposed LSPT within the realm of visual prompt tuning is confined to SSL-ViTs. This limitation constrains the real-world applicability of the proposed LSPT method. Additionally, are there any other baselines within this specific experimental context?

We understand the reviewer's concern about the confined scope of LSPT to SSL-ViTs. To address this, we expand our experimental setup to compare baselines on ViT-B/16 pre-trained on supervised ImageNet-21k in the Table below, thereby demonstrating the generality and applicability of LSPT across different types of vision transformers.

Dear Reviewer 7rGz,

Thank you for appreciating our approach. We will address your comments below.

While the proposed modules make intuitive sense, the explanations lack quantitative analysis or theorems to rigorously justify the designs. Some ablation studies analyze contributions but more analysis connecting the methods to mitigating forgetting could strengthen the approach.

We appreciate the reviewer's suggestion for a more rigorous quantitative analysis. We introduce deeper quantitative analysis on the forgetting gate in learned LSTM to further justify our design choices. Specifically, we calculate the forget gate to decide how much of the previous prompt will be forgotten and how much of the previous prompt will be used in next steps, as shown in Table below. The number in the Forgetting colomn means how much information are discarded on average before feeding to the next layer. These quantitative results further elucidate how the forgetting gate aids in reducing forgetting in LSPT compared to VPT and GaPT. This analysis include detailed examinations of the forgetting metrics across different prompt tuning methods, demonstrating the effectiveness of LSPT's approach.

To be noticed, compared to VPT and GaPT whose ratios of forgetting are fixed and decides merely by the architecture or hyperparameter, the ratio of forgetting for our LSPT is an average over all the layers and data, controlled dynamically by the learned LSTM. This flexibility and selectivity of rememberring and forgetting also contributes to the final improvement.

The spatial prompt coding uses a simple patch token averaging, which seems like a heuristic. More sophisticated ways to accumulate spatial/positional information may exist. This component could likely be improved.

We acknowledge the reviewer's concern regarding the simplicity of our spatial prompt coding method. To address this, we conduct additional ablation studies exploring more sophisticated methods for accumulating spatial and positional information. Specifically, we investigate alternative techniques like k-means clustering on spatial patch tokens to enrich the spatial coding process and provide a comparative analysis of these methods. The overall performance get an increase on all the compared datasets when we use k-means to incorporate spatial information, while the additional computational cost leads to a trade-off. To be noticed, more exploration space on this spatial prompt coding will leave for future work.

The long-term prompt coding relies on a single LSTM layer. Ablations could explore using multiple LSTM layers or comparing to other sequential modeling approaches like GRUs.

Thanks for the valuable suggestion! We extend our ablation studies to include experiments with multiple LSTM layers and compare their efficacy with other sequential modeling approaches, such as GRUs. Table below shows the comparison results with two LSTM layers and one GRU layers. We can observe that here is a trade-off between params and performance, and a single LSTM layer achieves the good balance on both. This is why our long-term prompt coding module using a single LSTM layer is not too complex. These results further help us to validate the choice of LSTM in our approach and explore potential improvements.