Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning

We are deeply grateful for the time and effort you invested in reviewing our manuscript. Your insightful comments and the connection you drew between our methods and existing DML ideas are highly appreciated.

Addressing the Need for Comprehensive Comparison with Existing DML Methods

In alignment with our response to Reviewer 1 (hvFX), we wish to further clarify the baseline methodologies used in our study. The "full fine-tuning" baseline, as discussed in both our main paper and Appendix, is based on the proxy-anchor approach [1], which is recognized as one of the state-of-the-art methods in DML. Additionally, we have conducted comparisons with other leading approaches, such as Hyp-ViT [2] and RS@k [3], which are also based on ViT architectures and considered state-of-the-art.

It is important to note that all these state-of-the-art methods [1][2][3], while varying in their optimization loss, share a commonality in their computational costs, as they are all fundamentally full fine-tuning methods. To provide a more detailed comparison, we have included an additional analysis in Table 6 in the Appendix. This table compares our methods with those mentioned and other existing DML methods, offering a comprehensive comparative view.

For an in-depth understanding of these comparisons and the rationale behind our approach, we kindly refer you to our detailed response to Reviewer 1 (hvFX), where we elaborate on these points further.

Addressing the comparison with other parameter-efficienct fine-tuning methods

We would like to draw your attention to our comprehensive analysis comparing our VPT method with several leading parameter-efficient techniques in the DML benchmark. This detailed comparison is outlined in Section 5.1.2 of our paper. The results of this comparative study are showcased in Table 1 of the main manuscript, where we assess and compare the performance of each method in terms of DML metrics (specifically R@1 and MAP@R), the number of adjustable parameters, and memory usage.

To ensure a fair and unbiased comparison, as elaborated in Section 5.1.3, we conducted an extensive hyperparameter search for each method. This rigorous approach ensures that our comparative analysis is not only comprehensive but also equitable, taking into account the varied characteristics and capabilities of each parameter-efficient method under consideration.

We trust that these clarifications address your concerns regarding the comparative analysis of our methods with other DML approaches. Your constructive feedback is invaluable in our pursuit of continual improvement of our manuscript.

[1] Kim, Sungyeon, et al. "Proxy anchor loss for deep metric learning." CVPR 2020.

[2] Ermolov, Aleksandr, et al. "Hyperbolic vision transformers: Combining improvements in metric learning." CVPR 2022.

[3] Patel, Yash, Giorgos Tolias, and Jiří Matas. "Recall@ k surrogate loss with large batches and similarity mixup." CVPR. 2022.

We are immensely grateful for the time and effort you have devoted to reviewing our manuscript. Your acknowledgment of our paper's contribution and novelty, coupled with your valuable insights and suggestions, is highly appreciated.

Addressing Limited Results on Some Datasets Compared with SOTA

In Section 5 of the Appendix, we have openly discussed the limitations of our proposed VPT methods. The suboptimal performance on the SOP and InShop datasets, which feature a higher number of classes and proxies, is a focus of our ongoing research and improvement. This admission underlines our commitment to transparency and the continuous development of our work.

Performance on Large vs. Small Pre-training Datasets

As elaborated in the Experimental section, we posit that the performance of VPT and other parameter efficiency methods is influenced by the distributional variances between pre-training and fine-tuning datasets. Smaller pre-training datasets like ImageNet-1K often display a more significant variance in distribution compared to target datasets. This observation is consistent with prior findings in VPT research [1], lending support to our analysis.

Reorganization of Results and Appendix Content

In response to your suggestion, we have included the results from the iNaturalist dataset in the main paper (now in Table 3). Due to space and format constraints, we are unable to relocate more results from the Appendix to the main text. We believe that the current structure of the manuscript strikes a balance between detail and conciseness, effectively presenting our findings.

Attention to Minor Details

All minor points raised in your review have been meticulously addressed and corrected. We believe these revisions significantly enhance the clarity and accuracy of our manuscript.

We believe that our responses and revisions comprehensively address your concerns and contribute to the refinement of our paper. We are sincerely thankful for the opportunity to improve our manuscript based on your constructive feedback and eagerly await any further guidance you might offer.

[1] Jia, Menglin, et al. "Visual prompt tuning." ECCV, 2022

Q1: What is the Computational Cost of the Proposed Method

As mentioned, we evaluate our method's computational cost in two aspects:

1. The total trainable parameters that are updated in each optimization step (shown in Table 1).

2. The actual running latency per optimization step (shown in Table 8 of the Appendix).

Table 1 in the main paper demonstrates that our method requires only about 5% of the trainable parameters needed for full fine-tuning methods. For running latency, our method significantly undercuts full fine-tuning methods, as listed in Table 8 of the Appendix.

Q2 and Q3: Comparison to State-of-the-Art DML Methods and Vanilla Proxy-Based DML

To better represent our "full fine-tuning method" as a stronger baseline compared to the vanilla proxy-anchor, we have added explanatory text in our revision. We have also updated Table 1 in the main paper and Table 8 in the Appendix for greater clarity. Additionally, a new table (Table 6 in Appendix) comparing our proposed method with existing DML works using ViT backbones [4][5] and earlier DML methods with CNN backbones [3][6][7][8] has been added.

We hope these revisions and additional explanations comprehensively address your concerns, thereby enhancing the clarity and impact of our work. We are deeply appreciative of the opportunity to refine our manuscript based on your thoughtful feedback and eagerly await any further suggestions you may have.

[1] Jia, Menglin, et al. "Visual prompt tuning." ECCV, 2022

[2] Chen et al. "Adaptformer: Adapting vision transformers for scalable visual recognition." NeurIPS 2022

[3] Kim, Sungyeon, et al. "Proxy anchor loss for deep metric learning." CVPR 2020.

[4] Ermolov, Aleksandr, et al. "Hyperbolic vision transformers: Combining improvements in metric learning." CVPR 2022.

[5] Patel, Yash, Giorgos Tolias, and Jiří Matas. "Recall@ k surrogate loss with large batches and similarity mixup." CVPR. 2022.

[6] Teh et al. "Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis." ECCV 2020

[7] Roth et al. "Non-isotropy regularization for proxy-based deep metric learning." CVPR 2022

[8] Wang et al. "Cross-batch memory for embedding learning.", CVPR 2020

Dear Reviewers,

We extend our sincerest thanks for your insightful and constructive comments on our paper. Your feedback is invaluable in helping us clarify and address the questions you have raised.

Comparison in Terms of Efficiency with Existing DML Methods

Aligning with the metrics in existing parameter-efficient works [1][2], we assess efficiency by comparing the number of tunable parameters in the model, which reflects computational costs per iteration. Most existing DML methods [4][5] rely on full fine-tuning optimization with varying loss functions or sampling strategies, making their efficiency comparable to our full fine-tuning baseline.

Furthermore, we have compared actual running latencies in Section C and Table 8 of our Appendix (formerly Supplementary Materials). This table shows the average running time per iteration in identical environments across 100 trials. Our results show that our proposed VPT and other parameter-efficient methods significantly reduce latency compared to full fine-tuning approaches. It's important to note that running latency is subject to hardware and software configurations. Hence, parameter-efficient works [1][2] often use tunable parameters as a universal metric for evaluating fine-tuning efficiency.

Consideration of More Existing DML Methods and a Vanilla Proxy-Based Baseline

In response to your concern, we have made the following clarifications and additions:

1. Our full fine-tuning approach, detailed in the main paper and Appendix, is based on the vanilla proxy-anchor (PA) [3] methodology, enhanced with a ViT backbone and larger ImageNet-21k pre-training datasets. We have included comparisons with state-of-the-art methods like Hyp-ViT [4] and RS@k [5], which also utilize a ViT backbone. These comparisons are specific to ViT backbones trained on ImageNet-21k, as earlier works [3][6][7][8] with CNN backbones and ImageNet-1k training show significantly lower DML benchmark performance.
2. Most current state-of-the-art methods, including Hyp-ViT [4] and RS@k [5], use full fine-tuning of the ViT with different losses or sampling strategies, similar to our PA baseline. These methods often require larger batch sizes (as indicated in Table 2), leading to higher latencies compared to our baseline. Therefore, their overall efficiency is close to or below our full fine-tuning PA baseline, while our VPT methods demonstrate higher DML performance, as shown in Table 2.
3. Although it's not entirely fair to compare with earlier works using different backbones and pre-training models, we have included performance and tunable parameters for some earlier works [6][7][8] and the vanilla proxy-anchor [3] as references. This comparison is also added to our Appendix as Table 6.