MemVLT: Vision-Language Tracking with Adaptive Memory-based Prompts

Dear Reviewer jufM,

Thanks for thoroughly reviewing our work. We appreciate your acknowledgment of our model's strong performance and the quality of our writing. Below, we address your concerns and provide detailed responses:

Weakness 1: novelty of our work

(1) Novelty in long and short-term memory modeling: To highlight the differences between our method and existing work based on long and short-term memory, we have provided direct comparisons in Sections 2.1 and 2.2. Additionally, please refer to our summary of distinctions in global response 5.

(2) Contributions of our technical solution:

1). Considering the strong feature representation capability of transformers, we used vanilla transformer layers [1] and transformer variants with multiple cross-attention layers [2][3] to achieve feature interaction among different elements. This is a commonly adopted approach in mainstream model design. However, it is important to emphasize that our primary contribution lies in the modeling of memory mechanisms using these basic network layers, rather than the design of these transformer layers.

2). Our CLS-based memory strategy shares similarities with the temporal modeling approaches in some contemporary visual trackers (e.g., AQATrack [1]) but also exhibits distinct differences, which we have directly compared in Section 2.2. For instance, in terms of memory storage, our proposed section-top storage strategy significantly outperforms the sliding windows approach used in AQATrack (see Table 4).

Weakness 2: regarding Figure 1(a)

(1) The purpose of the consistency curve in Figure 1(a) is to illustrate an inherent fact: static initial prompts (i.e., the template patch and language description) and dynamic targets are difficult to maintain consistently over time. Even without this curve, we can qualitatively observe this fact from Figure 1(a). The curve is a supplementary explanation, and this fact cannot be changed. Therefore, we cannot improve the consistency through model design. This challenge motivates our model design to introduce implicit dynamic memory to better guide tracking. Since these features cannot be directly measured, we cannot demonstrate our method's effectiveness by improving this consistency curve.

(2) As described in Section B.1, plotting the consistency curve only requires a pretrained backbone capable of extracting image features, such as ResNet or ViT. It is important to note that the backbone used here differs from the backbone of the tracker, as their purposes differ. To address your concerns, we have plotted the consistency curve based on ViT in the supplementary PDF, and you can see that the curves based on ResNet and ViT are consistent.

Weakness 3: ablation of backbone

We understand your concerns because previous research [3][4] has demonstrated that the HiViT-based backbone offers performance improvements over the ViT-based backbone, which is why we chose it. To address your concerns, we have conducted the relevant experiments and performed the following analysis:

(1) We find that our proposed memory mechanism brings a 4.3% AUC improvement (comparing #1 and #2), while the HiViT backbone offers AUC gains of 1.1% (comparing #1 and #3). This indicates that the performance improvement brought by our memory mechanism exceeds that of the HiViT-based backbone.

(2) The experimental results in #3 show that with a ViT-based backbone, our model still performs well, surpassing some of the latest VLTs, such as MMTrack [5] and QueryNLT [6] (see Table 1). This demonstrates the generalizability of our method.

Weakness 4: ablation of block layers

Thanks for your suggestion. We have conducted ablation experiments on the number of block layers. Due to limited time, we have temporarily supplemented the performance of SMG and transformer decoder with 1 and 5 layers and compared them with the existing model at 3 layers. It can be observed that too few block layers (comparing #1 and #2) significantly harms the model's performance, while continuously increasing the block layers (comparing #2 and #3) does not yield noticeable performance gains. Our provided model effectively reflects the convergence performance.

Question 1: meaning of symbols

We apologize for any confusion caused by the abbreviations. The meanings of MIM (Memory Interaction Module) and MSM (Memory Storage Module) are provided in the overview of the model (lines 117-127). We will make these abbreviations bold in the future to enhance the readability of the paper.

We hope this detailed response addresses your concerns. We would appreciate it if you could reconsider your rating. If you have additional feedback or require further clarification, please let us know.

[1] Attention is All You Need, Vaswani et al., in NeurIPS 2017.

[2] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, et al., in arXiv 2023.

[3] Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers, Xie et al., in CVPR 2024.

[4] Explicit Visual Prompts for Visual Object Tracking, Shi et al., in AAAI 2024.

[5] Towards Unified Token Learning for Vision-Language Tracking, Zheng et al., in TCSVT 2023.

[6] Context-Aware Integration of Language and Visual References for Natural Language Tracking, Shao et al., in CVPR 2024.

Thank you for your response, which addresses all of my concerns. I decided to raise my rating to accept.

Dear Reviewer jufM,

We are thrilled to receive your response and deeply honored that our reply has resolved all of your concerns. Your decision to raise the rating is incredibly encouraging for us.

We will incorporate your valuable suggestions and include the aforementioned ablation analyses in the revised manuscript.

Sincerely wishing you all the best ~~~

Dear Reviewer unPL,

Thanks for your careful review of our work. Your acknowledgment of our memory mechanism modeling and its promising performance is invaluable to us. We notice that you are particularly concerned about the model's ablation analysis. Therefore, we have conducted the corresponding experiments and hope that the following experimental analysis will address your concerns.

Weakness 1: more ablation analysis

（1）Ablation on backbone:

As you mentioned, previous research [1][2] has demonstrated that the HiViT-based backbone offers performance improvements over the ViT-based backbone, which is why we chose it. It is important to note that the primary contribution of our paper is the modeling of memory mechanisms rather than backbone design. Therefore, our ablation experiments focus on the former. To address your concerns, we have conducted the relevant experiments, with results shown in the table below:

Comparing #1 and #2, it is evident that the introduction of our memory mechanism improves AUC by 4.3%. Comparing #1 and #3, we can see that the HiViT backbone offers AUC gains by 1.1%, consistent with previous research conclusions [1][2]. Moreover, the performance improvement brought by our memory mechanism exceeds that of the HiViT-based backbone, demonstrating that the superior performance of our work primarily stems from the memory mechanism rather than the HiViT-based backbone.

（2）Ablation on image resolution: To our knowledge, vision-language trackers typically report results at a single resolution setting, as seen in DecoupleTNL [3] (256), QueryNLT [5] (320), and MMTrack [6] (384). This limits our comparison to the results provided by DecoupleTNL at a resolution of 256 and is also the reason why we did not perform resolution-related ablation experiments. We will clarify the input resolution in Table 1. To address your concerns, we supplement our experiments at the 256 resolution setting (tested on TNL2K):

As expected, reducing the image resolution somewhat degrades performance. However, MemVLT-256 still significantly outperforms some recent models with larger resolutions, such as MMTrack [6] (384) and QueryNLT [5] (320) (see Table 1). This demonstrates that our method maintains strong performance even at lower resolutions.

Weakness 2: generalization of the model

Thanks for your suggestion. We have conducted experiments under the vision-only tracking setting. The results demonstrate that our method can generalize to the vision-only tracking task. Specifically, we remove the text-related components from MemVLT, seamlessly transforming it into a vision tracker. As shown in the table below, our proposed Memory Interaction Module (MIM) and Memory Storage Module (MSM) incrementally improve the performance of the vision-only tracker, demonstrating the good generalizability of our method.

Weakness 3: method description

We apologize for any confusion caused by the description of our method. Due to the need for temporal modeling, the connections between various modules are relatively complex. For instance, when introducing the memory interaction module, we need to explain the meaning of the memory features used in the memory storage module. We suspect this might have caused the confusion. In the revised version, we will meticulously review our paper's wording and enhance the introduction of key concepts in method section. We hope this will provide readers with a reading experience similar to that of Reviewers BESZ and jufM, who highly commended the readability of our paper.

We believe this detailed response has clarified your concerns. We would appreciate it if you could reconsider your rating based on this additional information. Please reach out if you have further questions.

[1] Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers, Xie et al., in CVPR 2024.

[2] Explicit Visual Prompts for Visual Object Tracking, Shi et al., in AAAI 2024.

[3] Tracking by Natural Language Specification with Long Short-Term Context Decoupling, Ma et al., in ICCV 2023.

[4] Joint Visual Grounding and Tracking with Natural Language Specification, Zhou et al., in CVPR 2023.

[5] Context-Aware Integration of Language and Visual References for Natural Language Tracking, Shao et al., in CVPR 2024.

[6] Towards Unified Token Learning for Vision-Language Tracking, Zheng et al., in TCSVT 2023.

Dear Reviewer unPL,

Thanks for your further response.

Regarding your comments on "unfair comparison" and "unclear reason for accuracy improvement," we respectfully disagree. First, we highly value your feedback, which is why we promptly conducted additional model training and testing analyses as soon as we saw the concerns you raised. Second, based on the issues you mentioned, we recognize that you have a deep understanding and professional background in tracking tasks. However, we believe there may be some misunderstanding regarding our experimental settings and motivations, so we feel it is necessary to provide further clarification to address your concerns about the performance.

Question 1: Is the performance comparison between our model and others truly unfair?

1. Comparison with DecoupleTNL

We have summarized the performance of the DecoupleTNL and our model under different backbone and resolution settings in the table below. The conclusions are as follows:

(1) As you mentioned, previous research has demonstrated that a ViT-based backbone and a 256 resolution negatively impact tracker performance. Despite this, even under these challenging settings (#1), our model still achieves strong performance. Compared to DecoupleTNL, we observe a slight performance decrease on LaSOT (AUC -0.7%), but a significant performance improvement on TNL2K (AUC +4.5%). Considering the number of search frames across the two datasets, our model shows a higher average performance (AUC +1.5%).

(2) From a model perspective, the DecoupleTNL includes both a frame encoder and a video encoder as part of its visual backbone, whereas our model only uses a single image encoder. This means our model is at a disadvantage in terms of input information (i.e., no multi-frame video data input). Despite this, we chose to compare under the same resolution (256) in #4 to ensure as much fairness as possible. Clearly, the performance in #4 significantly exceeds that of #1.

2. What settings represent a model's final performance to demonstrate SOTA performance?

To compare models and demonstrate SOTA performance, it is essential to focus on the model's absolute performance. To our knowledge, almost all VLTs consider only one setting for absolute performance when comparing with other models to claim SOTA results. Given that it is common for visual trackers to show results under "additional settings", we have also provided such analyses.

However, it is important to note that when comparing with other models, these "additional settings" are typically limited to resolution. For example, in OSTrack and AQATrack, models' performances at 256 and 384 resolutions are presented to compare with others.

Therefore, the absolute performance of our model can be represented by the results in the table above (#4, #5). Clearly, our model outperforms existing VLTs significantly (see Tab.1), which sufficiently demonstrates that our model achieves SOTA performance.

Question 2: What contributes to our strong performance?

The explanation for this question will be provided in the next comments.

Dear Reviewer unPL,

We sincerely appreciate your positive response. Your decision to raise the rating is of great importance to us.

We highly value your constructive suggestions. In the revised manuscript, we will clarify our comparison conditions and include the ablation experiments related to backbone and resolution.

Wishing you all the best ~~~

Dear Reviewer GRLs,

We sincerely appreciate your time and effort in reviewing our work. We are grateful for your recognition of our cognitive science-inspired modeling approach, strong performance, and broader impact discussion. We notice that you are particularly concerned about the differences between our model and other works. Therefore, we have conducted a detailed comparative analysis to highlight the novelty of our method.

Weakness 1: Comparison with CiteTracker

(1) Differences in adaptive prompts:

1). CiteTracker is a vision tracker. Its adaptive prompts refer to discrete and closed-set text cues identified based on visual target information. This is primarily achieved through image attribute recognition.

2). Our proposed MemVLT is designed for the vision-language tracking task. It considers textual information that is diverse and open-set in language description, making it a more challenging task. Our adaptive prompts refer to initial multimodal prompts implicitly embedded with dynamic interaction information. Unlike CiteTracker, which only models adaptive textual prompts, our approach models both textual and visual adaptive prompts, aiding in better tracking guidance. Additionally, our adaptive prompts are implicitly obtained through prompt learning, which helps mitigate the impact of classification errors caused by explicit image attribute recognition.

(2) Performance comparison: As shown in the table below, our MemVLT significantly outperforms CiteTracker, demonstrating the effectiveness of our adaptive prompts mechanism.

Weakness 2: Comparison with Xmem

(1) Key differences in memory mechanism modeling:

1). Xmem uses the Atkinson-Shiffrin memory model, which emphasizes the representation of short-term, working, and long-term memories. A crucial modeling processing of Xmem is the unidirectional information transfer from short-term to long-term memory. In contrast, our memory model is based on the Complementary Learning Systems (CLS) theory, which emphasizes the interaction between short-term and long-term memories. The importance of bidirectional interaction has been highlighted in recent studies [1].

2). For memory storage and updating processes, Xmem adopts a top-k updating method based on usage frequency as the importance indicator. In contrast, our proposed section-top method not only considers the importance of memory units but also accounts for the temporal representation range(see Table 4).

(2) Different tasks: Xmem focuses on vision-only video object segmentation, while MemVLT concentrates on vision-language multimodal tracking. Due to the significant differences in input and output, a direct quantitative comparison is not feasible, and existing VLTs have not conducted performance comparisons with it.

Weakness 3: Comparison with recent SOTA trackers

(1) All-in-one: As shown in the table below, our tracker significantly outperforms All-in-one on all benchmarks except LaSOT_ext. Overall, the average performance of our model far exceeds that of All-in-one (AUC +4.5%). Although All-in-one uses a larger training dataset to align text and vision modalities, it neglects the modeling of dynamic temporal information. In contrast, MemVLT effectively models temporal information through the memory mechanism, resulting in superior tracking performance.

(2) Untrack: In fact, Untrack is not a VL tracker. Although it claims to handle tracking in any modality, it does not include our VLT task, making direct comparison infeasible. Additionally, we found a recent tracker with a similar motivation to Untrack that includes the VLT task, called OneTracker [2], and compared its performance with ours (see table below). Evidently, our model significantly outperforms OneTracker across all metrics. We speculate that the primary reason for this is OneTracker's lack of modeling dynamic temporal information.

Weakness 4: computational efficiency of MemVLT

We understand your concerns regarding the computational efficiency (e.g., fps, parameters). Actually, we have already included comparisons and analyses in our paper (see Table 2 and Section 4.2). Overall, our model is comparable to the latest trackers in terms of parameters and fps, while demonstrating superior performance.

We hope this detailed response addresses your concerns and would be grateful if you could reconsider your rating. Should you have any additional feedback or require further clarification, please do not hesitate to let us know.

[1] Organizing memories for generalization in complementary learning systems, Sun et al., in Nature 2023.

[2] OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning, Hong et al., in CVPR 2024.

Dear Reviewer GRLs,

Thanks for your response. We are honored that our reply has resolved most of your concerns. Your decision to raise the rating is of great significance to us.

We will incorporate your valuable suggestions and include the comparisons mentioned above in the revised manuscript.

Wishing you all the best !!!

Dear Reviewer BESZ,

Thanks for taking the time to review our work. We appreciate your recognition of our interesting insights, good writing, and strong performance. We hope the following responses will address your concerns.

Weakness 1 & 2: comparison with recent SOTA trackers

(1) MMA [1]: We found that MMA is not a tracker. It focuses on vision-language pre-training, with its primary downstream task being image classification. Due to the significant gap, a direct comparison is not feasible.

(2) CAI [2]: The tracker proposed in the CAI paper is called QueryNLT, which we have already compared in our paper (see Table 1). Our model outperforms QueryNLT across all benchmarks. For instance, on the TNL2K, our model achieves a 6.4% higher in AUC. QueryNLT modulates multimodal prompts by filtering tokens based on visual-text similarity. In contrast, our modulation approach embeds dynamic information into the multimodal prompts, avoiding the potential loss of important tokens due to similarity calculation errors. We suspect this accounts for the inferior performance of QueryNLT.

(3) OVLM [3]: The comparison results are shown in the table below. Our MemVLT significantly outperforms OVLM on LaSOT (AUC +5.2%) but slightly falls short on TNL2K (AUC -1.4%). Overall, our model demonstrates superior average performance (AUC +2.4%). Although OVLM involves memory mechanism modeling, there are key differences between our method (addressing weakness 2):

1). Memory Representation: OVLM assumes the text prompt is a precise long-term cue, using vision and text features to represent short and long-term memories, respectively. However, it overlooks the misalignment between text information and video targets (see Fig 1(a)), which can mislead the tracker [4]. Our method addresses this by using both text and vision features to comprehensively represent short and long-term memories, and modulate them with dynamic information.

2). Memory Interaction: OVLM focuses solely on the unidirectional modulation of short-term memory features by textual long-term memory, whereas our method achieves bidirectional modulation between long-term and short-term memories. The importance of bidirectional interaction has been highlighted by recent research [5].

In summary, our model provides a more comprehensive representation and interaction of memory information, resulting in superior performance.

Question 1: novelty in long and short-term memory modeling

(1) To highlight the differences between our method and existing trackers based on long and short-term memory, we have provided direct comparisons in Sections 2.1 and 2.2. Additionally, please refer to our summary of distinctions in global response 5.

(2) For the VLT task, introducing textual cues brings new challenges. Due to the modality differences between text and vision cues, previous visual memory modeling methods may not directly apply to textual memory. For example, acquiring the memory unit in TrDiMP[6] relies on RoI processing in the search image, which cannot be applied to the text modality. Our tracker proposes a novel approach to represent textual memory features based on the prompt learning method. The table below also demonstrates the necessity of specifically designing textual memory information.

Question 2: generalizability of our memory mechanism

As introduced in the global response 4, we conduct experiments under a vision-only tracking setting and find that our method can be generalized to the vision tracker. Specifically, we remove the text-related components from MemVLT, seamlessly transforming it into a vision tracker. As shown in the table below, the proposed Memory Interaction Module (MIM) and Memory Storage Module (MSM) incrementally improve the performance of the vision-only tracker. This validates that our method can generalize to a visual tracking framework.

Limitations : efficiency of MemVLT

We have already included comparisons and analyses of efficiency in our paper (see Table 2 and Section 4.2). Overall, our model is comparable to the latest trackers in terms of parameters and fps, while demonstrating superior performance. Additionally, our model achieves a speed of 32 fps, which meets the requirements for real-time processing.

We hope this detailed response addresses your concerns. We would appreciate it if you could reconsider your rating. If you have additional feedback or require further clarification, please let us know.

[1] MMA: Multi-Modal Adapter for Vision-Language Models, Yang et al., in CVPR 2024.

[2] Context-Aware Integration of Language and Visual References for Natural Language Tracking, Shao et al., in CVPR 2024.

[3] One-Stream Vision-Language Memory Network for Object Tracking, Zhang et al., in TMM 2023.

[4] Joint visual grounding and tracking with natural language specification, Zhou et al., in CVPR 2023.

[5] Organizing memories for generalization in complementary learning systems, Sun et al., In Nature 2023.

[6] Transformer meets tracker: Exploiting temporal context for robust visual tracking, Wang et al., in CVPR 2021.