Efficient Motion Prompt Learning for Robust Visual Tracking

We thank the reviewer for the positive feedback, and address the concerns below.

1. Sensitivity to the trajectory quality: The proposed adaptive weight mechanism in our method can mitigate the impact of noisy trajectories to some extent. As shown in Figure 3 (Page 8 in submitted paper), the predicted weights align with the accuracy of motion predictions which implicitly reflect the quality of input trajectories. |Success Rate|[0.0,0.1)|[0.1,0.2)|[0.2,0.3)|[0.3,0.4)|[0.4,0.5)|[0.5,0.6)|[0.6,0.7)|[0.7,0.8)|[0.8,0.9)|[0.9,1.0)| |-|-|-|-|-|-|-|-|-|-|-| |OSTrack_MPT|1%|21%|26%|32%|42%|63%|82%|94%|99%|100%|

   To further analyze the sensitivity of our MPT to varying noisy trajectories, we evaluate the success rate (IoU>0.5) under different quality trajectory inputs on LaSOTEXT, as shown in the table above. Here, [a,b) represents the range where the average IoU of the trajectory is no less than a and less than b. The following conclusions can be drawn:

   • The success rate increases significantly with the improvement of trajectory quality, especially when the trajectory IoU is in the range of [0.6, 1.0).
   • Our method shows a certain level of robustness to noisy trajectory inputs, obtaining a success rate of 21% even when the trajectory IoU is in the range of [0.1, 0.2).

   Thanks for your valuable advice, we will add the above analysis to our paper.

2. Deep analysis of success/failure cases: To deeply explore the success/failure cases of our MPT, we evaluate the proportion (Rate) of these two conditions on LaSOTEXT, along with the corresponding average trajectory IoU (Avg_IoU_traj) and average IoU of the previous tracking result (Avg_IoU_pre).

   Here, we define a success case when the predicted IoU of our MPT exceeds that of the visual baseline by more than 0.3. Conversely, when the predicted IoU of our MPT is lower than that of the visual baseline by more than 0.3, we define it as a failure case. As shown in the table below, success cases of our MPT usually occur when the historical trajectory IoU is around 0.72, while failure cases usually happen when the historical trajectory IoU is around 0.18. Additionally, our method is more likely to succeed when the tracking result of the previous frame is better, whereas poorer previous tracking results are more likely to lead to failure. This is mainly because the tracking result of the previous frame affects the current search area and the completeness of the target.

   	Rate	Avg_IoU_traj	Avg_IoU_pre
   Success	2%	0.72	0.76
   Failure	1%	0.18	0.1

   Moreover, we separately evaluate the distribution of trajectory IoU and previous frame IoU in both success and failure cases, shown in the following two tables. In success cases, the input trajectory IoU primarily falls within a relatively broader range of [0.6, 1.0), especially in [0.8, 0.9). In failure cases, the input trajectory IoU mainly falls within the range of [0.0, 0.2), particularly in [0.0, 0.1). This indicates that addressing misleading tracking issues caused by extremely poor-quality trajectories remains a challenge for our method. This is a limitation of our method that requires future research.

   Distribution_succ	[0.0,0.1)	[0.1,0.2)	[0.2,0.3)	[0.3,0.4)	[0.4,0.5)	[0.5,0.6)	[0.6,0.7)	[0.7,0.8)	[0.8,0.9)	[0.9,1.0)
   IoU_traj	1%	2%	3%	4%	6%	6%	10%	20%	34%	15%
   IoU_pre	2%	0%	1%	3%	4%	6%	9%	16%	37%	23%

   Distribution_fail	[0.0,0.1)	[0.1,0.2)	[0.2,0.3)	[0.3,0.4)	[0.4,0.5)	[0.5,0.6)	[0.6,0.7)	[0.7,0.8)	[0.8,0.9)	[0.9,1.0)
   IoU_traj	55%	12%	8%	6%	6%	5%	4%	3%	1%	0%
   IoU_pre	74%	11%	4%	2%	2%	2%	1%	1%	1%	1%

   Thanks for your valuable advice, we will add the above analysis to our paper.

3. Extension to MOT: Different from single-object tracking (SOT), typical multi-object tracking (MOT) frameworks involve object detection and association. Therefore, integrating our MPT into an MOT framework will differ from that of SOT. However, the core idea of our method remains highly compatible with MOT problem. Specifically, historical trajectories of each object can be utilized as prompts to enhance MOT performance. On the one hand, our MPT can be integrated into the object detection process to enable motion-aware detection, thereby improving detection accuracy. On the other hand, during the association process, our MPT can help maintain the coherence of the object trajectory, achieving more accurate matching.

   In addition, compared with SOT, MOT usually encounters more complex backgrounds with a higher number of distractors. As an advantage of our MPT, introducing motion prompts can potentially help address distractor challenges. However, complex scenes are more prone to generating noisy trajectories, which may also pose a risk of misleading tracking issues.

   Thanks for your valuable advice, we will add the above discussion to our paper.

We thank the reviewer for the constructive feedback, and address the concerns below.

1. More efficiency metrics: We have provided several efficiency metrics for both baselines and our methods in Table3 (Page 7 in submitted paper), including memory footprint (Params, and Mem), latency (FPS), and training time (Time_tr). Thanks for your advice, we will add the FLOPs metric to our paper to further demonstrate the efficiency, as shown in the table below. ||OSTrack |+MPT(Ours)|ARTrack|+MPT(Ours)| |-|-|-|-|-| |FLOPs (G)|29.1|33.7|40.3|41.9|

2. Impact of fine-tuning strategies: According to your advice, we supplement the ablation study in terms of fine-tuning strategies. Our method freezes all of the baseline parameters, and only fine-tunes parameters of MPT during training. Here, we also evaluate two variants. FinetuneAll fine-tunes both MPT and baseline parameters, while FinetuneHead fine-tunes parameters of MPT and the tracking head of the baseline, i.e., only freezing the backbone of the baseline model. As shown in the table below, our method slightly outperforms the other two variants, and is more efficient in terms of memory footprint and training time. The potential reason is that freezing the baseline parameters enables the model to focus on learning motion cues and the fusion mechanism. ||OSTrack-B256|+MPT(Ours)|FinetuneAll|FinetuneHead| |-|-|-|-|-| |VOT22(EAO)|0.530|0.572|0.567|0.548| |LaSOTEXT(AUC)|46.9|48.7|48.3|48.1| |Mem(GB)|35.0|6.1|37.5|7.7| |Training Time(h)|38|5|8|6|

   Thanks for your valuable advice, we will add the above experimental analysis to our paper.

3. Comparison with PEFT-based tracking methods, like LoRAT: PEFT methods primarily include adapter-based and prompt learning-based approaches, both of which demonstrate great efficiency in various downstream tasks. LoRAT successfully applies LoRA, a typical adapter-based PEFT method, to the tracking task. The main contribution of LoRAT lies in designing a LoRA-friendly visual tracking model to achieve efficient training and inference. In contrast, our work aims to build a joint tracking framework that incorporates both visual and motion cues, leveraging the temporal consistency of videos and complementing visual limitations. Additional motion inputs are naturally compatible with prompt learning-based PEFT, which is why we choose prompt learning to ensure efficiency.

   Furthermore, we integrate our MPT into LoRAT, as shown in the table below. Results demonstrate that our MPT improves LoRAT by an average performance of 1.5% across the two datasets, showing good complementarity of our method with such PEFT-based tracking methods.

   	LoRAT	+MPT(Ours)
   VOT22(EAO)	0.541	0.558
   LaSOTEXT(AUC)	49.9	51.2

   Thanks for your advice, we will add the above comparison to our paper.

4. Comparison with recent trackers: Since our method is a plug-and-play module, and can be flexibly integrated into varying visual trackers to achieve performance complementarity, especially in challenging scenarios, the relative performance improvement over baseline methods can fairly demonstrate the effectiveness of our method. Thank you for your advice. We will add the above experimental comparisons with LoRAT to our paper to showcase the effectiveness of our method on recent trackers.

5. Minor performance gains for ARTrack and on some datasets: Compared with ARTrack, we provide new insights into efficient temporal tracking. In addition, the apparent performance bias is primarily due to the varying distributions of easy and hard sequences across different datasets, which is further clarified through the quantitative analysis of performance bias in Table 6 (Page 13 of the submitted paper). Our method consistently exhibits robustness advantages in challenging scenarios, demonstrating its prospects for real-world tracking applications.

We thank the reviewer for the positive feedback, and address the concerns below.

1. Appearance-based prompt trackers: Taking PromptVT as an example, appearance-based prompt methods efficiently enhance appearance features using dynamic appearance information, thereby enabling the model to be robust to appearance changes during tracking. However, the tracking problem is still treated as a discrete image processing task. In contrast, our MPT leverages the temporal consistency of videos by incorporating continuous object trajectories into the visual tracking framework. Our method proves to be more effective in addressing visually challenging scenarios, such as occlusions and distractors. These two kinds of methods are complementary: while appearance-based prompt methods focus on enhancing visual features, our MPT can integrate historical motion cues into this visual framework, achieving a joint enhancement of both visual and motion information.

   Thanks for your valuable advice. We will add the above comparison and related references to our paper.

2. Sensitivity to the trajectory quality: The proposed adaptive weight mechanism in our method can mitigate the impact of noisy trajectories to some extent. As shown in Figure 3 (Page 8 in submitted paper), the predicted weights align with the accuracy of motion predictions which implicitly reflect the quality of input trajectories. |Success Rate|[0.0,0.1)|[0.1,0.2)|[0.2,0.3)|[0.3,0.4)|[0.4,0.5)|[0.5,0.6)|[0.6,0.7)|[0.7,0.8)|[0.8,0.9)|[0.9,1.0)| |-|-|-|-|-|-|-|-|-|-|-| |OSTrack_MPT|1%|21%|26%|32%|42%|63%|82%|94%|99%|100%|

   To further analyze the sensitivity of our MPT to varying noisy trajectories, we evaluate the success rate (IoU>0.5) under different quality trajectory inputs on LaSOTEXT, as shown in the table above. Here, [a,b) represents the range where the average IoU of the trajectory is no less than a and less than b. The following conclusions can be drawn:

   • The success rate increases significantly with the improvement of trajectory quality, especially when the trajectory IoU is in the range of [0.6, 1.0).
   • Our method shows a certain level of robustness to noisy trajectory inputs, obtaining a success rate of 21% even when the trajectory IoU is in the range of [0.1,0.2).

   Due to the character limit, please find additional deep analysis of success/failure cases in our response to Reviewer EZ8N (2)

3. Architecture of fusion decoder: According to your advice, we supplement two sets of ablation studies in terms of the architecture of the fusion decoder, as shown in the table below. ||OSTrack-B256|+MPT(Ours)|+OnewayCA|D3|D4| |-|-|-|-|-|-| |VOT22(EAO)|0.530|0.572|0.554|0.555|0.555| |LaSOTEXT(AUC)|46.9|48.7|43.8|47.7|49.2|

   • First, our method employs bidirectional Cross-Attention (CA) for better alignment and fusion. Compared with one-way CA, our method obtains an average performance improvement of 3.4% across the two datasets, which demonstrates the effectiveness of our fusion decoder architecture.

   • Second, our method implements a two-layer fusion decoder, improving the baseline by an average performance of 3.0% across the two datasets. Deeper variants (D3/4) exhibit slightly inferior or comparable performance, with average improvements of 1.7% and 2.4%, respectively. Therefore, the two-layer structure excels in both performance and efficiency. In addition to reducing efficiency, there are two potential risks for a deeper-layer fusion decoder. First, since the number of prompt tokens is much smaller than that of visual tokens, a deeper fusion process may lead to the dilution and loss of motion prompts, thereby weakening the effectiveness of the motion information. Second, deeper-layer decoders introduce more trainable parameters, increasing the risk of overfitting.

   Thanks for your valuable advice. We will add the above experimental analysis to our paper.