ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model

Q1: Line 127 mentions only the initial bounding box and search frame as inputs and does not mention template frames as inputs. However, the equation in line 128 writes template as input.

Thanks for your careful reading.

In the equation P^{t}_{VT} = \mathcal{F}_V$$_T$$(I^{t};I^{1}, G) , $I^t$ represents the image of the t-th frame, $G$ represents the initial bounding box, and $I^1$ is the first frame of the video with $G$ as the initial bounding box. We will clarify this in the revised manuscript.

Q2: In section 4.1, MixFormer-L and ARTrack-256 as visual coders for ChatTracker-L and ChatTracker-B? I have a question, why not use mixformer-B/L or ARTrack-B/L uniformly as visual encoders. Could the authors please provide more results for mixformer-B and ARTrack-L for performance comparison.

Thanks for your question.

At the time of submission, ARTrack[1] had not released the code and model weights for ARTrack-L384, so we did not use ARTrack-L as the visual encoder for ChatTracker-L.The choice of MixFormer-L[2] for ChatTracker-L leads to high accuracy, and ARTrack-B for ChatTracker-B provides a better trade-off between accuracy and speed.

We provide more results for MixFormer-B and ARTrack-L for performance comparison. In the table below, results of visual trackers with the integration of ChatTracker are marked by *.

Table.1 The comparison of results forVision Language trackers using ChatTracker-generated text (marked by*). The speed was tested on the same device.

Q3: In section 4.3 and Table 2, when applying the proposed module to other visual trackers, the paper does not describe how the linguistic description is fused with the visual features, could the authors please explain this process specifically.

In fact, we do not fuse the visual tracker into our ChatTracker at the feature level. When replacing visual trackers, we incorporate the different tracking results of the corresponding visual tracker, $P^t_{VT}$, into $P^t_{fore}$ as supplemental foreground proposals. At the same time, the foreground proposals generated by the GVLM using linguistic descriptions do not change.

We will provide a detailed description of how we replace the other visual tracking methods in the revised manuscript and we will release our code to help our readers better understand this process.

Q4: The LaSOT_ext and OTB_Lang test datasets are also included in the VL tracking task, and the authors are asked to provide performance on these datasets to help enrich the experimental results in this paper.

Thank you for your suggestion. Our performance on these two datasets is as follows, demonstrating that our method also achieves SOTA results on these datasets.

Table.2 Performance comparison of ChatTracker, UVLTrack, and ARTrack on the LaSOT_ext and OTB_Lang datasets.

Q5: Does the proposed approach require fine-tuning in the downstream tracking dataset?

No, we don't need to fine-tune MLLM in the downstream tracking dataset. We use the MLLM reflection to narrow the knowledge gap between the VL tracker and the MLLM. This allows ChatTracker to achieve favorable performance without fine-tuning.

Thanks for your suggestion. We will investigate the fine-tuning version of our proposed framework in future work.

[1]Xing Wei, et al. Autoregressive visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2023.

[2] Yutao Cui, et al. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2022.

[3]Heng Fan, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 2021.

[4]Xiao Wang, et al. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

[5]Zhenyang Li, et al. Tracking by natural language specification. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017

Dear Reviewer,

Thanks a lot for your recognition of our work.

Best wishes,

The Authors

Due to the word limit in the rebuttal, we provide the remaining citations here.

[1]OVLM: Huanlong Zhang, Jingchao Wang, Jianwei Zhang, Tianzhu Zhang, Bineng Zhong "One-stream Vision-Language Memory Network for Object Tracking" TMM 2023

[2]MMTrack: Yaozong Zheng, Bineng Zhong, Qihua Liang, Guorong Li, Rongrong Ji, Xianxian Li"Towards Unified Token Learning for Vision-Language Tracking" TCSVT 2023

[3]Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juehuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129:439–461, 2021.

[4]Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13763–13773, 2021.

[5]Shiyu Hu, Dailing Zhang, Meiqi Wu, Xiaokun Feng, Xuchen Li, Xin Zhao, and Kaiqi Huang. 2023. "A Multi-modal Global Instance Tracking Benchmark (MGIT): Better Locating Target in Complex Spatio-temporal and Causal Relationship." Paper presented at the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), Track on Datasets and Benchmarks.

[6]Y. Wu, J. Lim and M. -H. Yang, "Object Tracking Benchmark," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1834-1848, 1 Sept. 2015, doi: 10.1109/TPAMI.2014.2388226.

[7] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23151–23160, 2023.

[8]Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, and Yihong Gong. Autoregressive visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9697–9706, 2023.

[9]Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Jinpeng Zhang, and Mengxue Kang. Unifying visual and vision-language tracking via contrastive learning, 2024

Q1: SOTA trackers are missing in Table 1, such as OVLM[1], MMTrack[2].

Thank you for your valuable suggestion. We will include comparisons with OVLM and MMTrack in the revised manuscript. Additionally, we have found that compared to their variants with highest performance, our ChatTracker demonstrates better tracking performance. Furthermore, our method does not require manually annotated target text for initialization.

Table.1 Complete comparison of OVLM and MMTrack with ChatTracker's performance on LaSOT[3] and TNL2K[4] Datasets.

Q2: I want to know the model's performance on MGIT[5] and OTB99_Lang[6].

Thank you for your valuable suggestion. We evaluated ChatTracker-B on MGIT and OTB99_Lang.

• For the MGIT dataset, we used only the first frame's bounding box (BBOX) for initialization and used ChatTracker's self-generated language descriptions for tracking. For the comparison method JointNLT[7], we selected Mechanism A (i.e., the first frame's BBOX and the annotated text in the dataset) for comparison. For ARTrack-256[8], we used only the first frame's BBOX for initialization.

Table.2. Results on the MGIT dataset

• For the OTB99_Lang dataset, our ChatTracker used only the first frame's bounding box (BBOX) for initialization and utilized ChatTracker's self-generated language descriptions for tracking. The settings for JointNLT and ARTrack-256 are the same as those for MGIT.

Table.3: Results on the OTB99_Lang dataset

The results show that, thanks to more accurate text descriptions and the proposed vision-language tracking framework, our ChatTracker outperforms existing methods. We will include these results in the revised manuscript.

Q3 Why is it necessary to design ChatTracker-L and ChatTracker-B, but use different backbone networks?

• For the design of ChatTracker-L and ChatTracker-B:

ChatTracker-L is designed for better performance, while ChatTracker-B is designed to achieve a better trade-off between accuracy and speed. Here, L stands for Large, and B stands for Base.

• For the use of different backbone networks:

At the time of submission, ARTrack[1] had not released the training code and model weights for ARTrack-L384, so we use the Mixformer-L for ChatTracker-L.

Q4: There is a discrepancy between the data from UVLTrack[9] in Table 1 and the data provided by the official source. Also, I believe comparing the L model in Table 1 is unfair, as UVLTrack provides the L model, and annotations are needed for models of different scales.

Thank you for pointing this out. In Table 1, we reported the results for UVLTrack-B. As shown in the table below, our AUC scores on LaSOT, TrackingNet, and TNL2K are higher than UVLTrack-L. For other trackers in Table 1, we selected the variant with the highest performance for a fair comparison. We will replace the comparison results of UVLTrack-B in Table 1 with the UVLTrack-L version and annotate the scales of the compared methods in the revised manuscript. We have included the revised Table.1 in the uploaded PDF file.

Table.4 Comparison results of UVLTrack-L and ChatTracker on the LaSOT, TrackingNet, and TNL2K datasets

Q5: if the natural language description includes background information, is it reasonable to only crop out the target for calculating image-text similarity?

It is reasonable to only crop out the target for calculating image-text similarity.

When background information is present in the language description, the image-text similarity is lower compared to language descriptions without background information. Therefore, the higher the image-text similarity between the language descriptions and the cropped target, the less background information is included in the language descriptions. The less background information included in the language descriptions indicates that the text quality is higher.

Therefore, in our experiments, cropping the target from each frame and calculating its text-to-image similarity is reasonable. We will clarify this in the revised manuscript.

Q6: I believe the phrase "via Chatting with Multimodal Large Language Model" in the title overstates the contribution of the entire framework.

"via Chatting with Multimodal Large Language Model" refers to the iterative process where the MLLM generates language descriptions, and the GVLM provides feedback on the quality of these descriptions to the MLLM, helping it generate better language descriptions. This process is akin to chatting with the MLLM. We think this phrasing helps our readers better understand our method intuitively.

We are open to changing the title if there is a more suitable one. We would appreciate it if you could give more detailed suggestion or more explanation about how does the current one overstate the contribution.

Due to the word limit in the rebuttal, we provide the remaining citations in the following comment.

Dear Reviewer:

Thank you again for taking the time to review our work and for providing us with valuable feedback. We now address your additional comments.

Q1. MGIT provides flexible and diverse semantic labels. This benchmark is somewhat closer to the "chat" feature than current VLT benchmarks. Therefore, I suggest that the author conduct further experiments on this benchmark to analyze the effects and limitations of the method under different granularities and evaluation mechanisms, and consider adding this part of the analysis to the article to highlight the motivation and theme of the article.

We appreciate your valuable suggestion. MGIT is a high-quality vision-language benchmark. The MGIT dataset focuses on the impact of text annotations at different granularities on tracking performance.

However, our ChatTracker only uses the initial bounding box as input and does not utilize additional text descriptions as input. Although we agree that further applying our proposed method to MGIT is very interesting, employing different text annotation granularities is beyond the scope of this paper.

In the rebuttal, we have provided ChatTracker's results on the MGIT dataset with the initial bounding box and we will include these results in the revised manuscript.

Thank you again for your valuable suggestion. We will further investigate these text inputs into our framework in future work.

Q.2 In addition, please note that the author cited the wrong MGIT literature (Reference 5) in the second reply window for me. I am unsure what the relationship is between the work cited by the author and the rebuttal itself.

Thank you for your careful reading. We have corrected Reference [5] in the comment.

Q.3 Finally, NeurIPS officially recommends that the reply to each reviewer during the rebuttal should not exceed 6,000 characters (only one window). I noticed that the author's reply to 2 reviewers had exceeded the limit of one window. In a sense, this is unfair (for other authors of the same period who only use one window for rebuttal). I hope the authors can pay attention to these details.

Thanks again for your advice. For additional comments, we follow the [NeurIPS 2024] Clarification on author rebuttal email:

• Comments to paper and reviews will be fine. Comments can be seen in time. Please set the readers correctly when you post them. Reviewers are not required to take comments into consideration.

To better discuss the issues, we responded to the longest review (up to 11 questions) in one additional Comments and provided relevant references in another Comment.

We strictly follow the policy and instructions given by the NeurIPS 2024 PCs email. This email is sent to all authors and we don't think there is any fairness issue.

We hope our responses have fully addressed your concerns. If you have any other questions or need further clarification, please feel free to let us know.

Thank you very much!

Best wishes,

Authors

Due to the word limits, we provide the remaining citations here.

[1]Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juehuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129:439–461, 2021.

[2]Y. Wu, J. Lim and M. -H. Yang, "Object Tracking Benchmark," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1834-1848, 1 Sept. 2015, doi: 10.1109/TPAMI.2014.2388226.

[3] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23151–23160, 2023.

[4]Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Jinpeng Zhang, and Mengxue Kang. Unifying visual and vision-language tracking via contrastive learning, 2024

[5]Xin Li, Yuqing Huang, Zhenyu He, Yaowei Wang, Huchuan Lu, and Ming-Hsuan Yang. Citetracker: Correlating image and text for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9974–9983, 2023.

Due to the word limit in the rebuttal, we provide the remaining responses here.

Q7 Lines [204-205] - To confirm, the anchor sample is always the target template, but do you sample positive samples from tracker result or foreground proposal? Visual tracking is mentioned in Line 202, but not in the Appendix.

No, we do not sample positive samples from the visual tracker's results during training. We only randomly choose a target patch from other frames of the same video as the positive sample. In Line 202, we use the trained $f(.)$​ (which is a neural network) to calculate the foreground score of the visual tracker's result. We will clarify this in the revised manuscript.

Q8 Line [221] - The IoU threshold for especially foreground seems a little low, what’s the reason for this? Do you have any statistics for what the accepted IoUs are from the foreground verification? Is it generally lower with the proposals?

• The IoU threshold for especially foreground seems a little low, what’s the reason for this? Is it generally lower with the proposals?

For the IoU threshold for foreground(i.e. $\theta_1$ in equation (5) ), we set $\theta_1$ to 0.3. This is to ensure there are enough positive words $T_{pos}^i$ as feedback during the early iterations, helping the iterative optimization process converge quickly.

For accepted IoUs (i.e., $\epsilon$ in line157) ，we choose the IoU threshold based on the experiments. We tested on the LaSOT dataset, and the results are shown in the table below. The results indicate that an IoU threshold (i.e., $\epsilon$ in line157) of 0.4 leads to the best results.

Table.2 The performance of chattracker on Lasot dataset with different IoU thresholds

• Do you have any statistics for what the accepted IoUs are from the foreground verification?

Yes, the average accepted IoU is 0.79. We calculated the distribution of accepted IoUs on the LaSOT dataset, and the results are shown in the table below.

Table.3 Data distribution of IoU values on the Lasot dataset.

Thank you for your question. We will add the relevant results and analysis in the revised manuscript.

Q9 Please use $P_{norm}$ or NP consistently across tables. Some tables report NP and P, others one or the other. It would be better to report consistently even if it’s just across smaller tables.

Thanks for pointing this out. In the revised manuscript, we will use $P_{norm}$ or NP consistently and report metrics consistently.

Q10 baseline tracker results in Table 3 are not matching the same tracker’s results in Table 1.

Thanks for pointing this out.

In Table 1, the baseline trackers (i.e., JointNLT[3] and UVLTrack[4] ) are initialized using both natural language and the initial bounding box to compare with the best-performing model.

In Table 3, the baseline trackers (i.e., JointNLT and UVLTrack) are initialized using only natural language. The purpose is to compare the effectiveness of ChatTracker-generated text versus manually annotated text for Vision-Language trackers without the influence of the initial bounding box.

We will clarify this in the revised manuscript.

Q11 Fig. 3 - not seeing any blue boxes in the image, even though CiteTracker is in the legend.

Thanks for pointing out this typo. CiteTracker[5] is not included in this comparision and we reuse the legend picture in this figure mistakenly. We revise the manuscript accordingly. For clarification, we have placed the revised figure in the .pdf file.

Q1. The potential limitations mentioned in the weaknesses should be addressed or discussed in the paper. Have the authors done any experiments where foreground/background prompts were updated while the video is being processed? What were the insights?

Thanks for pointing this out. Temporal changes to target and background are indeed one of the main challenges in the visual tracking domain. Our ChatTracker relies on a baseline tracker to consider potential temporal changes in the video. However, compared to traditional visual trackers, which rely only on a single image template to locate the target, natural language provides more temporal-invariant features, such as object categories and textures. Our ChatTracker incorporates MLLMs and designs a novel Reflection-based Prompt Optimization (RPO) module to obtain accurate target descriptions, providing better robustness when facing temporal changes.

Updating foreground and background prompts is not straightforward for two main reasons:

1. Accuracy of Predictions: During tracking, the tracker's predictions of the target are not always accurate, and there are no annotations of background objects. This makes it difficult to dynamically generate foreground and background prompts.
2. Trade-off Between Performance and Efficiency: If MLLMs are called multiple times during tracking to update foreground and background prompts, it incurs a certain computational overhead (although this overhead is decreasing as technology advances). Achieving a good balance between performance and efficiency requires extensive research.

We intend to delve into this challenge as part of our future work, and to address this, we have incorporated a discussion in the limitations section.

Q2. How often does the visual tracker proposal get picked over a GVLM proposal? Or does the visual tracker proposal mostly get used when there are no GVLM proposals?

A GVLM proposal gets picked 33.27% of the all time in our experiment. We analyzed 685,360 frames in the LaSOT dataset [1], of which 228,042 used the GVLM proposal as the result.

Q3. How dependent is the final performance on how well the Foreground Verification module is performing (since it’s only been trained on the LaSOT training set)?

Thanks for pointing this out. The Foreground Verification module is crucial to the whole framework and its final performance. To better answer your question, we design an additional ablation experiment using ChatTracker-L on the OTB-Lang dataset[2]. The experiment involves three versions of our trackers:

1. Complete ChatTracker-L: Uses the Foreground Verification module.
2. ChatTracker-L with Ground Truth: Does not use the Foreground Verification module but selects the proposal with the highest IoU with Ground Truth as the tracking result. This entry is used to establish the theoretical upper bound.
3. ChatTracker-L with Random Selection: Does not use the Foreground Verification module and randomly selects a proposal as the tracking result.

The results are shown in the table below. The results show that using the Foreground Verification module significantly outperforms the random selection group. This demonstrates that our Foreground Verification module is effective. Although there may still be room for improvement, our ChatTracker-L result is close to the theoretical upper bound. Thank you for your suggestion. We will discuss and add this in the supplementary.

Table.1 The performance comparison of Foreground Verification module ,random selection module in our framework on Lasot Dataset

Q4 Related works section can be expanded a bit.

Thanks for your suggestions. We will add a section in the revised manuscript to introduce related benchmarks for broader audiences to understand the insights. This section will include whether the LaSOT, TrackingNet, TNL2K, and other benchmarks contain text annotations, the methods used for text annotation, and potential reasons for low quality.

Q5 What’s the purpose of $T_{pos}^i$ in equation (6) ?

In equation (6):

$T_{neg}^i = \\{ w_m^i|for\ all\ n\ that\ S_z^{nm}>\theta_2 \wedge IoU(P_z^n,G)<\theta_3 \\} \backslash T_{pos}^i$

The purpose of $T_{pos}^i$ is to ensure that a word does not appear in both $T_{pos}^i$ and $T_{neg}^i$ .

$T_{pos}^i$ and $T_{neg}^i$ are used to construct a reflection prompt during the iterative optimization process to help the MLLM generate better language descriptions. If a word is present in both$T_{pos}^i$ and $T_{neg}^i$ , it may confuse the MLLM and prevent it from generating an effective response.

Q6 Lines [152-153] - What happens if a word matches to no proposals? Can we add that to negative words? Why not?

If a word matches no proposals, it will be ignored and not added to negative words.

$T_{neg}^i$ is used to inform the LLM that the GVLM has associated this word with objects in the background. The LLM will further enable the GVLM to identify the correct target based on this information.

When a word matches no proposals, it is considered meaningless (i.e., it doesn't contain any information that helps the LLM distinguish the target), so we do not add it to $T_{neg}^i$ .

For example, in the swing-17 example in Fig. 2, when the text description is “Human on a swing seat,” "Human" and "Swing" are categorized as negative words, and "seat" is categorized as positive. They all correspond to image regions in the figure. However, "on" and "a," which match no proposals, do not provide effective feedback to the MLLM. Therefore, we do not add them to negative words.

Due to the word limit in the rebuttal, we provide the remaining responses in the following comment.

Dear Reviewer,

Thank you once again for your valuable comments. We also agree with reviewer z5a5's advice on this further research direction. It would be very interesting to extend our framework in MGIT's multi-granular annotations and dataset.

Since analyzing the effect of multi-granular annotations is non-trivial and beyond the scope of this paper, we will investigate this direction in our future work.

Thank you very much!

Best wishes,

The Authors

Q1: The paper should illustrate the number of chat iterations in the reflection-based prompt optimization module and their performance effect. The example in Figure 5 shows that the module needs 2 iterations to get an accepted prompt but the overall analysis should be considered, especially in the images with complex scenes and objects.

• The paper should illustrate the number of chat iterations in the reflection-based prompt optimization module.

Thanks for your advice. The iterative refinement process averages 2.3 rounds. We illustrated the number of chat iterations in Section 4.1 (Line 229 - 230).

• The example in Figure 5 shows that the module needs 2 iterations to get an accepted prompt but the overall analysis should be considered, especially in the images with complex scenes and objects.

We conducted experiments using ChatTracker-B to study the performance effect of the maximum number of iterations. The results show that when the maximum number of iterations is less than 10, the tracker performance improves with an increase in iterations. When the number of iterations exceeds 10, the improvement in tracker performance becomes less significant. In our experiments, the default maximum number of iterations is set to 20.

For complex scenes and objects, we will add more visual examples in the revised manuscript to help our readers understand the impact of iteration numbers on tracker performance.

Table.1 Impact of the number of iteration rounds on tracker performance on Lasot[1] and TNL2k[2] datasets

[1]Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juehuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129:439–461, 2021.

[2]Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13763–13773, 2021.