DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM

1. The data generation pipeline as well as the data collection procedure are lack of novelty, by simply using previous DTLLM-VLT and VLT and SOT benchmarks.
2. The quality of the benchmark relies heavily on the performance of DTLLM-VLT, and there seems to be no correction/filtering strategy to compensate for this.
3. The evaluation of VLT methods should be more comprehensive, including more NLT approaches to further validate the effectiveness of DTVLT

1. While the dataset is a valuable addition to the field, offering some level of novelty, it falls short of significant innovation. The work’s main strength lies in diversifying existing benchmark annotations rather than introducing a groundbreaking methodology or framework. Although this level of novelty is acknowledged, the lack of an entirely new approach may limit the perceived impact of the benchmark within the research community. The work could benefit from a clearer distinction between its novel contributions and those of prior benchmarks to emphasize its unique value.
2. The dense captioning provided by the dataset primarily centers on spatial relationships, which, while useful, presents a relatively narrow challenge to models. By focusing mainly on spatial relations, the annotations risk being overly uniform in terms of complexity, potentially limiting the depth of the benchmark’s impact on model evaluation. A more varied approach, incorporating a broader spectrum of semantic relationships beyond spatial ones, could introduce more diverse challenges for model training and testing, offering a more robust assessment of model capabilities.
3. The paper does not introduce new baseline models, especially for dense caption-based tasks, to validate the proposed benchmark’s utility. The absence of baseline models that are specifically designed to adapt to the dense captioning structure leaves an open question about whether existing models are sufficient or if structural changes are required to adapt models without compromising previous task integrity. Designing or adapting baseline models could serve as a valuable proof of concept, demonstrating the benchmark’s effectiveness and providing the community with a clear starting point for future research on model adaptations for dense captioning.

(1) This paper appears more like a pipeline proposal, merely leveraging existing models to enhance an existing dataset, without focusing on the scientific questions that should be addressed when establishing this dataset. (2) The motivation is unclear, as Dtllm-vlt[1] seems to have already addressed the short-term tracking, long-term tracking, and global instance tracking challenges mentioned in this paper. (3) DTVLT heavily relies on LLM-generated text descriptions but lacks a quality control method for these generated texts. The automatically generated text may contain semantic biases, inconsistencies, overly long or redundant information, and may even be disconnected from the visual content, particularly in multi-modal tasks. Analyzing the quality of generated texts and investigating how to produce better descriptions would be more worthwhile.

[1] DTLLM-VLT: Diverse Text Generation for Visual Language Tracking Based on LLM.

1. In Table 3, did the author use the original implementations and pretrained weights for these trackers (MMTrack, JointNLT, and UVLTrack)? If so, it would be helpful for the author to discuss any discrepancies between their results and those reported in the original papers, as the performance results for these trackers in Table 3 do not align with the performance reported in their respective publications.

2. As this paper proposes a benchmark, additional trackers should be tested to fully evaluate it (i.e., VLT ("Divert More Attention to Vision-Language Tracking")).

3. I am skeptical about the necessity of the "Initial Concise" annotation by the LLM. This annotation seems less effective than the official language, as the performance decreases when directly testing the trackers, as shown in Table 3. Additionally, Figure A7 indicates that even with retraining, performance still declines with the "Initial Concise" annotation. Could the author provide a more detailed analysis of why they included the "Initial Concise" annotation despite its apparent limitations, and what specific value it adds to the benchmark.

4. When directly testing tracking performance with "Dense Concise" or "Dense Detailed" annotations—such as for the N-th test frame in a video, which language annotation does the author use? How their choice of annotation for each frame might impact the tracking results, and whether they considered alternative approaches.

5. In the training process using "Dense Concise" or "Dense Detailed' annotation, the model needs sample the N-th training frame from the given video. Which language annotation does the author use in this case. how the choice of annotation during training might affect the model's ability to generalize to different text granularities during testing.

6. In Figure A7, the performance with "Dense Detailed" annotation appears worse for the AUC metric. Could the author provide a hypothesis for why the "Dense Detailed" annotation leads to worse AUC performance, and to discuss the implications of this finding for the design of VLT systems.

• Tracking by natural language specification enhances human-machine interaction and has the potential to improve tracking outcomes. Although this paper introduces four levels of granularity in text annotations within the benchmark, the individual contributions of each text type to model performance remain ambiguous. As shown in Figure 5, models retrained on the DTVLT dataset exhibit improved performance on DTVLT; however, it is difficult to determine whether these models have merely memorized the patterns within DTVLT. Therefore, could you evaluate the models trained on DTVLT on an entirely different VLT dataset to assess model performance improvements? Additionally, could you train separate models on each of the four text granularity datasets and compare their performance to isolate the impact of each type?

• DTVLT provides significantly more diverse semantic information than previous datasets, with texts generated by large language models (LLMs), which differ from human language. Therefore, conducting real-world experiments to compare models trained with and without DTVLT is valuable for evaluating performance and robustness in practical applications. Have you considered conducting user studies or real-world deployment tests in specific domains like autonomous driving or surveillance to compare the performance of models trained with and without DTVLT?

• Considering the word distribution in DTVLT, as shown in Figure 4, terms like 'center', 'positioned', and 'black' constitute a large proportion, which may lead to bias in models trained on this dataset. It would be beneficial to analyze how this distribution impacts model performance. Could you conduct an ablation study to measure how removing or reducing the frequency of common terms like 'center', 'positioned', and 'black' affects model performance?

• In terms of bad cases, complex scenarios with multiple similar objects often pose difficulties. As shown in Figure A9, many objects match the detailed description, such as 'A zebra is standing in the middle of the field', which may make it harder for the model to accurately identify the target. Therefore, filtering out redundant information may enhance the quality of the dataset. Have you considered assessing the frequency of such ambiguous cases in the dataset and implementing a post-processing step to refine the generated descriptions?