DOVTrack: Data-Efficient Open-Vocabulary Tracking

We thank Reviewer 8J7k for acknowledging our empirical rigor and providing constructive feedback on generalizability:

1. Evaluation on Additional Dataset

The reviewer suggests to supplement experiments on one additional dataset with sparse annotations and analyze performance trends. Since KITTI-MOT only conducts the evaluation on two categories (person and vehicle), we use BDD100K dataset containing more categories. We conduct comprehensive experiments on BDD100K containing 8 categories to address this concern from two perspectives as below.

① Sparse annotation interval analysis: Following the reviewer's suggestion, to validate our method's effectiveness under different sparse annotation conditions, we conduct training experiments on BDD100K with different temporal-interval annotations. Since BDD100K is annotated at 5 fps, the minimum interval is 5 frames. We evaluate our method and baselines using the annotation intervals of 5, 15, and 30 frames.

These results demonstrate three key findings:

1. Our method with 30 frame intervals achieves comparable performance to baseline methods trained with dense 5 frame intervals, highlighting the effectiveness of our diffusion-based approach for handling sparse annotations.
2. With 15 and 5 frame intervals, our method significantly outperforms all existing approaches, demonstrating superior performance across different annotation densities.
3. The consistently good performance across different intervals validates the robustness and general applicability of our data-efficient training paradigm.

② Cross-dataset generalization evaluation: To further verify the generalization capability of our method, we conduct zero-shot experiments on BDD100K (training on TAO and testing on BDD100K) validation set without any training on this dataset.

We can see that our method achieves significant improvements across all metrics (TETA: +2.7%, LocA: +2.1%, AssocA: +2.3%, ClsA: +3.8% over the second-best method OVTR), demonstrating strong zero-shot cross-dataset generalization capability.

2. Challenging Scenarios Analysis

To comprehensively assess our method's robustness, we systematically selected the most challenging videos from the TAO validation set (93 videos of 988 videos) using multiple quantitative criteria. Videos satisfying any of the following conditions were included in our challenging subset (TAO-Hard):

(a) Heavy occlusion scenarios: We identified videos with substantial mutual occlusions by computing inter-trajectory IoU overlaps across consecutive frames. Videos were selected if object bounding boxes exhibit IoU > 0.4 with other objects for more than 20% of the total frames, indicating persistent occlusion challenges.

(b) Rapid motion patterns: We detected fast-moving objects by analyzing trajectory displacement patterns. Specifically, we calculated the ratio of bounding box center displacement to box diagonal length across consecutive frames. Videos with average displacement ratios > 0.3 were selected as they present significant temporal association challenges due to rapid object movement.

(c) Frequent entry/exit dynamics: We monitored object appearance and disappearance patterns by tracking trajectory start/end points within video sequences. Videos where objects exhibit frequent entry/exit behaviors (more than 3 entry/exit cycles per trajectory on average) were prioritized to evaluate our method's robustness to discontinuous tracking scenarios.

(d) High object density: We selected videos containing crowded scenes with high object density, specifically those maintaining an average of more than 8 concurrent objects per frame, which creates complex multi-object association challenges.

Using these systematic criteria, we identified videos presenting the challenging tracking conditions from the TAO validation set, which we refer to as TAO-Hard. The evaluation results are shown below:

From the results we can first see that, on the subset of challenging scenarios, our method still obtains the best performance with significant margins compared with other methods. The results also show that our method exhibits the smallest performance degradation across challenging scenarios. While all methods experience some decline in performance under extreme conditions, our approach shows the most minimal degradation (1.8% for novel and 1.2% for base classes), compared to baseline methods that suffer degradations of 1.9% -- 2.4%. This superior robustness stems from the proposed association and detection strategies that provide more coherent object representations, enabling better handling of challenging tracking scenarios.

3. Computational Scalability

Regarding the scalability of our method for larger category datasets, we discuss about the computational efficiency of key components in detail as below.

(a) Scalability of the diffusion model: The training and inference time of our diffusion model are essentially independent of the number of categories, meaning that an increase in category count does not incur additional computational overhead. The diffusion model operates in the feature space and is primarily influenced by the sampling steps, which determine the number of interpolated frames between adjacent frames. As indicated by our ablation results (Table 3 in the paper), optimal performance can be achieved with just three sampling steps, demonstrating that the associated overhead is minimal regardless of temporal gaps.

(b) Scalability of the dynamic group contrastive learning: To directly address the reviewer's concern about computational efficiency with larger category sets, we conducted comprehensive experiments with class counts ranging from 10 to 1200. We measured the average time taken for dynamic group contrastive learning per batch, which includes both the construction of dynamic groups and the calculation of contrastive learning loss.

The results demonstrate the good scalability of the proposed method. Specifically, even at 1200 classes, the average time consumed per batch is only 0.27 seconds, with the majority of this time spent on dynamic group construction. The time required for contrastive learning loss calculation remains below 0.1 seconds across all class counts. This demonstrates that our dynamic group contrastive learning approach scales efficiently and is not computationally prohibitive for very large category sets.

(c) Real-world applicability: Our approach is well-suited for real-world deployment as: 1) the computational overhead does not scale significantly with dataset size, 2) it reduces annotation requirements while maintaining performance, and 3) the method achieves the highest inference speed (15.3 FPS on a single 3090 GPU) among all comparison methods, making it promising for real-time applications.

Dear Reviewer 8J7k,

Thank you for your valuable feedback on empirical rigor and generalization. We have submitted a comprehensive rebuttal with additional experiments on the additional dataset, challenging scenario analysis, and computational scalability studies to address your concerns about generalization beyond TAO and real-world applicability.

As the discussion deadline approaches, we hope our expanded evaluation has addressed your questions. We welcome any further discussion you may find necessary.

Best regards,

The Authors

We thank Reviewer 8GVG for the thorough evaluation and insightful comments. We address each point below:

1. Association Head Training Details

We provide a comprehensive explanation of how generated intermediate features enhance association training from both quantitative and qualitative perspectives:

(1) Quantitative sample space expansion: Our diffusion model fundamentally transforms sparse association training by generating intermediate features $F\_{\text{asso}}^t$ at multiple timesteps between annotated frames. For each object trajectory originally containing only two annotated features ($F\_{\text{key}}$ and $F\_{\text{ref}}$ separated by 30 frames), our method generates $N$ intermediate features where $N$ corresponds to sampling steps (typically 3). This directly expands the positive sample set $Q^+(\mathbf{q})$ for each object identity from 2 to 5 features. Simultaneously, the generated features from different object trajectories enrich the negative sample set $Q^-$, creating a much denser and more comprehensive sample space $Q = Q^+ \cup Q^-$. Consequently, the total training samples increase by approximately 3×, providing substantially more positive and negative pairs for robust contrastive learning.

(2) Qualitative feature continuity enhancement: Beyond quantity expansion, our generated intermediate features $F\_{\text{asso}}^t$ exhibit superior temporal continuity compared to sparse annotations. The diffusion-based generation ensures smooth feature transitions through: 1) reconstruction loss enforcing accurate interpolations between keyframes; 2) smoothness loss guaranteeing consistent feature evolution; 3) endpoint loss ensuring precise alignment with reference features. This results in temporally coherent feature representations that better capture object state transitions, providing higher-quality training samples for association learning.

(3) Enhanced loss construction: Building upon the quantitative expansion from (1) and qualitative enhancement from (2), as mentioned in the implementation details of the paper, we employ the same association loss framework as OVTrack but with significantly enhanced effectiveness due to our enriched sample space. The complete tracking loss $\mathcal{L}\_{\text{track}}$ is formulated as:

$\mathcal{L}\_{\text{track}} = -\sum\_{\mathbf{q} \in Q} \frac{1}{|Q^+(\mathbf{q})|} \sum\_{\mathbf{q}^+ \in Q^+(\mathbf{q})} \log\left(\frac{\exp(\mathbf{q} \cdot \mathbf{q}^+/\tau)}{\text{PosD}(\mathbf{q}) + \sum\_{\mathbf{q}^- \in Q^-(\mathbf{q})} \exp(\mathbf{q} \cdot \mathbf{q}^-/\tau)}\right)$

where $\text{PosD}(\mathbf{q}) = \frac{1}{|Q^+(\mathbf{q})|} \sum\_{\mathbf{q}^+ \in Q^+(\mathbf{q})} \exp(\mathbf{q} \cdot \mathbf{q}^+/\tau)$ and $\tau$ is the temperature parameter. The enriched $Q^+(\mathbf{q})$ contains both original keyframe features and our generated intermediate features (from point 1), while $Q^-(\mathbf{q})$ includes negative samples from different trajectories. This formulation maximizes positive sample similarity (numerator) while minimizing negative sample similarity (denominator), leveraging both the expanded sample quantity from (1) and improved feature quality from (2) for more robust association learning.

2. Linear Interpolation Assumption

Our method uses linear interpolation for feature reconstruction, but this is only applied over very small temporal intervals (30 frames between consecutive annotations). It is well-justified by the calculus principles that any smooth function can be approximated linearly over a sufficiently small interval. For such short temporal intervals in video sequences, we use the linear interpolation assumption in this work.

For longer time spans, object motion patterns are inherently non-linear, involving complex transformations like turns, acceleration and deformation. Within the short intervals where we apply linear interpolation, such dramatic changes rarely occur. Moreover, the smoothness loss in Eq. (5) (in the main paper) further ensures realistic feature transitions.

3. Challenging Scenarios Analysis

To comprehensively assess our method's robustness, we systematically selected the most challenging videos from the TAO validation set using multiple quantitative criteria. Videos satisfying any of the following conditions were included in our challenging subset (TAO-Hard):

(a) Heavy Occlusion Scenarios: We identified videos with substantial mutual occlusions by computing inter-trajectory IoU overlaps across consecutive frames. Videos were selected if object bounding boxes exhibit IoU > 0.4 with other objects for more than 20% of the total frames, indicating persistent occlusion challenges.

(b) Rapid Motion Patterns: We detected fast-moving objects by analyzing trajectory displacement patterns. Specifically, we calculated the ratio of bounding box center displacement to box diagonal length across consecutive frames. Videos with average displacement ratios > 0.3 were selected as they present significant temporal association challenges due to rapid object movement.

(c) Frequent Entry/Exit Dynamics: We monitored object appearance and disappearance patterns by tracking trajectory start/end points within video sequences. Videos where objects exhibit frequent entry/exit behaviors (more than 3 entry/exit cycles per trajectory on average) were prioritized to evaluate our method's robustness to discontinuous tracking scenarios.

(d) High Object Density: We selected videos containing crowded scenes with high object density, specifically those maintaining more than 8 concurrent objects per frame on average, which creates complex multi-object association challenges.

Using these systematic criteria, we identified videos presenting the challenging tracking conditions from the TAO validation set, which we refer to as TAO-Hard. The evaluation results are shown below:

We can see that the results on all metrics show different levels of degradation on these challenging scenarios, which is natural. Take the association accuracy for example: these challenging scenarios with unpredictable object states, e.g., significant appearance changes, objects leaving and re-entering, make the tracking inherently more difficult. Nevertheless, our method still maintains superior performance compared to existing approaches, demonstrating its strong robustness across diverse scenarios.

We thank Reviewer Eere for the detailed feedback. We address each concern systematically in the following.

1. Controlled ablation studies on component effectiveness:

The reviewer asked about that since this paper has made improvements in both the detector approach and the associated data features, the evaluation on them should be conducted independently. To address this concern, we conduct systematic ablation studies that evaluate each component respectively to control the experiment conditions.

(1) Effectiveness of diffusion-based enhancement for different baselines: The reviewer asks: "If other methods also adopt the diffusion-model-based enhancement method proposed in this paper, can the method in this paper still achieve an advantage?" To answer this, we integrate our diffusion-based association enhancement (Sec.3.1) into existing OVMOT baselines while keeping other components unchanged.

Results demonstrate that diffusion-based association enhancement consistently improves association accuracy (AssocA) across different baselines by approximately 4.0% for novel classes and 2.5/3.7% for base classes, validating the universal effectiveness of this component for association. From the last row, we can see that our method (last row) still achieves the highest TETA scores. This is because that when using the same association modules as the baseline methods, our method also shows superior performance on the detection results (LocA and ClsA), which demonstrates that our method still has the advantages.

(2) Effectiveness of detection approach for different baselines: The reviewer further asks: "If other methods use the same detector improvements as this paper, can this paper still maintain the advantage?" To verify this, we apply our detection approach (Sec.3.2) to existing baselines.

The detection enhancement strategy consistently improves localization (LocA) and classification (ClsA) accuracies on different baselines, demonstrating its general applicability. Similarly, our method consistently outperforms the baselines with the same detector as ours, especially in association-related metric AssoA and the overall metric TETA. This clearly verifies that our method maintains the advantage compared to the baselines equipped with the same detection enhancement.

2. Discussion about the data scarcity

The reviewer suggests using open-vocabulary detectors or detection datasets. We clarify that existing OVMOT methods already follow a two-stage paradigm: detection training (which leverages open-vocabulary detectors/datasets) and association training (which requires video data). The challenge lies in the association training stage, where TAO is the only open-vocabulary video dataset but its sparse annotations are insufficient. OVTrack and SLAck have demonstrated that attempting to train robust OV trackers directly on TAO's sparse annotations fails to produce effective results.

To address this fundamental problem, OVTrack and subsequent methods (OVTR, MASA, etc.) use LVIS image pairs to simulate videos for association training. However, these image pairs lack motion continuity and appearance consistency, so the video data scarcity problem persists. SLAck is currently the only method that attempts to directly tackle this sparse annotation problem by using pseudo-labeling on TAO's intermediate unannotated frames. While it achieved improvements, it suffers from noise contamination and computational overhead.

Our diffusion-based approach provides a cleaner solution by generating reliable intermediate features in the feature space without requiring any intermediate unannotated frames. We achieve significant improvements over SLAck: 4.1% TETA improvement on novel classes (35.2% vs 31.1%) and 3.2% on base classes (40.4% vs 37.2%), with particularly substantial gains in association accuracy of 2.5% on novel classes and 4.5% on base classes.

3. Model complexity analysis

The reviewer asks about model complexity, parameters, and inference speed. Our method maintains comparable computational complexity to existing methods while achieving the highest FPS (15.3) among all compared approaches on a single 3090 GPU. Compared to OVTrack, we add minimal training overhead and no inference cost since the diffusion model only generates features during training.

4. Method clarity about association training

The reviewer requests clearer explanation of how generated features enhance association training. We provide a comprehensive explanation from both quantitative and qualitative perspectives:

(1) Quantitative sample space expansion: Our diffusion model fundamentally transforms sparse association training by generating intermediate features $F\_{\text{asso}}^t$ at multiple timesteps between annotated frames. For each object trajectory originally containing only two annotated features ($F\_{\text{key}}$ and $F\_{\text{ref}}$ separated by 30 frames), our method generates $N$ intermediate features where $N$ corresponds to sampling steps (typically 3). This directly expands the positive sample set $Q^+(\mathbf{q})$ for each object identity from 2 to 5 features. Simultaneously, the generated features from different object trajectories enrich the negative sample set $Q^-$, creating a much denser and more comprehensive sample space $Q = Q^+ \cup Q^-$. Consequently, the total training samples increase by approximately 3×, providing substantially more positive and negative pairs for robust contrastive learning.

(2) Qualitative feature continuity enhancement: Beyond quantity expansion, our generated intermediate features $F\_{\text{asso}}^t$ exhibit superior temporal continuity compared to sparse annotations. The diffusion-based generation ensures smooth feature transitions through: 1) reconstruction loss enforcing accurate interpolations between keyframes; 2) smoothness loss guaranteeing consistent feature evolution; 3) endpoint loss ensuring precise alignment with reference features. This results in temporally coherent feature representations that better capture object state transitions, providing higher-quality training samples for association learning.

(3) Enhanced loss construction: Building upon the quantitative expansion from (1) and qualitative enhancement from (2), as mentioned in the implementation details of the paper, we employ the same association loss framework as OVTrack but with significantly enhanced effectiveness due to our enriched sample space. The complete tracking loss $\mathcal{L}\_{\text{track}}$ is formulated as:

$\mathcal{L}\_{\text{track}} = -\sum\_{\mathbf{q} \in Q} \frac{1}{|Q^+(\mathbf{q})|} \sum\_{\mathbf{q}^+ \in Q^+(\mathbf{q})} \log\left(\frac{\exp(\mathbf{q} \cdot \mathbf{q}^+/\tau)}{\text{PosD}(\mathbf{q}) + \sum\_{\mathbf{q}^- \in Q^-(\mathbf{q})} \exp(\mathbf{q} \cdot \mathbf{q}^-/\tau)}\right)$

where $\text{PosD}(\mathbf{q}) = \frac{1}{|Q^+(\mathbf{q})|} \sum\_{\mathbf{q}^+ \in Q^+(\mathbf{q})} \exp(\mathbf{q} \cdot \mathbf{q}^+/\tau)$ and $\tau$ is the temperature parameter. The enriched $Q^+(\mathbf{q})$ contains both original keyframe features and our generated intermediate features (from point 1), while $Q^-(\mathbf{q})$ includes negative samples from different trajectories. This formulation maximizes positive sample similarity (numerator) while minimizing negative sample similarity (denominator), leveraging both the expanded sample quantity from (1) and improved feature quality from (2) for more robust association learning.

We appreciate the reviewer's valuable comments and suggestions, and commit to addressing these concerns in our final version.

Dear Reviewer Eere,

Thank you for your constructive feedback on our submission. We have submitted a detailed rebuttal addressing your concerns regarding experimental rigor, data scarcity arguments, model complexity analysis, and association training methodology clarity.

As the discussion deadline approaches, we hope our responses have addressed your questions. Please let us know if any further clarification would be helpful.

Best regards,

The Authors

Dear Reviewer Eere,

We hope our comprehensive rebuttal has addressed your primary concerns. To summarize our key responses:

• Experimental rigor: Following your suggestions, we conducted additional controlled ablation studies demonstrating: (1) Our diffusion-based association enhancement consistently improves association accuracy (AssocA) across different baselines; (2) Our detection approach consistently improves localization (LocA) and classification (ClsA) accuracies across baselines. Importantly, our method maintains superior overall performance even when baselines adopt identical components.

• Data scarcity discussion: We clarified that existing methods follow a two-stage paradigm (detection + association training), with the core challenge being association training due to TAO's sparse annotations. Our diffusion-based approach directly addresses this by generating reliable intermediate features, achieving the best results over current SOTA methods.

• Model complexity: We provided detailed analysis showing our method maintains the highest FPS (15.3) while adding minimal training overhead and no inference cost since diffusion only operates during training.

• Method clarity: We offered comprehensive explanations of how generated features enhance association training through both quantitative sample expansion (3× training samples) and qualitative feature continuity improvements.

Given that the discussion period is ending soon, we would greatly appreciate any additional feedback or concerns you might have. If our responses have sufficiently addressed your questions, we hope you would consider updating your evaluation.

Thank you for your time and valuable input.

Best regards,

The Authors

Dear Reviewer Eere,

We would like to kindly follow up regarding our previous responses to your feedback.

If there are any remaining questions or concerns, we would be glad to provide further clarification. Otherwise, if our responses have addressed your points, we would greatly appreciate your consideration in updating your evaluation.

Thank you again for your time and input.

Best regards,

The Authors

We thank Reviewer NEJd for the positive evaluation and constructive feedback. We address each concern below:

1. Theoretical Contributions

Our work mainly focuses on and takes the first step for the application of diffusion models on OVMOT, which also makes several key theoretical contributions to diffusion models:

(a) Novel diffusion paradigm: We pioneer the application of diffusion models to feature-level temporal interpolation in tracking, fundamentally departing from traditional pixel-level generation approaches. Unlike existing diffusion methods that focus on final denoised outputs, we innovatively utilize intermediate denoising results as meaningful feature representations, modeling the denoising process as temporal evolution of object features between sparse annotations.

(b) Sparse-to-continuous mapping rationale: We demonstrate theoretically and empirically that diffusion models can effectively bridge temporal gaps in sparse video annotations by simulating object state transitions. It provides the first systematic approach to transform sparse temporal annotations into continuous feature representations, enabling robust tracking learning.

(c) Multi-objective optimization for temporal sequence completion: We establish a theoretically grounded training framework that combines three complementary losses:

• Reconstruction loss: ensuring feature reliability through interpolation supervision
• Endpoint loss: guaranteeing accurate feature alignment
• Smoothness loss: enforcing temporal consistency

This loss combination also provides a reference for generating reliable intermediate object features with diffusion models.

2. Generalizability to Other Video Tasks

From the perspective of the original intention, our diffusion-based data generation is specifically designed for open-vocabulary tracking tasks, as it effectively simulates object motion changes and state transitions between sparse annotations. The diffusion model captures the temporal dynamics essential for tracking by modeling feature evolution across time gaps.

While primarily designed for tracking, the core principle of generating intermediate features between sparse annotations is task-agnostic and can be extended to other sparsely annotated video tasks, such as video action localization and motion trajectory prediction.

Thanks for the reviewer's comments, we are willing to explore these insights for other tasks in future work.

3. Comparison with SLAck

We provide quantitative evidence that our method outperforms SLAck's pseudo-labeling approach:

(a) Method and principle: SLAck generates pseudo labels by computing the IoU matching with each annotated frame (one for every 30 frames). We conducted a statistical analysis of SLAck's pseudo-labeling method and found that for each annotated frame, only the adjacent (previous and next) 7.8 frames on average are used for generating pseudo labels. In contrast, our diffusion-based approach can work across the entire temporal gaps (30 frames) between annotated frames.

(b) Results and comparison: We achieve 4.1% higher TETA score than SLAck on validation set for novel classes (35.2% vs 31.1%) and 3.2% for base classes (40.4% vs 37.2%). Our association accuracy (AssocA) significantly outperforms SLAck by 2.5% on novel classes (40.3% vs 37.8%) and 4.5% on base classes (42.1% vs 37.6%), demonstrating the effectiveness of our approach.

(c) Less demand and more cues: We achieve these improvements without using any intermediate un-annotated frames, which are necessary in SLAck. Additionally, SLAck's IoU-based pseudo-labeling approach cannot consider motion continuity, while our method fully considers motion continuity through reconstruction loss and smoothness loss, ensuring smooth feature transitions.

4. Detection Model Fine-tuning Necessity

The reviewer asks about the necessity of the fine-tuning stage and generalization performance.

(1) Effectiveness without detection fine-tuning: We conduct extra ablation experiments that demonstrate that even without detection fine-tuning (using only our diffusion-based association enhancement), our method still achieves state-of-the-art results. As shown in the table below, removing detection fine-tuning (on TAO), our method still yields competitive performance, especially for the association performance, indicating the effectiveness of our approach.

(2) Zero-shot generalization capability:

To demonstrate zero-shot generalization capability, we conducted experiments by applying our method on BDD100K validation dataset without any fine-tuning on BDD100K, showing superior cross-dataset transfer performance. We can see that our method achieves significant improvements across all metrics (TETA: +2.7%, LocA: +2.1%, AssocA: +2.3%, ClsA: +3.8% over the second-best method OVTR), demonstrating enhanced generalization capability.

(3) Comprehensive training paradigm rationale:

Overall, the proposed method benefits from but is not dependent on the detection fine-tuning (on specific dataset, e.g., TAO). The fine-tuning strategy of our method (complete version) constructs a comprehensive data-efficient training paradigm that maximizes the utility of limited sparse annotations. Fine-tuning with our proposed dynamic group contrastive learning and adaptive localization loss specifically addresses small dataset adaptation challenges while maintaining generalization, creating a holistic approach that leverages both association improvements (via diffusion) and detection improvements (via fine-tuning) to achieve optimal performance under data-constrained conditions.

5. Code Availability

We commit to releasing all the code to ensure reproducibility upon acceptance of this paper.

Dear Reviewer NEJd,

We sincerely appreciate your support for our work and positive evaluation. We have provided a comprehensive analysis in our rebuttal addressing your concerns about theoretical contributions, generalizability, comparison methodology, detection fine-tuning, and code availability.

Given the approaching discussion deadline, we hope our detailed responses have effectively addressed your questions. We welcome any follow-up discussion you may find helpful.

Best regards,

The Authors